Discrete representations in neural models of spoken language (2021)

Abstract: The distributed and continuous representations used by neural networks are at odds with representations employed in linguistics, which are typically symbolic. Vector quantization has been proposed as a way to induce discrete neural representations that are closer in nature to their linguistic counterparts. However, it is not clear which metrics are the best-suited to analyze such discrete representations. We compare the merits of four commonly used metrics in the context of weakly supervised models of spoken language. We compare the results they show when applied to two different models, while systematically studying the effect of the placement and size of the discretization layer. We find that different evaluation regimes can give inconsistent results. While we can attribute them to the properties of the different metrics in most cases, one point of concern remains: the use of minimal pairs of phoneme triples as stimuli disadvantages larger discrete unit inventories, unlike metrics applied to complete utterances. Furthermore, while in general vector quantization induces representations that correlate with units posited in linguistics, the strength of this correlation is only moderate.

## introduction the dominant machine learning paradigm for processing spoken language is based on neural network architectures, such as recurrent nets and transformers, inducing a hierarchy of hidden representations which are distributed and continuous. in contrast, human history has repeatedly seen the discovery and wide adoption of discrete, symbolic representations of speech in the form of writing. these systems commonly represent basic units of language such as morphemes, syllables or phonemes while discarding other information contained in the speech signal such as emotion or speaker identity. symbolic representation of speech has proven tremendously useful for storage and transmission of information, and it also plays a crucial role in systems dealing with spoken language such as spoken dialog systems: these typically employ an automatic speech recognition (asr) module to transcribe the speech signal into written form, which is then used as input to upstream language understanding modules. while some attempts have been made to train such systems end-to-end, pipelines are still very competitive (see @xcite , which strengthens the point that a symbolic encoding of spoken language contains most of the information relevant for this task. it may thus be desirable to incorporate similar representations in neural architectures. accordingly, multiple efforts have been made to design neural networks with discrete hidden representations and to apply them to spoken language data. this is evident in the recent editions of the zerospeech challenge @xcite on unit discovery, which have featured many such approaches. specifically, vector quantization (vq) has proven to be a simple and effective method to induce discrete neural representations (e.g., @xcite @xcite . vq layers are added to neural architectures in order to map continuous activation vectors onto a finite set of discrete units, often referred to as codes, via a dictionary (or codebook) associating these codes with their vector embeddings; the number of entries in the codebook is the codebook size. such symbolic codes have been claimed to correspond to phonemes and/or words. what is still lacking though is a detailed analysis of how much the reported equivalence is affected by details of the architectures such as the size and placement of the vq layers, learning objectives and dataset, as well as by the evaluation metrics used to quantify it. the present study aims to fill this gap. we study two approaches to modeling spoken language: learning driven by language-internal structure, and learning driven by grounding in the extra-linguistic world. these two approaches are exemplified by two types of models with vq layers: the self-supervised model for unit discovery of @xcite , and a visually grounded model similar to @xcite . the datasets used to train each model, zerospeech 2020 @xcite and flickr8k @xcite are also typical of the task they are used for. using these two models as our test cases, we systematically investigate the impact of the following factors: (i) the codebook size for the vq layer, and (ii) the level of placement of the vq layer. furthermore, we apply and check the consistency across four different metrics for evaluating the correspondence of the representations with phonemes. ## related work the domain of speech processing has seen recent efforts to modify existing neural architectures to enable the induction of discrete latent representations. these developments are promising for boosting performance, improving interpretability, and modeling the acquisition and processing of linguistic knowledge in humans. applying vector quantization (vq) techniques for this purpose was pioneered by @xcite , who propose generative models based on the variational auto-encoder (vae) architecture and use vq to induce discrete latent representations. they apply this method to images, videos, and speech, and show that the models can learn discrete latent representations without supervision. when applied to raw speech, the vq-vae architecture learns high-level discrete representations that are invariant to low-level features of the audio signal such as prosody and speaker identity, and mostly encode the content of the speech. classification of the discrete representations into phoneme classes (based on majority ground truth label) suggests they capture phonemes to some extent. learning discrete instead of (or in addition to) continuous representations can facilitate unit discovery in unsupervised models of speech. in the 2015 zero resource speech challenge @xcite , @xcite present a binarized auto-encoder, certain variants of which outperform its continuous counterpart. in the 2017, 2019, and 2020 editions of this challenge @xcite @xcite , following the work of van den @xcite , several models include at least one vq layer (see e.g. @xcite @xcite . these studies demonstrate the benefit of using vq layers for phoneme classification and for learning speaker-invariant representations, focusing on the abx phoneme discrimination metric to evaluate the encoding of phonemic information. however, little analysis on the impact of the size and configuration of the employed vq layers is performed. for example, van niekerk et al. (2020) train a vq-vae model for reconstructing audio waveforms, and a vq-cpc (contrastive predictive coding) model for predicting future acoustic units. they evaluate these architectures on the abx phoneme discrimination task, voice conversion and speaker classification and show that vq-cpc performs better than vq-vae overall, but they do not manipulate the configuration or the dimension of the vq layers. @xcite , however, report the impact on phoneme and speaker classification of systematic manipulation of vq-related factors. they train an autoregressive predictive coding (apc) model to predict upcoming frames, and use vq as a methodology to limit the model's capacity. using a frame-wise diagnostic classifier (namely linear logistic regression), they show that under restricted configurations (only one vq layer inserted at the in contrast to the work cited above, which discusses uni-modal speech models, @xcite use vq layers within the setting of learning spoken language via grounding in the visual modality, where the speech signal is associated to images (for an overview of visually grounded speech models, see @xcite . they hypothesize that discrete representations learned by such models are more likely to capture higher-level semantic information. their analyses suggest that the trained multimodal models can learn discrete linguistic units at both word and sub-word levels, with quantization layers inserted at the lower levels of the network showing correspondence to sub-word (i.e. phonemic) units, and those inserted at the higher level corresponding to word-level units. analyses are based on the zerospeech abx metric for phoneme encoding and f1 scores for word detection. similarly, @xcite propose a framework based on vq to discover discrete concepts in models of visually grounded language trained on video and text, video and audio or image and audio. evidence of a correspondence between the learned concepts and visual entities/actions or words are given but no detailed analysis is performed. when evaluating the encoding of phonemic information in continuous representations from models of spoken language, recent work has shown that different metrics may yield different outcomes. @xcite show that representational similarity analysis (rsa) and diagnostic classifier (dc) applied to pooled representations disagree with the results of dc applied on local representa-tions for rnn-based architectures, while they are all in agreement when applied to transformer-based representations. algayres et al. ( 2020 ) compare the aforementioned abx metric and the mean-averageprecision (map) metric (which uses representations to predict whether two stimuli have the same ground-truth label) to each other and to a downstream frequency estimation task. performance on the three metrics is correlated, but not to a high degree, and marked discrepancies are found for particular models. ## experimental results here we report experiments examining how vq layers encode phonemes in each target model according to different evaluation metrics. the impact of vq layers on performance of the visuallysupervised model in image retrieval is reported in section a.2 of the supplementary material. ## discussion to summarize, the different metrics we compared give divergent views of the same representations. this should not necessarily be interpreted as one metric being right while the others are flaweded. it is likely that the different metrics account for somewhat different properties of the representation. the differences between rsa and dc/nmi are probably related to the purity of the representation. rsa is based on the correlation of distances between pairs of stimuli and is thus sensitive to the presence of additional information in the representation. if the model's representation contains information that is not related to the representation that is the target of the analysis (in our case phonemes identification), this will be reflected in lower rsa scores. an obvious example of such information that a model of speech is likely to encode is speaker identity. as pointed out in section 4.1, the pattern of results obtained with a classifier trained to predict speaker identity supports this view. the encoding of speaker information could lead to comparatively lower scores with rsa, especially for level 1 and large codebooks where speaker identity is better encoded. in general, our results suggest that different metrics might be preferred depending on the question that one is trying to answer. rsa scores are a better indicator of the exact match between two representations while dc/nmi better evaluate the extent to which a given information can be extracted from a model's representation, irrespective of other sources of information that might be encoded at the same time. this could be confirmed through white-box experiments, where the different metrics would be applied to hand-crafted representations with different properties (e.g. in term of purity). we leave this to future work. the only remaining point of concern that arises from our results is the interaction between codebook size and input size. larger codebook sizes tend to be disadvantaged when short input segments are used as input, such as minimal pairs of phoneme triplets. this is particularly relevant for abx where the use of minimal pairs of phoneme triplets is a common practice. analysis of discrete representations using short segments should preferably be carried with diagnostic classifiers or nmi, especially if codebooks or different sizes are compared. it is also important to highlight that the importance of this effect is dependent on the architecture and the training objective of the model, as our experiments with a self-supervised model show. ## conclusion we compared four different metrics for discrete representations induced by vq layers in weaklysupervised models of spoken language, and while the results are broadly consistent, some differences did emerge. rsa tends to show a bigger gap between trained and untrained models as it is more sensitive to the purity of representations with respect to the information of interest. more surprising is the divergent results we observe when evaluation is performed on minimal pairs of phoneme trigrams: this is likely due to the skew of distance distributions with large codebooks sizes. this is an important finding as some previous work on discrete representations focused exclusively on the abx metric. in contrast, we recommend corroborating results with multiple analytical approaches, as currently their behavior in different settings is incompletely understood. overall, our findings do support the idea that vector quantization is an effective way to induce discrete representations, and that these correlate with symbolic representations assumed in linguistics. however, it is worth noting that the absolute values of our metrics measuring correspondence to phonemes are moderate at best. 7 it is thus important to keep in mind that these symbolic units are not exact analogs of the concepts familiar from linguistic theory and psycholinguistic studies.