Discrete representations in neural models of spoken language Anonymous ACL-IJCNLP submission
Abstract
The distributed and continuous representations used by neural networks are at odds with representations employed in linguistics, which are typically symbolic.
Vector quantization has been proposed as a way to induce discrete neural representations that are closer in nature to their linguistic counterparts.
However, it is not clear which metrics are the best-suited to analyze such discrete representations. We compare the merits of four commonly used metrics in the context of weakly supervised models of spoken language.
We perform a systematic analysis of the impact of (i) architectural choices, (ii) the learning objective and training dataset and (iii) the evaluation metric.
We ﬁnd that the different evaluation metrics can give inconsistent results. In particular, we ﬁnd that the use of minimal pairs of phoneme triples as stimuli during evaluation disadvantages larger embeddings, unlike metrics applied to complete utterances.
Introduction
The dominant machine learning paradigm for processing spoken language is based on neural network architectures, such as recurrent nets and transformers, inducing a hierarchy of hidden representations which are distributed and continuous.
In contrast, human history has repeatedly seen the discovery and wide adoption of discrete, symbolic representations of speech in the form of writing.
These systems commonly represent basic units of language such as morphemes, syllables or phonemes while discarding other information contained in the speech signal such as emotion or speaker identity.
Symbolic representation of speech has proven tremendously useful for storage and transmission of information, and it also plays a crucial role in systems dealing with spoken language such as spoken dialog systems: these typically employ an automatic speech recognition (ASR) module to transcribe the speech signal into written form, which is then used as input to upstream language understanding modules.
While some attempts have been made to train such systems end-to-end, pipelines are still very competitive (see Haghani et al., 2018;
Higy et al., 2020), which strengthens the point that a symbolic encoding of spoken language may be (near) optimal.
It may thus be desirable to incorporate similar representations in neural architectures.
Accordingly, multiple efforts have been made to design neural networks with discrete hidden representations and to apply them to spoken language data.
This is evident in the recent editions of the
ZeroSpeech challenge (Dunbar et al., 2019) on unit discovery, which have featured many such approaches.
Speciﬁcally, Vector Quantization (VQ) has proven to be a simple and effective method to induce discrete neural representations (e.g., van den
Oord et al., 2017; Harwath et al., 2020a; Chung et al., 2020).
The resulting symbolic codes have been claimed to correspond to phonemes and/or words, but a detailed analysis of how much the reported equivalence is affected by details of the architectures, learning objectives and dataset, as well as by the evaluation metrics used to quantify it is lacking.
The present study aims to ﬁll this gap.
We study two approaches to modeling spoken language: learning driven by language-internal structure, and learning driven by grounding in the extra-linguistic world.
These two approaches are exempliﬁed by two types of models with VQ layers: the self-supervised model for unit discovery of van Niekerk et al.
(2020), and a visually grounded model similar to Harwath et al.
(2020a).
The datasets used to train each model (Zerospeech 2020 and Flickr8K) are also typical of the task they are used for.
In addition to model type and dataset, we also systematically investigate the impact of the following factors: (i) the codebook size for the
VQ layer, and (ii) the level of placement of the VQ layer.
Furthermore, we apply and check the consistency across four different metrics for evaluating the correspondence of the representations with phonemes.
Findings
The self-supervised model shows high variability, but with some of the evaluation metrics (especially ABX and RSA, see Section 3.3) there is a trend for better correspondence to phonemes for smaller codebook sizes (32-128).
The visually grounded model, on the other hand, generally shows the closest match with phonemes for the largest codebook sizes (512 or 1024) when evaluated on full utterances.
In contrast, smaller codebooks score higher for the short utterance segments used by the ABX metric applied to minimal pairs of phoneme triples.
Thus, the conclusions drawn using a single metric, even a widely used one like the ZeroSpeech ABX metric, should be treated with caution, and further corroborated.
Related work
The domain of speech processing has seen recent efforts to modify existing neural architectures to enable the induction of discrete latent representations.
These developments are promising for boosting performance, improving interpretability, and modeling the acquisition and processing of linguistic knowledge in humans.
Applying Vector Quantization (VQ) techniques for this purpose was pioneered by van den Oord et al.
(2017), who propose generative models based on the variational auto-encoder (VAE) architecture and use VQ to induce discrete latent representations.
They apply this method to images, videos, and speech, and show that the models can learn discrete latent representations without supervision.
When applied to raw speech, the VQ-VAE architecture learns high-level discrete representations that are invariant to low-level features of the audio signal such as prosody and speaker identity, and only encode the content of the speech.
Classiﬁcation of the discrete representations into phoneme classes (based on majority ground truth label) suggests they capture phonemes to some extent.
Learning discrete instead of (or in addition to) continuous representations can facilitate unit discovery in unsupervised models of speech.
In the 2015 Zero Resource Speech Challenge (Versteegh et al., 2015), Badino et al. (2015) present a binarized auto-encoder, certain variants of which outperform their continuous counterpart. In the 2017, 2019, and 2020 editions of this challenge (Dunbar et al., 2017, 2019, 2020), following the work of van den Oord et al. (2017), several models include at least one VQ layer (see e.g. Chorowski et al., 2019;
Eloff et al., 2019; Tjandra et al., 2019, 2020).
These studies demonstrate the beneﬁt of using VQ layers for phoneme classiﬁcation and for learning speaker-invariant representations, focusing on the ABX phoneme discrimination metric to evaluate the encoding of phonemic information. However, little analysis on the impact of the size and conﬁguration of the employed VQ layers is performed.
For example, van Niekerk et al.
(2020) train a VQ-VAE model for reconstructing audio waveforms, and a VQ-CPC (Contrastive Predictive Coding) model for predicting future acoustic units.
They evaluate these architectures on the ABX phoneme discrimination task, voice conversion and speaker classiﬁcation and show that VQ-CPC performs better than VQ-VAE overall, but they do not manipulate the conﬁguration or the dimension of the VQ layers.
Chung et al.
(2020), however, report the impact on phoneme and speaker classiﬁcation of systematic manipulation of VQ-related factors.
They train an Autoregressive Predictive Coding (APC) model to predict upcoming frames, and use VQ as a methodology to limit the model’s capacity.
Using a frame-wise diagnostic classiﬁer (namely linear logistic regression), they show that under restricted conﬁgurations (only one VQ layer inserted at the top), phoneme prediction using discrete representations improves over using continuous representations learned by an APC model without VQ.
In this conﬁguration, larger code sizes lead to better performance in phoneme classiﬁcation but not in speaker identiﬁcation.
In contrast to the work cited above, which discusses uni-modal speech models, Harwath et al.
(2020a) use VQ layers in a model of visually grounded language learning, where the speech signal is associated to images.
They hypothesize that discrete representations learned by such models are more likely to capture higher-level semantic information.
Their analyses suggest that the trained multimodal models can learn discrete linguistic units at both word and sub-word levels, with quantization layers inserted at the lower levels of the network showing correspondence to sub-word (i.e. phonemic) units, and those inserted at the higher level
[Table]
[Table]
[Table] [Table]
regime corresponding to word-level units.
Analyses are based on the Zerospeech ABX metric for phoneme encoding and F1 scores for word detection.
When evaluating the encoding of phonemic information in continuous representations from models of spoken language, recent work has shown that different metrics may yield different outcomes.
Chrupała et al.
(2020) show that representational similarity analysis (RSA) and diagnostic classiﬁer (DC) applied to pooled representations disagree with the results of DC applied on local representations for RNN-based architectures, while they are all in agreement when applied to transformer-based representations.
Algayres et al. (2020) compare the aforementioned ABX metric and the mean-averageprecision (MAP) metric (which uses representations to predict whether two stimuli have the same ground-truth label) to each other and to a downstream frequency estimation task. Performance on the three metrics is correlated, but not to a high degree, and marked discrepancies are found for particular models. Table 1 summarizes some of the representative studies that use VQ layers and their speciﬁcations and reported analyses.
As can be seen from this summary, existing work on learning VQ-based discrete representations does not easily lead to a coherent picture due to the wide range of the training objectives and modeling architecture they use, the analyses they perform, the evaluation metrics they employ and the VQ-related factors they manipulate.
In this paper, we aim to provide this overview by consistently comparing discretized speech modeling approaches, architectural parameters, and evaluation metrics.
Methods 3. Vector quantization
We investigate evaluation metrics for the analysis of discrete representations induced through vector quatization.
Since phoneme classiﬁcation/identiﬁcation has been the dominant analysis task for discrete representations of speech, we use this as our main task.
We do so through the speciﬁc case of speech representations learned by two different models: a self-supervised model of speech trained to reconstruct the audio waveform, and a visually-supervised model of spoken language which maps audio representations of spoken utterances and visual representations of their corresponding images to a shared semantic space.
Both models employ VQ layers in their architecture to induce discretized representations.
A VQ layer takes as input a continuous distributed representation in the form of a vector h ∈ R d , and returns the closest of K prototype vectors contained in a trainable codebook { e 1 , e 2 , . .
, e K } where e i ∈ R d .
For a sequence of continuous vectors ( h 1 , h 2 , . . .
, h n ) the discrete codes are given by the sequence of indices of the prototype vectors returned by the
VQ layer.
Since the arg max operation needed to select the nearest vector is not differentiable, the gradient for backpropagation is approximated by using the straight-through estimator (Bengio et al., 2013), which replaces each non-differentiable operation with the identity function for the backward pass.
For further details, consult van den Oord et al.
3. Target models Self-supervised
Our self-supervised model is the
VQ-VAE model introduced in van Niekerk et al.
The model consists of an encoder built out of a stack of ﬁve convolutional layers, a bottleneck comprising a linear projection and a VQ layer, and a decoder comprising an embedding layer and a stack of upsampling and recurrent layers; the decoder attempts to reconstruct the original waveform.
For details of the architecture, see van Niekerk et al.
In the experiments reported here, we vary the size of the codebook, but keep the placement of the VQ layer constant as the encoder contains only one fully connected layer after which the VQ layer can be placed.
Visually-supervised
A visually-supervised model of spoken language with discrete representations was introduced by Harwath et al.
(2020a): they adapted an existing model (Harwath et al., 2020b) by inserting one or more VQ layers within the speech encoder stack.
We similarly adapt the architecture used in Merkx et al.
and
Chrupała et al.
(2020) by inserting a single VQ layer at one of three levels: either following the ﬁrst, second or third GRU layer of the speech encoder.
In addition to VQ layer placement, we also vary the size of the codebook.
Note that unlike Harwath et al.
(2020a)
we do not use a pre-training stage which by-passes the VQ layers; rather, we train the complete network from scratch.
This model thus consists of an image encoder, which takes as input image features extracted via a pre-trained ResNet-152 model (He et al., 2016) and maps these features via a learned afﬁne transform into a joint visual-language space.
The audio input are MFCC features with total energy and delta and double-delta coefﬁcients with combined size 39.
The speech encoder consists of one 1D convolutional layer (with 64 output channels) which subsamples the input by a factor of two, four bidirectional GRU layers (each of size 2048), with a VQ layer inserted between a single pair of GRU layers.
This stack is followed by a self-attention-based pooling layer.
The objective function is a version of the triplet loss with negative examples from the current batch.
The model is trained with the Adam optimizer (Kingma and Ba, 2015) with a cyclical learning rate schedule (Smith, 2017).
3. Evaluation methods
Different evaluation methods have been used for analyzing the nature of information captured by
VQ-based discrete representations in different studies.
We present here a thorough examination of their formal similarities and differences as well as their sensitivity to different conditions.
Below we introduce the methods commonly used to evaluate the learned representations.
Normalized Mutual Information (NMI) An information-theoretically motivated measure of the association between two random variables is mutual information. In the general case of vectorvalued neural representations, computing mutual information with the target annotation is intractable. In the special case where the representation is discrete valued, we can use the standard empirical estimate.
Given discrete random variables X with image X and Y with image Y (i.e. frame-wise codes and phoneme labels in our case), the mutual information I ( X ; Y ) is
[Equation]
It is often more informative to use mutual information normalized by the arithmetic mean of the entropies of the two random variables:
[Equation]
where H ( X ) is the entropy of X .
This deﬁnition of normalized mutual information (NMI) is equivalent to the V-measure (Rosenberg and Hirschberg, 2007).
Diagnostic Classiﬁer (DC)
A diagnostic model, also known as a probe , is a classiﬁer or regressor trained to predict some information of interest (such as a linguistic annotation) given a neural representation.
To the extent that the model successfully predicts the annotation, we conclude that the neural representation encodes this information.
Informally, such a diagnostic classiﬁer can be seen as quantifying the amount of easily-accessible – or in the extreme case, linearly decodable – information about the target annotation (Adi et al., 2017;
Alishahi et al., 2017; Hupkes et al., 2018; Conneau et al., 2018, among others).
As argued by Pimentel et al.
(2020), without the qualiﬁcation that information be easily accessible, probing should aim to approximate the mutual information between the neural representation and the target annotation, and thus should use the bestperforming probe possible.
Furthermore, it is not possible for the neural representation to contain more information about the target annotation than the source utterance itself, due to the information processing inequality, and thus, in the general case, probing with an unrestricted classiﬁer is not a wellfounded exercise.
In the special case of probing a discrete-valued variable the situation is simpler: the accuracy of a linear classiﬁer is closely related to the empirical estimate of the mutual information between the representation and the target annotation; see the formal argument in Appendix A.1 as well as our empirical results.
Representational Similarity Analysis (RSA)
RSA is a second-order technique originating in neuroscience (Kriegeskorte et al., 2008) where similarities between pairs of stimuli are measured in two representation spaces: e.g. neural activation pattern space and a space of symbolic linguistic annotations such as sequences of phonemes or syntax trees.
The correlation between these pairwise similarity measurements quantiﬁes how much the two representations are aligned.
This approach requires a similarity or distance metric for pairs of stimuli within each representation space, but does not need a way of mapping from one space to the other.
It generally does not have any trainable parameters.
As a consequence, it is sensitive to the purity of the representation with regard to the variable of interest.
See for example Bouchacourt and Baroni (2018); Chrupała and Alishahi (2019); Abnar et al.
; Abdou et al.
; Chrupała et al.
Fereidooni et al.
; Davis and van Schijndel (2020) for uses in NLP and speech processing.
Global versus local scope
[Equation] study the effect representation scope, i.e. activation vectors retrieved at the level of frames (local) or pooled over whole utterances (global), concluding that it can affect results. Following their recommendations, and for the sake of simplicity, we include one measure for each scope.
Since their ﬁndings suggest that local RSA lacks sensitivity, we use local DC (which is also the most widely used) as well as global RSA in our experiments. Global RSA has the double advantage of not using any trainable parameters and not requiring any alignments. ABX discriminability (ABX) The
ABX phoneme discriminability metric (Schatz, 2016) as used in the Zerospeech challenge (Dunbar et al., 2019) is based on triples of stimuli ( A, B, X ) where A and X belong to the same category and
B and X belong to different categories.
The ABX error is a function of d ( A, X ) and d ( B, X ) where d ( · , · ) is a distance metric for the representation being evaluated: [Equation]
[Equation]
The categories are determined by gold annotation: in the case of Zerospeech they are phoneme labels.
The stimuli are presented in context in the form of minimal pairs: ( A = /beg/ 1 ,
B = /bag/,
X = /beg/ 2 ), where /beg/ 1 and /beg/ 2 are two different utterances of this phoneme sequence.
In our use case, alignments between the gold phoneme transcriptions and the evaluated representations are required to extract the stimuli for ABX.
Relation to RSA
The ABX error is loosely related to the RSA score.
With RSA, pairwise distances are measured between gold representations of stimuli (e.g. their phonemic transcriptions) as well as between system representations of the same set of stimuli.
The correlation coefﬁcient between these two sets of distance measurements is the RSA score.
With RSA there is no notion of a stimulus triple, but rather the score reﬂects distances between all pairs of stimuli.
Likewise, the representation of stimuli according to the gold standard is typically not in the form of atomic categorical labels but can be any representation with an associated distance (or similarity) metric.
Thus RSA can be seen as more general than ABX, while being less controlled.
Distance metrics for RSA and ABX
When the neural representations of the stimuli evaluated are sequences of vectors, we need to make a choice regarding how to measure similarities or distances between them.
Here we focus on neural representations which take the form of sequences of symbolic codes, which makes measuring distances simple: a natural choice is the Levenshtein edit distance normalized by the length of the longer string.
We can
[Table]
thus apply the same edit-distance metric on both the neural representations and on the reference sequences of phonemes or words (for efﬁciency we collapsed code repetitions).
Summary of metrics
Table 2 summarizes the main characteristics of the evaluation metrics described above in the context of analyzing neural representations of speech, along the following facets: the need to arrange input in the form of minimalpaired stimulus triples, the need for alignment between input/codes and phonemic transcriptions, the presence of trainable parameters, and reliance on a distance metric between stimuli.
According to these criteria, RSA and NMI are the least restricted in their applicability, requiring only a distance function or alignment, respectively.
In addition to a distance metric, ABX needs minimal-paired stimulus triples to be extracted.
In addition to alignment DC has trainable parameters, and in the case of analyzing discrete codes, it behaves like an approximation to the NMI metric.
3. Evaluation procedure
We evaluate the induced discrete representations by applying the trained networks on the relevant examples – either full utterances, or speech segments corresponding to sequences of three phonemes (triplets) – and extract the sequences of codes from the VQ layer.
We do the same for randomly initialized (untrained) versions of the networks, in order to provide a baseline score, following the methodology of Chrupała et al.
In Section 4, we include in the plots both the baseline scores and the scores with the trained models.
For the selfsupervised target model we vary only the size of the codebook in the VQ layer, 3 using sizes 2 n for n ∈ [5 , 10] .
For the visually-supervised target model we used the same sizes and also vary the placement of the VQ layer between one of three levels (following ﬁrst, second or third GRU layer).
For both target models, each variant was trained three times with a different random initialization; the scatter plots in Section 4 show each of these runs, as well as a LOESS ﬁt (Cleveland, 1979) to each combination of codebook size and level.
3. Datasets Training target models
Following van Niekerk et al.
we train the self-supervised model on about 15 hours of speech from over 100 speakers provided by Zerospeech 2020.
The visuallysupervised model is trained on the Flickr8K Audio Caption Corpus (Harwath and Glass, 2015;
Rashtchian et al., 2010) 5 , which consists of 8, images each paired with ﬁve spoken captions.
The training portion of this dataset contains 6,000 images and about 34 hours of speech.
Evaluation
We encode the development captions of Flickr8K (5,000 captions) using the encoders of the trained and untrained target models, and for each utterance extract the sequence of codes output by the VQ layer.
We split this data in half, and use one half for training the DC, and the other half for computing the scores for DC, RSA and NMI.
As ABX (and one experiment with RSA) is not computed on full utterances but on phoneme trigrams, we prepare this data by sampling 1,000 captions from the Flickr8K development set, and cutting the audio into non-overlapping segments corresponding to a sequence of three phonemes.
We then use the ZeroSpeech code to generate minimal-pair stimulus triples.
In order to obtain reference phonemic transcriptions we use forced alignment with the Gentle toolkit, 6 based on Kaldi (Povey et al., 2011).
This fails for a small number of utterances, which we remove from the data.
3. Repository
Upon acceptance of the paper, the code enabling the replication of our experiments will be available on GitHub under Apache License 2.0.
Experimental results
Here we report experiments examining how VQ layers encode phonemes in each target model ac-
cording to different evaluation metrics.
The impact of VQ layers on performance of the visuallysupervised model in image retrieval is reported in Section A.2 of the Supplementary Material.
4. Visually-supervised representations
We extract codes from visually-supervised VQ models trained on the Flickr8K data while varying VQ layer placement and codebook size.
We evaluate how much these codes correspond to phonemes according to four metrics: DC, NMI, RSA and ABX.
These results are shown in Figure 1.
DC vs. NMI
As expected on theoretical grounds, diagnostic classiﬁers and mutual information give very similar results.
Overall, VQ layers at leveland 2 perform signiﬁcantly better than VQ layers at level 3 or equivalent untrained models.
Larger codebook sizes also tend to perform better than smaller ones.
DC and NMI vs. RSA
RSA differs from the other two metrics on three main aspects: (i) VQ layers at level 3 perform comparably to level 2, (ii) medium codebook sizes give better performances for VQ layers at level 1, and (iii) untrained models show very poor performance.
The last point can be explained by the sensitivity of RSA to the purity of the representations.
While representations extracted from untrained models can still contain meaningful information, it will be less explicit and mixed with information that is not relevant for the task of interest.
A similar explanation might also hold for the ﬁrst point; in particular, we observed that the VQ layers at level 1 retain much more information on the speaker than the other two levels (see Section A.4 of the Supplementary Material for related experiments).
ABX vs. rest
ABX is the most divergent metric.
While VQ layers at level 1 and 2 still perform better, the gap with equivalent untrained models is smaller.
The effect of codebook size is also much less pronounced and layer-speciﬁc.
The difference in patterns of results between the metrics might be due to the different testing stimuli used by ABX versus the other three metrics: whereas DC, NMI and RSA are tested on full utterance audio ﬁles, ABX is tested on small phoneme trigram ﬁles.
Role of stimulus size
To disentangle the impact of the metric and the size of the stimulus it is applied on, we run two additional experiments.
First, [Table] we re-calculate RSA using the same phoneme trigram ﬁles used for ABX.
The correlation coefﬁcient between ABX and this version of RSA is much stronger (see Table 3), suggesting that the type of stimulus used to test the model does play a role.
The other experiment goes in the opposite direction and brings the ABX evaluation closer to the other three metrics. Training the target model on full sentences but applying it to short segments could play a role. Thus, we run an additional set of experiments where we apply the models to full utterances and generate the representation used in
ABX by extracting the portion of the code sequence corresponding to each phoneme trigram from the full sequence of activations.
The correlation with
RSA is still low ( 0 . 14 ) indicating that the problem is intrisic to the evaluation relying on phoneme triplets and train/test mismatch. This impact of stimulus size on results is likely related to the fact that, with very short stimuli, most normalized edit distances will be maximum, or near maximum, and this will especially be the case for large codebook sizes, giving very skewed and long-tailed edit distance distributions (see details in Section A.3 in the Supplementary Material).
4. Self-supervised representations
We extract codes from a number of self-supervised
VQ-VAE models trained on the Zerospeechchallenge dataset with varying codebook sizes (note that the VAE model has only one possible placement for the VQ layer).
We compare how well these codes correspond to phonemes according to the same four metrics, as displayed in Figure 1.
In contrast to what we see for visually-supervised representations, here RSA and ABX scores are largely consistent and suggest that a larger codebook leads to weaker encoding of phonemic information. The effect is more pronounced for ABX though with [Figure] [Table]
[Figure]
[Table] [Figure] the largest codebook performing similarly to the baseline. DC and NMI are relatively insensitive to the size of the codebook.
The self-supervised model does not show the discrepancy observed with the visually-supervised model when RSA and ABX scores are run on testing input of different size.
This is conﬁrmed by the correlation coefﬁcients between the ABX score and RSA computed on complete utterances and phoneme triplets shown in Table 3.
The VQ layer in the self-supervised model only has access to a limited context, which provides enough information to subsequently reconstruct the input audio frame.
The visually-supervised architecture on the other hand builds a representation for the whole utterance, supported by recurrent layers.
This core difference may explain the pattern of results we report here.
Conclusion
We compared four different metrics for discrete representations induced by VQ layers in weaklysupervised models of spoken language, and while the results are overall consistent, some differences did emerge.
RSA tends to show a bigger gap between trained and untrained models as it is more sensitive to the purity of representations with respect to the information of interest.
More surprising is the divergent results we observe when evaluation is performed on minimal pairs phone trigrams: this is likely due to the skew of distance distributions with large codebooks sizes.
This is an important ﬁnding as some previous work on discrete representations focused exclusively on the ABX metric.
In contrast we recommend corroborating results with multiple metrics whenever possible, as currently their behavior in different settings is incompletely understood.
Overall, our ﬁndings do support the idea that vector quantization is an effective way to induce discrete representations, and that these correlate with symbolic representations assumed in linguistics.
However, it is worth noting that the absolute values of our metrics measuring correspondence to phonemes are moderate at best.
It is thus important to keep in mind that these symbolic units are not exact analogs of the concepts familiar from linguistic theory and psycholinguistic studies.
A Supplementary material A. Loss of a logistic diagnostic classiﬁer
The most general measure of the amount of information about the value of a random variable Y obtained through the observation of the value of random variable X is Mutual Information
I ( Y ; X ) .
Here we relate the loss of a logistic diagnostic classiﬁer predicting Y from X in the special case where both Y and X are discrete, with image Y and X respectively.
We can construct a logistic classiﬁer which outputs the empirical probability ˆ P ( Y = y | X = x ) with y ∈ Y and x ∈ X , by using a one-hot encoding of the categorical predictor variable X as x and by setting the classiﬁer coefﬁcients W as
[Equation]
The softmax of the logistic classiﬁer with these coefﬁcients simpliﬁes to the empirical estimates of conditional probabilities of Y :
[Equation]
The cross entropy of the predictions of this classifer is then:
[Equation]
where y n and x n are the values taken by the random variables Y and X for the n th example.
The loss is equivalent to the empirical estimate of the conditional entropy
H
( Y | X ) , and related to the mutual information between I ( Y ; X ) via:
[Equation]
To the extent that the scores of the logistic diagnostic classiﬁer and normalized mutual information applied to the same data are not perfectly correlated, this would be due to the stochasticity of training, regularization, as well as the use of accuracy rather than cross entropy to measure performance.
A. Overall performance of the visually supervised model
Since the visually-supervised model is trained and optimized for matching images and their corresponding spoken captions, we measure the recall@10 of retrieving the correct image for a given [Figure] Model Input
Size
Skew
Kurtosis SS complete [Table] [Table]
VS complete
[Table]
VS triplet
[Table] [Table] VS complete [Table]
VS triplet
[Table]
spoken utterance as a function of the size of the learned codebook and the placement of the VQ layer.
Figure 2 shows these results.
We observed the following patterns:
• Most models perform worse than a model without the VQ layer, with the exception of the models with the VQ layer at level 1 and a codebook of size 512 or 1024 .
• Performance on the image retrieval task is negatively correlated with the level of placement of the VQ layer.
• VQ layers with larger codebooks perform better.
A. Edit distance distribution
Table 4 presents skew and excess kurtosis of edit distance distributions for both target models, using codebooks of size 32 and 1024 and trained on long and short segments, conﬁrming the hypothesis that the combination of short segments and large codebooks leads to skewed distributions.
A. Speaker identiﬁcation
Figure 3 shows accuracy of diagnostic classiﬁers trained on code sequences encoded as vectors of code frequencies.
For visually-supervised models, speaker identity is represented to some degree in untrained models, for codebooks of all sizes and at all levels, and most strongly for large codebook sizes at level 1.
After training, we observe differentiated patterns for VQ layers at level 1 versus 2 and 3.
While speaker identity is emphasized in codebooks at level 1 compared to the untrained models, it is weakened for subsequent layers, to the point of being effectively removed at level 3.
These results indicate that VQ layers at level 1 represent speaker-dependent information, possibly encoding acoustic rather than phonemic information.
The self-supervised models show results similar to the visually-supervised models with the VQ layer at level 1 when trained, but capture nearly no speaker information before training
[Figure]