B EYOND C ATEGORICAL L ABEL R EPRESENTATIONS FOR I MAGE C LASSIFICATION
creativemachineslab.com/label-representation.html
A BSTRACT
We ﬁnd that the way we choose to represent data labels can have a profound effect on the quality of trained models.
For example, training an image classiﬁer to regress audio labels rather than traditional categorical probabilities produces a more reliable classiﬁcation.
This result is surprising, considering that audio labels are more complex than simpler numerical probabilities or text.
We hypothesize that high dimensional, high entropy label representations are generally more useful because they provide a stronger error signal.
We support this hypothesis with evidence from various label representations including constant matrices, spectrograms, shufﬂed spectrograms, Gaussian mixtures and uniform random matrices of various dimensionalities.
Our experiments reveal that high dimensional, high entropy labels achieve comparable accuracy to text (categorical) labels on the standard image classiﬁcation task, but features learned through our label representations exhibit more robustness under various adversarial attacks and better effectiveness with a limited amount of training data.
These results suggest that label representation may play a more important role than previously thought
I NTRODUCTION
Image classiﬁcation is a well-established task in machine learning.
The standard approach takes an input image and predicts a categorical distribution over the given classes.
The most popular method to train these neural network is through a cross-entropy loss with backpropagation.
Deep convolutional neural networks (Lecun et al., 1998; Krizhevsky et al., 2012; Simonyan & Zisserman, 2014;
He et al., 2015; Huang et al., 2016) have achieved extraordinary performance on this task, while some even surpass human level performance.
However, is this a solved problem?
The state-of-theart performance commonly relies on large amounts of training data (Krizhevsky, 2009;
Russakovsky et al., 2015; Kuznetsova et al., 2018), and there exist many examples of networks with good performance that fail on images with imperceptible adversarial perturbations (Biggio et al., 2013; Szegedy et al., 2013; Nguyen et al., 2014).
Much progress has been made in domains such as few-shot learning and meta-learning to improve the data efﬁciency of neural networks.
There is also a large body of research addressing the challenge of adversarial defense.
Most efforts have focused on improving optimization methods, weight initialization, architecture design, and data preprocessing.
In this work, we ﬁnd that simply replacing standard categorical labels with high dimensional, high entropy variants (e.g. an audio spectrogram pronouncing the name of the class) can lead to interesting properties such as improved robustness and efﬁciency, without a loss of accuracy.
Our research is inspired by key observations from human learning.
Humans appear to learn to recognize new objects from few examples, and are not easily fooled by the types of adversarial perturbations applied to current neural networks.
There could be many reasons for the discrepancy between how humans and machines learn.
One signiﬁcant aspect is that humans do not output categorical probabilities on all known categories.
A child shown a picture of a dog and asked “what is this a picture of?” will directly speak out the answer — “dog.”
Similarly, a child being trained by a parent is shown a picture and then provided the associated label in the form of speech.
These observations raise the question: Are we supervising neural networks on the best modality?
[Figure]
In this paper, we take one step closer to understanding the role of label representations inside the training pipeline of deep neural networks.
However, while useful properties emerge by utilizing various label representations, we do not attempt to achieve state-of-the-art performance over these metrics.
Rather, we hope to provide a novel research perspective on the standard setup.
Therefore, our study is not mutually exclusive with previous research on improving adversarial robustness and data efﬁciency.
An overview of our approach is shown in Figure 1.
We ﬁrst follow the above natural observation and modify the existing image classiﬁers to “speak out” the predictions instead of outputting a categorical distribution.
Our initial experiments show surprising results: that neural networks trained with speech labels learn more robust features against adversarial attacks, and are more data-efﬁcient when only less than 20% of training data is available.
Furthermore, we hypothesize that the improvements from the speech label representation come from its property as a speciﬁc type of high-dimensional object.
To test our hypothesis, we performed a large-scale systematic study with various other high-dimensional label representations including constant matrices, speech spectrograms, shufﬂed speech spectrograms, composition of Gaussians, and high dimensional and low dimensional uniform random vectors.
Our experimental results show that high-dimensional label representations with high entropy generally lead to robust and data efﬁcient network training.
We believe that our ﬁndings suggest a signiﬁcant role of label representations which has been largely unexplored when considering the training of deep neural networks.
Our contributions are three fold.
First, we introduce a new paradigm for the image classiﬁcation task by using speech as the supervised signal.
We demonstrate that speech models can achieve comparable performance to traditional models that rely on categorical outputs.
Second, we quantitatively show that high-dimensional label representations with high entropy (e.g. audio spectrograms and composition of Gaussians) produce more robust and data-efﬁcient neural networks, while highdimensional labels with low entropy (e.g. constant matrices) and low-dimensional labels with high entropy do not have these beneﬁts and may even lead to worse performance.
Finally, we present a set of quantitative and qualitative analyses to systematically study and understand the learned feature representations of our networks.
Our visualizations suggest that speech labels encourage learning more discriminative features.
R ELATED W ORKS
Data Efﬁciency and Robustness Data efﬁciency has been a widely studied problem within the context of few-shot learning and meta-learning (Thrun & Pratt, 2012; Vilalta & Drissi, 2002; Vanschoren, 2018;
Wang et al., 2019).
Researchers have made exciting progress on improving methods of optimization (Ravi & Larochelle, 2016; Li et al., 2017), weight initialization (Finn et al., 2017;
Ravi & Larochelle, 2017), and architecture design (Santoro et al., 2016a;b).
There is also a large body of research addressing the challenge of adversarial defense.
Adversarial training is perhaps the most common measure against adversarial attacks (Goodfellow et al., 2014;
Kurakin et al., 2016; Szegedy et al., 2013; Shaham et al., 2018; Madry et al., 2017).
Recent works try to tackle the problem by leveraging GANs (Samangouei et al., 2018), detecting adversarial examples (Meng & Chen, 2017;
Lu et al., 2017; Metzen et al., 2017), and denoising or reconstruction (Song et al., 2017;
Liao et al., 2017).
Most of these techniques for improving data efﬁciency and adversarial robustness study the problem from the perspective of model, optimizer and data.
Relatively little research has been conducted on the labels themselves or more speciﬁcally, their representation.
Hosseini et al.
(2017) augmented categorical labels with a new NULL class to allow the model to classify and reject perturbed examples.
Papernot et al.
(2015) utilizes model distillation (Hinton et al., 2015) for adversarial defense.
Papernot & McDaniel (2017) further augment the label used to train the distilled model with the predictive uncertainty from the original model.
Nevertheless, the method of Hosseini et al.
(2017) requires adversarial examples to train on while that of defensive distillation has been shown to be vulnerable to substitute model black-box attacks (Papernot et al., 2017).
Label Smoothing The closest approach to our work is Label Smoothing (LS) (Szegedy et al., 2016).
Here we highlight key differences between our approach and LS.
First, LS applies to discriminative outputs where both correct and incorrect class information are presented during training, while our output is generative and only correct class information is presented.
That is, our outputs are not a distribution over classes.
Although LS has been shown to improve adversarial robustness (Goibert & Dohmatob, 2020), it has not been shown to be effective for low-data learning.
As we will show in our experiments, LS does not help when the amount of training data is limited, while our label representations lead to signiﬁcant improvements.
Therefore, our high-dimensional, high-entropy labels provide beneﬁts beyond those provided by label smoothing.
B EYOND A CCURACY : E MERGENCE OF R OBUSTNESS AND E FFICIENCY
It is well-known that deep neural networks with similar accuracy on the same task may perform very differently under different evaluation scenarios.
Additionally, real-world applications rely on more considerations than just accuracy.
Robustness and data efﬁciency are two practical challenges for deep neural networks.
We test the emergence of those properties under various label representations.
3. R OBUSTNESS UNDER A DVERSARIAL A TTACKS
We evaluate the robustness of all the trained models using the fast gradient sign method (FGSM)
(Goodfellow et al., 2014) and the iterative method (Kurakin et al., 2016) across multiple widely used convolutional networks.
We choose these attacks because their adversarial images I adv are usually indistinguishable from the original images I or do not signiﬁcantly affect human evaluation, but they can be very challenging for neural networks to correctly classify.
When a loss function J is involved in generating the adversarial images, we use the cross-entropy loss for the text model and the smooth L1 loss for the speech model.
FGSM is a fast one-step attack that generates an adversarial image by adding a small adversarial perturbation to the original image.
The perturbation is based on the gradient of the loss with respect to the original image.
The maximum magnitude of the perturbation is maintained by (cid:15) :
[Equation]
We test both untargeted and targeted versions of FGSM.
The untargeted attacks increase the loss between the predicted class and the true class Y true :
[Equation]
whereas the targeted attacks decrease the loss between the predicted class and a target class Y target :
[Equation]
We choose a random incorrect class as the target class for each input image, and the same target classes are used to test different models.
All I adv are normalized after the perturbation.
Iterative Method As an extension to FGSM, the iterative method applies multiple steps of gradientbased updates.
In our experiments, we initialize the adversarial image I adv to be the original image
I so that I
adv
=
I .
Then we apply FGSM for 5 times with a small step size α =
(cid:15)/ 5 .
The untargeted update for each iteration becomes
[Equation]
and the targeted update becomes
[Equation]
where Clip I,(cid:15) denotes clipping the total perturbation
I N adv − I in the range of [ − (cid:15), (cid:15) ] .
We use the same targeted classes from FGSM for the evaluation on the iterative method.
3. L EARNING E FFICIENCY WITH L IMITED D ATA
We take the most straightforward approach to evaluate data efﬁciency.
We start with only 1% of the original training data.
We always use the full amount of testing data.
We then gradually increase the amount of training data to 2% , 4% , 8% , 10% , and 20% of the original to perform extensive multi-scale evaluation.
E XPERIMENTAL S ETUP
Dataset We evaluate our models on the CIFAR-10 and CIFAR-100 datasets (Krizhevsky, 2009).
We use the same training, validation and testing data split ( 45 , 000 / 5 , 000 / 10 , 000 ) for all of our experiments.
We also keep the same random seed for data preprocessing and augmentation.
Therefore, we present apple to apple comparisons for all label representations.
Speech Label Generation We generate the speech labels shown in Figure 1 by following the standard practice as in recent works (Naranjo-Alcazar et al., 2019; Zhang et al., 2019):
• We ﬁrst generate the English speech audio automatically with a text-to-speech (TTS) 1 system from the text labels in the corresponding dataset.
Therefore, all the speech labels are pronounced consistently by the same API with the same parameters for controlled experiments.
We leave the exploration of different languages and intonations as future
work. • We save each audio ﬁle in the WAVE format with the 16-bit pulse-code modulation encoding, and trim the silent edges from all audio ﬁles.
• Since different speech signals may last for various lengths, we preprocess 2 each speech label to generate a Log Mel spectrogram to maintain the same dimension.
We use a sampling rate of 22 , 050 Hz, 64 Mel frequency bands, and a hop length of 256 .
Another advantage of this preprocessing into spectrograms is that we can then utilize convolutional neural networks as the speech decoder to reconstruct our speech labels.
Meanwhile, we convert the amplitudes to the decibel scale.
Finally, the spectrograms are shaped into a N × N matrix with values ranging from − 80 to 0 , where N is double the dimension of the image input.
Our resulting speech spectrogram can be viewed as a 2D image.
Given the ﬁrst step in this procedure, note that for a given class (e.g. ”bird”) there is only one corresponding spectrogram.
Therefore, the improved robustness we observe is not a result of any label-augmentation.
Other Labels For a deeper understanding of which properties of speech labels introduce different feature learning characteristics, we replicate all the experiments using the following highdimensional label variants.
We also show visualizations for all the label variants in Figure 1.
• shufﬂed-speech
We cut the speech spectrogram image into 64 slices along the time dimension, shufﬂe the order of them, and then combine them back together as an image.
Although the new image cannot be converted back to a meaningful audio, it preserves the frequency information from the original audio.
More importantly, this variant does not change the entropy or dimensionality of the original speech label.
• constant-matrix
In contrast, the constant matrix represents another extreme of highdimensional label representation, with all elements having the same value and zero entropy.
The constant-matrix label has the same dimension as the speech label, and values are evenly spaced with a range of 80 (which is also the range of the speech labels).
• Gaussian-composition
We obtain this representation by plotting a composition of 2D Gaussians directly as images.
Each composition is obtained by adding 10 Gaussians with uniformly sampled positions and orientations.
• random/uniform-matrix
We also adopt a matrix with same dimensions as the above highdimensional labels by randomly sampling from a uniform distribution.
We additionally construct a uniform random vector with low dimensionality to inspect whether dimensionality matters.
Throughout the paper, we will use random matrix and uniform matrix interchangeably to refer to the same representation.
• BERT embedding We obtain the BERT embedding from the last hidden state of a pretrained BERT model (Devlin et al., 2018; Wolf et al., 2020).
We remove outlines larger than twice the standard deviation and normalize the matrix to the same range of our other high-dimensional labels.
The BERT embedding results in a 64 × 64 matrix.
• GloVe embedding Similarly, we directly use the pretrained GloVe (Pennington et al., 2014) word vectors.
This results in a 50-dimensional label vector.
Models We take three widely used convolutional networks as our base model: VGG19 (Simonyan & Zisserman, 2014), ResNet-32 and ResNet-110 (He et al., 2015).
Here we deﬁne that a categorical classiﬁcation model has two parts: an image encoder I e and a category (text) decoder I td .
Traditionally, I td represents fully connected layers following various forms of convolutional backbones
I e .
Finally, a softmax function is applied to the output from the last layer of I td to convert the predictions to a categorical probability distribution.
The prediction is retrieved from the class with the highest probability.
Similarly, we deﬁne that the models for our high-dimensional labels consists of two parts: an image encoder and a label decoder I ld .
Throughout the paper, we use the same image encoder but replace the category decoder I td with one dense layer and several transpose convolutional layers as I ld .
All the layers inside I ld are equipped with batch normalization (Ioffe & Szegedy, 2015) and leaky ReLU with a 0 .
01 negative slope.
Overall, the majority of parameters of the network comes from the image encoder which is shared across both categorical labels and other label representations.
The decoder for high-dimensional labels increases the number of parameters by a very limited amount (see Appendix for all the numbers).
Thus, our experiments are well-controlled with respect to the number of parameters.
Learning We train the categorical model to minimize the traditional cross-entropy objective function.
We minimize Equation 6 for other models with high-dimensional labels.
We use the smooth
L1 loss (Huber loss)
L s shown in Equation 7.
Here, y i is the predicted label matrix while s i stands for the ground-truth matrix.
[Equation]
We optimize all the networks using stochastic gradient descent (SGD) with back-propagation, and we select the best model based on the accuracy on the validation set.
Evaluation For categorical models, a prediction is considered correct if the class with the highest probability indicates the same category as the target class.
For high-dimensional labels, we provide two types of measurements.
Given the model output, we select the ground-truth label that minimizes the distance to the output.
We refer to this as the ”nearest neighbor” (NN) choice.
The other criteria is to check whether the smooth L1 loss is below a certain threshold.
We use Amazon Mechanical
Turk (Sorokin & Forsyth, 2008) to validate that generated speech below our threshold is correctly identiﬁable by humans.
In our experiments, we ﬁnd 3 .
5 is a reasonable threshold.
The human evaluations on the speech label demonstrate that our metric captures both the numerical performance and the level of interpretability of the generated speech output.
Note that we mainly rely on the NN method for evaluation and only refer to the threshold method to demonstrate the qualitative results from the speech label.
R ESULTS AND D ISCUSSION 5. D O A LL THE M ODELS L EARN TO C LASSIFY I MAGES ?
We report the classiﬁcation accuracy for all the labels in Table 1.
Speech labels, shufﬂed-speech and
Gaussian-composition labels achieve comparable accuracy with the traditional categorical labels, while the constant-matrix performs slightly worse than these representations.
This suggests that the constant-matrix label is harder to train with.
We verify this observation by visualizing the training curves for all label representations on CIFAR-100 with the ResNet-110 image encoder in Figure 2.
The training curves show that the constant-matrix model takes longer to converge than the others, and converges to a higher loss
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table]
[Figure] [Table]
5. F EATURE R OBUSTNESS
In order to evaluate the robustness of the models we take all the trained models (Table 1) as the starting points for adversarial attacks with the FGSM and the iterative method.
We apply an (cid:15) from 0 to 0 .
3 with a 0 .
05 increment to the normalized images by following the original FGSM setup (Goodfellow et al., 2014) and test the model accuracy for each (cid:15) value.
We only run attacks on the original correctly classiﬁed images and mark the original misclassiﬁed images as incorrect when we compute the accuracy for all (cid:15) values.
We provide the accuracy computed on the subset of test images that are initially correctly classiﬁed in the Appendix, though the ranking among different models remains the same.
Figure 3 shows the test accuracy under various attacks.
Although the accuracy of all models decreases as the attack becomes stronger (larger (cid:15) ), the models with speech labels, shufﬂed speech labels, and composition of Gaussian labels perform consistently much better than the models with traditional categorical labels across all three image encoders, for all types of attacks, and on both CIFAR datasets.
Uniform random matrix labels perform similarly well in this setting (see the Appendix for details).
Interestingly, models with constant matrix labels perform worse than all other models with high-dimensional labels, suggesting that there are some inherent properties that enhance model robustness beyond simply high dimensionality.
[Table]
Uniform-Random CompositeSpeechShuffleLowdimCategoryConstant FGSM (Untargeted) FGSM (Targeted)Iterative (Untargeted)Iterative (Targeted)
R e s N e t R e s N e t
V GG
Epsilon
T e s t
A cc u r a c y ( % ) 5. F EATURE E FFECTIVENESS
With the CIFAR-10 dataset, we train models with various label representations using 1% , 2% , 4% , 8% , 10% , and 20% of the training data.
For each amount of data, we train with the VGG19, ResNet32, and ResNet-110 image encoders with ﬁve different random seeds.
To conduct controlled experiments, we use the exact same training procedures and hyperparameters as the full-data experiments, so that the only difference is the amount of training data.
All models are evaluated on the same validation set and test set.
Figure 4 reports the test accuracy.
Similar to the results from the robustness evaluation, speech labels, shufﬂed speech labels, composition of Gaussian labels, and uniform random labels achieve higher accuracies than the models with categorical labels for both VGG19 and
ResNet-110, and comparable results for ResNet-32.
The results demonstrate that the speech models are able to learn more generalizable and effective features with less data.
This property is extremely valuable when the amount of training data is limited.
Additionally, our results suggest that label-smoothing does not provide further beneﬁts when the amount of training data is limited, as discussed above.
Lastly, the performance of the models trained on constant matrix labels is consistent with that in the robustness experiment: it performs worse than all other high-dimensional labels.
We provide further analysis in the next section.
Uniform-Random
CompositeSpeechShuffleBERTLowdimGloVeCategory
[Figure]
5. W HAT IS S PECIAL ABOUT A UDIO L ABELS ?
Our experiments for robustness and data efﬁciency suggest that high-dimensional labels hold some interesting inherent property beyond just high-dimensionality that encourage learning of more robust and effective features.
We hypothesize that high-dimensional label representations with high entropy provide stronger learning signals which give rise to better feature representations.
To verify our hypothesis [Table] we measure several standard statistics over various label representations, shown in Table 2.
Speciﬁcally [Table] we measure the normalized L1 and L2 distance between pairs of labels for each representation.
We further measure the entropy for each individual label
[Table]
also have higher entropy than other label forms. Interestingly, although the Manhattan and Euclidean distance between pairs of labels do not show any particularly useful patterns, the average entropy of the speech labels, the shufﬂed speech labels, and composition of Gaussian labels are all higher than that of the constant matrix and original categorical labels.
The ranking of the entropy between these two groups exactly matches the performance in our robustness and data efﬁciency experiments shown in Figure 3 and Figure 4.
This correlation suggests that high dimensional labels with high entropy may have positive impacts on robustness and data-efﬁcient training.
We further validate that the beneﬁts come from both high dimensionality and high entropy by training a model with low-dimensional and high-entropy labels.
We generated these labels by sampling from a uniform distribution, following the same procedure as the uniform-random matrix label described previously.
We found that while models trained with this label perform comparably to ones trained with high-dimensional high-entropy labels in terms of adversarial robustness (see
Appendix), high-dimensional and high-entropy label models outperform the low-dimensional highentropy model in terms of data efﬁciency, as shown by the curve ”Low-dim” in Figure 4.
We ﬁnd a similar result for categorical models trained with label smoothing, which has been previously shown to improve adversarial robustness (Goibert & Dohmatob, 2020).
In fact, high dimensionality is a prerequisite for high entropy because the maximum entropy is limited by the dimensionality of the label.
Note that the model trained with label smoothing uses the standard cross-entropy loss, meanwhile the low-dimensional high-entropy model is trained with the Huber loss.
As a result, we argue that the loss is not responsible for the improved performance of models trained with high-dimensional, high-entropy labels.
5. V ISUALIZATIONS
Throughout the training process, we visualized the learned features immediately after the image encoder layer of ResNet-110 with t-SNE (van der Maaten & Hinton, 2008), both for audio and categorical models on CIFAR-10 test set.
The results are shown in Figure 5.
We observe that the embedding of the learned features evolve as the training progresses.
Compared with the feature embedding of the categorical model, the embedding of the speech models shows the formation of clusters at earlier stages of training.
More separated clusters are also obtained towards convergence.
We provide further visualizations and Grad-CAM interpretations in the Appendix.
C ONCLUSION
We introduce a novel paradigm for the traditional image classiﬁcation task by replacing categorical labels with high-dimensional, high-entropy matrices such as speech spectrograms.
We demonstrate comparable accuracy on the original task with our speech labels, however, models trained with our speech labels achieve superior performance under various adversarial attacks and are able to learn in a more data efﬁcient manner with only a small percentage of training data.
We further study the inherent properties of high dimensional label representations that potentially introduce the advantages.
Through a large scale systematic study on various label representations, we suggest that high-entropy, high-dimensional labels generally lead to more robust and data efﬁcient training.
Our work provides novel insights for the role of label representation in training deep neural networks.
A CKNOWLEDGMENTS
This research is supported by NSF NRI 1925157 and DARPA MTO grant L2M Program HR001118-2-0020.
A A PPENDIX A. T HRESHOLD V ALIDATION
We deployed a large-scale study on Amazon Mechanical Turk 4 to validate our choice of 3 . 5 as a classiﬁcation threshold for the speech model.
In particular, we asked workers to listen to the outputs of the speech model and choose from the set of classes (with a ”None” class for unintelligible output) the one which best ﬁts the output.
We assigned 3 workers to evaluate each output from the VGG19 speech model on CIFAR-10 test set.
We chose a restrictive selection criteria to ensure maximum quality of responses.
Only workers with a ≥ 99% approval rating and at least 10 , 000 approvals were selected.
To measure agreement between humans and our speech model, for each sample in the test set we determine the decision made by the model using our pre-selected threshold (loss < 3 . 5 is correct, while loss ≥ 3 . 5 is incorrect).
Then we compare these decisions to those of the human workers.
When we count each of the three workers independently, we ﬁnd that humans agree with the model 99 .
4% of the time.
When we take a majority vote ( 2 / 3 humans agreeing) we ﬁnd that humans agree with the model 99 . 8% of the time.
We conclude that 3 . 5 is a reasonable threshold for evaluating the model.
A. S UBSET R OBUSTNESS
Additional robustness evaluation is computed using the subset of the test images that are initially correctly classiﬁed by the models without any adversarial attacks (Figure 6).
All test accuracy starts at 100% and decreases with stronger attacks.
The strength of the attacks is described by the value of epsilon
[Figure]
A. A DDITIONAL R ESULTS ON R OBUSTNESS E VALUATION
We here include the full results on robustness evaluation on CIFAR-10 dataset in Figure [Figure]
[Figure] [Figure] [Figure] I MPLEMENTATION D ETAILS
We train the categorical models for 200 epochs with a starting learning rate of 0 . 1 , and decay the learning rate by 0 .
1 at epoch 100 and 150 .
The high-dimensional models are trained for 600 epochs with the same initial learning rate, and we drop the learning rate by 0 .
1 at epoch 300 and 450 .
All models are trained with a batch size of 128 using the SGD optimizer with 0 .
9 momentum and 0 .weight decay.
One exception is when train categorical models with the VGG19 image encoder, we use a larger weight decay, 0 . 0005 .
We implement our models using PyTorch Paszke et al.
All experiments are performed on a single GeForce RTX 2080 Ti GPU.
The limited-data experiments are conducted using the same settings as the full data experiments.
A.4. A RCHITECTURE P ARAMETERS
We provide the parameter counts for the all models in Table 3 and Table 4.
The majority of the parameters come from the image encoders.
High-dimensional models have slightly more parameters than categorical models due to the high-dimensional label decoder (Table 5).
[Figure]
[Table]
A. D ATASET
To demonstrate the effectiveness of our proposed method, we evaluate our models on the CIFAR10 and CIFAR-100 datasets Krizhevsky (2009).
For each dataset, we train different models on the same training set and evaluate the models on the same validation set using the same random seeds for fair comparisons.
To preprocess the training images, we randomly crop them with a padding size 4 and perform random horizontal ﬂips.
All CIFAR images are normalized with mean (0 . 4914 , 0 .
and standard deviation (0 . 2023 , 0 . 1994 , 0 . 2010) of the training set.
[Table]
[Table]
CIFAR-10 consists of 60 , 000 images of size 32 × 32 uniformly distributed across 10 classes.
The dataset comes with 50 , 000 training images and 10 , 000 test images.
We use a 45 , 000 / 5 , 000 training/validation split.
CIFAR-100 also comprises 60 , 000 images of size 32 × 32 , but it has 100 classes each containing 600 images.
The dataset is split into 50 , 000 training images and 10 , 000 test images.
We randomly select 5 , 000 images from the training images to form the validation set.
A. V ISUALIZATIONS
In addition to the progressive t-SNE plots we presented in the main context, we plot the embedding of all three types of image encoders of models trained on speech labels, categorical labels, and constant matrix labels in Figure 8.
We only show the results from the models with highest accuracy.
Similarly, we observe that speech models have better separation of clusters.
The feature embedding of the constant model is worse than that of the speech model, further conﬁrming that the speech representation is unique in addition to its dimensionality.
Grad-CAM We visualize activations from beginning, intermediate, and ﬁnal layers in the image encoders for both speech and categorical models in Figure 9.
We see that the input activations for the speech model conform more tightly than those of the categorical model to the central object of each image at all three stages.
This may suggest that the speech model learns more discriminative features than the categorical model.
These features are also visually more easily understandable to humans.
[Figure]
[Figure]