E FFECTS OF L INGUISTIC L ABELS ON L EARNED V ISUAL R EPRESENTATIONS IN C ONVOLUTIONAL N EURAL N ETWORKS : L ABELS MATTER !
Anonymous authors
Paper under double-blind review
A BSTRACT
We investigated how the visual representations learned by CNNs is affected by training using different linguistic labels (e.g., basic-level labels only, superordinate-level only, or both at the same time), and how these differentlytrained models compare in their ability to predict the behavior of humans tasked with selecting the object that is most different from two others in a triplet.
CNNs used identical architectures and inputs, differing only with respect to the labels used to supervise the training.
In the absence of labels, we found that models learned very little categorical structure, suggesting that this structure cannot be extracted purely from the visual input.
Surprisingly, models trained with superordinate labels (vehicle, tool, etc.) were most predictive of the behavioral similarity judgments.
We conclude that the representations used in an odd-one-out task are highly modulated by semantic information, especially at the superordinate level.
I NTRODUCTION
A critical distinction between human category learning and machine category learning is that only humans have a language.
A language means that human learning is not limited to a one-to-one correspondence between a visual input and a category label.
Indeed, the users of a language are known to actively seek out categorical relationships between objects and use these relationships in making perceptual similarity judgments and in controlling behavior (Hays, 2000; Lupyan & Lewis, 2017).
A premise of our work is that a language provides a semantic structure to labels, and that this structure contributes to the superior efﬁciency and ﬂexibility of human vision compared to any artiﬁcial systems (Pinto et al., 2010).
Of course, the computer vision literature on zero-shot and fewshot learning has also made good progress in leveraging semantic information (e.g., image captions, attribute labels, relational information) to increase the generalizability of a model’s performance (Lampert et al., 2013; Sung et al., 2018; Lei Ba et al., 2015).
Still, this performance pales in comparison to the human ability for classiﬁcation, where zero-shot and few-shot learning is the norm, and efﬁciently-acquired category knowledge is easily generalized to new exemplars (Ashby & Maddox, 2005; Ashby & Ell, 2001).
One reason why machine learning lags behind human performance may be because of a failure to fully consider the semantic structure of the ground-truth labels used for training, which can be heavily biased by basic or subordinatelevel categories.
This might result in models learning visual feature representations that may not be best for generalization to new, higher-level categories.
For example, ImageNet (Deng et al., 2009) contains 120 different dog categories, making the models that are trained using these labels dog experts, creating an interesting but highly atypical semantic structure.
Here we study how the linguistic structure of labels inﬂuences what is learned by models trained on the same visual inputs.
Speciﬁcally, we manipulated the labels used to supervise the training of CNN models, each having the same architecture and given identical visual inputs.
For example, some of these models were trained with basic-level labels only, some with only superordinate-level labels, and some with both.
We then compare visual representations learned by these models, and predict human similarity judgements that we collected using an Odd-one-out task where people had to select which of three object images was the most different.
With this dataset, and using categorical representations extracted from our trained models, we could predict human similarity decisions with up to 74% accuracy, which gives us some understanding of the labels needed to produce human-like representations.
Our study also broadly beneﬁts both computer vision and behavioral science (e.g., psychology, neuroscience) by suggesting that the semantic structure of labels and datasets should be carefully constructed if the goal is to build vision models that learn visual features representations having the potential for human-like generalization.
For behavioral science, this research provides a useful computational framework for understanding the effect of training labels on the human learning of category relationships in the context of thousands of naturalistic images of objects.
R ELATED WORK
NEW
2. SEMANTIC LABEL EMBEDDING
Although many computer vision models perform well in image classiﬁcation, generalization tasks such as zero-shot and few-show learning remain challenging.
Several studies have attempted to address this problem by embedding semantic information into a model’s representations using text description (Lei Ba et al., 2015), attribute properties (Lampert et al., 2013; Akata et al., 2015; Chen et al., 2018), and relationships between objects (Sung et al., 2018; Annadani & Biswas, 2018).
More related to our work, some studies even directly leveraged the linguistic structure of labels.
For example, Lei et al.
(2017) and Wang & Cottrell (2015) found that training CNNs with coarse-grained labels (e.g., basic-level categories) improve classiﬁcation accuracy for ﬁner-grained labels (e.g., subordinate-level labels).
Also, Frome et al. (2013) re
-trained a CNN to predict the word vectors learned by a word embedding model, instead of using one-hot labels, and found improved zero-shot predictions; the model was able to predict thousands of novel categories that were never seen with 18% accuracy.
These results suggest that different semantic structures of labels, such as word hierarchy, order of learning, or semantic similarity between words, affect learned visual representations in CNNs to differing degrees.
The current study provides a more systematic investigation of this question.
2. U NDERSTANDING HUMAN VISUAL REPRESENTATION
The human visual system is unparalleled in its ability to learn feature representations for objects that are robust to large changes in appearance.
This tolerance to variability, not only enables accurate object recognition, but also facilitates generalization to new exemplars and categories(DiCarlo et al., 2012).
Understanding how humans learn these visual representations is, therefore, an enormously important question, but one that is difﬁcult to study because human learning in the real world is affected and confounded by many factors that are difﬁcult to control experimentally.
Recently, work has addressed this issue by computationally modeling and simulating human representation.
For example, Hebart et al.
(2019) studied human visual representations by ﬁtting probabilistic models to human similarity judgements, and found that human visual representations are composed of semantically interpretable units, with each conveying categorical membership, functionality, and perceptual attributes.
Peterson et al.
(2018), the study most similar to ours, trained CNNs with labels that differed in hierarchy (e.g., subordinate-level vs basic-level).
They found that training on coarser-grained labels (either as standalone or as coming after ﬁner-grained) induces a more semantically structured representation, and produces more human-like generalization performance.
The current study builds on this earlier work by 1) including CNNs trained with no labels (autoencoder) or very ﬁne-grained labels (word vector), 2) testing on a large-scale dataset of human similarity judgements, and 3) comparing superordinate vs basic levels.
M ODEL T RAINING
Our goal is to study how linguistic labels change the visual representations learned by CNNs.
To do this, we trained equivalently designed CNNs for classiﬁcation, but each with different linguistic labels as groundtruth.
In addition, we trained a Convolutional autoencoder, which encodes the images using the same convolutional structure as the other models but, instead of being supervised to predict the class of the image, the objective of this model is to generate an output image that is the same as the input.
This Conv.
Autoencoder therefore represents a model that was not trained with any linguistic label, in contrast to the other models that were each trained with some type of linguistic labels.
The description of each model and the labels used for training are provided below.
• Conv.
Autoencoder : Autoencoder with Convolutional encoder and decoder trained to output the same image as input • Basic labels : CNN model trained with one-hot encoding of basic-level categories, n= • Superordinate labels : CNN model trained with one-hot encoding of superordinate-level categories, n= • Basic + Superordinate : CNN model trained with two-hot encoding of both basic and superordinate-level categories, n=40(10+30) • Basic then Superordinate : CNN model trained with one-hot encoding of basic-level categories ﬁrst (n=30), and then ﬁnetuned with one-hot encoding of superordinate categories (n=10) • Superordinate then Basic : CNN model trained with one-hot encoding of superordinatelevel categories ﬁrst (n=10), and then ﬁnetuned with one-hot encoding of basic categories (n=30)
NEW
• Basic FastText vectors : CNN model trained with basic-level word vectors extracted from FastText word embedding model (Bojanowski et al., 2017), dimension= • Superordinate FastText vectors : CNN model trained with superordinate-level word vectors extracted from FastText word embedding model (Bojanowski et al., 2017), dimension=
NEW
The identical CNN architecture was used for each model in our labeling manipulation, except for the output layer and its activation function.
This general pipeline is described in Figure 1.
Our
CNN models consist of ﬁve blocks of two Convolutional layers, each followed by Max pooling and Batch normalization layers.
For all Convolutional and Max pooling operations, zero padding was used to produce output feature maps having the same size as the input.
Rectiﬁed linear units (ReLU) were used to obtain an activation function after each convolution.
The ﬂattened output of the ﬁnal Convolutional layer, the ”bottleneck” feature that we later extract and use as a model’s visual representation (dim=1568), was then fed into one fully connected dense layer.
For Conv.
Autoencoder, the same Convolutional architecture was used for encoding and decoding, with the hidden layer in the model (dim=1568) serving as the bottleneck feature for analysis.
The ﬁnal predicted output, ”label vector” is either one-hot or word embedding according to the model’s target labels.
Output activation functions differed depending on what label vector was used: a sigmoid function for Basic + Superordinate CNN, a linear function for the Conv.
Autoencoder and FastText vectors CNNs, and a softmax for the rest of CNNs.
All models were trained and validated on the images of 30 categories from the IMAGENETdataset (Deng et al., 2009), and tested on images of the same 30 categories from the THINGS dataset (Hebart et al., 2019).
These 30 basic-level categories were grouped into 10 higher-level, superordinate categories, which included: ’mammal’, ’bird’, ’insect’, ’fruit’, ’vegetable’, ’vehicle’, ’container’, ’kitchen appliance’, ’musical instrument’, and ’tool’.
A list of all 30 categories, with their superordinates, are provided in the Supplementary 7.1.
All input images were converted from
RGB to BGR and each channel was zero-centered with respect to the ImageNet images.
Different loss functions were used for training different models: Binary Crossentropy loss for Basic + Superordinate CNN, and Mean Squared Error loss for both Conv.
Autoencoder and FastText vectors
CNNs, and Categorical Crossentropy loss was used for the rest of the CNNs.
All models were trained using Adam optimization (Kingma & Ba, 2014), with a mini-batch size of 64.
During training, early stopping was implemented and the model with the lowest validation loss was used for the following analysis.
B EHAVIORAL D ATA
To compare the visual representations learned by our trained models with those of humans, we collected human similarity judgements in an Odd-one-out task, as in Zheng et al.
Participants were shown three images of objects per trial, a triplet, and were asked to choose which object was most different from the other two.
Each triplet consisted of three exemplar objects from the 30 categories used for our model training.
All exemplar objects came from Zheng et al.
(2019), except for ’crate’, ’hammer’, ’harmonica’, and ’screwdriver’, which were replaced with new exemplars to increase image quality and category representativeness.
There are 4060 possible triplets that can be
3 x 3 C on v 2 D , 2 x 2 M a x
P oo l 2 D
B a t c h N o r m 3 x 3 C on v 2 D , 3 x 3 C on v 2 D , 2 x 2 M a x
P oo l 2 D
B a t c h N o r m 3 x 3 C on v 2 D , 3 x 3 C on v 2 D , 2 x 2 M a x
P oo l 2 D
B a t c h N o r m 3 x 3 C on v 2 D , 3 x 3 C on v 2 D , 2 x 2 M a x
P oo l 2 D
B a t c h N o r m 3 x 3 C on v 2 D , 3 x 3 C on v 2 D , 2 x 2 M a x
P oo l 2 D
B a t c h N o r m 3 x 3 C on v 2 D ,
Input Image: 244x244x
F l a tt ened La y e r ,
F u ll y C
onne c t ed
La y e r ,
Labe l V e c t o r s
Encoder
Classifier
Bottleneck Feature generated from all 30 categories, but we collected behavioral data on only a subset of these to reduce time and cost of data collection.
This subset includes 1) the ten triplets having objects coming from the same superordinate category e.g., ’orangutan’, ’lion’, ’gazelle’ 2) all 435 triplets where two objects came from the same superordinate category e.g., ’orangutan’, ’lion’, ’minivan’, and 3)triplets where all objects came from different categories e.g., ’orangutan’, ’minivan’, ’lemon’, yielding 1820 unique triplets in total.
Amazon Mechanical Turk (AMT) workers participated in this task, each making responses on ∼ 200 triplets.
After removing responses with reaction times below 500ms, we collected 9697 similarity judgements where each triplet was viewed by 5.6 workers, on average (min=4, max=51).
[Table]
E XPERIMENTS 5. E VALUATING MODEL PERFORMANCE
Although our goal was not to compete with state-of-the-art vision models in classiﬁcation, we evaluated classiﬁcation accuracy to see effects of different labels on learning, thereby conﬁrming that the visual features learned by our models represented category knowledge.
To evaluate classiﬁcation accuracy, we report top@k, the percentage of accurately classiﬁed test images where the true class was in the model’s
the top K predictions in Table 1.
Average precision, and average recall over all categories are also reported in the Supplementary 7.3.
All metrics were computed on the THINGS test dataset (Hebart et al., 2019).
Because the FastText vectors CNN predicts a word vector, not a class, we approximated its classiﬁcation performance by calculating cosine similarity between predicted and true word vectors and choosing the corresponding class from top@k similarities.
Classiﬁcation results cannot be generated from Conv.
Autoencoder, but we include examples of images generated from this model in the Supplementary 7.2 to show that the model worked.
As can be seen in Table 1, the top@5 classiﬁcation accuracy for all trained models was good (all > .82), although there is room for improved classiﬁcation for FastText vectors CNN.
5. E XPLORING V ISUAL R EPRESENTATIONS
To explore how the different linguistic labeling schemes affected the learned visual representations, we extracted and analyzed the bottleneck features from each model (i.e., the 1568-dimensional output of the last Convolutional layer; see Figure 1).
We ﬁrst measured the representational similarity of all objects in the training dataset (IMAGENET 2012;
Deng et al., 2009) both between and within each category.
These representational distributions were visualized using t-SNE (Maaten & Hinton, 2008) and are attached in Supplementary 7.5.
We also analyzed the similarity between categorical
FIX representations by plotting a similarity matrix in Figure 2.
To create categorical representations, we simply averaged the obtained bottleneck features from all training images per category, creating in a sense ”prototypical” representation for each class.
Clustering Quality
To investigate how model’s category representations are dense and well separated, we computed the ratio of between-category dispersion and within-category dispersion using cosine distance (1-cosine angle of two feature vectors).
Between-category dispersion is the average cosine distance between the center(mean) of different categories.
Within-category dispersion is the average cosine distance between every exemplar and the center of each category.
Comparing the models in Table 2 revealed that using distributed word vectors as targets, especially Superordinate FastText vectors, produced the highest between-to-within ratio, suggesting the most tightly clustered representations.
Interestingly, the Basic + Superordinate CNN model, which was trained with both basic and superordinate labels at the same time, learned more scattered and less distinguishable categorical representations compared to other label-trained models.
Lastly, Conv.
Autoencoder produced the lowest between-towithin ratio, suggesting that even if a model learns visual features that are good enough to generate input-like images, these visual representations may still be poorly discriminable not only in basic level categories, but also in superordinate level categories.
Widely distributed features of Conv.
Autoencoder from T-SNE plots in Supplementary 7.5 further supported that the visual input alone is not sufﬁcient to produce any clusterable structure or category representations.
Similar trend was observed in the other clustering quality measures as reported in the Supplementary 7.4.
[Table]
Basic then Superordinate
[Table]
Superordinate then Basic
[Table]
Basic FastText vectors 0.
Superordinate FastText vectors 0.
Visualization of Categorical Representations
Figure 2 visualizes cosine similarity matrices for the category representations learned by the models to explore whether the hierarchical semantic structure of the 30 categories is captured (e.g., every basic-level category belongs to one of ten superordinate categories).
For more complete comparison, we also analyzed categorical representations extracted from SPoSE (Zheng et al., 2019),
FastText (Bojanowski et al., 2017), and VGG16 early layer (i.e., the output from the ﬁrst max pooling layer; Simonyan & Zisserman, 2014).
SPoSE model’s category representations were trained on human similarity judgements.
This serves as an approximation of human perceived similarity, which can be combination of semantic and visual similarities.
While FastText similarity represents semantic similarity between categories in basic-level terms, VGG16 early layer similarity represents lower level visual similarity.
Whereas little effect of category hierarchy can be seen in VGG16 early layer or Conv.
Autoencoder features, various semantic structure can be observed in the other models (e.g., the emergent bright yellow squares in the ﬁgure).
Upon closer analysis, these categorical divisions seemed to occur for: 1) nature vs non-nature, 2) edible vs non-edible, and 3) the superordinate categories.
Surprisingly, basic-level structures are still observed in the Figure 2f (e.g., ﬁne-grained lines in the diagonal), where the model is trained only on the superordinate-level labels.
This suggests that guidance from superordinate labels was often as good or better as guidance from much ﬁnergrained basic-level labels, which is consistent with the previous ﬁnding that training with coarser labels induce more hierarchical structure in visual representations (Peterson et al., 2018).
5. P REDICTING H UMAN V ISUAL B EHAVIOR
Finally, we evaluated how well the visual representations learned by the models could predict human similarity judgements in an Odd-one-out task (See Section 4).
For each triplet, responses were generated from the models by comparing the cosine similarities between the three visual object representations and selecting the one most dissimilar from the other two.
Three kinds of visual representations were computed and compared: 1) IMAGENET categorical representations, where features were averaged over ∼ 1000 images per category from the IMAGENET training dataset (Deng et al., 2009), 2) THINGS categorical representations, where features were averaged over ∼ 10 images per category from the THINGS dataset (Hebart et al., 2019), and 3) Single Exemplar representation, where only one feature per category was generated for the 30 exemplar images used in the behavioral data collection.
Together with accuracy from SPoSE (Zheng et al., 2019), FastText (Bojanowski et al., 2017), and VGG16 early layer (Simonyan & Zisserman, 2014), three baseline models of accuracy are reported below, which constitute upper and lower bounds.
• Null Acc : Accuracy achieved by predicting that every sample is the most frequent class in the dataset (lower bound, 36%).
• Bayes Acc : Accuracy achieved by predicting that every sample is the most frequent class in each unique triplet set (upper bound, 84%).
• SPoSE Acc : Accuracy achieved using the SPoSE model (Zheng et al., 2019), a probabilistic model that is directly trained on human responses on all triplets from 1854 THINGS objects (80%).
As shown in Figure 3, triplet prediction accuracy was highest when models used IMAGENET category representations and lowest when single exemplar representations were used, even if exemplar image is the one that participant actually saw during the experiment.
This shows that when humans do visual similarity ratings, they not only evaluate visual inputs but also use rich and abstract semantic information learned from viewing myriad exemplars.
Comparing individual model performance, the highest accuracy (74%) was obtained by the model trained with superordinate labels.
This performance is particularly impressive, considering 1) how coarsely grained superordinate labels are (dim=10) compared to Basic labels (dim=30), Basic + Superordinate labels (dim=40), or FastText vectors (dim=300), and 2) that this model is not trained on the actual human triplet data, as was the case for the SPoSE model whose performance was about 80%.
These results suggest that the representations used by humans in an Odd-one-out task are highly semantic, reﬂecting category structure especially at the superordinate level.
However, this may be only because the setting of odd-one-out task has caused people to use superordinate label information.
For example, when the participants are given a triplet like (’orangutan’, ’lion’, and ’lemon’),
[Table]
[Figure]
[Table]
[Figure]
[Table]
hummingbirdvulturegooseantbeegrasshopperbananaorangelemonzucchiniartichokecucumbertrolleyminivantaxicratemailboxbucketcoffee
[Table]
[Table] [Table]
[Table]
r r e
[Table]
r i
[Table]
r
[Table]
t
[Table]
r
[Table]
a r m o n i
[Table]
l
[Table]
t e d r
[Table]
m s c r e w d r
[Table]
v
[Table]
v e l
[Table]
hummingbirdvulturegooseantbeegrasshopperbananaorangelemonzucchiniartichokecucumbertrolleyminivantaxicratemailboxbucketcoffee
[Table]
n
[Table]
e
[Table]
n
[Table]
t u r e g
[Table]
e a n t b
[Table]
r a
[Table]
o
[Table]
e r
[Table]
a n a n
[Table]
r
[Table]
n
[Table]
e
[Table]
m
[Table]
n
[Table]
u
[Table]
h i n i
[Table]
r t
[Table]
m
[Table]
r t r o
[Table]
m i n
[Table]
n
[Table]
i
[Table]
r
[Table]
t e m
[Table]
b
[Table]
b u c
[Table]
e t c o
[Table]
_ p
[Table]
t t
[Table]
a s t e r r e f r i
[Table]
r a t
[Table]
r
[Table]
a r m o n i
[Table]
l
[Table]
t
[Table]
d r u m s
[Table]
r e w d
r
i v e r
[Table]
o v e l
[Table]
hummingbirdvulturegooseantbeegrasshopperbananaorangelemonzucchiniartichokecucumbertrolleyminivantaxicratemailboxbucketcoffee
[Table]
g r a ss h o pp e r b a n a n a o r a n g e l e m o n z u cc h i n i a r t i c h
o k
e c u c u m b e r t r o ll e y m i n i v a n t a x i c
r a t e m a il b o x b u c k e t
c o ff ee _ p o t t o a s t e r r e f r
i g e r a t
o r h a r m o n i c a f l u t e
d r u m s c r e w d r i v e r s h o v e l h a mm e r orangutanliongazelle hummingbirdvulturegooseantbeegrasshopperbananaorangelemonzucchiniartichokecucumbertrolleyminivantaxicratemailboxbucketcoffee _pottoaster
[Table]
Basic
o r a n g u t a n li o n g a z e ll e hu
mm i n gb i r d v u l t u r e g
oo s e a n t b ee g r
a ss
h
o pp e r b a n a n a
o
r
a n g e l e m o n z u cc h i n i a r t
i c h
o k
e c u c u m b e r t r o ll e y m i n i v a n t a x i c
r a t e m a il b o x b u c k e t
c o ff ee _ p o t t o a s t e r r e f r
i g e r a t
o r h a r m o n i c a f l u t e
d r u m s c r e w d r i v e r s h o v e l h a mm e r
[Table]
mm
[Table]
gb
[Table]
d
[Table]
o pp e
[Table]
b a n a n a o r a n g e l e m o n z u cc h i n i a r t i c h
o k
e c u c u m b e r t r o ll e y m i n i v a n t a x i c
r a t e m a il b o x b u c k e t
c o ff ee _ p o t t o a s t e r r e f r
i g e r a t
o r h a r m o n i c a f l u t e
d r u m s c r e w d r i v e r s h o v e l h a mm e
[Figure]
S P o S E F a s t T e x t V GG
[Figure]
e a r
[Figure]
C o n v .
A u t o e n c o d e r B a s i
c S u
p e r o r d i n a t e B a s i c + S u
p e r o r d i n a t e B a s i c t
h e n S u
p e r o r d i n a t e S u
p e r o r d
i n a t e t
h e n B a s i
c B a s i
c F a s t T e
x t v e c t
o r s S u
p e r o r
[Figure]
, ’fruit’, ’vegetable’, ’vehicle’, ’container’, ’kitchen appliance’, ’musical instrument’
and ’tool’.
they are prone to choose ’lemon’ because it is the most odd one in superordinate-level.
In fact, when the number of superordinate categories in a triplet is two as in the example above, 90% of human responses can be predicted just by telling which one is the odd superordinate category.
To investigate how much this task setting would affect the results, we broke down the triplet data based on the number of superordinate categories that a triplet belongs to and reported prediction performance for each split, as shown in the Figure 3.
Interestingly, the model trained with superordinate labels alone still performed the best (63%) when superordinate-level information was not very helpful, where all three images in a triplet come from three different superordinate categories, e.g, (’mammal’,’fruit’,’vehicle’).
Moreover, the superordinate labels CNN (59%) outperformed the basic labels CNN (56%) even when the images were to be compared at the basic level, where all three images in a triplet come from the same superordinate categories, e.g, (’lemon’,’orange’,’banana’). This implies humans leverage the guidance from coarser superordinate labels in shaping categorical visual representation in both basic and superordinate levels
[Figure]
C ONCLUSION
To be able to generalize to unseen exemplars, any vision system have to learn statistical regularities that make members of the same category more similar to one another than members of other categories.
But where do these regularities come from?
Are they present in the bottom-up (visual) input to the network?
Or does learning the regularities require top-down guidance from category labels?
If so, what kinds of labels?
To investigate this problem, we manipulated the visual representations learned by CNNs by supervising them using different types of labels, and then evaluated these models in their ability to predict human similarity judgements.
We found that the type of label used during training profoundly affected the visual representations that were learned, suggesting that there is categorical structure that is not present in the visual input and instead requires top-down guidance in the form of category labels.
We also found guidance from superordinate labels was often as good or better as guidance from much ﬁner-grained basic-level labels.
Models trained only on superordinate class labels such as ”musical instrument” and ”container” were not only more sensitive to these broader classes than models trained on just basic-level labels, but exposure to just superordinate labels allowed the model to learn within-class structure, distinguishing a harmonica from a ﬂute, and a screwdriver from a hammer.
This ﬁnding is consistent with the previous work that revealed that training with coarser labels induce more semantically structured visual representations (Peterson et al., 2018).
More surprisingly, models supervised using superordinate labels (vehicle, tool, etc.) were best in predicting human performance on an triplet odd-one-out task.
CNNs trained with superordinate labels not only outperformed other models when the odd-one-out came from a
[Table]
different superordinate category (which is not surprising), but also when all three objects from a triplet came from different superordinate categories (e.g., when choosing between a banana, a bee, and a screwdriver).
Our ongoing work into how different types of labels shape visual representations is exploring the effect of labels speciﬁc to different languages (e g., English vs. Mandarin), and how these may translate to differential human and CNN classiﬁcation performance.
A CKNOWLEDGMENTS
Details regarding research support will be added post-review.
S UPPLEMENTARY M ATERIAL 7. L IST OF 30 C ATEGORIES
[Table]
7. C ONV . A UTOENCODER P REDICTIONS
7. A VERAGE PRECISION AND AVERAGE RECALL SCORES FOR THE TRAINED MODELS .
The scores were sample-wise averaged (i.e., averaged over samples) for Basic + Superordinate CNN, and macro-averaged (i
[Table]
e.,averaged over categories) for the other models
[Table]
then Superordinate
[Table]
then
[Table]
7. O THER CLUSTERING QUALITY MEASURES
SC: Silhouette Coefﬁcients; CH: Calinski-Harabasz Index; DB: Davies-Bouldin Index; BW:
Between-to-within class dispersion in cosine distance; The arrow indicates in which direction of metric value represent more dense and well separated clusterings.
NEW
[Table]
FIX
[Figure] [Figure]