M EASURING THE I NTERPRETABILITY OF U NSUPERVISED R EPRESENTATIONS VIA Q UANTIZED R EVERSE P ROBING
Anonymous authors
Paper under double-blind review
A BSTRACT
Self-supervised visual representation learning has attracted signiﬁcant research interest.
While a common way to evaluate self-supervised representations is through transfer to various downstream tasks, we instead investigate the problem of measuring their interpretability, i .
e .
understanding the semantics encoded in raw representations.
We formulate the latter as estimating the mutual information between the representation and a space of manually labelled concepts.
To quantify this we introduce a decoding bottleneck: information must be captured by simple predictors, mapping concepts to clusters in representation space.
This approach, which we call reverse linear probing , provides a single number sensitive to the semanticity of the representation.
This measure is also able to detect when the representation contains combinations of concepts ( e .
g .
“red apple”) instead of just individual attributes (“red” and “apple” separately).
Finally, we suggest that supervised classiﬁers can be used to automatically label large datasets to enrich the space of concepts used for probing.
We use our method to evaluate a large number of self-supervised representations, ranking them by interpretability, highlight the differences that emerge compared to the standard evaluation with linear probes and discuss several qualitative insights.
I NTRODUCTION
Relying on black-box models such as deep networks comes sometimes with signiﬁcant methodological and ethical challenges.
This is particularly true for unsupervised and self-supervised models which are learned without human supervision.
While these models perform increasingly well in downstream applications, often outperforming supervised counterparts, there is very little understanding of what they learn, making their real-life deployment risky.
In this paper, we thus consider the problem of characterizing the meaning of data representations, with a particular focus on unsupervised and self-supervised representations of images.
Given a representation f mapping images x to representation vectors f ( x ) ∈ R D , our goal is to ﬁnd whether these vectors contain human-interpretable information.
This is usually done by ﬁnding a relationship or correspondence between f ( x ) and human-provided descriptions y ( x ) of the images, essentially translating the information from the representation space to a concept space.
Consider for example the popular linear probing method (Alain and Bengio, 2017).
Given a large dataset ˆ X of images with corresponding manual annotations y ( x ) , one learns linear classiﬁers ( probes ) to map the feature vectors f ( x ) to labels y ( x ) , measuring the resulting classiﬁcation accuracy.
If the predictor is accurate, then one can argue that the representation captures the corresponding concept y .
One possible issue with this type of approaches is that one can only discover in the representation concepts that are represented in the available data annotations.
In order to maximize the semantic coverage of the analysis, it is thus customary to combine annotations for several types of attributes, such as object classes, textures, colour, etc.
(Bau et al., 2017).
In this case, y is a vector of image attributes, and one can compute several linear probes to predict each individual attribute y i .
By doing so, however, attributes are treated independently, which may be unsatisfactory.
In order to obtain a single number assessing the overall semanticity of a representation, one must combine the prediction accuracies of several independent classiﬁers, and there is no natural way of doing so ( e . g .
a simple average would not account for the different complexity of the different prediction tasks).
Furthermore, the representation might be predictive of combinations of attributes, i .
e .
it might understand the concept of “red apple” without necessarily understanding the individual concepts, “red” and “apple”.
While it is in principle possible to test any combination of attributes via linear probing, most attribute combinations are too rare to generate signiﬁcant statistics for this analysis.
In this paper, we thus propose a complementary assessment strategy to address these shortcomings.
In contrast to linear probes, we start by considering the reverse prediction problem, mapping label vectors y ( x ) to representation vectors f ( x ) .
A key advantage is that the entire attribute vector is used for this mapping, which, as we show later, accounts for attribute combinations more effectively.
Next, we consider the challenge of deriving a quantity that allows to compare representations based on the performance of these reverse predictors.
Obviously a simple metric such as the average Lprediction error is meaningless as its magnitude would be affected by irrelevant factors such as the scale of the representation vectors.
To solve this problem in a principled manner, we consider instead the mutual information between the concepts and quantized representation vectors.
This approach, which we justify from the viewpoint of information theory, essentially measures whether the representation groups the data in a human-interpretable manner.
We use our approach to evaluate a large number of self-supervised and supervised image representations.
Importantly, we show qualitatively that (a) all methods capture interpretable concepts that extend well beyond the underlying label distribution ( e . g .
ImageNet labels, which are not used for training), (b) while some clusters that form in the representation space are purely semantic, others carry information about scenery, material, textures, or combined concepts, and (c) context matters. We also show that more performant methods recover more of the original label distribution, i . e .
learn better “ImageNet concepts”, and rely less on lower-level concepts. Quantitatively, we observe that our interpretability measure results in a similar but not identical ranking for state-of-the-art methods and that clustering-based approaches produce generally more interpretable representations. [Equation]
ELATED W ORK
Self-supervised representation learning (SSL)
In this paper, we focus on self-supervised methods that learn representations from image data.
Early self-supervised learning approaches devise numerous pretext tasks — such as colorization, predicting image rotations or solving image puzzles — to learn useful representations from unlabeled images (Gidaris et al., 2018;
Pathak et al., 2016; Noroozi and Favaro, 2016; Doersch et al., 2015; Larsson et al., 2016; Zhang et al., 2017; 2016).
More recent approaches follow the contrastive learning paradigm (Chen et al., 2020a; Frankle et al., 2020; Kalantidis et al., 2020;
He et al., 2020; Chen et al., 2020b; H ´ enaff et al., 2019;
Oord et al., 2018; Misra and Maaten, 2020; Tian et al., 2019; 2020; Wu et al., 2018), where the goal is to discriminate instances belonging to the same image from negative samples.
Other methods (Zbontar et al., 2021; Chen and He, 2020; Grill et al., 2020; Caron et al., 2021) eliminate the need for negative samples.
While there exists some theoretical approaches for explaining why the constrastive learning works (Arora et al., 2019; Wang and Isola, 2020; Tschannen et al., 2020; Tsai et al., 2020), in most cases the focus lies on showing empirical improvement in downstream tasks.
Clustering A related part of literature deals with unsupervised image clustering.
Early approaches include using autoencoders (Hinton and Salakhutdinov, 2006), agglomerative clustering (Bautista et al., 2016) and partially ordered sets of hand-crafted features (Bautista et al., 2017).
More recent methods combine representation learning with clustering, using mutual information (Ji et al., 2019;
Hu et al., 2017), K -means (Caron et al., 2018; Zhan et al., 2020), prototypes (Li et al., 2021) or optimal transport (Asano et al., 2020; Caron et al., 2020).
Other methods build on strong feature representations and, at a second stage, cluster and further reﬁne the network (Yan et al., 2020a; Van Gansbeke et al., 2020; Lee et al., 2019).
These methods produce pseudo-labels which can be used to evaluate representations by measuring the correlation to the ground-truth labels.
Evaluation and analysis of SSL There are two main directions in evaluating self-supervised learning methods.
The ﬁrst one uses pre-trained representations as an initialization for subsequent supervised tasks, e . g .
for object detection.
The second approach is to train a linear classiﬁer on the embedding spaces which are kept frozen after pre-training.
As a result, several benchmarking suites with various datasets and tasks have been proposed (Zhai et al., 2019; Goyal et al., 2019b; Van Horn
Image collection
Frozen Encoder ...
Feature extraction and quantization 𝜇 𝐾
Concept extraction
Image collection ...
Expert Expert
Expert N ...
Reverse linear probing ...
ℎ θ et al., et al.
et al.
et
Interpretability Although a large amount of work has studied feature representations in Convolutional Neural Networks (CNNs), it is heavily focused on models trained with full supervision.
Zeiler and Fergus (2014) and Zhou et al.
(2014) analyze networks by visualizing the most activating patches, while activation maximization methods (Mahendran and Vedaldi, 2016a;b; Nguyen et al., 2017; 2016; Olah et al., 2017; Simonyan et al., 2014) generate inputs to activate speciﬁc neurons.
Another line of work focuses on understanding what information is present in intermediate representations of CNNs, usually mapping activations to high-level concepts.
Alain and Bengio (2017) introduce linear probes as a means to understanding the dynamics of intermediate layers, by predicting the target labels from these layers.
Escorcia et al.
(2015) also use a linear formulation to study the relationship of mid-level features and visual attributes, while Oramas et al. (2019) adopt this to predict task-speciﬁc classes from the features (similar to Alain and Bengio (2017)), select relevant features and generate a visual explanation . Similarly, Zhou et al.
(2018) decompose feature vectors into a set of elementary and interpretable components and show decomposed Grad-CAM heat maps (Selvaraju et al., 2017) for concepts that contribute to the prediction.
Furthermore, Kim et al.
(2018) relate feature vectors and human-interpretable concepts using a set of user-provided examples for each concept and Bau et al. (2017) propose to quantify the interpretability of individual units by measuring the overlap between each unit and a set of densely annotated concepts, Finally,
Ghorbani et al.
(2019) propose a method to automatically assign concepts to image segments and
Yeh et al.
(2020) analyze the completeness of these concepts for explaining the CNN.
With the exception of (Laina et al., 2020; Bau et al., 2017; Fong and Vedaldi, 2018), relatively little work has been done to understand the emergence of interpretable visual concepts in self-supervised representations speciﬁcally.
In particular, Laina et al.
(2020) quantify the learnability and describability of unsupervised image groupings, by measuring how successful humans are at understanding the learned concept from a set of provided examples.
Instead, our approach does not depend on human input and is thus easily scalable to a wide range of methods and number of classes.
M ETHOD
We are interested in measuring the semanticity of data representations.
A representation f is a map f : X → R D that encodes data samples x ∈ X as vectors f ( x ) ∈ R D .
In this paper, we assume that the data are images X
= R 3 × H × W and f is a deep neural network, but other choices are possible.
The “meaning” or semantic content of an image x can be characterized by obtaining a label or description y ( x ) of the image from a human annotator.
If the representation f ( x ) captures the meaning of the image too, then it should be predictive of the description y
( x ) .
The mutual information I ( f ( x ) , y ( x )) provides a natural measure of predictivity, and it is thus tempting to use this quantity as a measure of the overall semanticity of the representation.
However, information as such is not a useful measure in our context.
To see why, note that, due to the data processing inequality ( I ( f ( x ) , y ( x )) ≤ I ( x, y ( x )) ), the information is maximized by observing the raw image, i .
e .
by the identity representation f ( x ) = x .
Since data processing cannot increase information content, a useful representation must preserve information while also making it easier to decode and act on it.
There are many possible deﬁnitions of what constitutes ‘simple decoding’.
Here we propose a deﬁnition that stems from the interpretation of differential entropy as the limit of discrete entropy for quantized versions of the corresponding variables (Cover and Thomas, 2006).
In our case, the description y ( x ) is already discrete, but the representation vectors are continuous.
We thus propose to vector-quantize the representation and to use the resulting discrete information as a measure of semanticity.
Intuitively, in the limit of inﬁnitely-ﬁne quantization, this quantity reduces (up to a normalizing term) to the mutual information above.
For a ﬁnite number of clusters, however, a large value of information means that the representation groups images in a way that makes sense to a human observer.
In this case, the ‘simple decoding’ amounts to grouping images based on the L 2 distance between their representation vectors.
Formally, we use a vector quantization algorithm such as K -means
(Lloyd, 1982) to learn a quantizer function µ K : R D
→ { 1 , . . .
, K } for the representation vectors and estimate the mutual information
I ( µ K ( f ( x )) , y ( x )) by rewriting it as:
[Equation]
In the above equation, the ﬁrst term denotes the entropy of the cluster assignments.
In practice, given a sample dataset ˆ X of images, we ﬁrst run K -means to compute the quantizer µ K , and then compute the frequency of cluster assignments f K ( x ) , x ∈ X to calculate the entropy.
The second term in Eq.
(1) is the conditional entropy:
[Equation]
where p is the true joint distribution between variables f K ( x ) and y ( x ) .
While this is difﬁcult to compute, we can upper-bound it by considering an auxiliary posterior distribution q interpreting the conditional entropy as the cross-entropy loss of a predictor h θ , parametrized by θ , that maps labels y ( x ) to clusters f K ( x ) .
We minimize the gap by learning this predictor in the usual way.
In practice, learning the predictor may result in overﬁtting, which would lead to an over-optimistic estimate of the mutual information.
We address this issue by learning the predictor h θ on a subset ˆ
X (cid:48)
⊂ ˆ X of the data and evaluating its cross-entropy loss on the remaining subset
ˆ X − ˆ X (cid:48) .
Importantly, we consider a linear predictor ( probe ) for h θ , to further reduce the risk of overﬁtting.
We refer to this as quantized reverse probing, as it maps semantic concepts to quantized data representations.
Obtaining labelled data via automatic experts Similar to prior work on interpretation, our method requires a dataset x ∈
ˆ X of images equipped with manually-provided labels y ( x ) ∈ Y .
Because we do not know a-priori which concepts may be capture by a given representation f ( x ) under analysis, these labels should provide a good coverage for a number of different concepts (e.g., textures, object types, parts, etc.).
A good example of such a dataset is Broden (Bau et al., 2017).
Because it would be cumbersome, and maybe even infeasible, to annotate any target dataset with all possible visual concepts, we can further and cheaply increase the coverage of the label space by predicting the labels y ( x ) automatically via a battery of expert classiﬁers learned using full supervision.
The noise of these automated predictions is small compared to the noise in the correlation between the concepts and the unsupervised/self-supervised visual representations under study.
Number of clusters The number of clusters K controls the size of the bottleneck.
Mutual information increases monotonically as K increases, as we show in the Appendix (Fig. 6); when comparing representations, it is thus important to ﬁx a value of K and use it consistently.
For added convenience, in practice we report a normalised value of information (NMI). − 1 .− 0 .0 .0 .1 . x− 1 .− 0 .0 .0 .1 .x
DecisionBoundary h 1 h
[Figure]
− 1 .− 0 .0 .0 .1 .
x− 1 .− 0 .0 .0 .1 .x DecisionBoundary h 1 h [Figure] Relation to linear probing Linear probing is a common approach for assessing representations, related to our method.
A linear probe is a simple predictor
h i ◦ f
( x ) ≈
y
i ( x ) that maps the representation f ( x ) to a speciﬁc concept y i
(e.g., a binary attribute or a class label).
The idea is that the representation captures concept y i
if the probe performs well.
As a result, linear probing is a standard evaluation protocol in self-supervised representation to measure the (linear) classiﬁcation accuracy of the learned representations against a labelled set, commonly ImageNet (Russakovsky et al., 2015).
In interpretability studies, different formulations of linear probing (Alain and Bengio, 2017) are used to understand intermediate layers of neural networks.
Similar to our approach, the simplicity of the probes (imposed via linearity and sometimes sparsity) is a necessary bottleneck for interpretability: without such a constrain, the best possible probe would take the raw image as input. Our method is complementary to linear probes, with some advantages.
To discuss this point further, we show a simpliﬁed example in Fig.
2, where data points with color and shape attributes are encoded by two different representations, such that they form well-deﬁned nameable clusters, i .
e .
blue square, red square, blue circle, and red circle.
Forward probes ( X → Y ) can be trained as binary classiﬁers
h 1 , h 2 for each attribute, mapping a feature vector (in this case, coordinates) to the corresponding attribute value for each data point.
If attributes are linearly separable (Fig. 2a), then forward probes do well at separating the representation space. If an attribute is not linearly separable (color in Fig. 2b), the predictive accuracy of the corresponding probe h 1 reduces to chance, and consequently reduces the average score. As a result, linear probes cannot always assess the meaningfulness of clusters or, in other words, whether a cluster in representation space responds to a well-deﬁned concept.
On the contrary, if all data points in a cluster have consistent attributes, reverse probes ( Y → X ) can achieve a high score for, e .
g . , “red square” without requiring the concept “red” to be also encoded in the features. In this case, we can say that the model has discovered the concept of a “red square” without identifying, in isolation, the concept “red”. Therefore, reverse probing handles combinations of attributes naturally and allows to better assess the semanticity of clusters.
E XPERIMENTS
We use our method to evaluate a wide range of recent self-supervised representation learning and clustering techniques.
In the following, we compare these techniques according to the proposed criterion, which results in a new ranking for the state-of-the-art approaches.
Further, we show qualitatively how our method discovers elementary concepts which are encoded in the raw representations.
4. I MPLEMENTATION DETAILS
Attributes and expert models As a ﬁrst step in our approach, we collect a set of attributes which we expect to ﬁnd encoded in the self-supervised representations; these include semantic object cat-
[Table] egories, scene types and attributes, materials and textures and possibly other information about the photographic style or overall sentiment of an image (Table 1).
We thus look for relevant humanannotated datasets and expert models trained on these datasets, as a proxy to human knowledge.
This allows us to extract information related to such attributes on a different dataset or images in the wild; in our experiments we focus on ImageNet (IN-1k)
(Deng et al., 2009) as the target data due to the large availability of pre-trained self-supervised models on this dataset.
We concatenate all available attributes and form M -dimensional binary vectors denoting the presence or absence of each attribute in an image.
We provide full details about the expert models in Appendix A. Since we primarily focus on ImageNet, it is natural to also use the existing 1000 human-annotated labels.
As IN-1k already includes several ﬁne-grained categories, we have not included further experts from common ﬁne-grained recognition datasets, such as iNaturalist (Van Horn et al., 2018).
Linear model For each method, we train a linear model mapping a set of input attributes to clusters, which are obtained by K -means clustering on top of ﬁxed representations and evaluate on a held-out set.
Further training details are provided in Appendix A.
4. C OMPARISON OF SELF - SUPERVISED REPRESENTATIONS
We use reverse probing to evaluate recent self-supervised techniques in Table 2 and rank them based on their semanticity.
We obtain Jigsaw, SimCLR, ClusterFit from the VISSL model zoo (Goyal et al., 2021) and for the rest we use the ofﬁcial implementations.
For each method, we freeze the pre-trained model and extract feature vectors on the IN-1k data.
Using all categories from Table 1, we run 5 × K = 1000 -means to obtain different clusterings, each used to train a linear model.
In addition to a normalized measure of mutual information (NMI), we also report the adjusted mutual information (AMI), classiﬁcation accuracy (top-1) and mean average precision (mAP).
We also note the linear classiﬁcation accuracy, as reported by the respective source for each method.
For fairness of comparisons, we also group methods based on the number of training epochs.
For reference, we also evaluate the supervised counterparts in the same way as for the self-supervised methods.
Overall, we ﬁnd that there is a relatively strong correlation between our approach and the common linear evaluation protocol (Fig. 3).
However, we make the following additional observations:
Ranking Despite the strong correlation to linear classiﬁcation accuracy, we obtain a different ranking from the viewpoint of representational interpretability, using our approach.
In particular, state-of-the-art methods DINO, SwAV and BYOL fall slightly behind in our evaluation.
The most recent version of MoCo (v3) performs the best across three metrics, with DeepCluster-v2 and OBoW ranking in the second place (across ResNet-50-based models).
Number of pre-training epochs For some methods, learned weights are available at various epochs.
In terms of interpretability, we observe no beneﬁt in longer pre-training in the case of
DeepCluster-v2 and SwAV, despite the solid performance gains on standard benchmarks.
However, there is a noticeable gain for MoCo-v2, MoCHi and SimCLR between 200 and 800 epochs.
Notably, one of the best performing methods, OBoW is only trained for 200 epochs.
Clustering vs contrastive approaches Methods such as SeLa, OBoW, PCL and DeepCluster-v2, which use a clustering mechanism during training, have generally more interpretable representations, i .
e .
they rank higher than methods with similar linear classiﬁcation accuracy.
Architecture Interpretability increases with self-supervised methods that use a ViT (Touvron et al., 2021;
Dosovitskiy et al., 2021) backbone, showing a signiﬁcant boost in all metrics, even more so than linear classiﬁcation accuracy suggests.
However, a fair comparison between MoCo-v and DINO is not possible due to pre-training for a different number of epochs (300 vs. 800). Lin. Acc.
refers to the linear classiﬁcation accuracy reported by the respective source for each model.
The remaining metrics are computed for our approach and reported as mean ± std over 5 clusterings.
[Table] [Figure] [Table]
4. E XPERT BREAKDOWN
Our next goal is to understand the effect of different concepts on explaining the representation,
i
[Table]
e .
we want to answer the question: to which degree does the representation know certain concepts, e .
g . material?
As we are focusing on self-supervised representations here, the answer to this question is far from obvious.
We again measure this via the predictive power of our probe, which we now train individually for each group of concepts.
These vary in nature; from highly semantic object categories ( e .
g . dog , avocado ) to lower-level features such as material ( e .
g .
wood , brick ), texture ( e .
g .
bubbly , striped ) and color ( e .
g . red ).
In Fig.
4, we show how the different concept groups contribute to the overall performance for selected ResNet-50-based methods.
We train and evaluate reverse probes for each shown combination, e .
g .
IN-1 K +O BJECTS , then IN-1 K +O BJECTS +S CENE , etc.
We compare variants with and without using ground truth IN-1 K categories as part of the input.
We observe that for earlier methods, such as MoCo-v1, ImageNet categories alone are not sufﬁcient for accurately predicting f K ( x ) .
In fact, it appears that using only semantic categories from MSCOCO and OpenImages (w/o ImageNet) provides about as much information about the clusters.
This likely suggests that the discovered clusters do not reﬂect ﬁne-grained distinctions, since the most notable difference between IN-1 K and the other semantic experts are some ﬁne-grained categories ( e .
g . dog breeds).
On the other side of the spectrum, the best performing methods do indeed owe a big part of their performance to IN-1 K concepts, meaning that they are able to recover clusters that are closer to the original label distribu- [Figure] [Figure]
[Table] tion and that rely less on low-level concepts such as textures. This suggests that more performant methods learn features highly correlated with IN-1 K labels, but less with other semantic categories.
Finally, in Table 3, we also show the (decorrelated) effect of each individual attribute group in isolation but always in combination with the ground truth IN-1 K categories.
We observe that in absence of concept combinations, the models we probed use as much (and sometimes more) information about scene and material properties as information about semantic object categories.
Miscellaneous concepts (O THER ) do not appear relevant for more recent methods.
We again observe that the representations of the most recent, state-of-the-art MoCo-v3 share very little information with concepts other than IN-1 K , despite the lack of label information during pre-training.
4. Q UALITATIVE RESULTS
Since the clusters we examine are learned without supervision and expert models are not perfect predictors, it is crucial to verify whether the clusters align with human interpretation, besides quantitative scores.
By inspecting the probe’s coefﬁcients θ , one can qualitatively explore what concepts are representative for speciﬁc clusters and how exposing the linear model to different sets of experts affects its predictive accuracy.
In Fig.
5 we show pairs of clusters from MoCo-v2 for which we observed a signiﬁcant drop in the confusion (estimated from the confusion matrix) after including an additional concept group on top of the ﬁxed ImageNet label set.
For example, after including O BJECTS we notice that several pairs of clusters become more easily disambiguated thanks to the concept person , or related concepts ( human hand , human face , etc.).
In Fig.
5 (top left), both clusters contain the ImageNet label French horn and it is only possible to understand their differences when more information is available.
This ﬁnding supports our initial hypothesis there could be interpretable properties in the network that may remain undetected since they are not annotated a-priori in the benchmark data.
Importantly, since ImageNet is only annotated with a single label per image, combined concepts, such as “person playing the french horn” cannot be discovered otherwise.
Similarly, when adding S CENE concepts (top right), clusters are distinguished by scene [Figure] [Table] type; one contains volcanos , while the other contain mountains and creeks (categories from
Places-365). With M ATERIAL , we can predict the shipwreck cluster more accurately, while T EXTURE can explain the difference in the more stratified appearance of cliffs for the cluster on the right.
It therefore appears that, despite the object-centric nature of ImageNet, self-supervised representations rely signiﬁcantly on context and concept combinations.
This likely also explains why in (Van Horn et al., 2021)
ImageNet self-supervised models do as well as or better than the supervised baseline at context, counting and gestalt tasks
[Table]
4. T RANSFER TO OTHER DATASETS
We ﬁnally evaluate the representations learned by selected methods by transferring them to a different domain.
For this experiment, the self-supervised methods are not ﬁne-tuned or adapted on the new data.
We perform the quantization on representations extracted on images from Places-365 and use all concepts from Table 5 but IN-1 K (now using the available ground truth for Places-365).
We report the performance in Table 4 and observe that the same ranking generally holds.
C ONCLUSION
We have introduced reverse linear probing, a measure of representation interpretability that allows to explain representations by combining multiple concepts.
Driven by the need to better understand and characterize the representations learned by self-supervised methods, the proposed measure has a rigorous foundation in information theory and provides complementary information to existing benchmarks as well as insights into the semanticity and other interesting properties of such models.
The proposed model is also considerably faster to train, as all concepts can be pre-extracted on any given image collection.
Code and data will be released upon acceptance.
A I MPLEMENTATION D ETAILS
We provide all information about the concepts and experts used in our experiments in full detail.
Expert models are trained on various tasks and image datasets to extract information about the scene and its objects, material, texture and miscellaneous (other).
Then, the trained models are applied to the target dataset ( e .
g .
ImageNet (Russakovsky et al., 2015)).
While the tasks differ in nature — e .
g . segmentation (dense), detection, classiﬁcation — in all cases we convert the predictions to binary vectors.
For example, in the case of segmentation and detection, we return only a (binary) label vector denoting whether a class is present in a given image, discarding dense/location information.
We concatenate the experts’ binary predictions forming the concept vector y ( x ) for image x , i .
e .
[Equation]
for m expert models with y ( i ) denoting the i -th expert.
In the following, we provide some expertspeciﬁc details and summarize them in Table 5.
Detection on Open Images.
We use the Tensorﬂow Object Detection API and publicly available
Faster R-CNN model (Ren et al., 2016), trained on Open Images (600 object categories).
We apply the model to our target datasets and keep only object categories predicted with a conﬁdence higher than 0.5.
Segmentation on MS COCO.
We use a DeepLab-v2 model (Chen et al., 2017) trained for segmentation on MS COCO
(Lin et al., 2014; Caesar et al., 2018a), which includes 91 ‘thing’ and 91 ‘stuff’ categories.
‘Thing’ categories include common, countable objects with a well-deﬁned shape, such as person, dog, bicycle, etc.; ‘stuff’ categories typically include background regions, such as sky, snow, wall, grass, etc.
and are thus closely related to material properties.
While we experimented with conﬁdence and minimum area thresholds for a predicted class to be considered in our scenario, we found that simply returning all predicted categories performed best.
Scene Classiﬁcation and Scene Attributes.
Although ImageNet is an object-centric dataset, a lot of images are related to speciﬁc scene types, e .
g .
outdoor scenery (mountains, seaside) or indoor scenery (houses, shops).
In fact, some ImageNet categories can even further subdivided by scene type, for example dogs sitting indoors or playing in the park .
It is thus crucial to consider scene classiﬁcation in our investigations.
We use a pre-trained ResNet-50 model provided by the ofﬁcial repository of the Places-365 database (Zhou et al., 2017) and assign each image of our target dataset to a scene category.
Further, we use the provided uniﬁed model (wide ResNet-18) to predict multiple scene attributes per image.
The model is trained on the SUN Attribute Database (Patterson and
Hays, 2012), which contains 102 categories describing scene properties, e .
g .
man-made vs. natural or enclosed vs. open area.
Texture and Material Classiﬁcation.
For further detecting material concepts, we train a Deep
Encoding Network (DEP) (Xue et al., 2020) on the Materials in Context (MINC) dataset (Bell et al., 2015), which consists of 23 categories (include a background class).
We then do the same on the
Describable Textures Dataset (DTD) (Cimpoi et al., 2014), which consists of close-up images fortexture categories.
DEP uses ResNet-50 (He et al., 2016) as backbone and is trained for single-label classiﬁcation.
We apply the trained models to our target datasets, thus returning a single label per image in each case.
Text Detection.
Next, we wish to identify the presence of text in the images to investigate whether this is a deciding factor in clustering the image features ( e .
g .
camera dates or watermarks).
As a text detection expert, we use CRAFT (Baek et al., 2019) (ofﬁcial implementation) without the reﬁnement step.
We only return one bit representing whether the image contains text or not, rather than character recognition.
Overall, we found that text as a concept did not contribute signiﬁcantly in our experiments.
Sentiment Analysis.
We use a VGG-19 classiﬁer (Simonyan and Zisserman, 2014) trained on the
Twitter for Sentiment Analysis Dataset (T4SA) to classify the sentiment for each image into one of three classes (positive/neutral/negative) and use the sentiment prediction as an additional concept.
As with text, we found that sentiment does not signiﬁcantly affect the performance.
We found the predictions of the expert model to be somewhat unreliable; this is also likely due to the fact that sentiment is a rather subjective quality.
[Table]
Photography.
We have observed differences in the photographic style, quality and type of images in ImageNet, e .
g .
ranging from macro photography, usually for ﬂowers or insects, to vector illustrations.
In order to probe the representations for style and image quality, we use the recent CLIP model (Radford et al., 2021), trained on 400 million image-text pairs collected from the web and create a set of candidate sentences: “a high quality photo”, “a noisy, grainy image”, “a blurry image of low quality”, “macro photography”, “a photo with out of focus background, bokeh effect”, “an animated picture, a vector illustration”, “a painting”, “a portrait”.
We use the pre-trained CLIP model to compute image and sentence features (normalized to unit norm) and use the dot product to ﬁnd the similarity between each image and each one of the sentences.
We pick the sentence with the highest similarity as the predicted class for each image if the score is at least 0.5, else we assign a background class.
Since CLIP operates on free-form text rather than a ﬁxed set of categories, its use is not limited to this type of queries; it can be used as an expert where a labelled dataset might unavailable.
Training details for linear model We evaluate representations from a number of pre-trained selfsupervised feature extractors (Table 2).
For ResNet-50 models, we evaluate features at the output of the average pooling layer (2048-d).
For ViT-Base models we evaluate the [CLS] token of the last self-attention layer (768-d).
For each model, we pre-extract and store features for the entire training set of ImageNet and, prior to clustering, we standardize them to zero mean and unit standard deviation.
We then train K -means for 100 steps using faiss (Johnson et al., 2017) and choose the best out of 5 runs.
We divide the data into trainval and test sets by splitting all cluster assignments with a 80/20 ratio.
Finally, we train the linear model with the cluster assignments as targets and concept vectors as inputs (where the concepts are pre-extracted on a given image collection, e .
g .
ImageNet).
We train for up to 100 epochs with batch size 512 and optimize using SGD with a momentum of 0. and initial learning rate of 3.5 which is further reduced by a factor of 10 at epochs 60 and 80.
We also add L2-regularization with weight 3 × 10 − 6 .
Computation Given a dataset ( e . g .
ImageNet) and a set of experts ( e .
g .
the ones listed in Table 5), all labels can be pre-computed for all images in the dataset.
Feature vectors for a given model can be also pre-extracted and stored for the whole dataset.
Our method computes cluster assignments using the fast K -means implementation of faiss (which takes less than 5min for 256k 2048-d vectors on 4 NVIDIA RTX A1000). Training of the linear model converges in less than 100 epochs in a matter of minutes on a single GPU (1 epoch takes 2sec). We should note that standard linear probing on ImageNet typically requires full images (including online augmentations) and multiple forward passes through the frozen feature extractor, which makes it signiﬁcantly slower to train. B R ELATION TO T OPIC M ODELS
Probabilistic topic models, such as LDA (Blei et al., 2003), often used in natural language processing, aim to recover the latent semantic structure, i .
e .
a set of abstract “topics”, from a collection of documents.
The goal is to ﬁnd the collection of the topics that best represent the corpus, with each document typically containing multiple topics in different proportions.
However, the unsuper- vised nature of these models and the lack of a gold standard makes their evaluation a long-standing problem.
For this reason, several automatic “topic coherence” measures have been proposed that evaluate the degree of semantic similarity of the top words in a topic (Mimno et al., 2011; Aletras and Stevenson, 2013; Newman et al., 2010; R¨oder et al., 2015), while human evaluations are also common (Chang et al., 2009; Morstatter and Liu, 2018).
There is a similarity between topic coherence and the way we evaluate models.
However, this analogy breaks down rather quickly.
To discuss this, we can directly map the terms of topic models (topics, documents and words) to our scenario (clusters, images and concepts) in this order.
The semantic coherence of a topic (cluster) is often measured by the co-occurrence of words (concepts) across documents (images).
One immediate difference that arises is that in topic modeling the original space (corpus) is authored by humans and can be thus considered interpretable.
Measures of coherence are introduced to understand whether the topics found by a model are also interpretable, i .
e .
evaluating the model producing the topics.
In our case, we do not know the degree to which the original space (representation) is interpretable and this is precisely what we aim to quantify with our method.
Another conceptual difference is that topic models operate on a higher level of abstraction: a topic is a collection of words that describe a higher-level idea deﬁned by this collection of words.
This is sensible as the goal in topic modelling is to discover which words describe higher-level ideas and can be likely grouped together vs. irrelevant words.
In our case, this is not necessary, as the expert annotations already deﬁne the relevant concepts.
A further abstraction is unnecessary and in fact it may even be undesirable because we are interested in the minimum number of concepts that may explain a cluster.
For example, a topic of “animals” may be coherent, yet a cluster of “animals” lacks speciﬁcity because it may contain a multitude of classes.
Finally, the main mechanism in evaluating topic coherence is co-occurrence of words in the general corpus (documents).
In our case, this translates to visual concepts co-occurring in the same image and models relationships between concepts in the images/the real world, whereas we are interested in the relationship between clusters and concepts as they have been learned by the model.
C F URTHER A NALYSIS
In addition to the results and discussion in the main paper, we provide more investigations and insights.
C. M UTUAL INFORMATION BETWEEN CONCEPTS AND I MAGE N ET CATEGORIES
One interesting question that arises is to which extent the elementary concepts we are considering are predictive of ImageNet labels.
We answer this question by training a linear model to predict the ground truth label (instead of a pseudo-label) for each image from its concepts.
This results in topaccuracy of 46.8% (NMI: 64.2, AMI: 54.2) — and signiﬁcantly less if we exclude ‘object’ concepts that overlap with ImageNet labels.
This suggests that using additional concepts provides information which is complementary to the ﬁxed label set of ImageNet, further justifying our approach.
C. V ARYING K
As discussed in Section 3, the mutual information between a representation f ( x ) and a ﬁxed set of concepts y ( x ) is maximized when f ( x ) =
x , i .
e .
any processing of x cannot increase the amount of information.
It is thus expected that with an increasing number of clusters ( K ), I ( f K ( x ) , y ( x )) will monotonically increase, as we approximate f ( x ) ( i .
e .
when each sample is its own cluster).
We verify this empirically in Fig.
For a fair comparison across methods, we have ﬁxed K = 1000 for all experiments on ImageNet experiments in the main paper.
To better understand the effect of the number of clusters, in Fig.
7 we also show the performance of most methods for K ∈ { 500 , 1000 , 1500 , 2000 , 2500 , 3000 } , measuring NMI, AMI and top-1 accuracy, for the predictions of the reverse linear probe trained with all concepts.
Importantly, we observe that for K ≥ 1000 the relative ranking of methods remains mostly consistent regardless of the number of clusters.
The top three methods, MoCo-v3, OBoW and DeepCluster-v2 perform similarly and converge for larger
[Figure]
[Figure] K , followed by SeLa-v2, SwAV, BarlowTwins and DINO. Further, we observe that BYOL scales gracefully for larger K , reaching the performance of the top methods; the opposite is observed for PCL-v2 which does not scale as well.
Though it is important to study the effect of varying K , we should also note that our results do not appear sensitive to it
[Figure]
as similar trends are observed for all methods.
C. D ISCUSSION ON CLUSTERING - BASED METHODS
We have observed that methods that employ a clustering mechanism during training have generally more interpretable representations.
A natural question that arises is whether clustering-based evaluation naturally favors such approaches.
However, clustering has already been shown to work well even for contrastive approaches (Zheltonozhskii et al., 2020), which is also further conﬁrmed by our experiments.
In our evaluations, the highest ranked models are MoCo-v3 (1k epochs) and OBoW (200 epochs).
MoCov3 uses a contrastive objective, while OBoW follows a different approach from previous methods with a bag-of-words prediction task that puts emphasis on contextual information and local feature statistics.
Our results align with the intuition behind OBoW (Gidaris et al., 2021) that encoding local concepts via a bag-of-visual-words approach yields richer, context-aware representations.
Another example, SeLa, uses Sinkhorn-Knopp optimization to compute the cluster assignments, while
DeepCluster-v2 uses spherical K-means.
All these clustering mechanisms differ from the one used speciﬁcally in our experiments.
Moreover, SeLa and DeepCluster-v2 train with K = 3000 prototypes, and there is no reason to believe that their representations are optimal for all values of K , and as we discussed above, the relative ranking of methods does not depend on the choice of K .
[Table]
[Table]
. C. K - MEANS FOR SELF - LABELLING METHODS
We now look speciﬁcally at self-labelling methods, SeLa (Asano et al., 2020) and SCAN (Van Gansbeke et al., 2020), to investigate the difference between (a) the pseudo-labels predicted directly by each method and (b) clustering the learned representations with K -means, i .
e .
same as the experiments reported in the main paper.
SeLa follows an optimal transport approach that yields clusters of approximately equal size.
We found that this weakens the quality of its self-labels but results in stronger clusters after K -means and consequently in improved performance in our evaluations.
Since SeLa is originally trained with 3000 classes 1 , in Table 6 we compare its pseudo-labels with K = 3000 -means clustering in Table 6.
SCAN builds on top of a representation learning method, i .
e .
MoCo (He et al., 2020) and, with a learnable clustering approach that removes adverse effects of low-level features, it produces more semantic pseudo-labels, improving upon K -means.
Thus SCAN (self-label) results in improved performance in Table 6.
C. T ESTING ON O BJECT N ET
Next we provide an evaluation of selected self-supervised methods using the reversed probes on the challenging test set ObjectNet (Barbu et al., 2019), evaluating the transferability of the quantized representations.
Speciﬁcally, we use the the reversed probes trained for each method on ImageNet to assign ObjectNet samples to clusters, measuring the success of the classiﬁcation (accuracy, NMI,
AMI) against the corresponding K -means assignments, i .
e .
we measure if we can successfully predict the cluster assignments from predicted concepts, without any further training/adaptation.
Here, we use the probes trained with all experts as input, except for the real ImageNet labels to account for
ObjectNet categories that do not appear in ImageNet.
We notice a signiﬁcant drop in performance which suggests that a domain gap is indeed present, though the ranking stays again roughly the same (with the exception of OBoW).
D Q UALITATIVE E XAMPLES
Finally, in Figs. 8, 9, 10 and 11 we present several additional qualitative examples that highlight the usefulness of reverse probing.
Similar to Fig.
5 in the main paper, we begin by showing selfsupervised clusters which are similar; they often contain images belonging to the same ImageNet categories.
As a result, when training a (reverse) linear probe from ImageNet labels to pseudo-labels (cluster assignments), pairs of clusters with overlapping concepts will typically have high confusion (computed from the confusion matrix).
We then add a concept group on top of the ImageNet labels and train a second linear probe, e .
g .
in Fig.
9 we include concepts from Places-365 and SUN Attributes.
We can then identify pairs of clusters for which the inclusion of these concepts signiﬁcantly reduced the ambiguity of the mapping, therefore reducing the confusion.
Finally, for each pair of clusters we show the difference in linear model’s coefﬁcients as a word cloud, with larger differences denoted by larger font size.
In other words, we show the concepts that are most important to distinguish the two clusters (blue for (i) and red for (ii)).
In this way, we were able to discover and understand ﬁne-grained differences between clusters and even problematic cases when considering only ImageNet labels.
For example, we ﬁnd that several
ImageNet categories appear in combination with people ( e .
g .
musical instruments).
The quantized space of self-supervised representations (for all methods) appears to separate the corresponding images based on whether they contain people or not (examples shown in Fig.
8 (a), (c) and (d)).
Noteworthy are the clusters in Fig.
8 (d), both of which correspond to the same ImageNet label ( hockey puck ), although they are visually very different.
Including additional concepts, such as person or sports equipment , justiﬁes their distinction into separate clusters and aligns with human intuition — perhaps even more so than assigning them to the same class.
In Fig.
9 we show examples where clusters with images from the same ImageNet category differ by scene-related concepts such as ﬂying vs. perched birds, wolves and orcas in different environments and even the inside vs. outside of a building.
Material categories (Fig. 10) are helpful to understand environment or context and whether images contain hands ( skin ).
Finally, texture (Fig. 11) can be used to tease apart details at a macro level, e .
g . cut and whole fruits or peacocks with open or closed plumage.
Overall, texture becomes less relevant models which are better at semantic discrimination.
We also found that other concepts, such as text, sentiment and image quality do not signiﬁcantly affect the clustering.
To summarize, we have proposed reverse linear probing as a way to understand whether interpretable clusters form in the representation space of self-supervised methods.
While we provide a quantiﬁable measure for this — more interpretable clusters will be predicted with higher accuracy — we also show that we can qualitatively identify how concepts are encoded in the quantized representations through the linear probe’s coefﬁcients.
We ﬁnally show that such concepts are often complementary to ImageNet labels and can in fact correctly push clusters which are seemingly semantically close — based on a ﬁxed label set — farther apart ( e .
g .
Fig. 9(d)).
[Figure] [Figure]
[Figure] [Figure]
Clusters where texture helps are usually of lower quality and more recent methods depend less on textures