V ISUAL R ECOGNITION WITH D EEP N EAREST C ENTROIDS
Anonymous authors
Paper under double-blind review
A BSTRACT
We devise deep nearest centroids (DNC), a conceptually elegant yet surprisingly effective network for large-scale visual recognition, by revisiting Nearest Centroids, one of the most classic and simple classiﬁers.
Current deep models learn the classiﬁer in a fully parametric manner, ignoring the latent data structure and lacking explainability.
DNC instead conducts nonparametric, case-based reasoning; it utilizes sub-centroidsoftrainingsamplestodescribeclassdistributionsandclearlyexplains theclassiﬁcationastheproximityoftestdataandtheclasssub-centroidsinthe feature space.
Due to the distance-based nature, the network output dimensionality is ﬂexible, and all the learnable parameters are only for data embedding.
That means all the knowledge learnt for ImageNet classiﬁcation can be completely transferred for pixel recognition learning, under the “pre-training and ﬁne-tuning” paradigm.
Apart from its nested simplicity and intuitive decision-making mechanism, DNC can even possess ad-hoc explainability when the sub-centroids are selected as actual training images that humans can view and inspect.
Compared with parametric counterparts,DNCperformsbetteronimageclassiﬁcation(CIFAR-10,CIFAR-100,
ImageNet) and greatly boots pixel recognition (ADE20K, Cityscapes) with improved transparency, using various backbone network architectures (ResNet, Swin) and segmentation models (FCN, DeepLab V3 , Swin).
Our code will be released.
I NTRODUCTION
Deep learning models, from convolutional networks ( e .
g ., VGG [1], ResNet [2]) to Transformer-based architectures ( e .
g .,
Swin [3]), push forward the state-of-the-art on visual recognition.
With these advancements, parametric softmax classiﬁers, which learn a set of parameters, i .
e ., weight vector, and bias term, for each class, have become the de facto regime in the area (Fig. 1(b)).
However, due to the parameterizationnature,they suffer from several limitations: First , they lack simplicity and explainability.
The parameters in the classiﬁcation layer are abstract and detached from the physical nature of the problem being modelled[4].
Thus these classiﬁers are hard to naturally lend to an explanation that humans are able to process [5].
Second , the networks are trained purely to optimize the accuracy of predictions, neglecting the latent structure that data may possess.
For each class, only one single weight vector is learned in a fully parametric manner.
Thus they essentially assume unimodality for each class[6, 7], bearing no intra-class variation.
Third , as each class has its own set of parameters, deep parametric classiﬁers require the output space with a ﬁxed dimensionality (equal to the number of classes)
As a result, their transferability is limited; when using ImageNet [9] trained image classiﬁers initialize segmentation networks ( i .
e ., pixel classiﬁers), the last classiﬁcation layer, whose parameters are valuable knowledge learnt from the image classiﬁcation task, has to be thrown away.
In light of the foregoing discussions, we are motivated to present deep nearest centroids (DNC), a powerful, nonparametric classiﬁcation network (Fig.1(d)).
Nearest Centroids, which has historical roots dating back to the dawn of artiﬁcial intelligence [10–15], is arguably the simplest classiﬁer.
Nearest
Centroidsoperates on an intuitive principle: given a data sample, it is directly classiﬁed to the class of training examples whose mean (centroid) is closest to it.
Apart from its internal transparency, Nearest
Centroids is a classical form of exemplar-based reasoning [5, 12], which is fundamental to our most effective strategies for tactical decision-making[16] (Fig.1(c)).
Numerous past studies[17–19] have shown that humans learn to solve new problems by using past solutions of similar problems.
Despite itsconceptual simplicity, empiricalevidenceincognitivescience,andeverpopularity [20–23],Nearest
Centroids, and particularly its utility in large datasets with high-dimensional input spaces, are widely unknown or ignored by current community.
Inheriting the intuitive power of Nearest Centroids,
our
[Figure]
Training dataset
[Figure]
DNC is able to serve as a strong yet interpretable backbone for large-scale visual recognition; it is fully aware of the limitations of parametric counterparts while shows even better performance.
Speciﬁcally, DNC summarizes each class into a set of sub-centroids (sub-cluster centers) by clustering of training data inside the same class, and assigns each test sample to the class with the nearest subcentroid.
DNCisessentiallyan experience -/ distance -basedclassiﬁer–itmerelyreliesontheproximity of test query to local means of training data (“quintessential” past observations) in the deep feature space.
Assuch,DNClearnsvisualrecognitionbydirectlyoptimizingtherepresentation,insteadofdeep parametric models needing an extra softmax classiﬁcation layer after feature extraction.
For training,
DNC alternates between two steps: i) class-wise clustering for automatically discovering class subcentroids, and ii) classiﬁcation prediction for supervised representation learning, through retrieving the nearest sub-centroids.
However, since the feature space evolves continually during training, computing the sub-centroids is expensive – it requires a pass over the full training dataset after each batch update and limits DNC’s scalability.
To solve this, we use a Sinkhorn Iteration[24] based clustering algorithm
[25] for fast cluster assignment.
We further adopt momentum update with an external memory for online estimating the sub-centroids (whose amount is more than 1K on ImageNet [9]) with small-batch size ( e .
g ., 256).
Consequently, DNC can be efﬁciently trained by simultaneously conducting clustering and stochastic optimization on large datasets with small batches, only slowing the training speed slightly ( e .
g ., ∼ 5% on ImageNet).
DNC enjoys a few attractive qualities: First , improved simplicity and transparency .
The intuitive workingmechanismandstatisticalmeaningofclasssub-centroidsmake DNC elegant and easy to understand.
Second , automated discovery of underlying data structure .
By within-class deterministic clustering, the latent distribution of each class is automatically mined and fully captured as a set of representative local means.
In contrast, parametric classiﬁers learn one single weight vector per class, intolerant of rich intra-class variations.
Third , direct supervision of representation learning .
DNC achieves classiﬁcation by comparing data samples and class sub-centroids on the feature space.
With such distance-based nature, DNC blends unsupervised sub-pattern mining (class-wise clustering) and supervised representation learning (nonparametric classiﬁcation) in a synergy: local signiﬁcant patterns are automatically mined to facilitate classiﬁcation decision-making; the supervisory signal from classiﬁcation directly optimizes the representation, which in turn boosts meaningful clustering.
Forth , better transferability .
DNC learns by only optimizing the feature representation, thus the output dimensionality no longer needs to be as many as the classes.
With this algorithmic merit, all the useful knowledge (parameters) learnt from a source task ( e .
g ., ImageNet [9] classiﬁcation) are stored in the representation space, and can be completely transferred to target tasks ( e .
g ., Cityscapes [26] segmentation).
Fifth , ad-hocexplainability .
If further restricting the class sub-centroids to be samples (images) of the training set, DNC can explain its prediction based on IF · · · Then rules and allow users to intuitively view the class representatives, and appreciate the similarity of test data to the representative images (detailed in § 3&4.3).
Such ad-hoc explainability [27] is valuable in safety-sensitive scenarios, and differs DNC from most existing network interpretation techniques [28–30] that only investigate post-hoc explains and thus fail to elucidate precisely how a model works [31, 32].
DNC is an intuitive yet general classiﬁcation framework; it is compatible with different visual recognition network architectures and tasks.
Weexperimentallyshow: In § 4.1 ,withResNet[2]andSwin[3] network architectures, DNC outperforms parametric counterparts on image classiﬁcation, i .
e ., 0.230.24 % top-1 accuracyonCIFAR-10[33]and 0.24-0.32 %onImageNet[9],by trainingfromscratch .
In § 4.2 , when using our ImageNet-pretrained, nonparametric versions of ResNet and Swin as backbones, our pixel-wise DNC classiﬁer greatly improves the segmentation performance of FCN [34],
DeepLab V3 [35], and UperNet [36], on ADE20K [37] ( 1.6-2.5 % mIoU ) and Cityscapes [26] ( 1.11.9 % mIoU ).
These results verify DNC’s strong transferability and high versatility.
In § 4.3 , after constraining class sub-centroids as training images of ImageNet, DNC becomes more interpretable, with only 0.12 % sacriﬁce in top-1 accuracy (but is still 0.17 % better than the parametric counterpart).
These results are particularly impressive, considering the nonparametric and transparent nature of
DNC.
We feel this work brings fundamental insights into related ﬁelds.
Our code will be released.
R ELATED W ORK
Distance-/Prototype-based Classiﬁers.
Among the numerous classiﬁcation algorithms ( e . g ., logistic regression [38], Na¨ıve Bayes [39], random forest [40], support vector machines [41], and deep neural networks (DNNs) [42]), distance-based methods are particularly remarkable, due to their intuitive working mechanism.
Distance-based classiﬁers are nonparametric and exemplar-driven, relying on similarities between samples and internally stored exemplars/prototypes.
Thus they conduct case-based reasoning that humans use naturally in problem-solving, making them appealing and interpretable [17, 43].
k -Nearest Neighbors ( k -NN)
[10, 11] is a form of distance-based classiﬁers; it uses all training data as exemplars [44, 45].
Towards network implementation of k -NN [46–48],
Wu et al .
[49] made notable progress; their k -NN network outperforms parametric softmax based
ResNet[2] and the learnt representation works well in few-shot settings.
However, k -NN classiﬁers (including the deep learning analogues) cost huge storage space and pose heavy computation burden ( e . g ., persistently retaining the training dataset and making full-dataset retrieval for each query)
[50, 51], and the nearest neighbors may not be good class representatives [43].
Nearest
Centroids [12–15] is another famous distance-based classiﬁer yet has neither of the deﬁciencies of k -NN [21, 43].
Nearest Centroids selects representative class centers, instead of all the training data, asexemplars.
Guerriero et al .[52] also investigate the idea of bringing Nearest Centroids into DNNs.
However, they simply abstract each class into one single class mean, failing to capture complex class-wise distributions and showing weak results even in small datasets [33].
The idea of distance-based classiﬁcation also stimulates the emergence of prototypical networks , which mainly focus on few-shot [53, 54] and zero-shot [55, 56] learning.
However, they often associate to each class only one representation (prototype) [57] and their prototypes are usually ﬂexible parameters [51, 53, 56] or deﬁned prior to training [8, 55].
In DNC, a prototype (sub-centroid) is either a generalization of a number of observations or intuitively a typical training visual example.
Via clustering based sub-class mining, DNC addresses two key properties of prototypical exemplars: sparsity and expressivity [58, 59].
In this way, the representation can be learnt to capture the underlying class structure, hence facilitating large-scale visual recognition while preserving transparency.
Neural Network Interpretability.
As the black-box nature limits the adoption of DNNs in decisioncritical tasks, there has been a recent surge of interest in DNNs’ interpretability.
However, most interpretation techniques only produce posteriori explanations for already-trained DNNs, typically by analysis of reverse-engineer importance values [28–30, 60–65] and sensitivities of inputs [66–69].
As many literature outlined, post-hoc explanations are problematic and misleading[32, 43, 70, 71];they cannotexplainwhat actually makesaDNNarriveatitsdecisions[72].Topursue ad-hoc explainability, some attempts have been initiated to develop explainable DNNs, by deploying more interpretable machineries into black-box DNNs [73–75] or regularizing representation with certain properties ( e .
g ., sparsity [76], decomposability [77], monotonicity [78]) that can enhance interpretability.
DNC intrinsically relies on class sub-centroid retrieving.
The theoretical simplicity makes it easy to understand; when anchoring the sub-centroids to available observations, DNC can derive intuitive explanations based on the similarities of test samples to representative observations.
It simultaneously conducts representation learning and case-based reasoning, making it self-explainable without post-hoc analysis[27].
DNC relates to concept -based explainable networks[4, 5, 72–74, 79, 80] that refer to human-friendly concepts/prototypes during decision making.
These methods, however, necessitate nontrivial architectural modiﬁcation and usually resort to pre-trained models, not to mention serving as backbone networks.
In sharp contrast, DNC only brings minimal architecture change to parametric classiﬁer based DNNs and yields remarkable performance on ImageNet[9] with training from scratch and ad-hoc explainability.
It provides solid empirical evidence, for the ﬁrst time as far as we know, for the power of case-based reasoning in large-scale visual recognition.
[Figure]
pppppx y
Training dataset
7 L
D EEP N EAREST C ENTROIDS (DNC)
Problem Statement.
Consider the standard visual recognition setting.
Let X be the visual space ( e .
g ., image space for recognition, pixel space for segmentation), and Y = { 1 , · · · , C } the set of semantic classes.
Given a training dataset { ( x n , y n ) ∈X×Y} Nn =1 , the goal is to use the N training examples to ﬁt a model (or hypothesis) h : X (cid:55)→Y that accurately predicts the semantic classes for new visual samples.
Parametric SoftmaxClassiﬁer.
Current commonpractice is toimplement h as DNNs and decompose it as h =
l ◦ f .
Here f : X (cid:55)→ F is a feature extractor ( e .
g ., convolution based or Transformer-like networks) that maps an input sample x ∈ X into a d -dimensional representation space F ∈ R d , i .
e ., x = f ( x ) ∈ F ; and l : F (cid:55)→ Y is a parametric classiﬁer ( e .
g ., the last fully-connected layer in recognition or last 1 × 1 convolution layer in segmentation) that takes x as input and produces class prediction ˆ y = l ( x ) ∈Y .
Concretely, l assigns a query x ∈X to the class ˆ y ∈Y according to:
[Equation]
where s c ∈ R indicates the unnormalized prediction score ( i .
e ., logit ) for class c , w c ∈ R d and b c ∈ R are learnable parameters – class weight and bias term for c .
Parameters of l and f are learnt by minimizing the softmax cross-entropy loss:
[Equation]
Though highly successful, the use of the parametric classiﬁer l has drawbacks as well: i)
The weight matrix W =( w 1 , · · · , w C ) ∈ R d × C andbiasvector b =( b 1 , · · · , b C ) ∈ R d in l are learnable parameters, cannot providing any information about what makes the model h reach its decisions.
ii) l makes the loss L only depend on the relative relation among logits, i .
e ., { s c } Cc =1 , and cannot directly supervise on the representation x [81, 82].
iii) W and b are learnt without considering any underlying data distribution.
iv)
The ﬁnal output dimensionality is constrained as many as the classes, i .
e ., C .
During transfer learning, as different visual recognition tasks typically have distinct semantic label spaces (with different number of classes), the classiﬁer from a pretrained model has to be abandoned, even though the learnt parameters W and b contain valuable knowledge from the source task.
The question naturally arises: might there be a simple way to address the limitations of current de facto , parametric classiﬁer based visual recognition regime?
Here we show that this is indeed possible, even coming with better performance.
DNC Classiﬁer.
Our DNC (Fig. 2) is built upon the intuitive idea of Nearest Centroids, i .
e ., assign a sample x to the class
ˆ y ∈Y with the closest class center:
[Equation]
where (cid:104)· , ·(cid:105) is a distance measure, used as the negative cosine similarity: (cid:104) u , v (cid:105) = − u (cid:62) v / (cid:107) u (cid:107)(cid:107) v (cid:107) .
Here ¯ x c is the mean vector of class c , x cn is a training sample of c , i .
e ., y cn
= c , and N c is the number of training samples in c .
As such, the feature-to-class mapping F (cid:55)→ Y is achieved in a nonparametric manner and understandable from user’s view, in contrast to the parametric classiﬁer l that learns“non-transparent” parametersforeachclass.
Itmakesmoresenseifusingmultiplesub-centroids (local means) per class instead of just one, which is in particular true for challenging visual recognition where complex intra-class variations cannot be simply described by the simple assumption of unimodality of data of each class.
When representing each class c as K sub-centroids, denoted by { p ck ∈ R d } Kk =1 , the C -way classiﬁcation for sample x takes place as a winner-takes-all rule:
[Equation]
Clearly, estimating class sub-centroids needs clustering of training samples within each class.
As class sub-centroids are sub-cluster centers in the latent feature space F , they are locally signiﬁcant visual patterns and can comprehensively represent class-level characteristics.
DNC can be intuitively understood as selecting and storing prototypical exemplars for each class, and ﬁnding classiﬁcation evidence for a previously unseen sample by retrieving the most similar exemplar.
This also aligns with the prototype theory in psychology [18, 83, 84]: prototypes are a typical form of cognitive organisation of real world objects.
DNC thus emulates the case-based reasoning process that we humans are accustomed to[27].
For instance, when ornithologists classify a bird, they will compare it with those typical exemplars from known bird species to decide which species the bird belongs to [43].
Sub-centroid Estimation.
To ﬁnd informative sub-centroids that best represent classes, we perform deterministic clustering within each class on the representation space F .
More speciﬁcally, for each class c , we cluster all the representations { x cn ∈ R d } N c n =1 into K clusters whose centers are used as the sub-centroids of c , i .
e ., { p ck ∈ R d } Kk =1 .
Let X c =[ x c 1 , · · · , x cN c ] ∈ R d × N c and P c =[ p c 1 , · · · , p cK ] ∈
R d × K denotethefeatureandsub-centroidmatrixes,respectively.
Thedeterministicclustering, i .
e ., the mapping from X c to P c , can be denoted as Q c =
[ q c 1 , · · · , q cN c ] ∈ { 0 , 1 } K × N c , where n - th column q cn ∈{ 0 , 1 } K is an one-hot assignment vector of n - th
sample x cn w.r.t the K clusters.
Q c is desired to maximize the similarity between X c and P c , leading to the following binary integer program (BIP):
[Figure]
[Equation]
where 1 K is a K -dimensional all-ones vector.
As in [25], we relax Q c to be a transportation polytope [24]: Q (cid:48)
c
= { Q c ∈ R K × N c + | ( Q c ) (cid:62) 1 K = 1 N c , Q c 1 N c = N c K 1 K } ,casting BIP(5) into an optimal transport problem.
In Q (cid:48) c ,besidesthe one-hot assignment constraint ( i .
e ., ( Q c ) (cid:62) 1 K = 1 N c ),an equipartition constraint ( i .
e .,
Q c 1 N c = N c K 1 K ) is added to inspire N c samples to be evenly assigned to K clusters.
This can efﬁciently avoid degeneracy, i .
e ., mapping all the data to the same cluster.
Then the solution can be given by a fast version [24] of Sinkhorn-Knopp algorithm [85], in a form of a normalized exponential matrix:
[Equation]
where the exponentiation is performed element-wise, α ∈ R K and β ∈ R N c are two renormalization vectors, which can be computed using a small number of matrix multiplications via SinkhornKnopp Iteration [24], and ε = 0 .
05 trades off convergence speed with closeness to the original transport problem.
In short, by mapping data samples into a few clusters under the constraints Q (cid:48) c , we pursue sparsity and expressivity [58, 59], making class sub-centroids representative of the dataset.
Training of DNC = Supervised Representation Learning+Automatic Sub-class Pattern Mining.
Ideally, according to class-wise cluster assignments { Q c } Cc =1 , we can get totally CK sub-centroids { p ck } C,Kc,k
=1 , i .
e ., mean feature vectors of the training data in the CK clusters.
Then the training target becomes as:
[Equation]
Comparing(2)and(7),astheclasssub-centroids { p ck } c,k are derived solely from data representations,
DNC learns visual recognition by directly optimizing the representation f , instead of the parametric classiﬁer l .
Moreover, with such a nonparametric, distance-based scheme, DNC builds a closer link to metric learning [86–89]; DNC can even be viewed as learning a metric function f to compare data samples { x n } n , under the guidance of the corresponding semantic labels { y n } n .
During training, DNC alternates two steps iteratively: i) class-wise clustering (5) for automatic subcentroid discovery, and ii) sub-centroid based classiﬁcation for supervised representation learning (7).
Throughclustering,DNC probesunderlying data distribution of each class, and produces informative sub-centroids by aggregating statistics from data clusters.
With the sematic label, DNC directly optimizes the representation by adjusting the arrangement between sub-centroids and data samples.
The enhanced representation in turn helps to ﬁnd more informativesub-centroids,beneﬁtingclassiﬁcation eventually.
As such, DNC conducts unsupervised sub-class pattern discovery during supervised representation learning , distinguishing it from most (if not all) of current visual recognition models.
Since the latent representation f evolves continually during training, class sub-centroids should be synchronized, which requires performing class-wise clustering over all training data after each batch update.
This is highly expensive on large datasets, even though Sinkhorn-Knopp iteration[24] based clustering (6) is highly efﬁcient.
To circumvent the expensive, ofﬂine sub-centroid estimation, we adopt momentum update and online clustering.
Speciﬁcally, at each training iteration, we conduct class-wise clustering on the current batch and update each sub-centroid as:
[Equation]
where µ ∈ [0 , 1] is a momentum coefﬁcient, and ¯ x ck ∈ R
d is the mean feature vector of data assigned to ( c, k ) -cluster in current batch.
As such, the sub-centroids can be up-to-date with the change of parameters.
Thougheffectiveenoughinmost cases, batch-wise clustering could not extend to a large number of classes, e . g ., when training on ImageNet [9] with 1K classes using batch size 256, not all the classes/clusters are present in a batch.
To solve this, we store features from several prior batches in a memory, and do clustering on both the memory and current batch.
DNC can be trained by gradient backpropagation in small-batch setting, with negligible lagging ( ∼ 5% training delay on ImageNet).
Versatility.
DNC is a general framework; it can be effortless integrated into any parametric classiﬁer based DNNs, coming with minimal architecture change, i .
e ., removing the parametric softmax layer.
However, DNC changes the classiﬁcation decision-making mode, reforms the training regime, and makes the reasoning process more transparent, without slowing the inference speed.
DNC can be appliedtovariousvisualrecognitiontasks,includingimageclassiﬁcation( § 4.1)andsegmentation( § 4.2).
Transferability.
As a nonparametric scheme, DNC can handle an arbitrary number of classes with ﬁxed output dimensionality ( d ); all the knowledge learnt on a source task ( e . g ., ImageNet classiﬁcation with 1K classes) are stored as a constant amount of parameters in f , and thus can be completely transferred for a new task ( e .
g ., Cityscapes[26] segmentation with 19 classes), under the“pre-trainingand ﬁne-tuning” paradigm.
In a similar setting, the parametric counterpart has to discard 2M parameters during transfer learning ( d =2048 when using ResNet101 [2]).
See § 4.2 for related experiments.
Ad-hoc Explainability.
DNC is a transparent classiﬁer that has a built-in case-based reasoning process, as the sub-centroids are summarized from real observations and actually used during classiﬁcation.
So far we only discussed the case where the sub-centroids are considered as average mean feature vectors of a few training samples with similar patterns.
When restricting the sub-centroids to be elements of the training set ( i .
e ., representative training images), DNC naturally comes with human-understandable explanations for each prediction, and the explanations are loyal to the internal decision mode and not created post-hoc.
Studies regarding ad-hoc explainability are given in § 4.3.
E XPERIMENT 4. E XPERIMENTS ON I MAGE C LASSIFICATION
Dataset.
The evaluation for image classiﬁcation is carried out on CIFAR-10[33] and ImageNet[9].
Network Architecture.
For completeness, we craft DNC on popular CNN-based ResNet50/100 [2] and recent Transformer-based Swin-Small/-Base
Note that, we only remove the last linear classiﬁcation layer, and the ﬁnal output dimensionality of DNC is as many as the last layer feature of the parametric counterpart, i .
e ., 2,048 for ResNet50/100, 768 for Swin-Small, and 1,024 for Swin-Base.
Training.
We use mmclassification 1 as codebase and follow the default training settings.
For
CIFAR-10, we train ResNet for 200 epochs, with batch size 128 . ThememorysizeforDNCmodelsis
setas100batches.
ForImageNet,wetrain100and300epochs with batch size 16 for ResNet and Swin, respectively.
The initial learning rates of ResNet and Swin are set as 0 . 1 and 0 . 0005 , scheduled by a step policy and polynomial annealing policy, respectively.
Limited by our GPU capacity,thememory sizesaresetas 1 , 000 and 500 batchesforDNCversionsofResNetandSwin,respectively.
Otherhyper-
[Table]
parametersareempiricallysetas: K =4 and µ =0 .
Models are trained from scratch on eight V100 GPUs.
Results on CIFAR-10.
Table 1 compares DNC with the parametric counterpart, based on the most representative CNN network architecture, i .
e ., ResNet.
As seen,DNCobtainsbetterperformance:DNCis 0.23 % higheronResNet50,and 0.24 %higheronResNet101, usingfewerlearnableparameters.
Withtheexactsame backbonearchitecturesandtrainingsettings,onecansafely attribute the performance gain to DNC.
[Table]
Results on ImageNet.
Table 2 illustrates again our compelling results over different vision network architectures.
In terms of top-1 acc.
our
DNC exceeds the parametric classiﬁer by 0.29 % and 0.28 %onResNet50andResNet101,respectively.
DNC also obtains promising results with
Transformer architecture: 83.26 % vs 83.02% on
Swin-S, 83.68 % vs 83.36% on Swin-B.
These results are impressive, considering the transparent, case-based reasoning nature of DNC.
One may notice another nonparametric, k -NN classiﬁer [49] reports 76.57% top-1 acc.
,basedonResNet50.However,[49]istrainedwith 130 epochs; in this setting, DNC gains 76.64%.
Moreover, as mentioned in § 2, [49] poses huge storage demand, i .
e ., retaining the whole ImageNet training set ( i .
e ., 1.2M images) to perform the k -NN decision rule, and suffers from very low efﬁciency, caused by extensive comparisons between each test image and all the training images.
These limitations prevent the adoption of [49] in real application scenarios.
In contrast,DNConlyreliesona small set of class representatives ( i .
e .,four sub-centroids per class) for classiﬁcation decision-making and causes no extra computation budget during network deployment.
4. E XPERIMENTS ON S EMANTIC S EGMENTATION
Dataset.
TheevaluationforsemanticsegmentationiscarriedoutonADE20K[37]andCityscapes[26].
Segmentation Network Architecture.
For comprehensive evaluation, we approach DNC on three famous segmentation models ( i .
e ., FCN [34], DeepLab V3 [35], UperNet [36]), using two backbone architectures ( i .
e ., ResNet101 [2] and Swin-B [3]).
For the segmentation models, the only architectural modiﬁcation is the removal of the “segmentation head” ( i .
e ., 1 × 1 conv based, pixel-wise classiﬁcation layer).
For the backbone networks, we respectively adopt parametric classiﬁer based and our nonparametric, DNC based versions, which are both trained on ImageNet[9] and reported in Table2, for initialization.
Thus for each segmentation model, we derive four variants from the different combinations of parametric and DNC versions of the backbone and segmentation network architectures.
Training.
We adopt mmsegmentation 2 as the codebase, and follow the default training settings.
We train FCN and DeepLab V3 with ResNet101 using SGD optimizer with an initial learning rate 0 . 1 , and UperNet with Swin-B using AdamW with an initial learning rate 6e-5.
For all the models, the learning rate is scheduled following a polynomial annealing policy.
As common practices [90, 91], we train the models on ADE20K train with crop size 512 × 512 and batch size 16 ; on Cityscapes train with crop size 769 × 769 and batch size 8 .
All the models are trained for 160 K iterations on both datasets.
Standard data augmentation techniques are used, including scale and color jittering, ﬂipping, and cropping.
The hyper-parameters of DNC are by default set as: K =10 and µ =0 . 999 .
[Table]
Performance on Segmentation.
As summarized in Table 3, our DNC segmentation models, no matter whether using DNC based backbones, obtain better performance over parametric competitors, i .
e ., FCN + ResNet101,
DeepLab V3 + ResNet101, UperNet +
Swin-B,acrossdifferent segmentation and backbone network architectures.
Taking as an example the famous FCN, the original version gains 39.91% and 75.6% mIoU on ADE20K and Cityscapes, respectively.
By comparison, with the same backbone – ResNet101,
DNC-FCN boosts the scores to 41.1% and 76.7%.
When turning to DNC pre-trained backbone – DNC-ResNet101, our DNC-FCN outperforms again its parametric counterpart – FCN, i .
e ., 42.3% vs 40.4% on ADE20K, 77.5% vs 76.3% on
Cityscapes.
Similar trends can be also observed for DeepLab V3 and UperNet.
Analysis on Transferability.
One appealing feature of DNC is its strong transferability, as DNC learns classiﬁcation by directly comparing data samples in the feature space.
The results in Tablealso evidence this point.
For example, DeepLab V3 – even a parametric classiﬁer based segmentation model – can achieve large performance gains, i .
e ., 44.6% vs 44.1% on ADE20K, 78.7% vs 78.1% on
Cityscapes, after using DNC-ResNet101 for ﬁne-tuning.
When it comes to DNC-DeepLab V3 , better performance can be achieved, after replacing ResNet101 with DNC-ResNet101, i .
e ., 45.7% vs 45.0% on ADE20K, 79.8% vs 79.1% on Cityscapes.
We speculate this is because, when the segmentation and backbone networks are both built upon DNC, the model only needs to adapt the original representation space to the target task, without learning any extra new parameters.
Also, these impressive results reﬂect the innovative opportunity of applying our DNC for more downstream visual recognition tasks, either as a new classiﬁcation network architecture or a strong and transferable backbone.
4. S TUDY OF A D - HOC E XPLAINABILITY
So far, we have empirically showed that, by inheriting the intuitive power of Nearest Centroids and the strong representation learning ability of DNNs, DNC can serve as a transparent yet powerful tool for visual recognition tasks with improved transferability.
When anchoring the class sub-centroids to real observations ( i .
e ., actual images from the training dataset), instead of selecting them as cluster centers ( i .
e ., mean features of a set of training images), one may expect DNC can gain stronger, adhoc explainability.
Next we will show this is indeed possible, only at negligible performance cost.
Experimental Setup.
We still train DNC version of ResNet50 on ImageNet train for 100 epochs.
In the ﬁrst 90 epochs, the model is trained in a standard manner, i .
e ., computing the class sub-centroids as cluster centers.
Then we anchor each sub-centroid to its closest training image, based on the cosine similarity of features.
In the ﬁnal 10 epochs, the sub-centroids are only updated as the features goo s e d e s k c
o
m pu t e r c a s tl e c on ve r ti b l e
IF ([ I , ]>
[ I , ] AND
AND [ I, ]>
[ I, ])
OR ([ I , ]>
[ I , ] AND
AND [ I, ]>
[ I, ])
OR ([ I , ]>
[ I , ] AND
AND [ I, ]>
[ I, ])
OR ([ I , ]>
[ I , ] AND
AND [ I, ]>
[ I, ]) THEN ( goose ) of their anchored training images.
Besides this, all the other training settings are as normal.
In this way, we can get a more interpretable DNC-ResNet50.
Interpretable Class Subcentroids.
The top of Fig.
plots our discovered subcentroid images for four
ImageNet classes.
These representative images are diverse in appearance, viewpoints, illuminations, scales, etc. , characterizing their corresponding classes and allowing humans to view and understand.
[Table]
Performance with Improved Interpretability.
We then report the score of our DNC-ResNet50 based on the interpretable class representatives on ImageNet val .
As shown in Table 4, enforcing the class sub-centroids as real training images only brings marginal performance degradation ( e .
g ., 76.49%
→ 76.37% top-1 acc.)
, while coming with better interpretability.
More impressively, our explainable
DNC-ResNet50 even outperforms the vanilla black-box ResNet50 , e . g ., 76.37% vs 76.20%.
Explain Inner Decision-Making Mode based on IF · · · Then Rules.
With the simple Nearest Centroids mechanism, we can use the representative images to form a set of IF · · · Then rules [4], so as to intuitively interpret the inner decision-making mode of DNC for human users.
In particular, let ˆ I denote a sub-centroid image for class c , ˇ I 1: T representative images for all the other classes, and I a query image.
One linguistic logical IF · · · Then rule can be generated for ˆ I :
[Equation] ˆ I ] >
[ I, ˇ I 1 ] AND [ I, ˇ I ] >
[ I, ˇ I 2 ] AND · · · AND [ I, ˆ I ] > [ I [Equation]
ˇ I T ] (cid:1)
THEN ( class c ) , [Equation] 9) where [ · , · ] stands for similarity, given by DNC.
The ﬁnal rule for class c is created by combining all the rules of K sub-centroid images ˆ I 1: K of class c (see Fig.
3 bottom):
IF [Equation] [Equation] [Equation]
[ I, ˇ I 1 ] AND · · · AND [ I, ˆ I 1 ] >
[ I, ˇ I T ] (cid:1)
OR (cid:0)
[ I, ˆ I 2 ] >
[ I, ˇ I 1 ] AND · · · AND [ I, ˆ I 2 ] >
[ I, ˇ I T ] (cid:1)
OR · · · OR (cid:0)
[ I, ˆ I K ] > [ I, ˇ I 1 ] AND · · · AND [ I, ˆ I K ] >
[ I, ˇ I T ] (cid:1) THEN ( class c ) .
[Equation]
[Figure]
t ou c an ho r nb ill v u lt u r e m agp
i e [Figure]
DNC s prediction: white wolf
Groundtruth: kuvasz w h it e w o lf k u v a sz
G r e a t
P y r e n ee s i ce
b e a r
[Figure]
[Table]
.2 % (c) Momentum coefﬁcient ( µ )
Interpret Prediction Based on (Dis)similarity to Sub-centroid Images.
Based on the interpretable class representatives, DNC can explain its predictions by letting users view and verify its computed (dis)similarity between query and class sub-centroid images.
As shown in Fig. 4(a), an observation is correctly classiﬁed, as DNC thinks it looks (more) like a particular exemplar of “ toucan ”.
However, inFig.4(b),DNCstrugglestoassigntheobservationtotwoexemplarsfrom“ whitewolf ”and “ kuvasz ” respectively, and makes a wrong decisionﬁnally.
ThoughusersareunclearhowDNCmapsanimageto feature, they can easily understand the decision-making mode [43] ( e .
g .
, why is one class predicted over another), and verify the calculated (dis)similarity – the evidence for classiﬁcation decision.
4. D IAGNOSTIC E XPERIMENT
To perform extensive ablation experiments, we train ResNet101 classiﬁcation and DeepLab V3 segmentation models for 100 epochs and 80 K iterations, on ImageNet and ADE20K, respectively.
Class Sub-centroids.
Table 5a studies the impact of the number of class sub-centroids ( K ) for each class.
When K = 1 , each class is represented by its centroid – the average feature vector of all the training samples of the class (Eq. 3), without clustering.
The corresponding baseline obtains 77 .
31% top-1 acc.
and 43 .
2% mIoU for classiﬁcation and segmentation, respectively.
For classiﬁcation, increasing K from 1 to 4 leads to better performance ( i .
e ., 77 .
31% → 77 . 80% ).
This supports our hypothesis that one single class weight/center is far enough to capture the underlying data distribution and proves the efﬁcacy of our clustering based sub-class pattern mining.
We stop using K >as the required memory exceeds the computational limit of our hardware.
We ﬁnd similar trends on segmentation; using more sub-centroids ( K : 1 → 10 )brings noticeable performance boost: 43.2% → 44.3%.However,increasing K above10provides marginalorevennegativegain.
Thismaybebecause over-clustering ﬁnds some insigniﬁcant patterns, which are trivial or harmful for decision-making.
External Memory.
We next study the inﬂuence of the external memory, only used in image classiﬁcation.
As shown in Table 5b, DNC gradually improves the performance as the increase of the memory size.
It reaches 77 .
80% top-1 acc.
at size 1000 .
However, the results are still not reaching the performance saturating point, but rather the upper limit of our hardware’s computational budget.
Momentum Update.
Last, we ablate the effect of the momentum coefﬁcient µ (Eq.
8) that controls the speed of sub-centroid online updating.
From Table 5c we ﬁnd the behaviors of µ are consistent in both tasks.
In particular, DNC performs well with larger coefﬁcients ( i .
e ., µ ∈ [0 . 999 , 0 . 9999] ), signifying the importance of slow updating.
The performance degrades when µ ∈ [0 . 9 , 0 . 99] , and encounters a large drop when µ =
( i .
e ., only using batch sub-centroids as approximations).
C ONCLUSION
We present deep nearest centroids (DNC), building upon the classic idea of classifying data samples according to nonparametric class representatives.
Compared to classic parametric models, DNC has merits in: i) systemic simpli- city by bringing the intuitive Nearest Centroids mechanism to DNNs; ii) automated discovery of latent data structure using within-class clustering; iii) direct supervision of representation learning, boosted by unsupervised sub-pattern mining; iv) improved transferability thatlosslesstransferslearnableknowledgeacrosstasks; and v) ad-hocexplainabilitybyanchoringclass exemplars with real observations.
Experiments conﬁrm the efﬁcacy and enhanced interpretability.
SUMMARY OF THE APPENDIX
This appendix contains additional details for the ICLR 2023 submission, titled Visual Recognition with Deep Nearest Centroids” .
The appendix is organized as follows:
• § A provides the pseudo code of DNC.
• § B introduces more quantitative results on image classiﬁcation.
• § C provides additional semantic segmentation results on COCO-Stuff [92] dataset. • § D presents the corresponding error bars on Tables 1, 2, and 3.
• § E investigates the potential of DNC in sub-categories discovery.
• § F reports additional diagnostic experiments for further investigations on K , update ratio of external memory, ImageNet capacity, and feature size.
• § G depicts more visual examples regarding the ad-hoc explainability of DNC.
• § H plots qualitative semantic segmentation results.
• § I discusses our limitations, societal impact, and directions of our future work.
A P SEUDO C ODE OF DNC AND C ODE R ELEASE
The pseudo-code of DNC is given in Algorithm 1.
To guarantee reproducibility, our full implementation shall be publicly released upon paper acceptance.
B M ORE E XPERIMENTS ON I MAGE C LASSIFICATION
Additional Results on CIFAR-10 [33].
CIFAR-10 dataset contains 60 K ( 50 K/ 10 K for train / test )
32 × 32 colored images of 10 classes.
Table 6 reports additional comparison results on
CIFAR-10, based on ResNet-18 [2] network architecture.
As seen, in terms of top-1 accuracy,
our
DNC is 0.94 % higher than the parametric counterpart, under the same training setting.
[Table]
Additional Results on CIFAR-100 [33].
Table 7 reports comparison results on CIFAR-100, based onResNet50 and ResNet101 network architectures[2].
CIFAR-100 dataset has 100 classes withtraining images and 100 testing images per class.
We can ﬁnd that, DNC obtains consistently better performance, compared to the classic parametric counterpart.
Speciﬁcally, DNC is 0.10 % higher on
ResNet50, and 0.16 % higher on ResNet101, in terms of top-1 accuracy.
[Table] [Table]
[Equation]
=
0, :] P[c, n !
= 0, :] =
P_c return loss def online_clustering(L, iters=3, epsilon=0.05) Q = torch.exp(L / epsilon)
Q /= torch.sum(Q) for _ in range(R):
# row normalization
Q /= torch.sum(Q, dim=1, keepdim=
True) Q /=
K
# column normalization Q /= torch.sum(L, dim=0, keepdim=
True)
Q /= N # make sure the sum of each column to be 1 Q *= N return one_hot(Q)
AdditionalResultsonImageNet[9]usingLightweightBackboneNetworkArchitectures.
Tablereports performance on ImageNet [9], using two lightweight backbone architectures: MobileNetV2 [93], and Swin-T [3].
As can be seen, DNC, again, attributes decent performance.
In particular, our DNC is 0.28 % and 0.30 % higher on MobileNet-V2 and Swin-Tiny, respectively.
It is worth noticing that DNC can efﬁciently reduce the number of learnable parameters when the parametric classiﬁer occupies a massive proportion in the original lightweight classiﬁcation networks.
Taking
MobileNet-V2 as an example: our DNC reduces the number of learnable parameters from 3.50M to 2.22 M.
[Table]
C M ORE E XPERIMENTS ON S EMANTIC S EGMENTATION
For through evaluation, we conduct extra experiments on COCO-Stuff [92], a famous semantic segmentation dataset.
COCO-Stuff contains 9 K/ 1 K images for train/test of 80 object classes and 91 stuff classes.
Similar to § 4.2, we approach DNC on FCN[34],DeepLab V3 [35],and UperNet[36], using two backbone architectures, i .
e ., ResNet101 [2] and Swin-B [3].
All models are obtained by following the standard training settings of COCO-Stuff train , i .
e ., crop size 512 × 512 , batch size 16 , and 40 K iterations.
From Table 9 we can draw similar conclusions: in comparison with the parametric counterpart, DNC produces more precise segments and yields improved transferability.
[Table]
D E RROR B ARS
In this section
[Table]
we report standard deviation error bars on Tables 10,
[Table]
and 12 for our main experiments regarding image classiﬁcation ( § 4.1) and semantic segmentation ( § 4.2).
E E XPERIMENTS ON S UB -C ATEGORIES D ISCOVERY
Through unsupervised
[Table]
within-class clustering
[Table]
our DNC represents each class as a set of automatically discovered class sub-centroids ( i
[Table]
e ., cluster center).
This allows DNC to better describe
[Table] [Table]
[Table]
the underlying, multimodal data structure and robusty depict for rich intra-class variance.
In other words, our DNC can effectively capture sub-class patterns, which is conducive to algorithmic performance.
Such a capacity of sub-patter mining is also considered crucial for good transferable features – representations learnt on coarse classes are capable of ﬁne-grained recognition [94].
In order to quantify the ability of DNC for automatic sub-category discovery, we follow the experimental setup posed by [94] – learning the feature embedding using coarse-grained object labels, and evaluating the learned feature using ﬁne-grained object labels.
This evaluation strategy allows us to assess the feature learning performance regarding how well the deep model can discover variations within each category.
A conjecture is that deep networks that perform well on this test have an exceptional capacity to identify and mine sub-class patterns during training, which the proposed DNC seeks to rigorously establish.
In particular, the network is ﬁrst trained on coarse-grained labels with the baseline parametric softmax and with our non-parametric DNC using the same network architecture.
After training on coarse classes, we use the top-1 nearest neighbor accuracy in the ﬁnal feature space to measure the accuracy of identifying ﬁne-grained classes.
The classiﬁcation performance evaluated in such setting is referred as induction accuracy as in [94].
Next we provide our experimental results on
CIFAR100 [33] and ImageNet [9], respectively.
Performance of Sub-category Discovery on CIFAR100.
CIFAR100 includes both ﬁne-grained annotations in 100 classes and coarse-grained annotations in 20 classes.
We examine sub-category discovery by transferring representation learned from 20 classes to 100 classes.
As shown in Table 13, DNC consistently outperforms the parametric counterpart: DNC increases 0.12%, in terms of the standard top-1 accuracy, on both ResNet50 and ResNet101 architectures.
Nevertheless, when transferred to CIFAR100 ( i .
e ., 100 classes) using k -NN, a signiﬁcant loss occurs on the baseline: 53.22% and 54.31% top-1 acc.
on ResNet50 and ResNet101, respectively.
Our features, on the other hand, provide 14.24% and 13.82% improvement over the baseline, achieving 67.46% and 68.13% top-1 acc.
on 100 classes on ResNet50 and ResNet101, respectively.
In addition, in comparison the parametric model, our approach results in only a smaller drop in transfer performance, i .
e ., -18.87 vs -32.99 on ResNet50, and -18.47 vs -32.17 on ResNet101.
using CIFAR-20 pre-trained models.
Numbers reported with knearest neighbor classiﬁers.
See § E for more details
[Table]
Performance of Sub-category Discovery on ImageNet.
Table 14 provides experimental results of sub-category discovery on ImageNet val .
As in [94], 127 coarse ImageNet categories are obtained by top-down clustering of 1000 ImageNet categories on WordNet tree.
Training on the 127 coarse classes, DNC improves the performance of baseline by 0.10% and 0.03%, achieving 84.39% and 85.91% on ResNet50 and ResNet101, respectively.
When transferring to the 1000 ImageNet classes using k -NN, our features provide huge improvements, i .
e ., 8.98% and 9.29%, over the baseline.
[Table]
The promising transfer results on CIFAR100 and ImageNet serve as strong evidence to suggest that our DNC is capable of automatically discovering meaningful sub-class patterns – latent visual structures that are not explicitly presented in the supervisory signal, and hence handle intra-class variance and boost visual recognition.
F A DDITIONAL D IAGNOSTIC E XPERIMENT
Output Dimensionality.
As stated in § 4, the ﬁnal output dimensionality of our DNC is set as many as the one of the last layer of the parametric counterpart, for the sake of fair comparison.
However, owing to the distance-/similarity-based nature, DNC has the ﬂexibility to handle any output dimensionality.
In Table 15, we further study the inﬂuence of the output dimensionality of DNC.
As seen, when setting the ﬁnal output dimensionality as 1280, we can achieve 76.61% top-1 acc., which is higher than the initial 2048 dimension conﬁguration, i .
e ., 76.49%.
We attribute the reason for the better balance between memory capacity and feature dimensionality with the limitation of hardware computational budget – when reducing the ﬁnal output dimensionality, the expressibility of the ﬁnal feature is weakened but more image features can be stored in the external memory for more accurate sub-centroid estimation.
[Table]
[Table]
G A DDITIONAL S TUDY OF A D - HOC E XPLAINABILITY
Interpretable Class Sub-centroids.
In Fig.
5, we show more examples of sub-centroid images for eight ImageNet classes.
These representative images are automatically discovered by DNC, and can be intuitively viewed by users.
As seen, the class sub-centroid images are able to capture diverse characteristics of their classes, in the aspects of appearance, viewpoints, scales, illuminations, etc .
Interpret Prediction Based on (Dis)similarity to Sub-centroid Images.
Fig.
6 provides more results regarding the (dis)similarity-based interpretation of DNC prediction.
As seen, based on the similarity of test images to the class sub-centroid images, users can clearly understand the decision making mode and make veriﬁcation.
DNC’s compelling explainability enables it to establish trustworthiness with humans and empowers its potential in high stake applications.
[Figure]
DNC's prediction:
Groundtruth: 81.2% 13.0% … …
g r ey f o x k it f
o x r e d f o x 5.3%
DNC's prediction:
Groundtruth: t ap e p l a ye r c a ss e tt e p l a ye r c a ss e tt e
C D p l a ye r 19.7% 69.8% 6.9% 2.3%
DNC's prediction:
Groundtruth:
[Figure]
[Table] [Table]
[Figure]
[Table]
… … s t a ti on w agon g r ill e
[Figure]
DNC's prediction:
Groundtruth:
O x w a t e r b u ff a l o b
i s on
[Figure]
[Table] [Table]
[Figure]
[Table] [Table]
[Figure]
[Table]
DNC's prediction:
Groundtruth:
[Figure] [Figure]
DNC's prediction:
Groundtruth:
[Figure]
a m i ngo s poonb ill p e li c an b l a ck s t o r k 47.2% 51.0%
[Figure]
[Figure]
0002% siamese cat station wagon grey fox grey fox station wagon flamingo siamese cat dho l e spoonbill tape player cassette player water buffalo
Ox
I n d i a n e l e p h a n t
H Q UALITATIVE R ESULTS ON S EMANTIC S EGMENTATION
Fig.
7 and Fig.
8 illustrate a few representative visual examples of semantic segmentation results on ADE20K [37] and Cityscapes [26], respectively.
In comparison with the parametric counterpart, our approach produces more precise segments in different challenging scenes (for example, where objects with drastic photometric or geometric appearances).
For instance, UperNet [36]-Swin-B [3] confuses on neighbouring objects with similar colors ( e.g. , desk and chair) and leaves a large false-negative regions (see the ﬁrst image of Fig. 7); it also has difﬁculties in segmenting small scale objects ( e.g. , motorbike semantic parsing).
Among these examples, DNC consistently demonstrates supreme performance.
Essentially, we argue that the proposed DNC has a stronger ability to supervises the pixel embedding space via anchoring sub-centroids directly, leading to a better predictions on segmenting such hard cases.
I L IMITATION AND F UTURE W ORK
Limitation.
One limitation of our approach is that the Sinkhorn-Knopp algorithm runs in time (cid:101) O ( n 2 (cid:15) 3 ) which would reduce the training efﬁciency.
Though in practice, we ﬁnd 3 sinkhorn loops per training iteration is sufﬁcient enough for model representation, bringing a minor computational overhead ( i.e. , ∼ 5% training delay on ImageNet).
This also indicates possible directions for our future research.
Social Impact.
This work introduces DNC possessing the nature of nested simplicity, intuitive decision-making mechanism and even ad-hoc explainability.
On positive side, the approach advances model accuracy and is valuable in safety-sensitive applications by showing the advanced robustness on sub-categories discovery, e.g. , quality analytics, autonomous driving, etc. .
For potential negative social impact, our DNC struggles in handling out-of-distribution data, which is a common limitation of all the discriminative classiﬁers.
Hence its utility in open-world scenarios should be further examined.
Future Work.
In the future, in addition to investigating more powerful, time-efﬁciency online clustering algorithms with further advantages on training speed and testing accuracy, we will explore the possibility of unifying close-set and open-world visual recognition withinourframework.
Moreover, considering the similarity-/distance-based nature, the incorporation of metric learning based training objectives is also another promising direction for further boosting the performance.
[Figure] [Figure]