S CHEMA
I NFERENCE
FOR
I NTERPRETABLE
I MAGE
C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we study a novel inference paradigm, termed as schema inference ,that learns to deductively infer the explainable predictions by rebuilding the priordeep neural network (DNN) forwarding scheme, guided by the prevalent philosophical cognitive concept of schema .
We strive to reformulate the conventional modelinference pipeline into a graph matching policy that associates the extracted visualconcept of an image with the pre-computed scene impression, by analogy withthe human reasoning mechanism via impression matching.
To this end, we devisean elaborated architecture, termed as SchemaNet , as a dedicated instantiation ofthe proposed schema inference concept, that models both the visual semanticsof input instances and the learned abstract imaginations of target categories astopological relational graphs.
Meanwhile, to capture and leverage the compositional contributions of visual semantics in a global view, we also introduce auniversal Feat2Graph scheme in SchemaNet to establish the relational graphs thatcontain abundant interaction information.
Both the theoretical analysis and theexperimental results on several benchmarks demonstrate that the proposed schemainference achieves encouraging performance and meanwhile yields a clear pictureof the deductive process leading to the predictions.
Our code and model will bemade publicly available.
I NTRODUCTION
“Now this representation of a general procedure of the imagination for providing aconcept with its image is what I call the schema for this concept 1 .”
— Immanuel Kant
Deep neural networks (DNNs) have demonstrated the increasingly prevailing capabilities in visualrepresentations as compared to conventional hand-crafted features.
Take the visual recognition taskas an example.
The canonical deep learning (DL) form for image recognition is to yield an effectivevisual representation with a stack of non-linear layers along with a fully-connected (FC) classifierat the end (He et al., 2016;
Dosovitskiy et al., 2021;
Tolstikhin et al., 2021), where specifically theinner-product similarities are computed with each category’s embedding as the prediction.
Despitethe success of such a DL scheme, existing deep networks are typically required to simultaneouslyperceive low-level patterns as well as high-level semantics to make predictions (Zeiler & Fergus, 2014;
Krizhevsky et al., 2017).
As such, both the procedure of computing visual representations and thelearned category-specific embeddings are opaque to humans, leading to challenges in security-matterscenarios, such as autonomous driving and healthcare applications.
In this paper, unlike prior arts that merely obtain the targets in a black-box manner, here we striveto devise an innovative and generalized DNN inference paradigm by reformulating the traditionalone-shot forwarding scheme into an interpretable DNN reasoning framework, resembling what occursin deductive human reasoning.
Towards this end, inspired by the schema in Kant’s philosophythat describes human cognition as the procedure of associating an image of abstract conceptswith the specific sense impression, we propose to formulate DNN inference into an interactivematching procedure between the local visual semantics of an input instance and the abstract category
[Figure] [Figure]
imagination, which is termed as schema inference in this paper, leading to the accomplishment ofinterpretable deductive inference based on visual semantics interactions at the micro-level.
To elaborate the achievement of the proposed concept schema inference , we take here imageclassification, the most basic task in computer vision, as an example to explain our technical details.
At a high level , the devised schema scheme leverages a pre-trained DNN to extract feature ingredientswhich are, in fact, the semantics represented by a cluster of deep feature vectors from a specific localregion in the image domain.
Furthermore, the obtained feature ingredients are organized into aningredient relation grpah (IR-Graph) for the sake of modeling their interactions that are characterizedby the similarity at the semantic-level as well as the adjacency relationship at the spatial-level.
Wethen implement the category-specific imagination as an ingredient relation atlas (IR-Atlas) for alltarget categories induced from observed data samples.
As a final step, the graph similarity between aninstance-level IR-Graph and the category-level IR-Atlas is computed as the measurement for yieldingthe target predictions.
As such, instead of relying on deep features, the desired outputs from schemacontribute only from the relationship of visual words, as shown in Figure 1.
More specifically, our dedicated schema -based architecture, termed as SchemaNet in this paper,is based on vision Transformers (ViTs) (Touvron et al., 2021; Dosovitskiy et al., 2021), which isnowadays the most prevalent vision backbone.
To effectively obtain the feature ingredients , wecollect the intermediate features of the backbone from probe data samples clustered by the k -meansalgorithm.
IR-Graph is then established through a customized Feat2Graph module that models thediscretized array of ingredients as the graph vertices, and meanwhile builds the connections, whichindicates the ingredient interaction with the self-attention mechanism (Vaswani et al., 2017) of the
ViT and the spatial adjacency.
Eventually, the graph similarity of IR-Graph and IR-Atlas is evaluatedvia a shallow graph convolutional network (GCN).
Our work relates to several existing methods that mine semantic-rich visual words from a DNNbackbone for a self-explanatory model (Brendel & Bethge, 2019;
Chen et al., 2019; Nauta et al.,2021; Xue et al., 2022b).
In particular, Brendel & Bethge (2019) propose BagNet by using a
DNN as the visual word extractor that is analogous to bag-of-visual-words BoVW Yang et al SIFT
(Lowe, 2004)
Moreover Chen et al
Despite encouraging compositional however
[Figure]
In sum, our contribution is an innovative schema inference paradigm that reformulates the existingblack-box network forwarding into a deductive DNN reasoning procedure that allows for a clearinsight into how the class evidence is gathered and deductively leads to the target predictions.
This isachieved by building the dedicated IR-Graph and IR-Atlas with the extracted feature ingredients andthen performing graph matching between IR-Graph and IR-Atlas to derive the desired predictions.
We also provide a theoretical analysis to explain the interpretability of our schema inference results.
Experimental results on CIFAR-10/100, Caltech-101, and ImageNet demonstrate that the proposedschema inference scheme yields results superior to the state-of-the-art on general visual recognitiontasks.
Further, we demonstrate that by transferring to unseen tasks without fine-tuning the matcher,the learned knowledge of each class is, in fact, stored in IR-Atlas rather than the matcher, therebyexhibiting the high interpretability of the proposed method.
S CHEMA N ET
In this section, we introduce our proposed SchemaNet in detail.
The overall procedure is illustratedin Figure 2, including a Feat2Graph module that converts deep features to instance IR-Graphs, alearnable IR-Atlas, and a matching network for making predictions.
The main idea is to model aninput image as an instance-level graph in which nodes are local semantics captured by the DNNbackbone and edges represent their interactions.
Meanwhile, a category-level graph is maintainedas the imagination for each class driven by training samples.
Finally, by measuring the similaritybetween the instance and category graphs, we are able to interpret how predictions are made.
2. P RELIMINARY
We first give a brief review of the architecture of ViTs.
The simplest implementation of ViTs forimage classification is proposed by Dosovitskiy et al.
(2021), which treats an image as a sequence ofindependent 16 × 16 patches.
The embeddings of all patches are then directly fed to a Transformerbased network (Vaswani et al., 2017), a stack of multiple encoder layers with the same structure:a multi-head self-attention mechanism (MHSA) followed by a multilayer perceptron (MLP) withresidual connections.
Additionally, they append a class token (CLS) to the input sequence forclassification alone rather than the average pooled feature.
Following this work, Touvron et al.
(2021)propose DeiT appending another distillation token (DIST) to learn soft decision targets from theteacher model.
This paper will mainly focus on DeiT-based backbones due to the relatively simpleyet efficient network architecture.
Let X ∈ R ( n + ζ ) ×
d denote the input feature (a sequence of n visual tokens and ζ auxillary tokens,e.g. , CLS and DIST, each of which is a d -dimensional embedding vector) to a MHSA module.
Theoutput can be simplified as MHSA( X ) =
(cid:80)
Hh =1 ˜Ψ h XW h , where ˜Ψ h ∈ R ( n + ζ ) × ( n + ζ ) denotes theself-attention matrix normalized by row-wise softmax of head h ∈ { 1 , . . .
, H } , and
W h ∈ R
d × d is a learnable projection matrix.
As the MLP module processes each token individually, the visual tokeninteractions take place only in the MHSA modules, facilitating the extraction of their relationshipsfrom the perspective of ViTs.
For convenience, we define some notations related to the self-attention matrix.
Let ¯Ψ be the averageattention matrix of all heads and Ψ be the symmetric attention matrix:
Ψ = (¯Ψ + ¯Ψ ⊤ ) / 2 .
Wepartition Ψ into four submatrices:
[Equation]
where Ψ A ∈ R ζ × n is the attention to the auxiliary tokens, Ψ V ∈ R n × n represents the relationshipsof visual tokens, and Ψ ∗ ∈ R ζ × ζ .
Finally, let ψ CLS ∈ R n represent the attention values to the CLStoken extracted from Ψ A .
2. F EAT 2G RAPH
Shown in Figure 2, given a pre-trained backbone and an input image, the intermediate feature Xis converted to an IR-Graph G for inference by Feat2Graph module, which includes three steps:(1) discretizing X into a sequence of feature ingredients with specific semantics; (2) mapping theingredients to weighted vertices of IR-Graph; (3) assigning a weighted edge to each vertex pairindicating their interaction.
To begin with, we start by defining IR-Graph and IR-Atlas mathematically.
Definition 1 (IR-Graph) .
An IR-Graph is an undirected fully-connected graph G =
( E, V ) , in whichvertiex set V is the set of feature ingredients for an instance or a specific category, and edge set Eencodes the interactions of vertex pairs.
In particular, to indicate the vertex importance (for instance, “bird head” may contribute to theprediction of bird more than “sky” in human cognition), a non-negative weight λ ∈ R + is assigned toeach vertex.
Let Λ ∈ R | V | + = { λ i } | V | i =1 be the collection of all vertex weights.
Besides, the interactionbetween vertex i and j is quantified by a non-negative weight e i,j ∈ R + .
With a slight abuse ofnotation, we define E ∈ R | V |×| V | + as the weighted adjacency matrix in the following sections.
We define IR-Atlas to represent the imagination of all categories:
Definition 2 (IR-Atlas) .
An IR-Atlas ˆ G of C classes is a set of category-level IR-Graphs, in whichelement ˆ G c =
( ˆ E c , ˆ V c ) for class c has learnable vertex weights ˆ Λ c and edge weights ˆ E c .
Discretization.
The main purpose of feature discretization is to mine the most common patternsappearing in the dataset, each of which corresponds to a human understandable semantic, namedvisual word, such as “car wheel” or “bird head”.
However, local regions with the same semanticmay show differently because of scaling, rotation, or even distortion.
In our approach, we utilize the
DNN backbone to extract a relatively uniform feature instead of the traditional SIFT feature utilizedin (Yang et al., 2007).
To be specific, a visual vocabulary Ω = { ω i } Mi =1 ( ω i ∈ R d ) of size M isconstructed by k -means clustering running on the collection X of visual tokens extracted from theprobe dataset 2 .
Further, a deep feature X (CLS and DIST are removed) is discretized to a sequenceof feature ingredients by replacing each element x with the index of the closest visual word
[Equation]
Strictly, an ingredient is referred to as the index of visual word ω .
With the entire ingredient set
M = { 1 , . . .
, M } , the discretized sequence is denoted as ˜ X =
(˜ x 1 , . . . ,
˜ x n ) , where ˜ x i ∈ M iscomputed from Equation (2).
Feat2Vertex
After we have computed the discretized feature ˜ X , the vertices of this feature arethe unique ingredients: V =
Unique( ˜ X ) .
The importance of each vertex is measured from twocriterions: the contribution to the DNN’s final prediction and the appearance statistically.
Formally,for vertex v ∈ M , the importance is defined as
[Equation]
where Ξ ( v | ˜ X ) is the set of all the appeared position of ingredient v in ˜ X , ψ CLS i is the attentionbetween the i -th visual token and the CLS token, and α 1 , 2 ≥ 0 are learnable weights balancing thetwo terms.
When α 1 = 0 , Λ is equivalent to BoVW representation.
Feat2Edge.
With the self-attention matrix Ψ V extracted from Equation (1), the interactionsbetween vertices can be defined and computed with efficiency.
For any two different verticesu, v ∈ V , the edge weight is the comprehensive consideration of the similarity in the view of ViTand spatial adjacency.
The first term is the average attention between all repeated pairs Π [( u, v ) | ˜
X ] :
[Equation]
where Π [( u, v ) | ˜ X ] is the Cartesian product of Ξ ( u | ˜ X ) and Ξ ( v | ˜ X ) , and Ψ Ci,j is the attention betweenthe visual tokens at i and j positions.
Furthermore, the adjacency is
[Equation]
where function Pos( · ) returns the original 2D coordinates of the input visual token with regard to thepatch array after the patch embedding module of ViTs.
Eventually, the interaction between vertices uand v is the weighted sum of the two components with learnable β 1 , 2 :
[Equation]
Equations (4) and (5) are both invariance when exchanging vertex u and v , so our IR-Graph isequivalent to undirected graph.
It is worth noting that only semantics and their relationships are preserved in IR-Graphs for furtherinference rather than deep features.2. M
[Figure]
GraphConv
WAP
GraphConv
Sim
G C N
After converting an input image to IR-Graph, the matcherfinds the most similar category-level graph in IR-Atlas.
The overall procedure of is shown in Figure 3, which iscomposed of a GCN module and a similarity computingmodule (Sim) that generates the final prediction.
Detailedsettings of the matcher are in Appendix E.2.
For feeding IR-Graphs to the GCN module, we assign eachingredient m ∈ M ( i.e. , graph vertices) with a trainableembedding vector, each of which is initialized from and G -dimensional random vector drawn independently frommultivariate Gaussian distribution N ( 0 , I d G ) .
In the GCN module, we adopt GraphConv (Morris et al. with slight modifications for weighted edges.
Let F ∈ R | V |× d G be the input feature of all vertices to a GraphConv layer, the output is computed as
[Equation]
where σ denotes a non-linear activation function, Normdenotes feature normalization, and
W ∈ R d G × d G is alearnable projection matrix.
After passing through total L G layers of GraphConv, a weighted average pooling (WAP) layer summarizes the vertex embeddings
F ( L G ) weighted by Λ , yielding the graph representation z = ΛF ( L G ) .
Now we have z for G andˆ Z ∈ R C ×
d G =
(ˆ z 1 , . . .
, ˆ z C ) for all graphs in ˆ G .
By computing the inner-product similarities, thefinal prediction logits is defined as y =
ˆ Zz ⊤ .
Prediction interpretation.
Given an instance graph G and a category graph ˆ G , we now analyzehow the prediction is made based on their vertices and edges.
We start with analyzing the similarity score s
= ˆ zz ⊤ , where z = ΛF ( L G ) and ˆ z = ˆ Λ ˆ F ( L G ) are weighted sum of the vertex embeddingsrespectively. For the sake of further discussion, let f ( l ) v denotes the embedding vector of vertex voutput from the l -th GraphConv layer, and λ ( l ) v denotes its weight.
The original vertex embeddingof v is defined as f (0) v , which is identical for the same ingredient in any two graphs.
Besides, letΦ =
V ∩
ˆ V be the set of shared vertices in G and ˆ G .
Now, the computation of s can be expanded as
[Equation]
Since directly analyzing Equation (8) is hard, we show an approximation and proof it in Appendix B.
Theorem 1.
For a shallow GCN module, Equation (8) can be approximated by
[Equation]
Particularly, if L G = 0 and α 1 = 0 , our method is equivalent to BoVW with a linear classifier.
Further, as delineated in Corollary 1, the interpretability of the graph matcher with a shallow GCNcan be stated as: (1) the final prediction score for a category is the summation of all present classevidence (shared vertices ϕ in Φ ) represented by the vertex weights; (2) the shared neighbors (localstructure) connected to the shared vertex ϕ also contribute to the final prediction.
2. T RAINING S CHEMA N ET
Except for training SchemaNet by cross-entropy loss L CE between the predictions and ground truths,we further constrain the complexity of IR-Atlas.
More formally, the complexity is defined for bothedges and vertex weights of all graphs in IR-Atlas as [Equation] [Equation] [Equation]
(cid:88)
[Equation]
(10) where function H ( x ) computes the entropy of the input vector x ∈ R k + normalized by its sum of kcomponents, and ˆ E c,u denotes the weighted edges connected to vertex u in ˆ G c of class c .
The finaloptimization goal is
[Equation]
(11) where γ v and γ e are hyperparameters.
Sparsification.
Each graph in IR-Atlas is initialized as a fully-connected graph with random vertexand edge weights.
However, this requires O ( C | V | 2 ) memory space for storage and training the wholeset of edges.
To alleviate this issue, we initialize IR-Atlas by averaging the instance IR-Graphs foreach class and remove the edges connected to vertices whose weights are below a given thresholdδ t = 0 .
01 from the category-specific graph.
Such a procedure will not only dramatically decrease thelearnable parameters but also boost the final performance, as shown in Table 2.
The overall training procedure is shown in Appendix C.
E XPERIMENTS
3. I MPLEMENTATION
Datasets
We evaluate our method on CIFAR-10/100 (Krizhevsky et al., 2009), Caltech-101 (Liet al., 2022), and ImageNet (Deng et al., 2009).
Particularly, Caltech-101 has around 9k images, andwe manually split it into a training set with 7.4k images and a test set with 1.3k images.
Selection of the hyperparameters.
Several hyperparameters are involved in our method, includingλ v,e in Equation (11) for adjusting the sparsity of IR-Atlas.
We set λ v = 0 .
5 and λ e = 0 .
75 as thedefault value for the following evaluation and the sensitive analyses are presented in Appendix H. Theinitial values of learnable α 1 , 2 and β 1 , 2 are set to 0 .
5 , and we show their learning curves in Figure 9.
[Table] [Table]
Visual vocabulary.
Thanks to the relatively uniform representation of visual semantics generated by DNNs [Table] our visualvocabulary size M can be set to around 10 × C with competitive performance.
Table 1 shows the default choice of M ondifferent datasets.
Results with different vocabulary sizes arepresented in Appendix G. Furthermore, all the intermediatefeatures are extracted from the 9-th layer of the backbone ViTs,including DeiT-Tiny, DeiT-Small, and DeiT-Base.
3. E XPERIMENTAL R ESULTS
We evaluate our method and comparison baselines with the following settings:
• BoVW-SIFT: the traditional BoVW approach that utilizing SIFT feature for constructing the visual vocabulary (Yang et al., 2007).
• Base: the base ViT model directly trained on the benchmark datasets with initial weightsobtained from the official repository in (Touvron et al., 2021), which is our backbone.• Backbone-FC: the frozen backbone with a FC layer.
For the intermediate features extractedfrom our frozen backbone, we follow the CNN’s procedure that the global average pooledembeddings are fed to an FC layer for classification.
• BoVW-Deep: the BoVW approach with our extracted visual vocabulary backed ViTs.
• BagNet: the implementation of BagNet backed with the ViT, which is constructed following the “Base” setting.
• SchemaNet: our proposed SchemaNet without initialization.
• SchemaNet-Init: our proposed SchemaNet initialized by the average instance IR-Graphs.
The comparison results are shown in Table 2, where we demonstrate the top-1 accuracy, the numberof learnable parameters, as well as the FLOPs for each setting.
It is noticeable that the proposed
SchemaNet consistently outperforms the baseline methods.
Specifically, for the backbone of DeiTTiny, ours achieves significant improvement ( i.e. , about 25.7% and 35.3% absolute gain on CIFAR-10and CIFAR-100, respectively) over BagNet.
With a larger backbone such as DeiT-Small and DeiTBase, though the performance gap slightly shrinks, our SchemaNet with initializations still yieldsresults superior to BagNet, by an average absolute gain of 12.7%.
The reason is that, unlike BagNet
[Figure]
that uses summation to obtain the similarity map as the BoVW representation (without featurediscretization), “BoVW-Deep” is implemented with the discretized visual words.
As such, both ofthe approaches do not consider the visual word interactions, leading to the inferior performanceespecially when the size of the visual vocabulary expands on CIFAR-100 and Caltech-101.
Moreover, compared with “Backbone-FC” that merely relies on intermediate deep features, theproposed SchemaNet also demonstrates encouraging results, achieving about a 1.4% absolute gain.
Furthermore, when initialization and redundant vertices are removed as indicated in Section 3.1,our “SchemaNet-Init”, with an even less learnable number of parameters, still leads to a higheraccuracy.
The proposed method also delivers gratifying performance, on par with the prevalent deep
ViT especially on CIFAR-10, but gives a clear insight on the reasoning procedure.
We present more experimental results in Appendix F, including the comparison results on ImageNet (Appendix F.1), employing other attribution methods (Appendix F.2), attacking with adversarial images (Appendix F.3), the extendability analysis of our SchemaNet (Appendix F.4), and theinference cost of all components (Appendix F.5).
3. E VALUATION OF THE I NTERPRETABILITY
To figure out how interpretability contributes to the model prediction, we adopt the positive andnegative perturbation tests presented in (Chefer et al., 2021) to give both quantitative and qualitativeevaluations on interpretability..
For a fair comparison, the attention values to the CLS token ϕ CLS areextracted as the pixel relevance for BoVW-Deep, BagNet, and SchemaNet-Init with the DeiT-Tinybackbone.
During the testing stage, we gradually drop the pixels in relevance descending/ascendingorder (for positive and negative perturbation, respectively) and measure the top-1 accuracy of themodels.
We plot the accuracy curves in Figure 4, showing that: (1) in the positive perturbation test,our method behaves similarly to BoVW while outperforming BagNet; (2) in the negative perturbationtest, our method achieves better performance by a large margin (the area-under-the-curve of ours isabout 5.71% higher than BagNet on CIFAR-10, and 11.23% higher than BagNet on CIFAR-100).
3. V ISUALIZATION
In Figure 5, we show the examples of instance IR-Graphs and learned IR-Atlas with DeiT-Tinybackbone trained on Caltech-101 dataset (for visualization purposes, the number of ingredients isset to 256, which are shown entirely in Figure 11).
Each row shows examples of a specific class.
The first column shows the category graphs and excerpts of appeared ingredients in Figure 11 forquicker reference.
We further render the vertices, i.e. , ingredients, in the category graph with differentcolors, and the appeared ingredients in the instance images and graphs are rendered by the same colorcorresponding to the category graph. We interpret the visualization from three perspectives: the interpretability of the ingredients, theinterpretability of the edges, and the consistency between instance and category graphs.
(1) Thanksto the powerful representation learning capability of DNNs, the extracted ingredients are able torepresent explicit semantics (such as “fuselage”, “bird legs”, etc.) with robustness. Besides, the [Table] [Figure]
[Table]
[Figure]
learned weight of each vertex is also consistent with human intuition as the object ingredients gethigher weights than the background ones.
The learned edge weights tend to connect object partsto the adjacent background and other object parts while ignoring the background connections, whichhelps distinguish fine-grain categories with the perception of the surroundings.
(3) As the categorygraphs are driven by the instances, they learn to capture and model the general character of instancesand eventually form the abstract imaginations of all categories.
More examples are presented in Appendix I.
C ONCLUSION AND O UTLOOK
In this paper, we propose a novel inference paradigm, named schema inference , guided by Kant’sphilosophy towards resembling human deductive reasoning of associating the abstract concept imagewith the specific sense impression.
To this end, we reformulate the traditional DNN inferenceinto a graph matching scheme by evaluating the similarity between instance-level IR-Graph andcategory-level imagination in a deductive manner.
Specifically, the graph vertices are visual semanticsrepresented by common feature vectors from DNN’s intermediate layer.
Besides, the edges indicatethe vertex interactions characterized by semantic similarity and spatial adjacency, which facilitatecapturing the compositional contributions to the predictions.
Theoretical analysis and experimentalresults on several benchmarks demonstrate the superiority and interpretability of schema inference.
In future work, we will implement schema inference to more complicated vision tasks, such as visualquestion answering, that enables linking the visual semantics to the phrases in human language,achieving a more powerful yet interpretable reasoning paradigm.
A A DDITIONAL R ELATED W ORK
In this section, we introduce literature that is related to our method.
A. V ISUAL C ONCEPT A NALYSIS
Visual concept analysis aims to interpret the behavior of pre-trained deep models by extractingand tracking intuitive visual concepts during the DNN’s inference procedure.
Previous works usemasks (Zhang et al., 2020b), explanatory graphs (Zhang et al., 2020a) and probabilistic models (Konforti et al., 2020) to interpret the internal layers of convolutional neural networks (CNNs).
Further,
Deng et al.
(2022) propose to capture and analyze the interaction of visual concepts contributingcooperatively to the prediction results for CNNs.
However, they use Shapley values to computethe interaction, which is computationally expensive.
Consequently, we adopt vision Transformers (ViTs) (Dosovitskiy et al., 2021; Touvron et al., 2021) as the DNN backbone of our methodinstead of conventional CNNs thanks to the multi-head self-attention (MHSA) mechanism (Vaswaniet al., 2017) that explicitly encodes the interactions of visual tokens in the similarity level.
More recently, VRX (Ge et al., 2021) is proposed to use a graphical model to interpret the predictionof a pre-trained DNN, in which edges are constructed to represent the spatial relation of visualsemantics.
They further utilize a GCN module to make a prediction with the input structural conceptgraphs by training to mimic DNN’s prediction.
Nevertheless, their approach requires training a graphfor every input sample, which is extremely expensive for applying to large-scale datasets.
Besides,
VRX does not construct the class imagination that carries the category-specific knowledge, meaningthat the GCN predictor must learn to memorize and distinguish all the class-specific graphs, whichimpairs the overall interpretability.
Our proposed schema inference, however, explicitly creates aninteractive matching procedure between the instances and category imagination.
Moreover, our GNNmodel only captures the local structure for gathering class evidence, which is easier to accomplish.
A. V ISUAL C ONCEPT L EARNING
Visual concept learning refers to a range of methods that mine sematic-rich visual words from a DNN,which are further utilized to make interpretable predictions towards a self-explanatory model in reality.
Concept bottleneck models (CBMs) (Koh et al., 2020; Zarlenga et al., 2022; Deng et al., 2022; Wong& McPherson, 2021) link the neurons with human-interpretable semantics explicitly, encouraging thetrustworthiness of DNNs.
Human interventions of learned bottleneck layers can fix the misclassifiedconcepts to improve the model performance.
Another line of visual concept learning, part-prototypebased methods, collectively makes predictions on target tasks with DNN and semantic-rich prototypes,which can be divided into two schools according to their prototypes. (1) Bag-of-Visual-Word (BoVW)approaches (Brendel & Bethge, 2019; Gidaris et al., 2020; Tripathi et al., 2022) obtain prototypes withsemantics through passively clustering the hand-crafted features (Yang et al., 2007) or deep featuresinto a set of discrete embeddings (visual words) that relate to specific visual semantics.
The predictionprocedure of these approaches is analog to the BoW model in natural language processing (NLP),in that an interpretable representation is constructed based on the statistics of the occurrence ofthe visual words, and then fed to an interpretable classifier, such as the linear classifier or decisiontrees.
(2) In contrast, part-prototypical networks (ProtoPNets) (Chen et al., 2019) and the followingworks (Nauta et al., 2021; Xue et al., 2022a; Zhang et al., 2022; Peters, 2022; Rymarczyk et al., 2021)jointly train DNN (as the feature extractor) and parameterized prototypes.
Then decisions are madebased on the linear combination of similarity scores between prototypes and feature vectors at all thespatial locations.
Despite the relatively high performance, as discussed in (Brendel & Bethge, 2019; Hoffmann et al., 2021), the similarity of the learned prototypes and deep features in the embeddingspace may be significantly different in the input space.
As such, we only compare those approachesin which prototypes are generated based on clustering for the same level of interpretability.
To thebest of our knowledge, none of the existing works have explored a deductive inference paradigmbased on the interaction of visual semantics.
A. S CENE G RAPH G ENERATION
A scene graph can be generated from an input image to excavate the collection of objects, attributes, and relationships in a whole scene.
Typical graph-based representations and learning algorithms (Herzig et al., 2018; Chang et al., 2021; Shit et al., 2022; Zhong et al., 2021; Zareianet al., 2020) adopt graph neural networks (GNNs) (Kipf & Welling, 2016) to model the relationshipsbetween objects in a scene.
Similar to scene graphs, our proposed SchemaNet also utilizes GNNsto represent the relationships between ingredients related to local semantics.
However, the nodes in
GNNs of SchemaNet represent more fine-grained representations, i.e. , local semantics of objects,compared to those relating to the whole objects in scene graphs.
Moreover, in SchemaNet, thetrained GNN is responsible for estimating the similarity between an instance IR-Graph and category
IR-Graphs and making classification based on the similarity scores.
A. F EATURE A TTRIBUTION FOR E XPLAINABLE AI
In the “Feat2Vertex” module, we extract the attention to the CLS token as one important componentof the ingredient importance. Currently, plenty of approaches have been proposed for indicating localrelevance (forming saliency maps) to the DNN’s prediction, termed feature attribution methods. Mostexisting works can be roughly divided into three classes: perturbation-based, gradient-based, anddecomposition-based approaches.
Perturbation-based approaches (Strumbelj & Kononenko, 2010;
Zeiler & Fergus, 2014; Ancona et al., 2019; Zintgraf et al., 2017) compute attribution by evaluatingoutput differences via removing or altering input features.
Gradient-based approaches (Simonyanet al., 2014; Shrikumar et al., 2017; Sundararajan et al., 2017; Selvaraju et al., 2017; Feng et al.
,2022) compute gradients w.r.t.
the input feature through backpropagation.
Decomposition-basedapproaches (Bach et al., 2015; Montavon et al., 2017; Chefer et al., 2021) propagate the final predictionto the input following the Deep Taylor Decomposition (Montavon et al., 2017).
Besides, CAM (Zhouet al., 2016) and ABN (Fukui et al., 2019) provide interpretable predictions with a learnable attributionmodule.
As most previous methods focus on CNNs, Chefer et al.
propose ViT-LRP tailoredfor vision Transformers.
However, most of the methods mentioned above are designed to generate asaliency map for a particular class, making them inefficient in our implementation due to the O ( C )computational complexity or dependency on gradients.
B P ROOF OF T HEOREM
We first restate the theorem:
Theorem (Theorem 1 restated) .
For a shallow GCN module, Equation (8) can be approximated by
[Equation]
Particularly, if L G = 0 and α 1 = 0 , our method is equivalent to BoVW with a linear classifier.
For further discussion, we first prove the following lemma:
Lemma 1.
For random vectors f , g ∈ R d G drawn independently from multivariate Gaussiandistribution N ( 0 , I ) , we have
[Equation]
(13) where ∥ · ∥ F is the matrix Frobenius norm, and
W ∈ R d G × d G is a projection matrix.
Proof.
To begin with, we expand the term
fW ⊤ Wg ⊤ as
[Equation]
[Equation]
Therefore, the expectation
[Equation]
[Figure]
v
[Figure]
[Figure] 224 u        v [Figure] while
[Equation]
Particularly, if W is the identity matrix
[Equation]
Further, if the components of matrix W are drawn i.i.d.
from Gaussian distribution N (0 , 1) and areindependent with f ,
[Equation]
Now we start to proof Theorem 1.
Proof.
We start with a simple case that the depth of GCN module is zero.
As G and ˆ G share the same original vertex embeddings, we have
[Equation]
According to Lemma 1, the expectation of the same vertex term E (cid:2) ff ⊤ (cid:3) =
d G is more significantthan different vertices terms
ˆ λ u λ v f (0)
u
f (0) v ⊤ for u ̸
= v when G and ˆ G are constrained with sparsity.
In Figure 6(a), we visualize the vertex similarities f (0)
u f (0) v ⊤ , showing that the similarities betweendifferent vertices are significantly lower than that of the same vertex.
Consequently, Equation (8) in the case of L G = 0 can be simplified as
[Equation]
where Φ = V ∩ ˆ V is the set of shared vertices of G and ˆ G .
With an extra assumption that α 1 = 0 , λ ϕ therefore represents the count of visual word ϕ .
Combingthe learnable term ˆ λ ϕ ∥
f
(0) ϕ
∥ 22 as w ϕ , the prediction of the matcher is
[Equation]
21) where W FC ∈ R | V |× C is the learnable projection matrix of a linear classifier.
Now, we prove the whole theorem.
Firstly, for different vertices u ∈ ˆ V and v ∈ V , the summationterm
[Equation]
can be rewritten as the aggregation form with feature output from the ( L G − 1) -th layer:
[Equation]
where ˙ N ˆ G ( u ) represent the neighbors of vertex u in Graph ˆ G and with u itself, and e u,u = 1 as theself loop.
With the assumption
that ˆ f
( L G − 1) i and f ( L G − 1) j are whitened random vectors (we say a random vectorx ∈ R d is whitened if all the components are zero mean and independent from each other,
which canbe achieved by normalization), the summation term ˆ f ( L G − 1)
i
[Equation]
⊤ with vertexi =
j will be significantly larger than those with different vertices.
We further illustrate the vertexsimilarities output from the final layer in Figure 6(b), showing that ˆ f ( L G ) u f
( L [Equation] [Equation] ⊤ is conspicuous forvertices u = v .
Corollary 1.
Consequently, s u,v is significant if ˙ N ˆ G ( u ) and ˙ N G ( v ) have shared vertices, relativelarge edge weight product ˆ e u,i e v,j , and vertex weight product ˆ λ u λ v , which means u and v havesimilar local structure , particularly in the case that u and v are the same vertex.
In conclusion, Equation (8) can be simplified as the summation of the joint vertices Φ =
ˆ V ∩ V ofthe instance and category graph G and ˆ G .
C T RAINING A LGORITHM
The training algorithm of our proposed SchemaNet is shown in Algorithm 1 with initializing IR-Atlasand sparsification.
D E FFECTIVE R ECEPTIVE F IELD OF V I T S
In this section, we discuss the effective receptive field (ERF) of ViTs, which is crucial as the visualtoken in the intermediate layers of ViTs may relate to other positions due to MHSA, affecting theinterpretability of the ingredients (visual word semantics).
Although Luo et al.
(2016) propose tomeasure the ERF for CNNs, it cannot be directly implemented to Transformer-base models.
In thissection, we propose a relatively simple yet effective approach to measuring the ERF of ViTs.
Definition 3 (Transformer ERF) .
Supposed that visual sequence X ∈ R n × d is fed into a ViTbackbone with N layers, x ∗ is a randomly chosen anchor token in X , and ε ∈ R d is the randomvector drawn from Gaussian distribution N (0 , I ) normalizing to the unit vector.
The Transformer
ERF is the average Euclidean distance between the closest token y to x ∗ in the 2D token array andx ∗ so that when y is disturbed to ˆ y =
y
+ ϵ the output change of token x ∗ is less than a predefinedthreshold δ r > 0 .
Figure 7 shows the visualization heatmap w.r.t.
the change of the output anchor with different ViTbackbones adopted in our method.
As we can observe, for an input image size of 224 × 224 , theoutput tokens with relatively significant changes are mainly distributed in the circle with a radius of25 .
Therefore, in our method, all the ingredients are in charge of 50 × 50 patches corresponding tothe input image. procedure do 18: end 25: end
[Figure] map of the receptive field of an anchor token over its neighborhood region. Averaged visualization results of 64 random images of three ViTs (DeiT-Ti, DeiT-S, and DeiT-B) from Caltech-101 dataset are demonstrated. The full image size is 224 × 224 , and we crop a small region with a size of 96 × 96 with the anchor token as the center for better visualization.
E. IR-G RAPH Before an instance-level graph, G and a category-level graph ˆ G are fed to the graph matcher, andtheir vertex and edge weights are normalized as follows.
The vertex weights in Λ are divided by the sum of Λ
[Equation]
The edge matrix E is divided by the row-wise summation:
[Equation]
For making the edge matrix symmetric, the edge matrix is defined as
[Equation]
E. GCN S ETTINGS
Now we describe the detailed settings of the GCN module in the graph matcher.
We adopt LayerNorm (Ba et al., 2016) as the Norm( · ) function and rectified linear unit (ReLU) as the activationin Equation (7).
Besides, the embedding dimension d G is set to 256 for all the experiments.
E. T RAINING DETAILS
The matcher and IR-Atlas in our method are optimized by AdamW (Loshchilov & Hutter, 2019) witha learning rate of 10 − 3 , weight decay of 5 × 10 − 4 , and cosine annealing as the learning rate decayschedule.
We implement our method with Pytorch (Paszke et al., 2019) and train all the settings for50 epochs with the batch size of 64 on one NVIDIA Tesla A100 GPU.
All input images are resizedto 224 × 224 pixels before feeding to our SchemaNet.
We adopt ResNet-style data augmentationstrategies: random-sized cropping and random horizontal flipping.
F A DDITIONAL R ESULTS
This section contains additional experimental results, highlighting the efficiency, robustness, andextendability of our proposed schema inference.
F. R ESULTS ON I MAGE N ET
We further implement SchemaNet on mini-ImageNet (Vinyals et al., 2016) and ImageNet-1k (Denget al., 2009). We adopt DeiT-Small as the backbone, and the visual vocabulary size is set to 1024for mini-ImageNet and 8000 for ImageNet-1k (due to the memory constraint). Particularly, asimplementing 1000 fully-connected category-level IR-Graphs is expensive, we keep at most 500valid vertices for each class while all other vertices are pruned during the training process based onthe initialized vertex weights.
All other settings are identical to the experiments on Caltech-101.
Theresults are presented in Table 3, drawing consistent conclusions as the main results in Table 2.
[Table]
F. C OMPARISON R ESULTS OF U SING A TTRIBUTION M ETHODS
Instead of using raw attention extracted from the backbone in Equation (3), we further employ featureattribution methods for computing the ingredient importance.
Specifically, we compare the resultsusing raw attention, CAM (Zhou et al., 2016), and Transformer-LRP (Chefer et al., 2021) in Table 4.
We can observe that using raw attention is superior to the attribution methods in terms of accuracyand running time.
Such results can be explained that: (1) the attribution methods compute a saliencymap for each class (particularly, ViT-LRP conduct C -times backpropagation, consuming enormoustime), which is significantly slower than our implementation; (2) further, as the backbone predicts softprobability distributions other than one-hot targets, the computed saliency maps for similar categorieswill be highlighted to some extent, impairing the graph matcher.
[Table]
F. A DVERSARIAL A TTACKS
To analyze the robustness of our proposed SchemaNet
[Table]
we evaluate the pre-trained SchemaNet (from
ImageNet-1k described in Appendix F.1) on two popular adversarial benchmarks: ImageNet-A (Djolonga et al., 2021) and ImageNet-R (Hendrycks et al., 2021).
The comparison results are shownin Table 5, revealing that our method is more robust than the baselines.
[Table]
F. E XTENDABILITY
We here evaluate the extendability of our method, which is evaluated by extending to unseen taskswhile keeping the original framework frozen.
As such, only the category-level IR-Graphs of the newtasks are trained and inserted into the original IR-Atlas.
Specifically, the tasks are disjointly drawn from Caltech-101 dataset.
The “Base” task has 21 classes,and task 1 to 4 has 20 classes.
The backbone model, i.e. , DeiT-Tiny, is trained on the “Base” task andthen is kept unchanged for the new tasks.
The experimental results are presented in Table 6, revealing limited performance degradation forincoming new tasks.
Such results show that the graph matcher is only responsible for evaluating thegraph similarity, while the category knowledge is stored in the imagination, i.e. , IR-Atlas.
[Table]
We here present a performance bottleneck of our method, i.e. , the Feat2Graph module, in which afeature ingredient array is converted to IR-Graph without benefiting from GPU parallel acceleration.
64 128 256 512 1024 2048 M 55.
A cc
[Figure]
visual vocabulary size M .
5 L G 76.
A cc [Figure] [Figure] GCN various depth L G .
1.00 v , e 77.
A cc v e
[Figure]
[Figure]
Specifically, as duplicated visual words may exist after the feature discretization, which happensfrom time to time, we implement this module in C++ to achieve better random access performance.
The experiments are conducted on one NVIDIA Tesla A100 GPU platform with AMD EPYC 774264-Core Processor.
In Table 7, we show the average running time of each component with an input batch size of 64.
Wecan see that without acceleration in parallel, the running time for “Feat2Edge” is significantly longerthan others (about 1.75 ms per image).
However, the inference time is still acceptable for real-timeapplications.
[Table]
G A BLATION S TUDY
Figure ) shows SchemaNet accuracy when using different sizes ofvisual vocabulary. We can observe that even with a relatively small size, e.g. , M = 256 on CIFAR100, the performance is still competitive (about 7% absolute degradation).
Besides, for the case of
M = 2048 , however, the performance suffers from low generalizability of the vocabulary.
Number of GraphConv layers.
The effect of using different depths of GCN is illustratedin Figure 8(b).
As L G increases to 5, the accuracy on CIFAR-100 decreases rapidly, yielding thatdeep GCN matcher impairs the graph matching performance, which has been explored due to theover-smoothing caused by many convolutional layers (Zhao & Akoglu, 2020; Li et al., 2018). In ourmethod, however, the graph matcher with only two layers of GraphConv is proven to be adequate forhigh performance and interpretability. Ablation of weight components. The effect of λ CLS and λ bag defined in Equation (3), and e attn ande adj defined in Equation (6) are shown in Table 8.
By setting the corresponding weight ( α 1 , 2 and β 1 , 2 )to zero once at a time, we are able to analyze the components individually.
In general, removing anyterm will lead to varying degrees of performance degradation.
More significantly, when removingthe term λ bag , the accuracy decreases by 6.03% on CIFAR-100, revealing that the statistics of theingredients help filter noisy vertices.
The learning curves of the learnable weights are shown in Figure 9, from which we can observe thatthe model tend to adopt a relatively larger alpha 2 (for visual word count) while keep the λ CLS v termfor filtering noisy ingredients such as background ingredients.
Settings Accuracy CIFARCIFAR- Learnable α 1 , 2 , β 1 ,95.
Fixed α 1 = 0 , α 2 =95.
Fixed α 1 = 1 , α 2 =95.
Fixed β 1 = 0 , β 2 =95.
Fixed β 1 = 1 , β 2 =95.
{+Fixed α 1 , 2 = β 1 , 2 = 
+}
[Figure]
Figure 8(c) shows the sensitivity analysis of λ v and λ e in Equation (11) thatconstrains the complexity of IR-Atlas. We evaluate the top-1 accuracy performance on the CIFAR100 dataset with DeiT-Tiny as the backbone. The curve of λ v shows that neither dense nor extremesparse IR-Atlas impairs the performance, while our method is more robust when changing the valueof λ e .
Sparsification threshold.
Table 9 presents the SchemaNet-Init top-1 accuracy trained withdifferent sparsification threshold δ t .
We can observe that an appropriate value of δ t , e.g. , 0.01, willboost the model performance.
[Table]
I M ORE V ISUALIZATIONSI. V ISUALIZATION OF I NGREDIENTS
In Figure 11, we visualize the whole set of ingredients on Caltech-101 for visualizations in Figure 5and Figures 12 to 14.
We extract 256 ingredients on Caltech-101 for a better view.
For each clustercenter generated from k -means clustering, we select the top-40 tokens that are closest to it and showthe corresponding image patch ( 50 × 50 in pixel, delineated in Appendix D).
We can observe thatimage patches of the same cluster share the same semantics.
[Figure]
Ingredients
[Figure]
[Figure]
I. V ISUALIZATION OF IR-A TLAS AND I NSTANCE IR-G RAPHS
We further provide more visualization examples on Caltech-101 dataset, shown in Figures 12 to
Figure 10 shows two misclassified examples along with the classification evidence. In Figure 10(a),the object is misclassified to a fine-grained category “rooster” because of a noise ingredient #240,which should be #103.
Unfortunately, ingredient #240 is a crucial vertex in the rooster’s IR-Graph,contributing about 9.5 absolute gains in the logit, leading to misclassification.
Figure 10(b) shows amore complicated example.
We can observe that instead of discretizing the appeared human face tothe “face” ingredients, the backbone provides features closer to #17, which is more similar to theanimal’s face.
Moreover, some part of the object’s body is assigned to #240 rather than the panda’sbody (#204).
Thus, it creates a remarkable pattern, i.e. , the interaction between #17 and #240, whichis the crucial local structure in the llama’s graph.
As a result, #240 and its local structure contributeabout 14.9 gains in the logit.
Although our schema inference framework is capable of revealing whyan image is misclassified by highlighting the key points, in future work, we must explore a morecompatible feature extractor that can generate more robust local features.
I n g r e d i e n t I D 12345678910111213141516171819202122232425262728293031323334353637383940 Top
I n g r e d i e n t I D 12345678910111213141516171819202122232425262728293031323334353637383940 Top
F i gu r e 11 : I ng r e d i e n t
v i s u a li za ti on on
C a lt ec hd a t a s e t .
P l ea s e z oo m i n f o r a b e tt e r v i e w .
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]