L ABEL - FREE C ONCEPT B OTTLENECK M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Concept bottleneck model (CBM) are a popular way of creating more interpretable neural networks by having hidden layer neurons correspond to humanunderstandable concepts.
However, existing CBMs and their variants have two crucial limitations: first, the need to collect labeled data for each of the predefined concepts, which is time consuming and labor intensive; second, the accuracy of a CBM is often significantly lower than that of a standard neural network, especially on more complex datasets.
This poor performance creates a barrier for adoption in practical real world applications.
Motivated by these challenges, we propose Label-free CBM which is a framework to transform any neural network into an interpretable CBM without labeled concept data, while retaining a high accuracy.
Our Label-free CBM has many advantages, it is: scalable - we present the first CBM scaled to ImageNet, efficient - creating a CBM takes only a few hours even for very large datasets, and automated - training it for a new dataset requires minimal human effort.
I NTRODUCTION
Deep neural networks (DNNs) have demonstrated unprecedented success in a wide range of machine learning tasks such as computer vision, natural language processing, and speech recognition.
However, due to their complex and deep structures, they are often regarded as black-box models that are difficult to understand and interpret their behaviors.
Interpretable models are important for many reasons such as creating calibrated trust in our models, i.e. understanding when we should trust the models, and making deep learning models interpretable and explainable is an active yet challenging research topic.
One approach to make deep learning more interpretable is Concept Bottleneck Models (CBMs) (Koh et al., 2020).
CBMs typically have a concept bottleneck layer before the (last) fully connected layer of the neural network.
The concept bottleneck layer is trained to have each neuron correspond to a single human understandable concept .
This makes the final decision a linear function of interpretable concepts, greatly increasing our understanding of the decision making.
Importantly, CBMs have been shown to be useful in a variety of applications, including model debugging and timely intervention on the DNNs.
However, there are two crucial limitations of existing CBMs and their variants (Koh et al., 2020; Yuksekgonul et al., 2022;
Zhou et al., 2018): (i) the labeled data is required for each of the predefined concept, which is time consuming and expensive to collect; (ii) the accuracy of a CBM is often significantly lower than the accuracy of a standard neural network, especially on more complex datasets.
To address the above two main challenges, we propose a new framework named Label-free CBM, which is capable of transforming any neural network into an interpretable CBM without labeled concept data while preserving comparable accuracy of the original neural network model by leveraging foundation models (Bommasani et al., 2021).
Our Label-free CBM has many advantages:
• it is scalable - to our best knowledge, it is the first CBM that scales to ImageNet •
it is efficient - creating a CBM takes only a few hours even for very large datasets • it is automated - training it for a new dataset requires minimal human effort
R ELATED WORK
Post-hoc explanations (Samek et al., 2021).
The approach of post-hoc explanations includes some classic methods such as LIME (Ribeiro et al., 2016) and SHAP (Lundberg & Lee, 2017), which try to explain individual model decisions by identifying which parts of the input data (e.g. pixels) are the most important for a given decision.
However, these methods are based on local approximations of the DNN model and as such are not always accurate.
Further, the explanations at the granularity of input pixels may not always be helpful and could require substantial subjective analysis from human.
In contrast, our explanations in Section 4.4 are correct(not approximated) and explain predictions in terms of human-understandable concepts.
More interpretable final layer : (Wong et al., 2021) proposes making the FC layer sparse, and develop an efficient algorithm for doing so.
They show that sparse models are more interpretable in many ways, but it still suffers from the fact the previous layer features are not interpretable.
NBDT
Wan et al.
(2020) propose replacing the final layer with a neural backed decision tree for another form of more interpretable decisions.
CBM : Most related to our approach are Concept Bottleneck Models (Koh et al., 2020; Losch et al., 2019) which means creating a layer before the last fully connected layer where each neuron corresponds to a human interpretable concept.
CBMs have been shown to be beneficial by allowing for human test-time intervention for improved accuracy, as well as being easier to debug.
To reduce the training cost of a CBM in (Koh et al., 2020), a recent work (Yuksekgonul et al., 2022) proposed a post-hoc CBM that only need to train the last FC layer along with an optional residual fitting layer without the need to train the backbone from scratch by leveraging Test Concept Activation Vectors (TCAV)
(Kim et al., 2018) or the multi-modal CLIP model (Radford et al., 2021).
However, the post-hoc CBM does not fully address the problems of the original CBM as using TCAV still requires collecting annotated/labelled concept data and their use of CLIP model can only be applied to the
NN backbone having the same architecture of CLIP image encoder which limits the application of post-hoc CBM.
Besides, the performance of post-hoc CBMs without residual fitting layers is often significantly lower than the standard DNNs; although adding the residual layer can help to improve the accuracy, the residual layer is not interpretable and is thus undesired.
Similarly, an earlier work
Interpretable Basis Decomposition Zhou et al.
(2018) proposes learning a concept bottleneck layer based on labeled concept data for explanable decisions, even though they do not call themselves a
CBM.
Extensive comparison of our method and existing approaches is shown in Table 1.
Model editing/debugging : Our approach is related to a range of works proposing ways to edit networks, such as (Bau et al., 2020; Wang et al., 2022) for generative vision models, (Bau et al., 2020) for classifiers, or language models (Meng et al., 2022; Mitchell et al., 2021).
In addition (Abid et al., 2021) propose a way to debug model mistakes using TCAV activation vectors.
CLIP-Dissect : (Oikarinen & Weng, 2022).
CLIP-Dissect is a recent method for understanding the roles of hidden layer neurons by leraging the CLIP multimodal model (Radford et al., 2021).
It can provide a score of how close any neuron is to representing any given concept, which makes it useful as an optimization target for learning our interpretable projection in Step 3.
[Table]
L ABEL - FREE CBM: A NEW FRAMEWORK TO BUILD CBM
In this section
[Table]
we propose Label-free CBM
[Table]
a novel framework that builds a concept bottleneck model (CBM) in an automated, scalable and efficient fashion, which addresses the core limitations of existing CBMs (Koh et al., 2020; Yuksekgonul et al., 2022).
Given a neural network backbone,
Label-free CBM transforms the backbone into an interpretable CBM without the need of concept labels in the following 4 key steps as illustrated in Fig 2: (Step 1) Create the initial concept set and filter undesired concepts; (Step 2) Compute embeddings from the backbone and the concept matrix on the training dataset; (Step 3)
Learn W c to create a Concept Bottleneck Layer; (Step 4) Learn the weights W F of the sparse final FC layer to make predictions.
Note that the backbone model here can either be a model trained on our target task, or some other task.
In the following, we provide the details of Label-free CBM for Step 1 in Sec 3.1, Step 2 and 3 in Sec 3.2, and finally Step 4 in Sec 3.3 respectively.
3. S TEP 1: C ONCEPT SET CREATION AND FILTERING
In this step, we will describe how to create a concept set to serve as the basis of human-interpretable concepts in the concept bottleneck layer that will be constructed in the Step 3.
This step consists of two sub-steps: A. Initial concept set creation and B. Concept set filtering below.
A. Initial concept set creation : To be specific, a concept set refers to the set of concepts represented in the Concept Bottleneck layer.
In the original CBM paper (Koh et al., 2020), this is decided by domain experts as the set of concepts that are important for the given task.
However, since our objective is to automate the entire process of generating concept bottleneck models, it is not desirable to rely on having human experts.
Instead, we propose generating the concept set via GPT-3 (Brown et al., 2020) using the OpenAI API.
Somewhat surprisingly, GPT-3 has a good amount of domain knowledge of which concepts are important for detecting each class when prompted in the right way.
Specifically, we ask GPT-3 the following:
• List the most important features for recognizing something as a { class } : • List the things most commonly seen around a { class } • Give superclasses for the word { class } :
Note that the { class } in here refers to the class name in the machine learning tasks.
For GPT-3 to perform well on the above prompt, we provide two examples of the desired outputs for few-shot adaptation.
Note that those two examples can be shared across all datasets, so no additional user input is needed for generating concept set for a new dataset.
Full prompts and example outputs are illustrated in the Appendix Figures 7 and 8.
To reduce variance we run each prompt twice and combine the results.
This gives us a large, somewhat noisy set of initial concepts, which we improve by filtering in B. Concept set filtering .
Note that Posthoc-CBM (Yuksekgonul et al., 2022) proposes using a knowledge graph known as ConceptNet (Speer & Havasi, 2013) to generate initial concept set, but it doesn’t work for class names that consist of multiple words (i.e. ImageNet classes such as leatherback sea turtle ), and in general we found GPT-3 concepts to be of higher quality.
See
Appendix B.2 for comparison against ConceptNet generated concepts.
B. Concept set filtering :
Several filters are employed to improve the quality and reduce the size of our concept set, as stated below:
Concept length : We delete any concept longer than 30 characters in length, to keep concepts simple and avoid unnecessary complication.
Remove concepts too similar to classes :
We don’t want our CBM to contain the output classes themselves as that would defeat the purpose of the explanation.
To avoid this we remove all concepts that are too similar to the names of target classes.
We measure this with cosine similarity in a text embedding space.
In particular we use an ensemble of similarities of the CLIP ViT-B/16 text encoder (Radford et al., 2021)
(denoted as CLIP cos) as well as the all-mpnet-base-v2 sentence encoder, so our measure is a combination of visual and textual similarity.
For all datasts we deleted concepts with similarity > 0 .
85 to any target class.
Remove concepts too similar to each other : Similar to above, we don’t want duplicate or too similar concepts in the bottleneck layer.
We use the same embedding space as above, and remove any concept that has another concept with > 0 .
9 cosine similarity already in concept set.
Remove concepts not present in training data : To make sure our concept layer accurately presents its target concepts, we remove any concept that doesn’t activate CLIP highly.
Note that this cut-off is a dataset specific metric and we delete all concepts with average topactivation less than the cut-off. 5.
Remove concepts we can’t project accurately : This step is actually performed after stepand is described in section 3.2.
We discuss the reasoning behind the filters and their effects on our results in detail in Appendix B.1.
3. S TEP 2 AND 3: L EARNING THE C ONCEPT B OTTLENECK L AYER (CBL)
Once the concept set is obtained, the next step is to learn a projection from the backbone model’s feature space into a space where axis directions correspond to interpretable concepts.
Here, we present a way of learning the projection weights W c without any labeled concept data using CLIPDissect (Oikarinen & Weng, 2022).
To start, we need the set of target concepts that the bottleneck layer is expected to represent as C = { t 1 , ..., t M } , as well as the training dataset (e.g. images)
D = { x 1 , ..., x N } of the original task, where x i ∈ R d 0 .
Next we calculate and save the CLIP concept activation matrix P where P i,j = E I ( x i ) · E T ( t j ) and E I and E T are the CLIP image and text encoders respectively.
W c is initialized as a random M × d 0 matrix where d 0 is the dimensionality of backbone features f ( x ) .
We define f c ( x ) =
W c f ( x ) , where f c ( x i ) ∈ R M .
We use k to denote a neuron of interest in the projection layer, and its activation pattern is denoted as q k where q ⊤ k =
[ f c,k
( x 1 ) , . . .
, f c,k ( x N )] ⊤ , with q k ∈ R N .
To make the neurons in the CBL interpretable, we need to enforce the projected neurons to activate in correlation with the target concept, which can be formulated as optimizing W c to maximize the CLIP-Dissect similarity between activation pattern and the target concept.
To optimize this similarity, we have designed a new and fully differentiable similarity function sim ( q i , t i ) that can be applied to CLIP-Dissect, called cosine cubed , that still achieves very good performance in explaining the neuron functionality.
Our goal is to minimize the following objective L over W c in Equation (1):
[Equation]
Here ¯ q indicates vector q normalized to have mean 0 and standard deviation 1, and the cosine cubed similarity sim ( q i , t i ) is simply the cosine similarity between two activation vectors after both have been normalized and raised to third power element-wise.
The third power is necessary to make the similarity more sensitive to highly activating inputs.
As this is still a cosine similarity, it takes values between [ − 1 , 1] .
We optimize L using the Adam optimizer on training data, with early stopping when similarity on validation data starts to decrease.
Finally to make sure our concepts are truthful, we drop all concepts j with sim ( q j , t j ) < 0 .
45 on validation data at the end of training W c .
This is the concept set filter #5 from Sec 3.1.
This cutoff was selected manually as a good indicator of a neuron being interpretable.
During this filtering, the number of concept is reduced: M ← M − ∆ , where ∆ is non-negative integer representing the number of concepts being removed in this step.
Note that matrix W c has also to be updated accordingly by removing the rows that corresponds to the removed concepts, and with our notation, W c ∈ R M × d .
To simplify Figure 2, we omit plotting this concept removal step.
3. S TEP 4: L EARNING THE SPARSE FINAL LAYER
Now that the Concept Bottleneck Layer is learned, the next task is to learn the final predictor with the fully connected layer W F ∈ R d z × M where d z is the number of output classes.
The goal is to keep W F sparse, since sparse layers have been demonstrated to be much more interpretable, as shown in (Wong et al., 2021).
Given that both the backbone f ( x ) and concept projection W c are fixed, this is a problem of learning a sparse linear model.
This can be solved efficiently with the elastic net penalty:
[Equation]
(2) where R α
=
(1 − α ) 12 || W F
||
+ α || W F || 1 and b F denotes the bias of the FC layer.
We optimize
Equation (2) using the GLM-SAGA solver created by (Wong et al., 2021).
For the sparse models, we used α = 0 .
99 and λ was chosen such that each model has 25 to 35 nonzero weights per output class.
This level was found to still result in interpretable decisions while retaining good accuracy.
Depending on the number of features/concepts in the previous layer this corresponds to 0 .
7 - 15% of the weights of the model being nonzero.
R ESULTS 4. E VALUATION SETUP
To evaluate our approach, we train Label-free CBMs on 5 datasets.
These are CIFAR-10, CIFAR100 (Krizhevsky et al., 2009), CUB (Wah et al., 2011), Places365 (Zhou et al., 2017) and ImageNet (Deng et al., 2009).
This is a diverse set of tasks, where CIFAR-10/100 and ImageNet are general image classification datasets, CUB is a fine-grained bird-species classification dataset and Places is focused on scene recognition.
Their sizes vary greatly, with CUB having 5900 training samples,
CIFAR datasets 50,000 each and ImageNet and Places365 have 1-2 million training images.
CUB contains annotations with 312 concepts for each datapoint, such as has wing color:blue or has head pattern:spotted .
We used neither the concept names or the concept annotations to train our LFCBM models to showcase our ability to perform without labels, yet our method discovered similar concepts and is competitive with methods that utilize the available concept information.
[Table]
For the model backbone, for CIFAR and CUB we use the same backbones as (Yuksekgonul et al., 2022) for fair comparison, so we use CLIP(RN50) image encoder as the backbone for CIFAR
[Table]
and
ResNet-18 trained on CUB from imgclsmob for CUB.
For both ImageNet and Places365 we use
ResNet-50 trained on ImageNet as the backbone network.
The number of concepts each model uses is roughly proportional to the number of output classes for that task, as each class adds more initial concepts.
The number of concepts for our models are as follows: 128 for CIFAR-10,for CIFAR-100, 211 for CUB, 2202 for Places-365 and 4505 for ImageNet.
CUB has a smaller number of concepts because we only used the important features prompt.
All models are trained on a single Nvidia Tesla P100 GPU, and the full training run takes anywhere from few minutes to 20 hours depending on the dataset size.
The majority of runtime is taken by step 2, where we save activations for both the backbone and CLIP over the entire training dataset.
Fortunately, these results only need to be calculated once and can be reused for training new CBMs on the same dataset.
Once the activations have been saved, learning the model takes less than 4 hours on all datasets.
4. A CCURACY
Table 2 shows the performance of Label-free CBM on all 5 datasets.
We can see that our method can create a CBM with a sparse final layer on all datasets with little loss in accuracy even on very large datasets, including a model with 72% top-1 accuracy on the challenging ImageNet.
Our Label-free
CBM noticeably outperforms Post-hoc CBM on the datasets evaluated, but some rows for P-CBM are N/A as they do not provide results for those datasets and it is unclear how to scale their method to larger datasets.
We did not include methods with non-interpretable components (i.e. P-CBM-h or IBD) for comparison as we want to focus on fully interpretable CBM models.
Note that P-CBM uses the expert provided concept set for CUB200, yet we can outperform it using our fully GPTderived set of features.
The standard sparse models were finetuned by us by learning a sparse final layer directly after feature layer f ( x ) as described in Wong et al.
and also have 25-35 nonzero weights per class.
Sparse standard models have accuracies comparable to our CBM, indicating the
CBM does not cause loss in accuracy.
4. E XPLAINABLE DECISION RULES
Perhaps the biggest benefit of Concept bottleneck models is that their decisions can be explained as a simple linear combination of understandable features.
To showcase this, in Figure 3 we visualize the final layer weights for the classes “Orange” and “Lemon” on ImageNet, and classes ”Mountain” and ”Mountain Snowy” on Places365.
The visualization is a Sankey diagram of the final layer weights for two output classes, with the weight between a concept and output class displayed as the width of the line connecting them.
We have only included weights with absolute value greater than 0.05 (for comparison the largest weights are usually 0.5-1).
Negative weights are denoted as NOT concept.
The learned decision rules largely align with our intuitions.
For ImageNet, the concept “citrus” fruit is highly connected to both ”Orange” and ”Lemon”, while orange colors activate the “Orange” class and yellow colors and limes activate the “Lemon” class.
On Places, concepts like ”a high elevation” activate both classes, while “Mountain Snowy” is activated by snow and ice related concepts and “Mountain” is activated by concepts related to volcanoes like “lava” and “crater”.
Visualizations like this allows us to gain a global understanding of how our model behaves.
4. E XPLAINABLE INDIVIDUAL DECISIONS
In addition to global understanding, Concept Bottleneck models allow us to understand the reasoning behind individual decisions.
Since our decisions are linear functions of interpretable features they can be explained in a simple and accurate way.
After learning the Concept Bottleneck Layer, we normalize the activations of each concept to have mean 0 and standard deviation 1 on the training data.
Given normalized features, the contribution of feature j to output i on input x k can be naturally calculated as Contrib ( x k , i, j ) =
W F [ ij ] f c ( x k ) j .
Since our W F is sparse, most contributions will be 0 and the important contributions can be easily visualized with a bar plot.
Example visualizations for explaining the models predicted class are shown in Figure 4 and more visualizations are available in Appendix B.5.
These visualizations show the concepts with highest absolute value contribution.
Concepts with negative activation can still be important contributors, and they are shown as “NOT concept” in our visualizations.
We can see for example the CUB model correctly identifies “a red head” as the most important feature in recognizing this image as “Red headed Woodpecker”, while the fact that there is no “a rosy breast” present also increases models confidence.
C ASE STUDY : M ANUALLY IMPROVING AN I MAGE N ET MODEL
In this section we take a deep dive into our ImageNet Label-free CBM – we analyze the errors it makes and design ways to debug/improve our network based on these findings.
In particular, we show how we can inspect a handful of individual incorrect decisions and manually change our model weights to not only fix those predictions but also multiple other predictions, improving the networks overall accuracy.
As far as we know, this is the first example of manually editing a large well-trained neural network in a way that improves test accuracy on the same distribution.
5. T YPES OF MODEL ERRORS
During the course of our experiment, we were able to identify 4 different types of mistakes our model makes, with different strategies for fixing them:
Type 1: Incorrect/ambiguous label :
Despite looking like errors in terms of accuracy, these are simply a consequence of
having noisy
labels and are unavoidable.
We will not attempt to fix this type of mistakes.
Type 2: No sufficient concept in CBL : Despite having a large set of concepts, our CBL does not include all concepts important for detecting certain classes.
For example, detecting the difference between two species of snakes may require recognizing very fine-grained patterns that can’t be easily explained in words.
While these could often be fixed by adding new concepts to the concept set, we do not focus on it as it requires retraining the projection W c ′ and the final layer.
Type 3: Incorrect concept activations : Some of the time, the activations in our CBL are incorrect and do not match the image, causing the prediction to become incorrect.
Since the network up to the Concept Bottleneck Layer is mostly a black box, we can’t easily improve the predictions, but predictions can be improved with test time interventions as seen in Figure 5.
Type 4: Incorrect final layer weight W F : Sometimes even if all the concept activations are correct, the models final layer weights cause it to still make an erroneous prediction.
As the final layer is fully interpretable, fixing these errors is the main focus of our study described in section 5.2.
See Figure 10 in the Appendix for examples of Type 1 and Type 2 errors.
5. E DITING FINAL LAYER WEIGHTS
In this section we describe our procedure for manually editing final layer weights of our model.
This is done with the following procedure:
Find an incorrect prediction belonging to Type 4 errors : This is done by visualizing incorrect model predictions and their explanations, and identifying Type 4 errors,
i
e. the highly activating concepts looks correct and the label is correct and unambiguous.
Identify a concept to edit.
: Perhaps the most tricky step, to flip a prediction we need to select a concept that is highly activated
in this image, important for the ground truth class and not important for the incorrect class using our domain knowledge, and will have a small impact on other predictions.
For example to flip a prediction from ”cricket insect” to ”grasshopper” we chose the concept ”a green color”.
Change weights by sufficient magnitude : Next we choose magnitude of weight change
∆ w ,
and edit the weights in
the following way: W F [ gt,concept ]
[Equation]
[ gt,concept
[Equation]
Typically
we want ∆ w to be
slightly
more than required to flip the prediction on our example
but understanding how it affect other
predictions is hard and should be checked with a validation dataset.
Figure 6 shows two examples of model edits we performed, and their effects on model predictions on the validation set.
In total we identified 5 such beneficial edits during our quick exploration, the other 3 are shown in Figure 9.
After applying all of them, the total validation accuracy of our model goes from 71.98% to 72.02%, correcting 38 predictions while turning 17 incorrect.
While this might seem like a small difference, it is worth noting we only change 10/4.5 million total weights in the final layer, of which 34,550 are non-zero.
Our edits only affect 10 classes which have a total ofvalidation examples, so in effect we increase accuracy on this subset by 4.2% which is a significant boost.
While labor-intensive, we believe this editing approach is very promising for future use, where practitioners could for example notice an incorrect prediction in a production system, quickly devise a fix for that prediction, and check how it affects other predictions using validation data.
C ONCLUSION
We have presented Label-free CBM , a fully automated and scalable method for generating concept bottleneck models and used it to create the first high performing CBM on ImageNet.
In addition, we have demonstrated how our models are easily understandable in terms of both global decision rules and individual predictions, and how to use this understanding to manually improve model weights in a fully trained ImageNet model in a way that improves validation accuracy.
R EPRODUCIBILITY
We have described our algorithm in detail, including hyperameters and other details in sectionsand 4.
In addition we will release our code publicly prior to publication
Language models are few-shot learners.
A A PPENDIX A. L IMITATIONS
While our model performs well in terms of both interpretability and accuracy, it still has some limitations.
For example, using a model like GPT-3 to produce concepts is stochastic, and sometimes may fail to generate important concepts for detecting a certain class.
However our model is quite robust to small changes in the concept set as shown in sections B.1 and B.3.
In general, automatic ways of generating concept set may sometimes lack the required domain knowledge to include some important concepts, and would perhaps be best used in collaboration with a human expert.
Second, our model is aimed to extend Concept bottleneck models to large datasets where labeled concept data is not available.
However it works best on domains where CLIP performs well and as such may not perform as well on small datasets that require specific domain knowledge and labeled concept data is available, such as medical datasets.
For such datasets we believe it is still better to use a method that can effectively leverage the labels available such as original Concept Bottleneck
Models Koh et al.
(2020) or P-CBMYuksekgonul et al.
A. A DDITIONAL FIGURES
Figures 7, 8 provide examples of the full prompts we used for GPT-3, as well as GPT outputs.
Figure 9 shows the additional model edits performed in our ImageNet CBM experiment, and Figure 10 showcases Type 1 and Type 2 errors made by our ImageNet CBM.
B A PPENDIX : N EW R EBUTTAL EXPERIMENTS B. A BLATION STUDY AND FURTHER DISCUSSION ON CONCEPT FILTERING IN S EC 3.
In this section we study how each step in our proposed concept filtering affect the results of our method.
In general our use of filters has two main aims:
• To improve the interpretability of our models • Improve computational efficiency and complexity by reducing the number of concepts
The filters are not designed to improve performance of the models in terms of accuracy, in fact we would expect them to slightly reduce accuracy as a model with more concepts is larger and more powerful.
To evaluate the effect each individual filter has we trained models on both CIFAR10 and
ImageNet while deactivating one filter at a time, as well as one without using any filters at all.
The results are displayed in Table 3.
We can see that the accuracy of our models is not very sensitive to the filters at all, with ImageNet accuracy remaining constant and CIFAR10 slightly increasing its accuracy with less filters.
Two ImageNet results are marked as N/A as the large number of concepts caused some memory/computational issues for training the models which we were unable to resolve in the rebuttal timeframe.
Filters
[Table]
All filters
[Table]
No length filter
[Table]
No similarity to classes filter
[Table]
No similarity to other concepts filter
[Table]
No CLIP activation filter
[Table]
No projection accuracy filter
[Table]
No filters at all
[Table]
To further show the effects of each filter, we will showcase the full filtering procedure for our CIFAR10 model with all filters.
We start with a freshly generated initial concept set from GPT-3, which has a total of 177 concepts.
We then take the following steps to filter down concepts:
Delete concepts that are too long.
This step is important to make sure concepts are easy to visualize and simple non-convoluted concepts.
For CIFAR-10 this leads to deletion ofconcepts: - white spots on the fur (in some cases) , feline features (e.g., whiskers, ears) and legs with two toes pointing forward and two toes pointing backwar .
174 concepts remain.
Delete concepts too similar to output classes.
This step is required for the explanations to be informative, as it helps avoid trivial explanations such as: ”This is image is a cat because it is a cat”.
This step removes the following CIFAR-10 concepts (with the class name it is too close to shown in brackets): - a cat (cat), a deer (deer), a horse (horse), a plane (airplane), a car (automobile), car (automobile), cars (automobile), vehicle (automobile), animal (dog), truck driver (truck) 164 concepts remain 3.
Delete concepts too similar to other concepts.
In this step we aim to delete duplicate concepts from the concept set, i.e. two concepts that have the same semantic meaning should not both be part of the concept set.
Duplicate concepts will cause unnecessary computational cost and confusing explanations.
For CIFAR-10 we delete the following concepts (with the concept it is too similar to in brackets): - 4 wheels (four wheels), a food bowl (a bowl), a furry, four-legged animal (a large, fourlegged mammal), a gasoline station (a gas station), a large boxy body (a large body),
a
[Table]
sea
[Table]
154 concepts remain 4.
Remove concepts not present in training data.
In this step we remove all concepts that CLIP thinks are not really present in training data.
The purpose is to avoid learning neurons that don’t correctly represent their target concepts, as concepts missing from the dataset are unlikely to be learned correctly in the Concept Bottleneck Layer.
For CIFAR-10 this step deletes the following concepts: - a bit, a mechanic, legs, long legs, passengers, several seats inside, windows all around 147 concepts remain 5.
Remove concepts we can’t project accurately.
The purpose of this step is similar to the previous step, we want to remove all concepts that are not faithfully represented by the CBL.
To do this we evaluate the similarity score between the target concept and the activations of our new neuron using CLIP-Dissect Oikarinen & Weng (2022) on the validation data, and delete concepts with similarity less than 0.45 which was determined to be a good indicator of faithful concept representation.
For CIFAR-10 we delete the following concepts: - a bed, a coffee mug, a crew, a house, a wheel We are left with our 142 final concepts.
In total our method has 5 different cutoff parameters that can be tuned.
Since the main purpose of these filters is to make the network more interpretable, their values were mainly chosen through trial and error to find ones that produce the most useful and explainable models.
Most of these cutoffs are independent of the dataset chosen, and the only one we changed from one dataset to another was the value of cutoff for images being present in the dataset (filter 4).
We used a cutoff of 0.25 for
CIFAR-10 and CIFAR-100, 0.26 for CUB-200 and 0.28 for Places and ImageNet.
We found we had to change this cutoff as CLIP doesn’t activate very highly on low resolution images of CIFAR, and in general we get higher top5 activations on larger datasets.
Since the other 4 cutoffs are fixed, using our method for a new dataset won’t require a large hyperparameter search, and as seen in Tablethe overall performance is not very sensitive to specific filters.
To visualize the effect of our filters, in Figure 11 we visualize the final layer weights for CIFARclass ”automobile” trained with and without filters.
We can see the model with filters has found a reasonable decision rule, with largest weights corresponding to four wheels and a steering wheel .
On the other hand, the weights for the model without filters are quite problematic.
First, the 3 largest weights are cars , car and a car .
Not only do they have the same meaning as the final class itself (filter 2), thus lacking any explanatory power, they are duplicates of each other (filter 3), making the explanation unnecessarily compilicated.
Finally we see the unrelated concept bed which is not present in CIFAR-10 having a small positive weight, which should be removed by either filter 4 or 5.
B. A BLATION : E FFECT OF C ONCEPT SET
In this section we evaluate the effect our initial concept set generator has on our results.
In our paper we proposed a new method of generating descriptions using GPT-3 Brown et al.
(2020), and
in this section we compare those results to the case where we generate the initial concept set using ConceptNet Speer & Havasi (2013) as proposed by Yuksekgonul et al.
In general ConceptNet only works well when prompted with relatively common single word phrases, but many of class names for CUB200, Places365 and ImageNet consist of multiple words, such as the ImageNet class great white shark .
To overcome this issue we split multi-word class names into single word components and add concepts related to each word to the concept set, in this case great , white and shark .
After creating the initial concept set we use the same filtering procedure described in section 3.1B for both models.
Table 4 displays the accuracy comparison between models trained on different initial concept setst.
In general using GPT3 seems to provide a small (0.1-1.5%) accuracy boost on all datasets, with the exception of CUB200 where the ConceptNet model fails completely.
This is because ConceptNet is unable to generate concepts for the highly specific class names in CUB200, such as Groove billed Ani , while GPT-3 is not troubled by this.
[Table]
B. C ONSISTENCY OF CONCEPT SET GENERATION
Since the initial concept set is generated using GPT-3, it is a random process and in this section we explore how much this noise effects our final results.
When regenerating initial concept set and running our concept filtering pipeline for our ImageNet model, we got 4380 concepts, compared to the original 4523 concepts.
In terms of accuracy, the model learned with regenerated concept set reached an accuracy of 71.89%, compared to the average of 71.95% with the original concept set.
This indicates that the difference is very small in terms of accuracy.
However, the exact concepts used and model weights can be pretty different as shown in Figure 12, despite all being relevent (e.g. there are many concepts that appear on both concept sets, e.g. Image 49962 has important concepts of ”wipes”, ”toiletries” on both, though they have different corresponding contributions).
Figure 12 shows the explanations for decisions on two random images for the original model and model trained with regenerated concept set.
B. A DDITIONAL WEIGHT VISUALIZATIONS FOR RANDOM CLASSES
In figures 13 and 14 we have visualized the final layer weights leading to 4 randomly chosen output classes for both Places365 and ImageNet.
These were created using the same procedure as Figure 3, with minor differences such as focus on single class at a time and no manual editing of concept colors according to semantic similarity.
We can see that the concepts that have large weights are indeed relevant to the class, and overall the weights look reasonable.
B. A DDITIONAL EXPLANATIONS FOR RANDOM IMAGES
In figures 15, 16, 17 we display additional samples of decision explanations for 4 randomly chosen input images for each of Places365, ImageNet and CUB-200 respectively.
Overall, it can be seen that the concepts and explanations are of good quality, even with the CUB-200 dataset which may require very specific knowledge and concepts to create CBMs.
This result demonstrates the effectiveness of our Lf-CBM, which does not use the knowledge-specific concepts sets unlike existing works on
CBMs.
B. A BLATION : LF-CBM WITHOUT SPARSITY CONSTRAINT
To estimate the effect sparsity has on our models, we show the accuracy of LF-CBM models trained without sparsity constraints (LF-CBM (dense)) in Table 5.
We can see that not using a sparsity constraint greatly improves accuracy (except for CUB200 which starts to overfit), but it comes at a very large cost for interpretability as can be seen in Figure 18, where almost none of the decision is explained by the 10 most highly contributing concepts.
[Table]