Concept Embedding Models
Anonymous Author(s)
AffiliationAddress email
Abstract
Deploying AI-powered systems requires trustworthy models supporting effectivehuman interactions, going beyond raw prediction accuracy.
Concept bottleneckmodels promote trustworthiness by conditioning classification tasks on an intermediate level of human-like concepts.
This enables human interventions which cancorrect mispredicted concepts to improve the model’s performance.
However, existing concept bottleneck models are unable to find optimal compromises betweenhigh task accuracy, robust concept-based explanations, and effective interventionson concepts—particularly in real-world conditions where complete and accurateconcept supervisions are scarce.
To address this, we propose Concept Embedding Models, a novel family of concept bottleneck models which goes beyondthe current accuracy-vs-interpretability trade-off by learning interpretable highdimensional concept representations.
Our experiments demonstrate that Concept
Embedding Models (1) attain better or competitive task accuracy w.r.t.
standardneural models without concepts, (2) provide concept representations capturingmeaningful semantics including and beyond their ground truth labels, (3) supporttest-time concept interventions whose effect in test accuracy surpasses that instandard concept bottleneck models, and (4) scale to real-world conditions wherecomplete concept supervisions are scarce.
Introduction
What is an apple ?
While any child can explain what an “apple” is by enumerating its characteristics,deep neural networks (DNNs) fail to explain what they learn in human-understandable terms despitetheir high prediction accuracy [1].
This accuracy-vs-interpretability trade-off has become a majorconcern as high-performing DNNs become commonplace in practice [2–4], thus questioning theethical [5, 6] and legal [7, 8] ramifications of their deployment.
Concept bottleneck models (CBMs, [9], Figure 1a) aim at replacing “black-box” DNNs by firstlearning to predict a set of concepts, i e., “interpretable” high-level units of information (e g.,“color” or “shape”)
[10] provided at training time [11, 12], and then using these concepts to learn adownstream classification task.
Predicting tasks as a function of concepts engenders user trust [13]by allowing predictions to be explained in terms of concepts and by supporting human interventions,where at test-time an expert can correct a mispredicted concept, possibly changing the CBM’soutput.
That said, concept bottlenecks may impair task accuracy [9, 14], especially when conceptlabels do not contain all the necessary information for accurately predicting a downstream task(i.e.
, they form an “incomplete” representation of the task [15]), as seen in Figure 1b.
In principle,extending a CBM’s bottleneck with a set of unsupervised neurons may improve task accuracy, asobserved by Mahinpei et al.
However, as we will demonstrate in this work, such a hybridapproach not only significantly hinders the performance of concept interventions, but it also affectsthe interpretability of the learnt bottleneck, thus undermining user trust [13].
Therefore, we argue that novel concept-based architectures are required to overcome the current accuracy/interpretability pitfalls of CBMs, thus enabling their deployment in real-world settings where concept annotationsare likely to be incomplete.
[Figure]
In this paper, we propose Concept Embedding Models (CEMs), a novel concept bottleneck model(described in Section 3) which overcomes the current accuracy-vs-interpretability trade-off found inconcept-incomplete settings (as shown in Figure 1c).
Furthermore, we introduce two new metrics forevaluating concept representations (Section 4) and use them to help understand why our approachcircumvents the limits found in the current state-of-the-art CBMs (Section 5).
Our experiments showthat CEM (1) attains better or competitive task accuracy w.r.t.
standard DNNs trained without conceptsupervision, (2) learns concept representations that capture meaningful semantics as least as well asvanilla CBMs, and (3) supports effective test-time concept interventions.
Background
Concept bottleneck models (CBMs, [9])
A concept bottleneck model learns a mapping fromsamples x ∈ X to labels y ∈ Y by means of: (i) a concept encoder function g : X → C whichmaps samples from the input space x ∈ X ⊆ R n to an intermediate space ˆc ∈ C ⊆
R k formed by kconcepts, and (ii) a label predictor function f : C → Y which maps samples from the concept spaceˆc ∈ C to a downstream task space ˆy ∈ Y ⊆ R l .
A CBM requires a dataset composed of tuples in
X × C × Y , where each sample consists of input features x (e.g., an image’s pixels), ground truthconcept vector c ∈ { 0 , 1 } k (i.e., a binary vector where each entry represents whether a concept isactive or not) and a task label y (e.g., an image’s class).
During training, a CBM’s is encouragedto align ˆc = g ( x ) and ˆy =
f
( g ( x )) to x ’s corresponding ground truth concepts c and task labels y ,respectively.
This can be done by (i) sequentially training first the concept encoder and then using itsoutput to train the label predictor, (ii) independently training the concept encoder and label predictorand then combining them to form a CBM, or (iii) jointly training the concept encoder and labelpredictor via a weighted sum of cross entropy losses.
Concept representations in CBMs For each sample x ∈ X , the concept encoder g learns kdifferent scalar concept representations ˆ c 1 , . . .
, ˆ c k .
Boolean and Fuzzy CBMs [9] assume that eachdimension of ˆc , which we describe by ˆ c i
= s (ˆ c )
[ i ] ∈ [0 , 1] , is aligned with a single ground truthconcept and represents a probability of that concept being active.
The element-wise activationfunction s : R → [0 , 1] can be either a thresholding function s ( x ) ≜ 1 x ≥ 0 .
5 ( Boolean CBM) orsigmoidal function s ( x ) ≜ 1 / (1 + e − x ) ( Fuzzy CBM) 1 .
A natural extension of this framework are
Hybrid CBMs [14], where ˆc ∈ R ( k + γ ) contains γ unsupervised dimensions and k supervised conceptdimensions which, when concatenated, form a shared concept vector (i.e., an “embedding”).
Concept Interventions in CBMs Interventions are one of the core motivations behind CBMs [9].
Through interventions, concept bottleneck models allow experts to improve a CBM’s task performanceby rectifying mispredicted concepts by setting, at test-time,
ˆ c i :=
c i (where c i is the ground truthvalue of the i -th concept).
Such interventions can significantly improve CBMs performance within
[Figure]
a human-in-the-loop setting [9].
Furthermore, interventions enable the construction of meaningful concept-based counterfactuals [7].
For example, intervening on a CBM trained to predict bird typesfrom images can determine that when the size of a “black” bird with “black” beak changes from to
, while all other concepts remain constant, then one may classify the bird as a
“raven” rather than a “crow”.
Concept Embedding Models
In real-world settings, where complete concept annotations are costly and rare, vanilla CBMs mayneed to hinder their task performance in order to preserve their interpretability [9].
While Hybrid
CBMs are able to overcome this issue by adding extra capacity in their bottlenecks, this comes atthe cost of their interpretability and their responsiveness to concept interventions, thus undermininguser trust [13].
To go beyond these pitfalls, we propose Concept Embedding Models (CEMs), aconcept-based architecture which represents each concept as a supervised vector.
Intuitively, usinghigh-dimensional embeddings to represent each concept allows for extra supervised learning capacity,as opposed to Hybrid models where the information flowing through their unsupervised bottleneckactivations is concept-agnostic.
In the following section, we introduce our architecture and describehow it learns a mixture of two semantic embeddings for each concept (Figure 2).
We then discuss howinterventions are performed in CEMs and introduce RandInt , a train-time regularisation mechanismthat incentivizes our model to positively react to interventions at test-time.
3.1 Architecture
For each concept, CEM learns a mixture of two embeddings with explicit semantics representing theconcept’s activity.
Such design allows our model to construct evidence both in favour of and againsta concept being active, and supports simple concept interventions as one can switch between the twoembedding states at intervention time.
We represent concept c i with two embeddings
ˆ c
+ i , ˆ c − i ∈ R m , each with a specific semantics: ˆ c + irepresents its active state (concept is true ) while ˆ c − i represents its inactive state (concept is false ).
To this aim, a DNN ψ ( x ) learns a latent representation h ∈ R n hidden which is the input to CEM’sembedding generators.
CEM then feeds h into two concept-specific fully connected layers, whichlearn two concept embeddings in R m , namely ˆ c
+ i =
ϕ
+ i ( h ) =
a ( W + i h
+
b
+ i ) and ˆ c − i = ϕ
− i ( h ) =
a ( W − i h
+ b − i ) .
2 Notice that while more complicated models can be used to parameterise our conceptembedding generators ϕ
+ i ( h ) and ϕ − i ( h ) , we opted for a simple one-layer neural network to constrain parameter growth in models with large bottlenecks.
Our architecture encourages embeddings ˆ c + iand ˆ c − i to be aligned with ground-truth concept c i via a learnable and differentiable scoring functions : R 2 m → [0 , 1] , trained to predict the probability ˆ p i ≜ s ([ˆ c + i , ˆ c − i ] T )
= σ (cid:0) W s
[ˆ c + i , ˆ c − i ]
T + b s (cid:1)of concept c i being active from the embeddings’ joint space.
For the sake of parameter efficiency,parameters W s and b s are shared across all concepts.
Once both semantic embeddings are computed,we construct the final concept embedding ˆ c i for c
i as a weighted mixture of ˆ c
+ i and ˆ c − i :
[Equation]
Intuitively, this serves a two-fold purpose: (i) it forces the model to depend only on ˆ c
+ i when the i -thconcept is active, i.e., c i = 1 (and only on ˆ c − i when inactive), leading to two different semanticallymeaningful latent spaces, and (ii) it enables a clear intervention strategy where one switches theembedding states when correcting a mispredicted concept, as discussed below.
Finally, all k mixedconcept embeddings are concatenated, resulting in a bottleneck g ( x ) =
ˆ c with k · m units (see endof Figure 2).
This is passed to the label predictor f to obtain a downstream task label.
In practice,following Koh et al.
[9], we use an interpretable label predictor f parameterised by a simple linearlayer, though more complex functions could be explored too.
Notice that as in vanilla CBMs, CEMprovides a concept-based explanation for the output of f through its concept probability vectorˆ p ( x ) ≜
[ˆ p 1 , · · · , ˆ p k ] , indicating the predicted concept activity.
This architecture can be trained in anend-to-end fashion by jointly minimising via stochastic gradient descent a weighted sum of the crossentropy loss on both task prediction and concept predictions:
[Equation]
where hyperparameter α ∈ R + controls the relative importance of concept and task accuracy.
3.2 Intervening with Concept Embeddings
As in vanilla CBMs, CEMs support test-time concept interventions.
To intervene on concept c i ,one can update ˆ c i by swapping the output concept embedding for the one semantically alignedwith the concept ground truth label.
For instance, if for some sample x and concept c i a CEMpredicted ˆ p i = 0 .
1 while a human expert knows that concept c i is active ( c i = 1 ), they can performthe intervention
ˆ p
i := 1 .
This operation updates CEM’s bottleneck by setting ˆ c i to ˆ c
+ i rather than 129 (cid:0) 0 .
1ˆ c
+ i +0 .
9ˆ c − i (cid:1) .
Such an update allows the downstream label predictor to act on information relatedto the corrected concept.
In addition, we introduce RandInt , a regularisation strategy exposing CEMsto concept interventions during training to improve the effectiveness of such actions at test-time.
RandInt randomly performs independent concept interventions during training with probability p int(i.e.
, ˆ p i is set to ˆ p
i :=
c i for concept c i with probability p int ).
In other words, for all concepts c i ,during training we compute embedding ˆ c i as:
[Equation]
while at test-time we always use the predicted probabilities for performing the mixing.
Duringbackpropagation, this strategy forces feedback from the downstream task to update only the correctconcept embedding (e.g., ˆ c + i
if c i = 1 ) while feedback from concept predictions updates both ˆ c
+ iand ˆ c − i .
Under this view, RandInt can be thought of as learning an average over an exponentiallylarge family of CEM models (similarly to dropout [18]) where some of the concept representationsare trained using only feedback from their concept label while others receive training feedback fromboth their concept and task labels.
Evaluating concept bottlenecks
To the best of our knowledge, while a great deal of attention has been given to concept-basedexplainability in recent years, existing work still fails to provide methods that can be used to evaluatethe interpretability of a concept embedding or to explain why certain CBMs underperform in theirtask predictions.
With this in mind, we propose (i) a new metric for evaluating concept quality inmultidimensional representations and (ii) an information-theoretic method which, by analysing theinformation flow in concept bottlenecks, can help understand why a CBM may underperform in adownstream task.
Concept Alignment Score (CAS)
The Concept Alignment Score (CAS) aims to measure how muchlearnt concept representations can be trusted as faithful representations of their ground truth conceptlabels.
Intuitively, CAS generalises concept accuracy by considering the homogeneity of predictedconcept labels within groups of samples which are close in a concept subspace.
More specifically, foreach concept c i the CAS applies a clustering algorithm κ to find ρ > 2 clusters, assigning to eachsample x ( j ) a cluster label π ( j ) i ∈ { 1 , · · · , ρ } .
We compute this label by clustering samples using theiri -th concept representations { ˆ c (1) i , ˆ c (2) i , · · · } .
Given N test samples, the homogeneity score h ( · ) [19]then computes the conditional entropy H of ground truth labels
C i =
{ c ( j ) i } Nj =1 w.r.t.
cluster labelsΠ i ( κ, ρ ) =
{ π ( j ) i } Nj =1 , i.e., h = 1 when H ( C i , Π i ) =
and h = 1 − H
( C i , Π i ) /H
( C i ) otherwise.
The higher the homogeneity, the more a learnt concept representation is “aligned” with its labels, andcan thus be trusted as a faithful representation.
CAS averages homogeneity scores over all conceptsand number of clusters ρ , providing a normalised score in [0 , 1] :
[Equation] [Equation]
To tractably compute CAS in practice, we sum homogeneity scores by varying ρ across ρ ∈ { 2 , 2 +δ, 2 + 2 δ, · · · , N } for some δ > 1 (details in Appendix A.1).
Furthermore, we use k-Medoids [20]for cluster discovery, as used in Ghorbani et al.
[10] and Magister et al.
[21], and use conceptlogits when computing the CAS for Boolean and Fuzzy CBMs.
For Hybrid CBMs, we use ˆ c i
≜[ˆ c
[ k : k + γ ] , ˆ c
[ i :( i +1)] ] T as the concept representation for c i given that the extra capacity is a shared 167 embedding across all concepts.
Information bottleneck The relationship between the quality of concept representations w.r.t.
theinput distribution remains widely unexplored.
Here we propose to analyse this relationship usinginformation theory methods for DNNs developed in Tishby et al.
[22] and Tishby and Zaslavsky [23].
In particular, we compare concept bottlenecks using the Information Plane method [22] to study theinformation flow at concept level.
To this end, we measure the evolution of the Mutual Information( I ( · , · ) ) of concept representations w.r.t.
the input and output distributions across training epochs.
We conjecture that embedding-based CBMs circumvent the information bottleneck by preservingmore information than vanilla CBMs from the input distribution as part of their high-dimensionalactivations.
If true, such effect should be captured by Information Planes in the form of a positivelycorrelated evolution of I ( X, ˆ C ) , the Mutual Information (MI) between inputs X and learnt conceptrepresentations ˆ C , and I ( ˆ C, Y ) , the MI between learnt concept representations ˆ C and task labels Y .
In contrast, we anticipate that scalar-based concept representations (e.g., Fuzzy and Bool CBMs), willbe forced to compress the information from the input data at concept level, leading to a compromisebetween the I ( X, ˆ C ) and I ( ˆ C, Y ) .
Further details on our implementation are in Appendix A.2.
Experiments
In this section, we address the following research questions:
• Task accuracy — What is the impact of concept embeddings on a CBM’s downstream
task performance?
Are models based on concept embeddings still subject to an information
bottleneck [22]?• Interpretability — Are CEM concept-based explanations aligned with ground truth con-
cepts?
Do they offer interpretability beyond simple concept prediction and alignment?
• Interventions — Do CEMs allow meaningful concept interventions when compared toHybrid or vanilla CBMs?
5.1 Setup
Datasets For our evaluation, we propose three simple benchmark datasets of increasing concept complexity (from Boolean to vector-based concepts): (1) XOR (inspired by [24]) in whichtuples ( x 1 , x 2 ) ∈ [0 , 1] 2 are annotated with two Boolean concepts { 1 c i > 0 .
5 } 2 i =1 and labeled asy =
c 1 XOR c 2 ;
(2) Trigonometric (inspired by [14]) in which three latent normal random variables
{ b i } 3 i =1 are used to generate a 7-dimensional input whose three concept annotations are a Booleanfunction of { b i } 3 i =1 and task label is a linear function of the same
; (3) Dot in which four latent randomvectors v 1 , v 2 , w 1 , w 2 ∈ R 2 are used to generate two concept annotations, representing whetherlatent vectors v i point in the same direction of reference vectors w i , and task labels, representingwhether the two latent vectors v 1 and v 2 point in the same direction.
Furthermore, we evaluate ourmethods on two real-world image tasks: the Caltech-UCSD Birds-200-2011 dataset (CUB, [16]),preprocessed as in [9], and the Large-scale CelebFaces Attributes dataset (CelebA, [25]).
In our
CUB task we have 112 complete concept annotations and 200 task labels while in our CelebAtask we construct 6 balanced incomplete concept annotations and each image can be one of 256classes.
Therefore, we use CUB to test each model in a real-world task where concept annotationsare numerous and they form a complete description of their downstream task.
In contrast, our CelebAtask is used to evaluate the behavior of each method in scenarios where the concept annotationsare scarce and incomplete w.r.t.
their downstream task.
Further details on these datasets and theirproperties are provided in Appendix A.3.
Baselines We compare CEMs against Bool, Fuzzy, and Hybrid Joint-CBMs as they all provideconcept-based explanations for their predictions and allow concept interventions at test-time.
Notethat this set excludes architectures such as Self-Explainable Neural Networks [26] and Concept
Whitening [12] as they do not offer a clear mechanism for intervening on their concept bottlenecks.
To ensure fair comparison, we use the same architecture capacity across all models .
We empiricallyjustify this decision in Appendix A.9 by showing that the results discussed in this section do notchange if one modifies the underlying model architecture.
Similarly, we use the same values of αand m within a dataset for all models trained on that dataset and set p int = 0 .
25 when using CEM(see Appendix A.6 for an ablation study of p int showing how intervention improvement plateausaround this value).
When using Hybrid CBMs, we include as many activations in their bottlenecks astheir CEM counterparts (so that they both end up with a bottleneck with km activations) and use a Leaky-ReLU activation for unsupervised activations.
Finally, in our evaluation we include a DNNwithout concept supervision with the same capacity as its CEM counterpart to measure the effect ofconcept supervision in our model’s performance.
For further details on the model architectures andtraining hyperparameters, please refer to Appendix A.4.
Metrics We measure a model’s performance based on four metrics: First, we measure task andconcept classification performance in terms of both task and mean concept accuracy .
Second, weevaluate the interpretability of learnt concept representations via our concept alignment score .
Toeasily visualise the accuracy-vs-interpretability trade-off, we plot our results in a two-dimensionalplane showing both task accuracy and concept alignment.
Third, we study the information bottleneckin our models via mutual information (MI) and the Information Plane technique [27].
Finally, wequantify user trust [13] by evaluating a model’s task performance after concept interventions.
Allmetrics in our evaluation, across all experiments, are computed on test sets using 5 random seeds, fromwhich we compute a metric’s mean and 95% confidence interval using the Box-Cox transformationfor non-normal distributions.
5.2 Task Accuracy
CEM improves generalisation accuracy (y-axis of Figure 3)
Our evaluation shows thatembedding-based CBMs (i.e., Hybrid-CBM and CEM) can achieve competitive or better downstream accuracy than DNNs that do not provide any form of concept-based explanations, and caneasily outperform Boolean and Fuzzy CBMs by a large margin (up to +45% on Dot).
This effectis emphasised when the downstream task is not a linear function of the concepts (e.g., XOR and
Trigonometry) or when concept annotations are incomplete (e.g., Dot and CelebA).
At the sametime, we observe that all concept-based models achieve a similar high mean concept accuracy acrossall datasets (see Appendix A.7).
This suggests that, as hypothesised, the trade-off between conceptaccuracy and task performance in concept-incomplete tasks is significantly alleviated by the introduction of concept embeddings in a CBM’s bottleneck.
Similar results can be observed when trainingour baselines using only a fraction of the available concepts in CUB as seen in Appendix A.10.
Finally, notice that CelebA showcases how including concept supervision during training (as in CEM) can lead to an even higher task accuracy than the one obtained by a vanilla end-to-end model( +5% compared to “No concepts” model).
This result further suggests that concept embeddingrepresentations enable high levels of interpretability without sacrificing performance.
[Figure] CEM overcomes the information bottleneck (Figure 4)
The Information Plane method indicates,as hypothesised, that embedding-based CBMs (i.e., Hybrid-CBM and CEM) do not compress inputdata information, with I ( X, ˆ C ) monotonically increasing during training epochs.
On the other hand,
Boolean and Fuzzy CBMs, as well as vanilla end-to-end models, tend to “forget” [27] input datainformation in their attempt to balance competing objective functions.
Such a result constitutes aplausible explanation as to why embedding-based representations are able to maintain both high taskaccuracy and mean concept accuracy compared to CBMs with scalar concept representations.
In fact,the extra capacity allows CBMs to maximise concept accuracy without over-constraining conceptrepresentations, thus allowing useful input information to pass by.
In CEMs all input informationflows through concepts, as they supervise the whole concept embedding.
In contrast with Hybridmodels, this makes the downstream tasks completely dependent on concepts, which explains thehigher concept alignment scores obtained by CEM (as discussed in the next subsection).
[Figure]
;2    I ( X , C )    7ULJRQRPHWU\    I ( X , C )    'RW
  
I ( X , C )    &8%    I ( X , C )   
&HOHE$ 1RFRQFHSWV %RROHDQ&% )X]]\&% +\EULG&% &(0
RXUV
5.3 Interpretability
CEM learns more interpretable concept representations (x-axis of Figure 3) Using the proposed
CAS metric, we show that concept representations learnt by CEMs have alignment scores competitiveor even better (e.g., on CelebA) than the ones of Boolean and Fuzzy CBMs.
The alignment score alsoshows, as hypothesised, that hybrid concept embeddings are the least faithful representations—withalignment scores up to 25% lower than CEM in the Dot dataset.
This is due to their unsupervisedactivations containing information which may not be necessarily relevant to a given concept.
Thisresult is a further evidence for why we expect interventions to be ineffective in Hybrid models (as weshow shortly).
CEM captures meaningful concept semantics (Figure 5)
Our concept alignment results hintat the possibility that concept embeddings learnt by CEM may be able to offer more than simpleconcept prediction.
In fact, we hypothesise that their seemingly high alignment may lead to theseembeddings forming more interpretable representations than Hybrid embeddings, which can leadto these embeddings serving as better representations for different tasks.
To explore this, we traina Hybrid-CBM and a CEM, both with m = 16 , using a variation of CUB with only 25% of itsconcept annotations randomly selected before training, resulting in a bottleneck with 28 concepts (see
Appendix A.11 for details).
Once these models have been trained to convergence, we use their learntbottleneck representations to predict the remaining 75% of the concept annotations in CUB using a simple logistic linear model.
The model trained using the Hybrid bottleneck notably underperfomswhen compared to the model trained using the CEM bottleneck (Hybrid-trained model has a meanconcept accuracy of 91.83% ± 0.51% while the CEM-trained model’s concept accuracy is 94.33% ±0.88%).
This corroborates our CAS results by suggesting that the bottlenecks learnt by CEMs areconsiderably more interpretable and can therefore serve as powerful feature extractors.
We can further explore this phenomena qualitatively by visualising the embeddings learnt for a singleconcept using its 2-dimensional t-SNE [28] plot.
As shown in colour in Figure 5a, we can see thatthe embedding space learnt for a concept
ˆ c i (here we show the concept “has white wings”) forms twoclear clusters of samples, one for points in which the concept is active and one for points in which theconcept is inactive.
When performing a similar analysis for the same concept in the Hybrid CBM(Figure 5b), where we use the entire extra capacity ˆ c [ k : k + γ ] as the concept’s embedding representation,we see that this latent space is not as clearly separable as that in CEM’s embeddings, suggesting thislatent space is unable to capture concept-semantics as clearly as CEM’s latent space.
Notice that
CEM’s t-SNE seems to also show smaller subclusters within the activated and inactivated clusters.
As
Figure 5c shows, by looking at the nearest Euclidean neighbours in concept’s c i embedding’s space,we see that CEM concepts clearly capture a concept’s activation, as well as exhibit high class-wisecoherence by mapping same-type birds close to each other (explaining the observed subclusters).
These results, and similar qualitative results in Appendix A.12, suggest that CEM is learning ahierarchy in its latent space where embeddings are separated with respect to their concept activationand, within the set of embeddings that have the same activation, embeddings are clustered accordingto their task label.
[Figure]
CEM supports effective concept interventions and is more robust to incorrect interventions(Figure 6) When describing our CEM architecture, we argued in favour of using a mixture oftwo semantic embeddings for each concept as this would permit test-time interventions whichcan meaningfully affect entire concept embeddings.
In Figure 6 left and center-left, we observe,as hypothesised, that using a mixture of embeddings allows CEMs to be highly responsive torandom concept interventions in their bottlenecks.
Notice that although all models have a similarconcept accuracy (see Appendix A.7), we observe that Hybrid CBMs, while highly accurate withoutinterventions, quickly fall short against even scalar-based CBMs once several concepts are intervenedin their bottlenecks.
In fact, we observe that interventions in Hybrid CBM bottlenecks have littleeffect on their predictive accuracy, something that did not change if logit concept probabilities wereused instead of sigmoidal probabilities.
However, even interventions performed by human experts arequite rarely perfect. For this reason, we simulate incorrect interventions (where a concept is set to thewrong value) to measure how robust the model is to such errors. We observe (Figure 6 center-rightand right) that CEM’s performance deteriorates as more concepts are incorrectly intervened on (asopposed to hybrid-CBMs), while it can withstand a few errors without losing much performance(as opposed to Bool and Fuzzy-CBMs).
We suggest that this is a consequence of CEM’s “incorrect”embeddings still carrying important task-specific information which can then be used by the labelpredictor to produce more accurate task labels, something worth exploring in future work.
As aresult, users can trust CEMs to better handle a small number of accidental mistakes made by humanexperts when intervening in its concept activations.
Finally, by comparing the effect of interventions in both CEMs and CEMs trained without RandInt, we observe that RandInt in fact leads to a modelthat is not just significantly more receptive to interventions, but is also able to outperform evenscalar-based CBMs when large portions of their bottleneck are artificially set by experts (e.g., as in CelebA).
This, as shown in Appendix A.8, comes without a significant computational training costsfor CEM.
Interestingly, such a positive effect in concept interventions is not observed if RandInt isused when training our other baselines (see Appendix A.14 for an explanation).
This suggests thatour proposed architecture can not only be trusted in terms of its downstream predictions and conceptexplanations, as seen above, but it can also be a highly effective model when used along with expertsthat can correct mistakes in their concept predictions.
For further details, including an exploration ofperforming interventions with Sequential and Independent CBMs, please refer to Appendix A.13.
[Figure]
Discussion
Relations with the state-of-the-art Concept bottleneck models engender user trust [13] by (i)forcing information flow through concept-aligned activations during training and (ii) by supportinghuman interventions in their concept predictions.
This allows CBMs to circumvent the well-knownunreliability of post-hoc methods [29, 30, 10], such as saliency maps [31–33], and invites theiruse in settings where input features are naturally hard to reason about (e.g., raw image pixels).
Inaddition, CBMs encourage human interactions allowing experts to improve task performance byrectifying mispredicted concepts, as opposed to other concept-based interpretable architectures suchas Self-Explainable Neural Networks [26] and Concept Whitening [12].
However, our experimentsshow that all existing CBMs are limited to significant accuracy-vs-interpretability trade-offs.
Inthis respect, our work reconciles theoretical results with empirical observations: while theoreticalresults suggest that explicit per-concept supervisions should improve generalisation bounds [34], incontrast Koh et al.
[9], Chen et al.
[12], and Mahinpei et al.
[14] empirically show how learning withintermediate concepts may impair task performance in practice.
The Information Plane method [27]reveals that the higher generalisation error of existing concept bottleneck models might be explainedas a compression in the input information flow caused by narrow architectures of Boolean and
Fuzzy CBMs.
In contrast, CEM represents the first concept-based model which does not need tocompromise between task accuracy, concept interpretability or intervention power, thus filling thisgap in the literature.
Furthermore, through an ablation study on CEM’s embedding size shown in
Appendix A.15, we see that one does not need to increase the embedding size drastically to beginto see the benefits of using a CEM over a vanilla CBM or an end-to-end black box DNN.
We note,however, that as stronger interpretable models are deployed, there are risks of societal harm whichwe must be vigilant to avoid.
Conclusion Our experiments provide significant evidence in favor of CEM’s accuracy/interpretability and, consequently, in favor of its real-world deployment.
In particular, CEMsoffer: (i) state-of-the-art task accuracy, (ii) interpretable concept representations aligned with humanground truths, (iii) effective interventions on learnt concepts, and (iv) robustness to incorrect conceptinterventions.
While in practice CBMs require carefully selected concept annotations during training,which can be as expensive as task labels to obtain, our results suggest that CEM is more efficientin concept-incomplete settings, requiring less concept annotations and being more applicable toreal-world tasks.
While there is room for improvement in both concept alignment and task accuracyin challenging benchmarks such as CUB or CelebA, as well as in resource utilization during inference/training (see Appendix A.8), our results indicate that CEM advances the state-of-the-art for theaccuracy-vs-interpretability trade-off, making progress on a crucial concern in explainable AI.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] Section 3 and Section 5 reflect our paper’s contributions
and scope as indicated in our introduction
(b) Did you describe the limitations of your work?
[Yes] See our conclusion in Section 6for a discussion on some of the limitations of our approach.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
We
discuss this in Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] We have read the ethics review guidelines and ensured that your paper
conforms to them.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We uploaded azip file with our code and documentation in the supplemental material and intend
to
make our repository public if the paper is accepted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] All training details are provided in Appendix A.
(c) Did you report error bars (e.g., w.r.t.
the random seed after running experiments multiple times)?
[Yes] Error bars are reported for all metrics and represent 95%
confidence intervals of the mean, unless specified otherwise.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Computational resources we used
are specified in Appendix A.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] References we used
[Table] are listed in Appendix A.5 (code), Appendix A.4 (models), and Appendix A.3 (data). [Table]
(b) Did you mention the license of the assets?
[Yes] We specified the licence of our new [Table]
assets in Appendix A.5.
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table] We included our new assets (code, data, and models) in the supplemental material. [Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]
using/curating?
[N/A] All new datasets are synthetic.
[Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable [Table]
information or offensive content?
[N/A] All new datasets are synthetic. [Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A] All new datasets are synthetic. [Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A] All new datasets are synthetic.
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A] All new datasets are synthetic. [Table]