GlanceNets: Interpretable, Leak-proof Concept-based Models
Abstract
There is growing interest in concept-based models (CBMs) that combine highperformance and interpretability by acquiring and reasoning with a vocabulary of high-level concepts.
A key requirement is that the concepts be interpretable.
Existing CBMs tackle this desideratum using a variety of heuristics based on unclear notions of interpretability, and fail to acquire concepts with the intended semantics.
We address this by providing a clear definition of interpretability in terms of alignment between the model’s representation and an underlying data generation process, and introduce GlanceNets, a new CBM that exploits techniques from disentangled representation learning and open-set recognition to achieve alignment, thus improving the interpretability of the learned concepts.
We show that GlanceNets, paired with concept-level supervision, achieve better alignment than state-of-the-art approaches while preventing spurious concepts from unintentionally affecting its predictions.
The code is available at https://github.com/ema-marconato/glancenet.
Introduction
Concept-based models (CBMs) are an increasingly popular family of classifiers that combine the transparency of white-box models with the flexibility and accuracy of regular neural nets [1–5].
At their core, all CBMs acquire a vocabulary of concepts capturing high-level, task-relevant properties of the data, and use it to compute predictions and produce faithful explanations of their decisions [6].
The central issue in CBMs is how to ensure that the concepts are semantically meaningful and interpretable for (sufficiently expert and motivated) human stakeholders.
Current approaches struggle with this.
One reason is that the notion of interpretability is notoriously challenging to pin down, and therefore existing CBMs rely on different heuristics—such as encouraging the concepts to be sparse [1], orthonormal to each other [5], or match the contents of concrete examples [3]—with unclear properties and incompatible goals.
A second, equally important issue is concept leakage , whereby the learned concepts end up encoding spurious information about unrelated aspects of the data, making it hard to assign them clear semantics [7].
Notably, even concept-level supervision is insufficient to prevent leakage [8].
Prompted by these observations, we define interpretability in terms of alignment : learned concepts are interpretable if they can be mapped to a (partially) interpretable data generation process using a transformation that preserves semantics.
This is sufficient to unveil limitations in existing strategies, build an explicit link between interpretability and disentangled representations, and provide a clear and actionable perspective on concept leakage.
Building on our analysis, we also introduce GlanceNets (aliGned LeAk-proof coNCEptual Networks), a novel class of CBMs that combine techniques from disentangled representation learning [9] and open-set recognition (OSR)
[10] to actively pursue alignment – and guarantee it under suitable assumptions – and avoid concept leakage.
Contributions:
Summarizing, we: ( i ) Provide a definition of interpretability as alignment that facilitates tapping into ideas from disentangled representation learning; ( ii ) Show that concept leakage can be viewed from the perspective of out-of-distribution generalization; ( iii ) Introduce GlanceNets, a novel class of CBMs that acquire interpretable representations and are robust to concept leakage; ( iv )
Present an extensive empirical evaluation showing that GlanceNets are as accurate as state-of-the-art
CBMs while attaining better interpretability and avoiding leakage.
Concept-based Models: Interpretability and Concept Leakage
Concept-based models (CBMs) comprise two key elements: (i)
A learned vocabulary of k high-level concepts meant to enable communication with human stakeholders [11], and (ii) a simulatable [12] classifier whose predictions depend solely on those concepts.
Formally, a CBM f : R d → [ c ] , with [ c ] := { 1 , . . .
, c } , maps instances x to labels y by measuring how much each concept activates on the input, obtaining an activation vector z ( x ) :
= ( z 1 ( x ) , . . .
, z k ( x ))
T ∈ R k , aggregating the activations into per-class scores s y ( x ) using a linear map [1, 3, 5], and then passing these through a softmax, i.e.,
[Equation]
Each weight w yj ∈ R encodes the relevance of concept z j for class y .
The activations themselves are computed in a black-box manner, often leveraging pre-trained embedding layers, but learned so as to capture interpretable aspects of the data using a variety of heuristics, discussed below.
Now, as long as the concepts are interpretable , it is straightforward to extract human understandable local explanations disclosing how different concepts contributed to any given decision ( x , y ) by looking at the concept activations and their associated weights, thus abstracting away the underlying computations.
This yields explanations of the form { ( w yj , z j ( x )) : j ∈ [ k ] } that can be readily summarized 1 and visualized [13, 14].
Importantly, the score of class y is conditionally independent from the input x given the corresponding explanation, i.e., s y ( x ) ⊥⊥ x | E ( x , y ) , ensuring that the latter is faithful to the model scores.
GlanceNets inherit all of these features.
Heuristics for interpretability.
Crucially, CBMs are only interpretable insofar as their concepts are.
Existing approaches implement special mechanisms to this effect, often pairing a traditional classification loss (such as the cross-entropy loss) with an auxiliary regularization term.
Alvarez-Melis and Jaakkola [1] acquire concepts using an autoencoder augmented with a sparsification penalty encouraging distinct concepts to activate on different instances.
Chen et al.
[5] apply geometric transforms to learn mutually orthonormal concepts that thus encode complementary information and attain comparable activation ranges.
These mechanisms – sparsity and orthogonality, respectively – alone cannot prevent capturing features that are not semantic in nature.
A second group of CBMs tackle this issue by constraining the concepts to match concrete cases, in the hope that these are better aligned with human intuition [15].
For instance, prototype classification networks [2], part-prototype networks [3], and related approaches [16–18] model concepts using prototypes in embedding space that perfectly match training examples or parts thereof.
Depending on the embedding space, which ultimately determines the distance to the prototypes, concepts learned this way may activate on elements unrelated to the example they match, leading to unclear semantics [19].
Closest to our work, concept bottleneck models (CBNMs) [20, 4] align the concepts using conceptlevel supervision – possibly obtained from a separate source, like ImageNet [21] – either sequentially or in tandem with the top-level dense layer.
From a statistical perspective, this seems perfectly sensible: if the supervision is unbiased and comes in sufficient quantity, and the model has enough capacity, this strategy appears to guarantee the learned and ground-truth concepts to match.
Concept leakage in concept-bottleneck models.
Unfortunately, concept-level supervision is not sufficient to guarantee interpretability.
Mahinpei et al.
[7] have demonstrated that concepts acquired by CBNMs pick up spurious properties of the data.
In their experiment, they learn two concepts zand z 5 , meant to represent the 4 and 5 MNIST digits, using concept-level supervision, and then show that – surprisingly – these concepts can be used to classify all other digits (i.e., MNIST images that are neither 4 ’s nor 5 ’s) as even or odd significantly better than random guessing.
This phenomenon, whereby learned concepts unintentionally capture information about unobserved concepts, is known as concept leakage .
Intuitively, leakage occurs because in CBNMs the concepts end up unintentionally capturing distributional information about unobserved aspects of the input, failing to provide well-defined semantics.
However, a clear definition of leakage is missing, and so are strategies to prevent it.
In fact, separating concept learning from classification and increasing the amount of supervision for the observed concepts (here, 4 and 5 ) is not enough [8].
A key contribution of our paper is showing that leakage can be understood from the perspective of domain shift and dealt with using open-set recognition [10].
Disentangling Interpretability and Concept Leakage
The main issue with heuristics used by CBMs is that they are based on unclear notions of interpretability.
In order to develop effective algorithms, we propose to view interpretability as a form of alignment between the machine’s representation and that of its user.
This enables us to identify conditions under which interpretability can be achieved, build links to well-understood properties of representations, and leverage state-of-the-art learning strategies.
[Figure]
Interpretability.
We henceforth focus on the (rather general) generative process shown in Fig.
2: the observations X ∈ R
d are caused by n generative factors G ∈ R n , themselves caused by a set of confounds C (including the label Y [22]).
Notice that the generative factors can be statistically dependent due to the confounds C , but as noted by Suter et al.
[23], the total causal effect [24, Def. 6.12] between G i and G j is zero for all i ̸
= j .
The generative factors capture all information necessary to determine the observation [23, 25], so the goal is to learn concepts Z ∈ R k that recover them.
The variable T is also a confounding factor, but it is kept separate from C as it relates to concept leakage, and will be formally introduced later on.
We posit that a (learned) representation is only interpretable if it supports symbolic communication between the model and the user, in the sense that it shares the same (or similar enough) semantics to the user’s representation.
The latter is however generally unobserved.
We therefore make a second, critical assumption that some of the generative factors G I ⊆
G are interpretable to the user, meaning that they can be used as a proxy for the user’s internal representation.
Naturally, not all generative factors are interpretable [26], but in many applications some of them are.
For instance, in dSprites [27] the generative factors encode the position, shape and color of a 2D object, and in
CelebA [28] the hair color and nose size of a celebrity.
Human observers have a good grasp of such concepts.
Interpretability as alignment.
Under this assumption, if the variables Z J ⊆
Z are aligned to the generative factors G I by a map α : g (cid:55)→ z J that preserves semantics, they are themselves interpretable.
Now, defining what a semantics-preserving map should look like is challenging, but constructing one is not: the identity is clearly one such map, and so are maps that permute the indices and independently rescale the individual variables.
One desirable property is that α does not “mix” multiple G ’s into a single Z .
E.g., if Z blends together head tilt, hair color, and nose size, users will have trouble pinning down what it means.
This property can be formalized in terms of disentanglement [29, 23, 9].
This is however insufficient: we wish the map between G i and its associated factor Z j to be “simple”, so as to conservatively guarantee that it preserves semantics.
This makes alignment strictly stronger than disentanglement.
Motivated by these desiderata, we say that Z J is aligned to G I if it satisfies: ( i ) Disentanglement.
There exists an injective map between indices π : [ n I ] → [ k ] , where [ n I ] identifies the subset of generative factors indexes in G I , such that, for all i, i ′ ∈ [ n I ] , i ̸
= i ′ , and
j = π
( i ) , it holds that fixing G i is enough to fix Z j regardless of the value taken by the other generative factors
G i ′ , and ( ii ) Monotonicity.
The map α can be written as α ( g ) =
( µ 1 ( g π (1) ) , . . . , µ n ( g π ( n I ) ))
T , where the µ i ’s are monotonic transformations.
This holds, for instance, for linear transformations of the form
A ( g π (1) , . . .
, g π ( n I ) ) T , where A ∈ R n I × k is a matrix with no non-zero off-diagonal entries.
This second requirement can be relaxed depending on the application.
Notice that we do not require each G i to map to a single Z j (a property known as completeness [29]):
Z J is interpretable even if it contains multiple – perhaps slightly different, but aligned – transformations of the same G i .
Measuring alignment with DCI.
Disentanglement can be measured in a number of ways [30], but most of them provide little information about how simple the map α is.
In order to estimate alignment, we repurpose DCI, a measure of disentanglement introduced by Eastwood and Williams [29], see also ?? .
According to this metric, a representation Z J is disentangled if there exists a regressor that, given z J , can predict g I with high accuracy using few z
i ’s to predict each g i .
Following [29], we use a linear regressor with parameters B ∈ R k × n I on the test set – assuming that it is annotated with the interpretable generative factors and corresponding learned representations – and then measure how diffuse the weights associated to each latent factor are.
We do this by normalizing them and computing their average Shannon entropy over all G i ’s, i.e.,
[Equation]
Hence, DCI gauges the degree of mixing that a linear map can attain using the learned representation
Z , and as such it indirectly measures alignment, with B approximating the inverse of A .
Achieving alignment with concept-level supervision.
It has been shown that disentanglement cannot be achieved in the purely unsupervised setting [31].
This immediately entails that alignment is also impossible in that setting, highlighting a core limitation of approaches like self-explainable neural networks [1].
However, disentanglement can be attained if supervision about the generative factors is available, even only for a small percentage of the examples [32].
As a matter of fact, supervision is used in representation learning to achieve identifiability , a stronger condition than – and that entails both of – disentanglement and alignment [33].
Thus, following CBNMs, we seek alignment by leveraging concept-level supervision.
Interpretability and concept leakage.
Intuitively, concept leakage occurs when a model is trained on a data set on which:
( i ) Some generative factors G V ⊂ G vary , while the others G F =
G \ G V are fixed , and
ii )
The two groups of factors are statistically dependent
For instance, in the even vs. odd experiment 4 and 5 play the role of G V and the other digits of G F .
CBNMs with access to supervision on G V tend to acquire a latent representation that approximates these factors.
But, because of ( ii ), this representation correlates with the fixed factors G F .
This immediately explains why additional supervision on G V cannot prevent leakage, but rather has the opposite effect: the better a latent representation matches G V , the more information it conveys about
G F .
In contrast with previous assessments [7, 8], we observe that this phenomenon can be viewed as a special form of domain shift: the training examples are sampled from a ground-truth distribution p ( X , G | T = 1) in which G F is approximately fixed, e.g., p ( G F | T = 1)
=
δ ( g ′ F ) for some vector g ′ F , while in the test set, the data is sampled from a different distribution p ( X , G | T = 0) in which G F is no longer fixed.
In the MNIST task, for instance, when T = 1 no concept besidesand 5 can occur, while all concepts except 4 and 5 can occur when T = 0 .
Here, T ∈ { 0 , 1 } selects between training and test distribution, see Fig.
Now, CBMs have no strategy to cope with domain shift and thus cannot disambiguate between known training and unknown test concepts.
Motivated by this, we propose then to tackle concept leakage by designing a CBM specifically equipped with strategies for detecting – at inference time – instances that do not belong to the training distribution using OSR [10].
The idea is to estimate the value of the variable T at inference time, essentially predicting whether an input was sampled from a distribution similar enough to the training distribution, and therefore can be handled by a model learned on this distribution, or not.
This strategy proves very effective in practice, as shown by our empirical evaluation (Section 5.2).
Addressing Alignment and Leakage with GlanceNets
GlanceNets combine a VAE-like architecture [34, 35] for learning disentangled concepts with a prior and classifier designed for open-set prediction [36].
In order to accommodate for non-interpretable factors, the latent representation of GlanceNets Z is split into two: (i) k concepts Z J , aligned to the interpretable generative factors G I , that are used for prediction, and (ii) ¯ k opaque factors Z ¯ J that are only used for reconstruction.
Specifically, a GlanceNet comprises an encoder q ϕ
( Z | X ) and a decoder p θ ( X | Z ) , both parameterized by deep neural networks, as well as a classifier p W ( Y | Z J ) feeding off the interpretable concepts only.
The overall architecture is shown in Fig.
Following other CBMs, the classifier is implemented using a dense layer with parameters W ∈ R v × k followed by a softmax activation, i.e., p W ( Y | z J ) :
= softmax( W z J ) , and the most likely label is used for prediction.
The class distribution is obtained by marginalizing over the encoder’s distribution:
[Equation]
Equality holds because Y ⊥⊥ X | Z J .
In order to expedite the computation, we follow the general practice of approximating the integral as softmax( W E q ϕ
( z | x ) [ z J ]) =
softmax( W [ µ ϕ ( x )] J ) .
In contrast to regular VAEs, GlanceNets associate each class to a prototype in latent space through the prior p ( Z | Y ) , which is conditioned on the class and modelled as a mixture of gaussians with one component per class.
The encoder, decoder, and prior are fit on data so as to maximize the evidence lower bound (ELBO) [37], defined as E p D ( x ,y )
[ L ( θ, x , y ; β )] with:
[Equation]
Here, p D ( x , y ) is the empirical distribution of the training set D
= { ( x i , y i ) :
i = 1 , . .
, m } .
The first term of Eq.
(4) is the likelihood of an example after passing it through the encoder distribution.
The second term penalizes the latent vectors based on how much their distribution differs from the prior and encourages disentanglement.
As mentioned in Section 3, learning disentangled representations is impossible in the unsupervised i.i.d.
setting [31].
Following Locatello et al.
[32], and similarly to CBNMs, we assume access to a (possibly separate) data set (cid:101)
D
= { ( x ℓ , g I,ℓ ) } containing supervision about the interpretable generative factors G I and integrate it into the ELBO by replacing the per-example loss L in Eq.
(4) with:
[Equation]
where γ > 0 controls the strength of the concept-level supervision.
Following [32], the term Ω( z , g ) penalizes encodings sampled from q ϕ ( z | x ) for differing from the annotation g .
Specifically, we implement this term using the average cross-entropy loss Ω( z , g ) :
= − (cid:80) k
g k log σ ( z k ) + (1 − g k ) log(1 − σ ( z k )) , where the annotations g k are rescaled to lie in [0 , 1] and σ is the sigmoid function.
Dealing with concept leakage.
In order to tackle concept leakage, GlanceNets integrate the OSR strategy of Sun et al.
[36], indicated in Fig.
1 by the “osr” block.
This strategy identifies out-of-class inputs by considering the class prototype µ y :=
E p ( z | y )
[ z ] in R k defined by the prior distribution and the decoder p θ ( x | z ) .
Recall that the prior is fit jointly with the encoder, decoder, and classifier by optimizing the ELBO.
Once learned, a GlanceNet uses the training data to estimate: ( i ) a distance threshold η y , which defines a spherical subset in the latent space B y
=
{ z : ∥ µ y − z
∥ < η y } centered around the prototype of class y (i.e., the mean of the corresponding Gaussian mixture component), and ( ii ) a maximum threshold on the reconstruction error η thr .
If new data points have reconstruction error above η thr or they do not belong to any subset B y , they are inferred as open-set instances, i.e., ˆ T = 0 .
In practice, we found that choosing the thresholds as to include 95% of training examples to work well in our experiments.
Further details are available in Appendices ??
and ?? .
4.1 Benefits and Limitations
GlanceNets can naturally be combined with different VAE-based architectures for learning disentangled representations [38], including β -TCVAEs [39], InfoVAEs [40], DIP-VAEs [41], and
JL1-VAEs [42].
Since our experiments already show substantial benefits for GlanceNets building on β -VAEs [43], we leave a detailed study of these extensions to future work.
Like CBNMs, GlanceNets foster alignment by leveraging supervision on the interpretable generative factors [32], possibly derived from an external data set [20].
However, GlanceNets can be readily adapted to a variety of different kinds of supervision used for VAE-based models, including partially annotated examples [26], group information [44], pairings [45, 46] and other kinds of weak supervision [47, 48], as well as feedback from a domain expert [49].
On the other hand, CBNMs are incompatible with these approaches.
One limitation inherited from VAEs by GlanceNets is the assumption that the interpretable generative factors are disentangled from each other [23].
In practice, GlanceNets work even when this does not hold (as in our even vs. odd experiment, see Section 5.2).
However, one direction of future work is to integrate ideas from hierarchical disentanglement [50].
Empirical Evaluation
In this section, we present results on several tasks showing that GlanceNets outperform CBNMs [20] in terms of alignment and robustness to leakage, while achieving comparable prediction accuracy.
All experiments were implemented using Python 3 and Pytorch [51] and run on a server with 128 CPUs, 1TiB RAM, and 8 A100 GPUs.
GlanceNets were implemented on top of the disentanglement-pytorch [52] library.
All alignment and disentanglement metrics were computed with disentanglement_lib [31].
Code for the complete experimental setup is available on GitHub at the link: https://github.com/ema-marconato/glancenet.
Additional details on architectures and hyperparameters can also be found in the Supplementary Material.
5.1 GlanceNets achieve better alignment than CBNMs
In a first experiment, we compared GlanceNets with CBNMs on three classification tasks for which supervision on the generative factors is available.
In order to evaluate the impact of this supervision on the different competitors, we varied the amount of training examples annotated with it from 1% to 100% .
For each increment, we measured prediction performance using accuracy, and alignment using the linear variant of DCI [29] discussed in Section 3.
Data sets.
We carried out our evaluation on two data sets taken from the disentanglement literature and a very challenging real-world data set.
dSprites [27] consists of 64 × 64 black-and-white images of sprites on a flat background, where each sprite is determined by one categorical and four generative factors, namely shape , size , rotation , position_x , and position_y .
The images were obtained by discretizing and enumerating the generative factors, for a total of 3 × 6 × 40 × 32 ×
D S P R
I TE
S
M P
I
D
C
ELE B
A images.
MPI3D [53] consists of 64 × 64 RGB rendered images of 3D shapes held by a robotic arm.
The generative factors are object_color , object_shape , object_size , camera_height , background_color , and the horizontal and vertical position of the arm.
The data contains 6 × 6 × 2 × 3 × 4 × 40 × 40 examples.
CelebA [28] is a collection of 178 × 218 RGB images of over 10k celebrities, converted to 64 × 64 by first cropping them to 178 × 178 and then rescaling.
Images are annotated with 40 binary generative factors including hair color, presence of sunglasses, etc .
Since we are interested in measuring alignment, we considered only those 10 factors that CBNMs can fit well (in the Appendix).
We also dropped all those examples for which hair color is not unique (e.g., annotated as both blonde and black ), obtaining approx.
127 k examples.
CelebA is more challenging than dSprites and MPI3D, as it does not include all possible factor variations and the generative factors – although disentangled – are insufficient to completely determine the contents of the images.
For dSprites and MPI3D, we used a random 80/10/10 train/validation/test split, while for
CelebA we kept the original split [28].
We generated the ground-truth labels y as follows.
For dSprites, we labeled images according to a random but fixed linear separator defined over the continuous generative factors, chosen so as to ensure that the classes are balanced.
For MPI3D and CelebA, we focused on the categorical factors instead.
Specifically, we clustered all images using the algorithm of [54], for a total of 10 andclusters for MPI3D and CelebA respectively, and then labeled all examples based on their reference cluster.
This led to slightly unbalanced classes containing different percentages of examples, ranging from 5% to 16% in MPI3D and from 21% to 29% in CelebA.
Architectures.
For dSprites and MPI3D, we implemented the encoder as a six layer convolutional neural net, while for CelebA we adapted the convolutional architecture of Ghosh et al.
We employed a six layer convolutional architecture for the decoder in all cases, for simplicity, as changing it did not lead to substantial differences in performance.
In all cases, as for all CBMs (see Section 2), the classifier was implemented as a dense layer followed by a softmax activation.
The very same architectures were used for both GlanceNets and CBNMs, for fairness.
For each data set, we chose the latent space dimension as the total number of generative factors, where categorical ones are one hot encoded.
In particular, we used 7 latent factors for dSprites, 21 for MPI3D and 10 for CelebA.
Further details are included in the Supplementary Material.
Results and discussion.
The results of this first experiment are reported in Fig.
The behavior of both competitors on dSprites and MPI3D was extremely stable, owing to the fact that these data sets cover an essentially exhaustive set of variations for all generative factors, so we report their hold-out performance on the test set.
Since for CelebA variance was non-negligible, we ran both methodstimes varying the random seed used to initialize the network and report the average performance across runs and its standard deviation.
In addition to alignment, we also report explicitness [29], which measures how well the linear regressor employed by DCI fits the generative factors.
The higher, the better.
Details on its evaluation are included in Suppl.
Material.
The plots clearly show that, although the two methods achieve high and comparable accuracy in all settings, GlanceNets attain better alignment in all data sets and for all supervision regimes than
CBNMs, with a single exception in CelebA using low values of supervision, for a total of 13 wins out of 15 cases.
In all disentanglement data sets, there is a clear margin between the alignment achieved by GlanceNets and that of CBNMs: performances vary up to maximum of 15% in dSprites, and a minumum of 8% in MPI3D. In CelebA, the gap is evident with full supervision (almost 8% of difference in alignment), and GlanceNets still attain overall better scores in the 25% and 50% regime.
On the other hand, performance are lower, but comparable, with 10% supervision.
The case at 1% refers to an extreme situation where both CBNMs and GlanceNets struggle to align with generative factors, as is clear also from the very low explicitness.
In dSprites and MPI3D, both GlanceNets and CBNMs quickly achieve very high alignment at 1% supervision, as expected [32], whereas better results in CelebA are obtained with growing supervision.
Also, both models display similar stability on this data set, as shown by the error bars in the plot.
5.2 GlanceNets are leak-proof
Next, we evaluated robustness to concept leakage in two scenarios that differ in whether the unobserved generative factors are disentangled with the observed ones or not, see Section 3.
In both experiments, we compare GlanceNets with a CBNM and a modified GlanceNet where the OSR component has been removed (denoted CG-VAE).
Leakage due to unobserved entangled factors.
We start by replicating the experiment of Mahinpei et al.
[7]: the goal is to discriminate between even and odd MNIST images using a latent representation
Z =
( Z 4 , Z 5 ) obtained by training (with complete supervision on the generative factors) only on examples of 4 ’s and 5 ’s. Leakage occurs if the learned representation can be used to solve the prediction task better than random on a test set where all digits except 4 and 5 occur.
During training, we use the digit label for conditioning the prior p ( Z | Y ) of the GlanceNet.
More qualitative results are collected in ?? .
[Figure]
Fig.
4 (a, b) illustrates the latent representations of the training and test set output by a GlanceNet: since the two digits are mutually exclusive, the model has learned to map all instances along the ( z 4 , z 5 ) diagonal.
This is where OSR kicks in: if an input is identified as open-set, T is predicted as 0 by the OSR component and the input is rejected.
In all leakage experiments, we implement rejection by predicting a random label.
Since MNIST is balanced, we measure leakage by computing the difference in accuracy between the classifier and an ideal random predictor, i.e., 2 · | acc − 12 | : the smaller, the better.
The results, shown in Fig.
4 (c), show a substantial difference between GlanceNet and the other approaches.
Consistently with the values reported in [7], CBNMs are affected by a considerable amount of leakage, around 28% .
This is not the case for our GlanceNet: most (approx.
85% ) test images are correctly identified as open-set and rejected, leading to a very low (about 2% ) leakage, 26% less than CBNMs.
The results for CG-VAE also indicate that removing the open-set component from GlanceNets dramatically increases leakage back to around 30% .
This shows that alignment and disentanglement alone are not sufficient, and that the open-set component plays a critical role for preventing leakage.
Leakage due to unobserved disentangled factors.
Next, we analyze concept leakage between disentangled generative factors using the dSprites data set.
To this end, we defined a binary classification task in which the ground-truth label depends on position_x and position_y only.
In particular, instances within a fixed distance from (0 , 0) are annotated as positive and the rest as negative, as shown in Fig.
5 (a).
In order to trigger leakage, all competitors are trained (using full concept-level supervision, as before) on training images where shape , size and rotation vary, but position_x and position_y are almost constant (they range in a small interval around (0 . 5 , 0 . 5) , cf.
Fig. 5).
leakage occurs if the learned model can successfully classify test instances where position_x and position_y are no longer fixed.
More qualitative results can be found in ?? .
[Figure]
hyperplane
For both competitors, we encode shape using a 3D one-hot encoding and size and rotation as continuous variables.
During training, we use the shape annotation for conditioning the prior p ( Z | Y ) of the GlanceNet.
The first two PCA components of the latent representations acquired by our GlanceNet and by a CBNM are shown, rotated so as to be separable on the first axis, in Fig.
5 (b, c): in both cases, it is possible to separate positives from negatives based on the obtained representations in the five latent dimensions.
As shown in Fig.
5 (d), this means that both CBNM and CG-VAE suffer from very large leakage, 80% and 98% , respectively.
In contrast, OSR allows us to correctly identify and reject almost all test instances, leading to negligible leakage even in this disentangled setting.
Related Work
Concept-based explainability.
Concepts lie at the heart of AI [56] and have recently resurfaced as a natural medium for communicating with human stakeholders [11].
In explainable AI, this was first exploited by approaches like TCAV [57], which extract local concept-based explanations from black-box models using concept-level supervision to define the target concepts.
Post-hoc explanations, however, are notoriously unfaithful to the model’s reasoning [58–60].
CBMs, including GlanceNets, avoid this issue by leveraging concept-like representations directly for computing their predictions.
Existing CBMs model concepts using prototypes [2, 3, 16, 17] or other representations [1, 20, 4, 5], but they seek interpretability using heuristics, and the quality of concepts they acquire has been called into question [61, 19, 7, 8].
We show that disentangled representation learning helps in this regard.
Disentanglement and interpretability.
Interpretability is one of the main driving factors behind the development of disentangled representation learning [62–64].
These approaches however make no distinction between interpretable and non-interpretable generative factors and generally focus on properties of the world , like independence between causal mechanisms [9] or invariances [43].
Interpretability, however, depends on human factors that are not well understood and therefore usually ignored [12, 65].
The link between disentanglement and interpretability has never been made explicit.
Importantly, in contrast to alignment, disentanglement does not require that the map between matching generative and learned factors preserves semantics.
We remark that other VAE-based classifiers either do not tackle disentanglement or are unconcerned with concept leakage [66, 67, 36].
Disentanglement and CBMs.
Neither the literature on disentanglement nor the one on CBMs have attempted to formalize the notion of interpretability or to establish a proper link between the latter and disentanglement.
The work of Kazhdan et al.
[68] is the only one to compare techniques for disentangled representation learning and concept acquisition, however it makes no attempt at linking the two notions.
Our work fills this gap.
Acknowledgments and Disclosure of Funding
The research of ST and AP was partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Refer to sec.(c)
Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We described the main procedure to obtain our same results.
More details will be available on the Suppl.
Material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] The most important details are addresses in the main text 3.
Those not relevant to our discussion, like values of the hyperparameters, number of epochs and training heuristics, are included in the Suppl.
Material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We reported the errorbars for CelebA experiment, see Fig.(d)
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We use existing libraries and open data sets.
Sources are cited in the experiments section (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]