GlanceNets: Interpretable, Leak-proof Concept-based Models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
There is growing interest in concept-based models (CBMs) that combine highperformance and interpretability by acquiring and reasoning with a vocabulary ofhigh-level concepts.
A key requirement is that the concepts be interpretable.
Existing CBMs tackle this desideratum using a variety of heuristics based on unclearnotions of interpretability, and fail to acquire concepts with the intended semantics.
We address this by providing a clear deﬁnition of interpretability in terms ofalignment between the model’s representation and an underlying data generationprocess, and introduce GlanceNets, a new CBM that exploits techniques from disentangled representation learning and open-set recognition to achieve alignment, thusimproving the interpretability of the learned concepts.
We show that GlanceNets,paired with concept-level supervision, achieve better alignment than state-of-the-artapproaches while preventing spurious information from unintendedly leaking intothe learned concepts.
Introduction
Concept-based models (CBMs) are an increasingly popular family of classiﬁers that combine thetransparency of white-box models with the ﬂexibility and accuracy of regular neural nets [1–5].
Attheir core, all CBMs acquire a vocabulary of concepts capturing high-level, task-relevant propertiesof the data, and use it to compute predictions and produce faithful explanations of their decisions [6].
The central issue in CBMs is how to ensure that the concepts are semantically meaningful andinterpretable for (sufﬁciently expert and motivated) human stakeholders.
Current approaches strugglewith this.
One reason is that the notion of interpretability is notoriously challenging to pin down,and therefore existing CBMs rely on different heuristics—such as encouraging the concepts to besparse [1], orthonormal to each other [5], or match the contents of concrete examples [3]—withunclear properties and incompatible goals.
A second, equally important issue is concept leakage ,whereby the learned concepts end up encoding spurious information about unrelated aspects of thedata, making it hard to assign them clear semantics [7].
Notably, even concept-level supervision isinsufﬁcient to prevent leakage [8], cf.
Fig.
Prompted by these observations, we deﬁne interpretability in terms of alignment : learned conceptsare interpretable if they can be mapped to a (partially) interpretable data generation process using atransformation that preserves semantics.
This is sufﬁcient to unveil limitations in existing strategies,build an explicit link between interpretability and disentangled representations, and provide a clear andactionable perspective on concept leakage.
Building on our analysis, we also introduce GlanceNets(aliGned LeAk-proof coNCEptual Networks), a novel class of CBMs that combine techniques fromdisentangled representation learning [9] and open-set recognition (OSR) [10] to actively pursuealignment – and guarantee it under suitable assumptions – and avoid concept leakage.
Contributions:
Summarizing, we: ( i ) Provide a deﬁnition of interpretability as alignment thatfacilitates tapping into ideas from disentangled representation learning; ( ii ) Show that concept leakagecan be viewed from the perspective of out-of-distribution generalization; ( iii ) Introduce GlanceNets, anovel of CBMs that acquire interpretable representations and are robust to concept leakage; ( iv )
Present an extensive empirical evaluation showing that GlanceNets are as accurate as state-of-the-art CBMs while attaining better interpretability and avoiding leakage.
Concept-based Models: Interpretability and Concept Leakage
Concept-based models (CBMs) comprise two key elements: (i)
A learned vocabulary of k high-levelconcepts meant to enable communication with human stakeholders [11], and (ii) a simulatable [12]classiﬁer whose predictions depend solely on those concepts.
Formally, a CBM f : R d → [ c ] , with[ c ] := { 1 , . . .
, c } , maps instances x to labels y by measuring how much each concept activates on theinput, obtaining an activation vector z ( x ) :
= ( z 1 ( x ) , . . .
, z k ( x ))
T ∈ R k , aggregating the activationsinto per-class scores s y ( x ) using a linear map [1, 3, 5], and then passing these through a softmax, i.e.,
[Equation]
Each weight w yj ∈ R encodes the relevance of concept z j for class y .
The activations themselves arecomputed in a black-box manner, often leveraging pre-trained embedding layers, but learned so as tocapture interpretable aspects of the data using a variety of heuristics, discussed below.
Now, as long as the concepts are interpretable , it is straightforward to extract human understandablelocal explanations disclosing how different concepts contributed to any given decision ( x , y ) bylooking at the concept activations and their associated weights, thus abstracting away the underlyingcomputations.
This yields explanations of the form { ( w yj , z j ( x )) : j ∈ [ k ] } that can be readilysummarized 1 and visualized [13, 14].
Importantly, the score of class y is conditionally independentfrom the input x given the corresponding explanation, i.e., s y ( x ) ⊥⊥ x | E ( x , y ) , ensuring that thelatter is faithful to the model scores.
GlanceNets inherit all of these features.
Heuristics for interpretability.
Crucially, CBMs are only interpretable insofar as their conceptsare.
Existing approaches implement special mechanisms to this effect, often pairing a traditionalclassiﬁcation loss (such as the cross-entropy loss) with an auxiliary regularization term.
Alvarez-Melis and Jaakkola [1] acquire concepts using an autoencoder augmented with a sparsiﬁcationpenalty encouraging distinct concepts to activate on different instances.
Chen et al.
[5] apply geometrictransforms to learn mutually orthonormal concepts that thus encode complementary information andattain comparable activation ranges.
These mechanisms – sparsity and orthogonality, respectively –alone cannot prevent capturing features that are not semantic in nature.
A second group of CBMs tackle this issue by constraining the concepts to match concrete cases, inthe hope that these are better aligned with human intuition [15].
For instance, prototype classiﬁcationnetworks [2], part-prototype networks [3], and related approaches [16–18] model concepts usingprototypes in embedding space that perfectly match training examples or parts thereof.
Depending onthe embedding space, which ultimately determines the distance to the prototypes, concepts learned thisway may activate on elements unrelated to the example they match, leading to unclear semantics [19].
Closest to our work, concept bottleneck models (CBNMs) [20, 4] align the concepts using conceptlevel supervision – possibly obtained from a separate source, like ImageNet [21] – either sequentiallyor in tandem with the top-level dense layer.
From a statistical perspective, this seems perfectlysensible: if the supervision is unbiased and comes in sufﬁcient quantity, and the model has enoughcapacity, this strategy appears to guarantee the learned and ground-truth concepts to match.
Concept leakage in concept-bottleneck models.
Unfortunately, concept-level supervision is notsufﬁcient to guarantee interpretability.
Mahinpei et al.
[7] have demonstrated that concepts acquiredby CBNMs pick up spurious properties of the data.
In their experiment, they learn two concepts z 4and z 5 , meant to represent the 4 and 5 MNIST digits, using concept-level supervision, and then showthat – surprisingly – these concepts can be used to classify all other digits (i.e., MNIST images thatare neither 4 ’s nor 5 ’s) as even or odd signiﬁcantly better than random guessing.
This phenomenon,whereby learned concepts unintentionally capture information about unobserved concepts, is knownas concept leakage .
Intuitively, leakage occurs because in CBNMs the concepts end up unintentionally capturing distributional information about unobserved aspects of the input, failing to provide well-deﬁned semantics.
However, a clear deﬁnition of leakage is missing, and so are strategies to prevent it.
In fact, separatingconcept learning from classiﬁcation and increasing the amount of supervision for the observed concepts (here, 4 and 5 ) is not enough [8].
A key contribution of our paper is showing that leakage canbe understood from the perspective of domain shift and dealt with using open-set recognition [10].
Disentangling Interpretability and Concept Leakage
The main issue with heuristics used by CBMs is that they are based on unclear notions of interpretability.
In order to develop effective algorithms, we propose to view interpretability as a formof alignment between the machine’s representation and that of its user.
This enables us to identifyconditions under which interpretability can be achieved, build links to well-understood properties ofrepresentations, and leverage state-of-the-art learning strategies.
G C T X Z
Interpretability.
We henceforth focus on the (rather general) generative process shown in Fig.
2: the observations X ∈ R
d are caused by ngenerative factors G ∈ R n , themselves caused by a set of confounds C(including the label Y [22]).
Notice that the generative factors can bestatistically dependent due to the confounds C , but as noted by Suteret al.
[23], the total causal effect [24, Def. 6.12] between G i and G jis zero for all i (cid:54) =
j .
The generative factors capture all informationnecessary to determine the observation [23, 25], so the goal is to learnconcepts Z ∈ R k that recover them.
The variable T will be introducedlater on.
We posit that a (learned) representation is only interpretable if it supports symbolic communication between the model and the user, in the sense that it shares the same (orsimilar enough) semantics to the user’s representation.
The latter is however generally unobserved.
We therefore make a second, critical assumption that some of the generative factors G I ⊆
G areinterpretable to the user, meaning that they can be used as a proxy for the user’s internal representation.
Naturally, not all generative factors are interpretable [26], but in many applications some of themare.
For instance, in dSprites [27] the generative factors encode the position, shape and color of a 2Dobject, and in CelebA [28] the hair color and nose size of a celebrity.
Human observers have a goodgrasp of such concepts.
Interpretability as alignment.
Under this assumption, if the variables Z J ⊆
Z are aligned tothe generative factors
G I by a map α : g (cid:55)→ z J that preserves semantics, they are themselvesinterpretable.
Now, deﬁning what a semantics-preserving map should look like is challenging, but constructing one is not: the identity is clearly one such map, and so are maps that permute theindices and independently rescale the individual variables.
One desirable property is that α doesnot “mix” multiple G ’s into a single Z .
E.g., if Z blends together head tilt, hair color, and nosesize, users will have trouble pinning down what it means.
This property can be formalized in termsof disentanglement [29, 23, 9].
This is however insufﬁcient: we wish the map between G i and itsassociated factor Z j to be “simple”, so as to conservatively guarantee that it preserves semantics.
This makes alignment strictly stronger than disentanglement.
Motivated by these desiderata, we say that Z J is aligned to G I if it satisﬁes:( i ) Disentanglement.
There exists an injective map between indices π : [ n I ] → [ k ] , where [ n I ]identiﬁes the subset of generative factors indexes in G I , such that, for all i, i (cid:48) ∈ [ n I ] , i (cid:54) =
i (cid:48) , andj =
π ( i ) , it holds that ﬁxing G i is enough to ﬁx Z j regardless of the value taken by the othergenerative factors G i (cid:48) , and( ii ) Monotonicity.
The map α can be written as α ( g ) =
( µ 1 ( g π (1) ) , . . . , µ n ( g π ( n I ) ))
T , where theµ i ’s are monotonic transformations.
This holds, for instance, for linear transformations of theform A ( g π (1) , . .
, g π ( n I ) ) T , where A ∈ R n I × k is a matrix with no non-zero off-diagonal entries.
This second requirement can be relaxed depending on the application.
Notice that we do not require each G i to map to a single Z j (a property known as completeness [29]):
Z J is interpretable even if it contains multiple – perhaps slightly different, but aligned – transformations of the same G i .
Measuring alignment with DCI.
Disentanglement can be measured in a number of ways [30],but most of them provide little information about how simple the map α is.
In order to estimatealignment, we repurpose DCI, a measure of disentanglement introduced by Eastwood and Williams[29].
According to this metric, a representation Z J is disentangled if there exists a regressor that,given z J , can predict g I with high accuracy using few z
i ’s to predict each g i .
Following [29], we usea linear regressor with parameters B ∈ R k × n I on the test set – assuming that it is annotated withthe interpretable generative factors and corresponding learned representations – and then measurehow diffuse the weights associated to each latent factor are.
We do this by normalizing them andcomputing their average Shannon entropy over all G i ’s, i.e.,
[Equation]
where ¯ b ji = b ji / (cid:80) [Equation]
(cid:48) [Equation] [ k
[Equation]
b [Equation]
and
[Equation]
Hence, DCI gauges the degree of mixing that a linear map can attain using the learned representation
Z , and as such it indirectly measures alignment, with B approximating the inverse of A .
Achieving alignment with concept-level supervision.
It has been shown that disentanglementcannot be achieved in the purely unsupervised setting [31].
This immediately entails that alignmentis also impossible in that setting, highlighting a core limitation of approaches like self-explainableneural networks [1].
However, disentanglement can be attained if supervision about the generativefactors is available, even only for a small percentage of the examples [32].
As a matter of fact,supervision is used in representation learning to achieve identiﬁability , a stronger condition than –and that entails both of – disentanglement and alignment [33].
Thus, following CBNMs, we seekalignment by leveraging concept-level supervision.
Interpretability and concept leakage.
Intuitively, concept leakage occurs when a model is trainedon a data set on which ( i ) some generative factors G V ⊂ G vary, while the others G F =
G \ G Vare ﬁxed, and ( ii ) the two groups of factors are statistically dependent.
For instance, in the evenvs.
odd experiment 4 and 5 play the role of G V and the other digits of G F .
CBNMs with accessto supervision on G V tend to acquire a latent representation that approximates these factors. But,because of ( ii ), this representation correlates with the ﬁxed factors G F .
This immediately explainswhy additional supervision on G V cannot prevent leakage, but rather has the opposite effect: thebetter a latent representation matches G V , the more information it conveys about G F .
In contrast with previous assessments [7, 8], we notice that this phenomenon can be viewed as aspecial form of domain shift: the training examples are sampled from a ground-truth distribution p ( X , G | T = 1) in which G F is approximately ﬁxed, e.g., p ( G F | T = 1)
=
δ ( g (cid:48) F ) for somevector g (cid:48) F , while in the test set, the data is sampled from a different distribution p ( X , G | T = 0) inwhich G F is no longer ﬁxed.
In the MNIST task, for instance, when T = 1 no concept besides 4and 5 can occur, while all concepts except 4 and 5 can occur when T = 0 .
Here, T ∈ { 0 , 1 } selectsbetween training and test distribution, see Fig.
Now, CBMs have no strategy to cope with domainshift and thus cannot disambiguate between known training and unknown test concepts.
Motivated by this, we propose then to tackle concept leakage by designing a CBM speciﬁcallyequipped with strategies for detecting – at inference time – instances that do not belong to the trainingdistribution using OSR [10].
The idea is to estimate the value of the variable T at inference time,essentially predicting whether an input was sampled from a distribution similar enough to the trainingdistribution, and therefore can be handled by a model learned on this distribution, or not.
This strategyproves very effective in practice, as shown by our empirical evaluation (Section 5.2).
Addressing Alignment and Leakage with GlanceNets
GlanceNets combine a VAE-like architecture [34, 35] for learning disentangled concepts with a priorand classiﬁer designed for open-set prediction [36].
In order to accommodate for non-interpretablefactors, the latent representation of GlanceNets Z is split into two: (i) k concepts Z J , aligned to theinterpretable generative factors G I , that are used for prediction, and (ii) ¯ k opaque factors Z ¯ J thatare only used for reconstruction.
Speciﬁcally, a GlanceNet comprises an encoder q φ ( Z | X ) and adecoder p θ ( X | Z ) , both parameterized by deep neural networks, as well as a classiﬁer p W ( Y | Z J )feeding off the interpretable concepts only.
The overall architecture is shown in Fig.
Following other CBMs, the classiﬁer is implemented using a dense layer with parameters W ∈ R v × kfollowed by a softmax activation, i.e., p W ( Y | z J ) :
= softmax( W z J ) , and the most likely label isused for prediction.
The class distribution is obtained by marginalizing over the encoder’s distribution:
[Equation]
Equality holds because Y ⊥⊥ X | Z J .
In order to expedite the computation, we follow the generalpractice of approximating the integral as softmax( W E q φ ( z | x ) [ z J ]) =
softmax( W [ µ φ ( x )] J ) .
In contrast to regular VAEs, GlanceNets associate each class to a prototype in latent space through theprior p
( Z | Y ) , which is conditioned on the class and modelled as a mixture of gaussians with onecomponent per class.
The encoder, decoder, and prior are ﬁt on data so as to maximize the evidencelower bound [37], deﬁned as E p D ( x ,y )
[ L ( θ, x , y ; β )] with:
[Equation]
Here, p D ( x , y ) is the empirical distribution of the training set D
= { ( x i , y i ) :
i = 1 , . .
, m } .
Theﬁrst term of Eq.
(4) is the likelihood of an example after passing it through the encoder distribution.
The second term penalizes the latent vectors based on how much their distribution differs from theprior and encourages disentanglement.
As mentioned in Section 3, learning disentangled representations is impossible in the unsupervised i.i.d.
setting [31].
Following Locatello et al.
[32], andsimilarly to CBNMs, we assume access to a (possibly separate) data set (cid:101)
D
= { ( x (cid:96) , g I,(cid:96) ) } containingsupervision about the interpretable generative factors G I and integrate it into the ELBO by replacingthe per-example loss L in Eq.
(4) with:
[Equation]
where γ > 0 controls the strength of the concept-level supervision.
Following [32], the term Ω( z , g )penalizes encodings sampled from q φ ( z | x ) for differing from the annotation g .
Speciﬁcally, weimplement this term using the average cross-entropy loss Ω( z , g ) :
= − (cid:80) k
g k log σ ( z k )
+ (1 −g k ) log(1 − σ ( z k )) , where the annotations g k are rescaled to lie in [0 , 1] and σ is the sigmoid function.
In order to tackle concept leakage, GlanceNets integrate the OSR strategy of Sun et al.
Thisstrategy identiﬁes out-of-class inputs by considering the class prototype µ y :=
E p ( z | y )
[ z ] in R kdeﬁned by the prior distribution and the decoder p θ ( x | z ) .
Once the encoder has been learned, GlanceNets use the training data to estimate: ( i ) a distance threshold η y , which deﬁnes a sphericalsubset in the latent space B y =
{ z : (cid:107) µ y − z
(cid:107) <
η y } centered around the prototype of class y , and( ii ) a maximum threshold on the reconstruction error η thr .
If new data points have reconstruction error above η thr or they do not belong to any subset B y , they are inferred as open-set instances, i.e.,ˆ T = 0 .
This procedure is illustrated in Fig.
In practice, we found that choosing the thresholds as toinclude the 95% of training examples to work well in our experiments.
Further details are availablein Appendices A and C.
4.1 Beneﬁts and Limitations
GlanceNets can naturally be combined with different VAE-based architectures for learning disentangled representations [38], including β -TCVAEs [39], InfoVAEs [40], DIP-VAEs [41], and
JL1-VAEs [42].
Since our experiments already show substantial beneﬁts for GlanceNets building onβ -VAEs [43], we leave a detailed study of these extensions to future work.
Like CBNMs, GlanceNets foster alignment by leveraging supervision on the interpretable generativefactors [32], possibly derived from an external data set [20].
However, GlanceNets can be readilyadapted to a variety of different kinds of supervision used for VAE-based models, including partially annotated examples [26], group information [44], pairings [45, 46] and other kinds of weaksupervision [47, 48], as well as feedback from a domain expert [49].
On the other hand, CBNMs areincompatible with these approaches.
One limitation inherited from VAEs by GlanceNets is the assumption that the interpretable generativefactors are disentangled from each other [23].
In practice, GlanceNets work even when this does nothold (as in our even vs. odd experiment, see Section 5.2).
However, one direction of future work is tointegrate ideas from hierarchical disentanglement [50].
Empirical Evaluation
In this section, we present results on several tasks showing that GlanceNets outperform CBNMs [5]in terms of alignment and robustness to leakage, while achieving comparable prediction accuracy.
All experiments were implemented using Python 3 and Pytorch [51] and run on a serverwith 128 CPUs, 1TiB RAM, and 8 A100 GPUs.
GlanceNets were implemented on top of thedisentanglement-pytorch [52] library.
All alignment and disentanglement metrics were computed with disentanglement_lib [31].
Code for the complete experimental setup is available inthe Supplementary Material and will be released on GitHub.
Additional details on architectures andhyperparameters can also be found in the Supplementary Material.
5.1 GlanceNets achieve better alignment than CBNMs
In a ﬁrst experiment, we compared GlanceNets with CBNMs on three classiﬁcation tasks for whichsupervision on the generative factors is available.
In order to evaluate the impact of this supervisionon the different competitors, we varied the amount of training examples annotated with it from 1%to 100% .
For each increment, we measured prediction performance using accuracy, and alignmentusing the linear variant of DCI [29] discussed in Section 3.
Data sets.
We carried out our evaluation on two data sets taken from the disentanglement literatureand a very challenging real-world data set.
dSprites [27] consists of 64 × 64 black-and-whiteimages of sprites on a ﬂat background, where each sprite is determined by one categorical and fourgenerative factors, namely shape , size , rotation , position_x , and position_y .
The imageswere obtained by discretizing and enumerating the generative factors, for a total of 3 × 6 × 40 × 32 × 32images.
MPI3D [53] consists of 64 × 64 RGB rendered images of 3D shapes held by a roboticarm.
The generative factors are object_color , object_shape , object_size , camera_height ,background_color , and the horizontal and vertical position of the arm.
The data contains6 × 6 × 2 × 3 × 4 × 40 × 40 examples.
CelebA [28] is a collection of 178 × 218 RGB images ofover 10k celebrities, converted to 64 × 64 by ﬁrst cropping them to 178 × 178 and then rescaling.
Images are annotated with 40 binary generative factors including hair color, presence of sunglasses,etc .
Since we are interested in measuring alignment, we considered only those 10 factors that CBNMscan
ﬁt well (in the Appendix).
We also dropped all those examples for which hair color is not unique(e.g.
, annotated as both blonde and black ), obtaining approx.
127 k examples.
CelebA is morechallenging than dSprites and MPI3D, as it does not include all possible factor variations and thegenerative factors – although disentangled – are insufﬁcient to completely determine the contents of D S P R I TE S
M P I D C
ELE B
A the images.
For dSprites and MPI3D, we used a random 80/10/10 train/validation/test split, while for
CelebA we kept the original split [28].
We generated the ground-truth labels y as follows.
For dSprites, we labeled images according toa random but ﬁxed linear separator deﬁned over the continuous generative factors, chosen so as toensure that the classes are balanced.
For MPI3D and CelebA, we focused on the categorical factorsinstead.
Speciﬁcally, we clustered all images using the algorithm of [54], for a total of 10 and 4clusters for MPI3D and CelebA respectively, and then labeled all examples based on their referencecluster.
This led to slightly unbalanced classes containing different percentages of examples, rangingfrom 5% to 16% in MPI3D and from 21% to 29% in CelebA.
Architectures.
For dSprites and MPI3D, we implemented the encoder as a six layer convolutionalneural net, while for CelebA we adapted the convolutional architecture of Ghosh et al.
Weemployed a six layer convolutional architecture for the decoder in all cases, for simplicity, as changingit did not lead to substantial differences in performance.
In all cases, as for all CBMs (see Section 2),the classiﬁer was implemented as a dense layer followed by a softmax activation.
The very samearchitectures were used for both GlanceNets and CBNMs, for fairness.
For each data set, we chosethe latent space dimension as the total number of generative factors, where categorical ones are onehot encoded.
In particular, we used 7 latent factors for dSprites, 21 for MPI3D and 10 for CelebA.
Further details are included in the Supplementary Material.
Results and discussion.
The results of this ﬁrst experiment are reported in Fig.
The behavior ofboth competitors on dSprites and MPI3D was extremely stable, owing to the fact that these data setscover an essentially exhaustive set of variations for all generative factors, so we report their hold-outperformance on the test set.
Since for CelebA variance was non-negligible, we ran both methods 7times varying the random seed used to initialize the network and report the average performanceacross runs and its standard deviation.
In addition to alignment, we also report explicitness [29], which measures how well the linearregressor employed by DCI ﬁts the generative factors.
The higher, the better.
Details on its evaluationare included in Suppl.
Material.
The plots clearly show that, although the two methods achieve high and comparable accuracy inall settings, GlanceNets attain better alignment in all data sets and for all supervision regimes than
CBNMs, with a single exception in CelebA using low values of supervision, for a total of 13 winsout of 15 cases.
In all disentanglement data sets, there is a clear margin between the alignment achieved by GlanceNets and that of CBNMs: performances vary up to maximum of 15% in dSprites,and a minumum of 8% in MPI3D. In CelebA, the gap is evident with full supervision (almost 8% ofdifference in alignment), and GlanceNets still attain overall better scores in the 25% and 50% regime.
On the other hand, performance are lower, but comparable, with 10% supervision.
The case at 1%refers to an extreme situation where both CBNMs and GlanceNets struggle to align with generativefactors, as is clear also from the very low explicitness.
In dSprites and MPI3D, both GlanceNets and CBNMs quickly achieve very high alignment at 1%supervision, as expected [32], whereas better results in CelebA are obtained with growing supervision.
Also, both models display similar stability on this data set, as shown by the error bars in the plot.
5.2 GlanceNets are leak-proof
Next, we evaluated robustness to concept leakage in two scenarios that differ in whether the unobserved generative factors are disentangled with the observed ones or not, see Section 3.
In bothexperiments, we compare GlanceNets with a CBNM and a modiﬁed GlanceNet where the OSRcomponent has been removed (denoted CG-VAE).
Leakage due to unobserved entangled factors.
We start by replicating the experiment of Mahinpeiet al.
[7]: the goal is to discriminate between even and odd MNIST images using a latent representation
Z =
( Z 4 , Z 5 ) obtained by training (with complete supervision on the generative factors) only onexamples of 4 ’s and 5 ’s. Leakage occurs if the learned representation can be used to solve theprediction task better than random on a test set where all digits except 4 and 5 occur.
During training,we use the digit label for conditioning the prior p ( Z | Y ) of the GlanceNet. More qualitative resultsare collected in Appendix D.
[Figure]
Fig.
4 (a, b) illustrates the latent representations of the training and test set output by a GlanceNet:since the two digits are mutually exclusive, the model has learned to map all instances along the( z 4 , z 5 ) diagonal.
This is where OSR kicks in: if an input is identiﬁed as open-set, T is predictedas 0 by the OSR component and the input is rejected.
In all leakage experiments, we implementrejection by predicting a random label.
Since MNIST is balanced, we measure leakage by computingthe difference in accuracy between the classiﬁer and an ideal random predictor, i.e., 2 · | acc − 12 | : thesmaller, the better.
The results, shown in Fig.
4 (c), show a substantial difference between GlanceNetand the other approaches.
Consistently with the values reported in [7], CBNMs are affected by aconsiderable amount of leakage, around 28% .
This is not the case for our GlanceNet: most (approx.85% ) test images are correctly identiﬁed as open-set and rejected, leading to a very low (about 2% )leakage, 26% less than CBNMs.
The results for CG-VAE also indicate that removing the open-setcomponent from GlanceNets dramatically increases leakage back to around 30% .
This shows thatalignment and disentanglement alone are not sufﬁcient, and that the open-set component plays acritical role for preventing leakage.
Leakage due to unobserved disentangled factors.
Next, we analyze concept leakage between disentangled generative factors using the dSprites data set.
To this end, we deﬁned a binary classiﬁcation task in which the ground-truth label depends on position_x and position_y only.
In particular,instances within a ﬁxed distance from (0 , 0) are annotated as positive and the rest as negative, asshown in Fig.
5 (a).
In order to trigger leakage, all competitors are trained (using full concept-levelsupervision, as before) on training images where shape , size and rotation vary, but position_xand position_y are almost constant (they range in a small interval around (0 . 5 , 0 . 5) , cf.
Fig.
5).leakage occurs if the learned model can successfully classify test instances where position_x andposition_y are no longer ﬁxed. More qualitative results can be found in Appendix E.
[Figure]
For both competitors, we encode shape using a 3D one-hot encoding and size and rotationas continuous variables.
During training, we use the shape annotation for conditioning the priorp ( Z | Y ) of the GlanceNet.
The ﬁrst two PCA components of the latent representations acquired byour GlanceNet and by a CBNM are shown, rotated so as to be separable on the ﬁrst axis, in Fig.
5 (b, c):in
both cases, it is possible to separate positives from negatives based on the obtained representationsin the ﬁve latent dimensions.
As shown in Fig.
5 (d), this means that both CBNM and CG-VAE sufferfrom very large leakage, 80% and 98% , respectively.
In contrast, OSR allows us to correctly identifyand reject almost all test instances, leading to negligible leakage even in this disentangled setting.
Related Work
Concept-based explainability.
Concepts lie at the heart of AI [56] and have recently resurfaced asa natural medium for communicating with human stakeholders [11].
In explainable AI, this wasﬁrst exploited by approaches like TCAV [57], which extract local concept-based explanations fromblack-box models using concept-level supervision to deﬁne the target concepts.
Post-hoc explanations,however, are notoriously unfaithful to the model’s reasoning [58–60].
CBMs, including GlanceNets,avoid this issue by leveraging concept-like representations directly for computing their predictions.
Existing CBMs model concepts using prototypes [2, 3, 16, 17] or other representations [1, 20, 4, 5],but they seek interpretability using heuristics, and the quality of concepts they acquire has been calledinto question [61, 19, 7, 8].
We show that disentangled representation learning helps in this regard.
Disentanglement and interpretability.
Interpretability is one of the main driving factors behindthe development of disentangled representation learning [62–64].
These approaches however makeno distinction between interpretable and non-interpretable generative factors and generally focuson properties of the world , like independence between causal mechanisms [9] or invariances [43].
Interpretability, however, depends on human factors that are not well understood and thereforeusually ignored [12, 65].
The link between disentanglement and interpretability has never been madeexplicit.
Importantly, in contrast to alignment, disentanglement does not require that the map betweenmatching generative and learned factors preserves semantics.
We remark that other VAE-basedclassiﬁers either do not tackle disentanglement or are unconcerned with concept leakage [66, 67, 36].
Disentanglement and CBMs.
Neither the literature on disentanglement nor the one on CBMs haveattempted to formalize the notion of interpretability or to establish a proper link between the latterand disentanglement.
The work of Kazhdan et al.
[68] is the only one to compare techniques fordisentangled representation learning and concept acquisition, however it makes no attempt at linkingthe two notions.
Our work ﬁlls this gap.
A Implementation details
A.1 GlanceNet and CBNM Architectures
In all experiments, we used exactly the same architecture and number of latent variables for both
GlanceNets and CBNMs to ensure a fair comparison.
Encoder architectures: • dSprites : We chose a rather standard architecture [52].
It comprises six 2D convolutional layers ofdepth 32 , 32 , 64 , 128 , 256 , and 256 , respectively, all with a kernel of size 4 , stride 2 , and padding1 , and followed by ReLU activations. The output is ﬂattened to a vector and passed through adense layer to obtain the mean µ ( x ) and (diagonal) variance σ ( x ) of the encoder distribution
N ( Z | µ ( x ) , diag( σ ( x ))) .
• MPI3D : We used the same architecture with slightly different convolutional depths of 32 , 32 , 64 ,64 , 128 , and 256 , changing also the kernel size to 3 and removing padding, as per [52]. • CelebA : We leveraged the architecture of Ghosh et al.
[55], which is a common reference for VAEmodels on CelebA-64 [69]. The encoder is composed of four convolutions of depth 128 , 256 , 512 ,1024 respectively, all with kernel size of 5 , stride of 2 , followed batch normalization and ReLUactivation. The models had exactly as many latent variables as generative factors for which supervision isavailable, which in our three data sets are 7 , 21 , and 10 , respectively.
Decoder architecture: All models share the same decoder architecture, obtained by stacking:
• A 2D convolution on the latent space with a ﬁlter depth of 256, kernel size of 1, and stride of 2,followed by the ReLU activation; • Five transposed 2D convolutions of depth 256 , 256 , 128 , 128 , 64 , 64 , and num_channels ,respectively, all with kernel of size 4 and stride 2.Here, num_channels is either 1 (dSprites) or 3 (MPI3D and CelebA).
The shape of the last layerwas chosen so as to match the dimension of the input image.
Additional details can be found in thevarious Tables in this appendix. A.2 Supervision and Training
Concept-level supervision.
Depending on the supervision provided, only a fraction of the inputswas made available during training with their generative factors.
In dSprites and MPI3D all generativefactors are matched by the models, whereas in the case of CelebA we restricted learning to those10 attributes that are best ﬁt by the CBNMs, namely: bald , black hair , brown hair , blondehair , eyeglasses , gray hair , male , no beard , smiling , and wearing hat .
Both CBNMs and
GlanceNets are jointly trained, meaning that optimization steps for the concepts and label supervisionare taken simultaneously.
Whenever concept supervision is lower than 100 %, for those exampleswithout concept annotations we trained both models using label supervision only. We did not evaluateother training strategies available for CBNMs (e.g., sequential training [20]) as these appear to bringno beneﬁt in terms of either performance nor leakage. Optimization setup. In all experiments, we used the Adam optimizer [70] with default parametersβ 1 = 0 .
9 and β 2 = 0 . For dSprites, we used a batch size of 64 and ﬁxed learning rate toη = 4 · 10 − 4 , while for MPI3D and CelebA we used a batch size of 100 and annealed the learning ratefrom 10 − 7 to η MPI = 10 − 3 and η CelebA = 10 − 4 , respectively. To prevent overﬁtting, in CelebAwe multiplied the learning rate by a factor of 0 . 95 in each epoch and apply early stopping on thevalidation set, with a patience of 10 epochs.
Prior to training, we selected a reasonable value for the following hyper-parameters:
• β : the weight of the KL divergence in Eq. (4).• γ : the weight of the loss on the generative factors in Eq. • λ : the weight of the cross-entropy loss over the label, which is left implicit in Eq.
[Table]
[Table] [Table]
[Table]
For dSprites, we found a good balance for λ = γ = 100 , while for MPI3D we achieved goodperformance with λ = 10 3 and γ = 7 · 10 3 . We adopted the same hyperparameters choice for CelebA, with the exception that we reduced the reconstruction error by 0 .
For all data sets,we cross-validated over different values of β
but we obtained better alignment performances withβ ≈ 1 . This happens because we inject supervision on the latent factors (which is absent in regularβ -VAEs [43]).
A.3 Implementation of leakage tests
MNIST. For this dataset, we considered only Multi-Latyer Perceptrons instead of convolutions. Boththe encoder and the decoder are composed by two linear layers with depth 128, and a dense layerconnected to the latent space and to the input space, respectively. Further details are on Table 5.
For the GlanceNet we considered a latent space of dimension 10 where the supervision on the 4 and 5digits is used to ﬁt the { z 4 , z 5 } latent factors.
These two, constitute the latent subspace where leakageoccurs, while the other are useful only for reconstruction.
Conversely, for the CBNM we consideredonly two latent factors.
During training of the latent encodings, we used stochastic gradient descent with learning rateη = 0 . 001 , reducing it by 0.95 in each epoch for both CBNMs and GlanceNets. The training wasperformed only on the 4 and 5 digits (in the usual training set partition for MNIST), for almost 50epochs. Afterwards, we considered the open-set representations, restricted to { z 4 , z 5 } , as inputs fortraining a logistic regression for parity recognition.
During the training, only the digits in the MNISTtraining set partition (exception made for 4 and 5) are considered, while performance are calculatedon the test set.dSprites. We adopted the same architecture in the upper section, except that we reduced the latentspace to 5 dimensions. As a reminder, during training all sprites are almost ﬁxed at the center, thereforeadditional factors of variations for its position are needless. The training was performed over 300epochs for both GlanceNets and CBNMs, with η = 4 · 10 − 4 .
After training, the representations ofthe open-set sprites (in which position is no longer ﬁxed) are used to ﬁt a logistic regression.
In thiscase, the labels depend on whether the sprite is located at bottom-left corner or at the upper-right one,for more information refer to Fig.
The classiﬁcation performance was evaluated on a held-out testset for both models, under an 80/20 train/test split. [Table] E NCODER (28 , 28) Flatten (784)
Linear dim= 128 , bias=
True
ReLU (128)
Linear dim=128, bias=
True ReLU (128) Linear dim=10+10, bias= True
D ECODER ( dim ( z ))
Linear dim= 128 , bias=
True
ReLU (128)
Linear dim= 128 , bias=
True
ReLU (128)
Linear dim= 728 , bias=
True (728)
Unsqueeze
In our case study, we are interested into DCI maps that linearly connect the z (cid:48) s to the g (cid:48) s . In orderto evaluate alignment performances, the inverse map α − 1 : R k
→ R n I is constructed from the latentspace to the span of the n I generative factors.
The latent representations and generative factors werenormalized in the [0 , 1] interval prior to learning. B.1 Alignment and explicitness The importance weights of this map are the absolute-values of the weights in the linear matrix ofα − 1 , indicated as B ∈ R k × n I in the main text, see Section 3. Then, the importance weights are used to evaluate the dispersion of the learned weights.
To this end, we measure each Shannon entropy H jon all k latent factors: [Equation] where ¯ b ji = b ji (cid:14) n I [Equation]
Then, the average alignment is calculated as:
[Equation] where ρ [Equation] [Equation]
and ranges in [0 , 1] . Similarly, the quantity: [Equation] and
[Equation] is the completeness of the latent representation, a measure akin to alignment (Eq. that quantities the degree to which each generative factor correlates with distinct latent factors. Alignment and completeness relate to different properties of the map: the higher the alignment , the more each Z j depends on variations of only a single G i .
On the other hand, learning multiple Z j ’s capturing a single G i reduces the completeness . As an illustrative example, consider the matrix: [Equation] [Equation]
[Equation]


From the above deﬁnitions, one gets alignment = 1 and completeness < 1 .
This follows sinceeach Shannon entropy for the alignment score is zero (as it is related to the rows),
whereas the
Shannon entropy for the completeness is greater than zero (it refers to the columns).
Moreover, eachlatent variable z i depends only on the variations of a single generative factor g j .
We also calculate the explicitness of the map α , which is related to the mean squared error (MSE) of the prediction.
Since the MSE for random guessing for a variable in the [0 , 1] interval is equal to 1/6, the explicitness becomes:
[Equation]
B.2 Empirical evaluation
For dSprites and MPI3D, all DCI quantities were calculated with the built-in evaluation code providedby disentanglement_lib , [31].
For CelebA, since the 40 attributes in CelebA are not exhaustivefor the image generation, we implemented computed DCI as follows: (i) we ﬁrst converted the Jattributes z J and g J connected to hair type to a single concept h and ﬁt the model with Lassoregression to predict g h from z .
Then, (ii) we trained a Logistic Regression with l 1 penalty to predictthe remaining g (cid:48) s .
Finally, we took both weights in (i) and in (ii) to compute the matrix B ∈ R 6 × 6 .
In this way, we determined alignment and explicitness for CelebA. We chose the lasso coefﬁcientλ = 0 .
01 for both regressions.
C Open-Set Recognition Mechanism
In this section, we provide additional details on the OSR mechanism introduced in Section 3.
Ourmethod adapts the one of Sun et al.
[36], which distinguishes between closed-set and open-set datapoints by combining a reconstruction check Γ r with a localization check Γ ls .
The overall OSR checkis given by: [Equation] After completing the training process, all the training instances are passed to the model to evaluatethe thresholds: • The reconstruction threshold η r is the maximum real number such that a ﬁxed percentage oftraining examples have reconstruction error less or equal to it.
At test time, given an instancex , let ˆ η = (cid:107) x − ˆ x (cid:107) 2 be the reconstruction error.
Then, Γ r = 1 (i.e., the check passes) if theempirical reconstruction error is less than the threshold, ˆ η < η r , otherwise Γ r = 0 . • The latent-space distance thresholds are evaluated for each class-prototype embedded inthe latent space µ y = E p ( z | y )
[ z ] .
For each of them, we ﬁrst evaluated the relative distancebetween point belonging to the class y and the prototype µ y .
Then, we evaluated a thresholdη y on the distances, as to include a ﬁxed percentage of training instances into the setB y =
{ z : (cid:107) µ y − z
(cid:107) <
η y } .
At test time, those points that do not belong to any set B y arepredicted as open-set instances,
i e. Γ ls = 0 , otherwise Γ ls = 1 .
In our experiments, the threshold are obtained by ﬁxing both reconstruction and latent space distance to keep the 95% of training data.
In the
case of η y , this quota has been reached singularly for each B y
, thus obtaining different values η y ’s from one another. Finally, combining both rejection methodswe are sure
the model
would predict as closed-set at least the 90% of training instances. D Concept Leakage in MNIST We report here additional details for the concept leakage test on MNIST, which has been originallyintroduced by Margeloiu et al.
The experiment has two stages:1. At train time, the model is trained to align its representations to the concepts of 4 and 5 ,by passing full supervision on them. Both CBNMs and GlanceNets are allotted two latentconcepts, which we denote ( Z 4 , Z 5 ) . There is no downstream classiﬁcation task in thisstage.2.
At test time, all MNIST images, excluding those of 4 ’s and 5 ’s, are encoded using thelearned encoder and used to learn a classiﬁer of even vs. odd digits.
The performance of theresulting classiﬁer, applied to non- { 4 , 5 } images, is then computed.
In this experiment, concept leakage occurs if the accuracy on the downstream task is above the 50% .
CBNM VAE [Figure] ET
D.1 Qualitative results
[Figure]
In Fig.
6, we show the latent space representations for different models on the MNIST leakagetest, for both closed-set and open-set data points.
To illustrate the contribution of our mixture prior, in addition to the CBNM and GlanceNet models, we also considered a simpler supervised VAEmodel. This model has the same encoder, decoder, and classiﬁer as the GlanceNet, but uses a regular Gaussian prior 4 . We found that this model achieved a similar level of leakage to CG-VAE.
We displayin Fig.
7 the reconstruction of a few random examples output by GlanceNet: the reconstructions ofall instances belonging to the open-set greatly deviate from the original. E Concept Leakage on dSprites In this section, we report additional material for the dSprites concept leakage experiment. Thisexperiment resembles the previous one on MNIST:1.
At training time, a CBM learns the representations of shape , size and rotation byreceiving supervision on all possible variations of these factors.
On the other hand, novariation of factors pos
x and pos _y are observed, in the sense that
the position of the training sprites is ﬁxed to the center of the image. We ﬁt the CBNM and the GlanceNet
with
latent factors to learn the representations. Again, no downstream classiﬁcation task appears at this stage. At test time, the encoder is kept ﬁxed for different variations varying of the factors pos _x
and pos _y are observed.
The downstream task in this phase amounts to recognizing whether a sprite lies in top-right or in the bottom-left corner of the image. In this experiment, concept leakage occurs if the accuracy on the downstream task is above the 50% . E.1 Qualitative results
We also include qualitative results for GlanceNet and on dSprites for closed set and open set datapoints.
In Fig.
8 we display the projections of train and test points on the two different latent subspaces
(see caption).
In both of them, positives and negatives representations are well separated from eachother, implying substantia leakage.
We also evaluated the reconstruction quality during training andtesting and reported some of them in Fig.
Notably, almost all points are recognized to be open setinstances thanks to the reconstruction threshold.
C L O
S E D
S ET
O P E
N S
ET
F Additional results for GlanceNets and CBNMs in CelebA
In this section, we discuss additional results for CBNMs vs. GlanceNets on the CelebA dataset.
Weﬁrst report the accuracy of the learned concepts on the supervised latent factors for both CBNMsand GlanceNets in CelebA. Then, we examine two variants of GlanceNets varying the dimension ofthe unsupervised factors in the latent space: a β -VAE with 20 latent factors and a β -TCVAE with40 latent factors, [39].
This variant includes an additional loss term to Eq.
(5) given by the Total
Correlation (TC) of the model posterior q φ ( z ) =
E p D ( x ) [ q φ ( z | x )] :
[Equation]
(10) where β denotes the strength hyper-parameter.
Both the β -VAE and the β -TCVAE receive supervisiononly on the 10 generative factors that are ﬁtted in the CBNM.
A the end of the section, we reporttraversals for the models with 40 latent factors.
F.1 Concepts Accuracy
We report the concepts accuracy for both CBNMs and GlanceNets in Fig.
10, with 10 latent dimensions and the TCVAE variant.
The difference in concept accuracy between GlanceNet (both variants)and CBNMs is negligible, with GlanceNets showing slightly higher variance when the percentage ofconcepts supervision is very small.
This highlights how, in terms of concept accuracy, the two classesof models are essentially indistinguishable, even though they are in terms of alignment.
F.2 Performances upon variations of the latent space dimension
Similarly to the analysis in Section 5.1, we show the behavior of the metrics upon increasing thedimension of the latent space.
The ﬁrst variant of GlanceNets, based on β -VAE, was ﬁtted withβ ≈ 1 , with a latent space of dimension 20.
The second variant is a TCVAE, trained with a weight ofthe total correlation β = 10 for all concepts supervisions, exception made for the 100 % run, wherewe found better results with β = 0 .
We measured alignment and explicitness for both variants of
GlanceNets by restricting on only those 10 latent factors where supervision were provided.
This isin line with the notion of alignment, since we are interested in measuring the interpretability of themodel, not the disentanglment among different components.
[Figure]
In Fig.
11 we report the results obtained, including the original variant with 10 latent dimensions.
Forthe β -VAE (20) and TCVAE (40) we can see the improvement provided by extending the latent space.
The latter achieves particular high values of alignment.
[Figure]
F.3 Latent traversals
[Figure]
We ﬁnally report in the traversals for some of the supervised attributes, obtained by the GlanceNet
TCVAE with full supervision on the concepts.
We excluded the traversals of the attributes HAT and
BALD since the generator failed to reproduce them faithfully.
The others are well captured by themodel, as we reported in Fig.