Original: Addressing Leakage in Concept Bottleneck Models
Revised: Addressing Leakage in Concept Bottleneck Models

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Concept bottleneck models (CBMs) enhance the interpretability of their predictionsby
Revised: Concept bottleneck models (CBMs) enhance the interpretability of their predictionsby

Original: ﬁrst predicting high-level concepts given features, and subsequently predictingoutcomes on the basis of these concepts.
Revised: ﬁrst predicting high-level concepts given features, and subsequently predictingoutcomes on the basis of these concepts.

Original: Recently, it was demonstrated that trainingthe label predictor directly on the probabilities produced by the concept predictoras opposed to the ground-truth concepts, improves label predictions.
Revised: Recently, it was demonstrated that trainingthe label predictor directly on the probabilities produced by the concept predictoras opposed to the ground-truth concepts, improves label predictions.

Original: However, thisresults in corruptions in the concept predictions that impact the concept accuracy aswell as our ability to intervene on the concepts – a key proposed beneﬁt of CBMs.
Revised: However, thisresults in corruptions in the concept predictions that impact the concept accuracy aswell as our ability to intervene on the concepts – a key proposed beneﬁt of CBMs.

Original: In this work, we investigate and address two issues with CBMs that cause thisdisparity in performance: having an insufﬁcient concept set and using inexpressiveconcept predictor.
Revised: In this work, we investigate and address two issues with CBMs that cause thisdisparity in performance: having an insufﬁcient concept set and using inexpressiveconcept predictor.

Original: With our modiﬁcations, CBMs become competitive in terms ofpredictive performance, with models that otherwise leak unintended information inthe concept probabilities, while having dramatically increased concept accuracyand intervention accuracy.
Revised: With our modiﬁcations, CBMs become competitive in terms ofpredictive performance, with models that otherwise leak unintended information inthe concept probabilities, while having dramatically increased concept accuracyand intervention accuracy.

Original:  Introduction
Revised:  Introduction

Original: Building interpretable machine learning models is important in several high-stake domains wherehumans must typically interact with the models to make meaningful decisions.
Revised: Building interpretable machine learning models is important in several high-stake domains wherehumans must typically interact with the models to make meaningful decisions.

Original: Among such models,
Revised: Among such models,

Original: Concept bottleneck models (CBMs) (Chen et al., 2020; Koh et al., 2020; Yeh et al., 2019; Lage& Doshi-Velez, 2020; Wang et al., 2017) propose explicitly aligning the intermediate layers of aneural network with some pre-deﬁned expert concepts by ﬁrst training a predictor of concepts, andsubsequently using this to predict the label (Figure 1 left).
Revised: Concept bottleneck models (CBMs) (Chen et al., 2020; Koh et al., 2020; Yeh et al., 2019; Lage& Doshi-Velez, 2020; Wang et al., 2017) propose explicitly aligning the intermediate layers of aneural network with some pre-deﬁned expert concepts by ﬁrst training a predictor of concepts, andsubsequently using this to predict the label (Figure 1 left).

Original: CBMs are examples of interpretablemodels that are well-suited for human-AI joint decision making.
Revised: CBMs are examples of interpretablemodels that are well-suited for human-AI joint decision making.

Original: A human supervisor can interpretand understand the label predictions by inspecting the concept predictions.
Revised: A human supervisor can interpretand understand the label predictions by inspecting the concept predictions.

Original: Moreover, when thesupervisor notices a misprediction among the concepts, they may intervene by correcting the conceptprediction and, in turn, the model is able to update its ﬁnal prediction accounting for the revisedconcept.
Revised: Moreover, when thesupervisor notices a misprediction among the concepts, they may intervene by correcting the conceptprediction and, in turn, the model is able to update its ﬁnal prediction accounting for the revisedconcept.

Original: Typically when training a CBM, the concept predictor and the label predictor are trained independentlyon ground-truth concepts.
Revised: Typically when training a CBM, the concept predictor and the label predictor are trained independentlyon ground-truth concepts.

Original: Unfortunately, these models can underperform in predictive performancecompared to a black-box predictor.
Revised: Unfortunately, these models can underperform in predictive performancecompared to a black-box predictor.

Original: Instead, the most predictive CBMs use soft concepts, meaningthat the inputs to the label predictor are the concept probabilities (the numerical values between 0 and1) as predicted by the concept predictor (Koh et al., 2020).
Revised: Instead, the most predictive CBMs use soft concepts, meaningthat the inputs to the label predictor are the concept probabilities (the numerical values between 0 and1) as predicted by the concept predictor (Koh et al., 2020).

Original: Soft concepts rely on this information-richrepresentation, which is in contrast with hard concepts, where label predictor only accepts binaryconcepts.
Revised: Soft concepts rely on this information-richrepresentation, which is in contrast with hard concepts, where label predictor only accepts binaryconcepts.

Original: While soft concepts may improve predictive performance, this improvement comes at a cost.
Revised: While soft concepts may improve predictive performance, this improvement comes at a cost.

Original: Theissue is that soft concepts allow the concept predictor to convey unintended information about thelabel that would otherwise not be available to the label predictor (Mahinpei et al., 2021; Margeloiuet al., 2021).
Revised: Theissue is that soft concepts allow the concept predictor to convey unintended information about thelabel that would otherwise not be available to the label predictor (Mahinpei et al., 2021; Margeloiuet al., 2021).

Original: For example, in an animal classiﬁer, when the concept predictor recognizes an image asa cat, it may predict a slightly higher likelihood to the concept ‘fur’ than the concept ‘tail’, and whenit recognizes the image as a dog, it may assign higher likelihood to the concept ‘tail’ than the concept
Revised: For example, in an animal classiﬁer, when the concept predictor recognizes an image asa cat, it may predict a slightly higher likelihood to the concept ‘fur’ than the concept ‘tail’, and whenit recognizes the image as a dog, it may assign higher likelihood to the concept ‘tail’ than the concept

Original: [Figure]
Revised: [Figure]

Original: Carnivore:Two-legged: ..
Revised: Carnivore:Two-legged: ..

Original: Claws: 0.90.
Revised: Claws: 0.90.

Original: Concepts c 1 ..
Revised: Concepts c 1 ..

Original: c K
Revised: c K

Original: Cat: 0.1 T-Rex: 0.
Revised: Cat: 0.1 T-Rex: 0.

Original: Label y 𝜃 𝜙
Revised: Label y 𝜃 𝜙

Original: Carnivore:Two-legged: ..
Revised: Carnivore:Two-legged: ..

Original: Claws: 0.90.
Revised: Claws: 0.90.

Original: Cat: 1.0 T-
Revised: Cat: 1.0 T-

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ‘fur’.
Revised: ‘fur’.

Original: Even though cats and dogs are indistinguishable based on these two concepts alone, the labelpredictor may still compare the two soft concept probabilities and make an accurate classiﬁcation.
Revised: Even though cats and dogs are indistinguishable based on these two concepts alone, the labelpredictor may still compare the two soft concept probabilities and make an accurate classiﬁcation.

Original: A ﬂexible predictor, such as a deep neural network, can quickly learn such encoding patterns andimprove its prediction using the information leaked by the concept predictor.
Revised: A ﬂexible predictor, such as a deep neural network, can quickly learn such encoding patterns andimprove its prediction using the information leaked by the concept predictor.

Original: Thus, the problem isdubbed leakage.
Revised: Thus, the problem isdubbed leakage.

Original: Leakage compromises the interpretability and trustworthiness of the model.
Revised: Leakage compromises the interpretability and trustworthiness of the model.

Original: The concept predictorno longer needs to faithfully predict the concepts to obtain high-quality label predictions.
Revised: The concept predictorno longer needs to faithfully predict the concepts to obtain high-quality label predictions.

Original: Instead, itonly needs to encode the class label in the soft concept probabilities.
Revised: Instead, itonly needs to encode the class label in the soft concept probabilities.

Original: A human supervisor can neverbe sure if the model is predicting a certain concept because it is likely to be present, or because it isencoding for something else.
Revised: A human supervisor can neverbe sure if the model is predicting a certain concept because it is likely to be present, or because it isencoding for something else.

Original: Moreover, since interventions remove this extra information from theconcept predictions, predictive performance conditional on interventions suffers signiﬁcantly.
Revised: Moreover, since interventions remove this extra information from theconcept predictions, predictive performance conditional on interventions suffers signiﬁcantly.

Original: How can one avoid leakage?
Revised: How can one avoid leakage?

Original: While hard concepts ensure that information is not leaked, these modelssuffer from poor performance in comparison to their soft counterparts.
Revised: While hard concepts ensure that information is not leaked, these modelssuffer from poor performance in comparison to their soft counterparts.

Original: In this work, we focus on theperformance disparity between soft and hard CBMs.
Revised: In this work, we focus on theperformance disparity between soft and hard CBMs.

Original: Speciﬁcally, we attribute this disparity to (i)having an insufﬁcient concept set and (ii) having an inﬂexible concept predictor.
Revised: Speciﬁcally, we attribute this disparity to (i)having an insufﬁcient concept set and (ii) having an inﬂexible concept predictor.

Original: The former issueoccurs as a result of the Markovian assumption under which CBMs operate: all the information aboutthe label in the input must be captured by the concepts.
Revised: The former issueoccurs as a result of the Markovian assumption under which CBMs operate: all the information aboutthe label in the input must be captured by the concepts.

Original: If this assumption is not met, the performanceof the hard CBM is limited by the information available in the concepts.
Revised: If this assumption is not met, the performanceof the hard CBM is limited by the information available in the concepts.

Original: The latter issue stems fromthe fact that CBMs predict concepts independently and cannot capture correlations between concepts.
Revised: The latter issue stems fromthe fact that CBMs predict concepts independently and cannot capture correlations between concepts.

Original: These correlations are important since some combinations of concepts may be unlikely or impossible(eg.
Revised: These correlations are important since some combinations of concepts may be unlikely or impossible(eg.

Original: the concepts ‘fur’ and ‘scales’ are mutually exclusive).
Revised: the concepts ‘fur’ and ‘scales’ are mutually exclusive).

Original: These two issues hinder the performanceof hard CBMs, while soft CBMs are able to mitigate them with leakage, at the cost of interpretabilityand intervenability.
Revised: These two issues hinder the performanceof hard CBMs, while soft CBMs are able to mitigate them with leakage, at the cost of interpretabilityand intervenability.

Original: We introduce two technical contributions that allow hard CBMs to match the predictive performanceof soft CBMs without compromising their resilience to leakage.
Revised: We introduce two technical contributions that allow hard CBMs to match the predictive performanceof soft CBMs without compromising their resilience to leakage.

Original: First, we circumvent the Markovianassumption by introducing a side-channel to learn a set of latent concepts alongside the known concepts that are needed for accurate predictions.
Revised: First, we circumvent the Markovianassumption by introducing a side-channel to learn a set of latent concepts alongside the known concepts that are needed for accurate predictions.

Original: Second, we introduce an autoregressive conﬁgurationfor the concept predictor that signiﬁcantly improves their ﬂexibility and it allows them to capturecorrelations in the concepts.
Revised: Second, we introduce an autoregressive conﬁgurationfor the concept predictor that signiﬁcantly improves their ﬂexibility and it allows them to capturecorrelations in the concepts.

Original: On several synthetic and real applications, we demonstrate that our side-channel models improvethe performance of hard CBMs in terms of accuracy, without affecting concept predictions orintervenability.
Revised: On several synthetic and real applications, we demonstrate that our side-channel models improvethe performance of hard CBMs in terms of accuracy, without affecting concept predictions orintervenability.

Original: In addition, our autoregressive concept predictor signiﬁcantly improves both conceptaccuracy and intervention accuracy compared to soft CBMs and non-autoregressive hard CBMs.
Revised: In addition, our autoregressive concept predictor signiﬁcantly improves both conceptaccuracy and intervention accuracy compared to soft CBMs and non-autoregressive hard CBMs.

Original: Summary of contributions:
Revised: Summary of contributions:

Original: • In our experiments, we demonstrate that leakage is detrimental to interpretability andintervenability in soft CBMs.
Revised: • In our experiments, we demonstrate that leakage is detrimental to interpretability andintervenability in soft CBMs.

Original: • Hard CBMs are resilient to leakage, but their performance is hindered by not meetingthe Markovian assumption and having an inﬂexible concept predictor.
Revised: • Hard CBMs are resilient to leakage, but their performance is hindered by not meetingthe Markovian assumption and having an inﬂexible concept predictor.

Original: We address theseshortcomings:– We circumvent the Markovian assumption by introducing a side-channel to learn a setof latent concepts.
Revised: We address theseshortcomings:– We circumvent the Markovian assumption by introducing a side-channel to learn a setof latent concepts.

Original: – We increase the ﬂexibility of the concept predictor by adapting an autoregressivearchitecture.
Revised: – We increase the ﬂexibility of the concept predictor by adapting an autoregressivearchitecture.

Original:  Background: Concept Bottleneck Models
Revised:  Background: Concept Bottleneck Models

Original: Consider a dataset D
Revised: Consider a dataset D

Original: = { x ( i ) , c
Revised: = { x ( i ) , c

Original: ( i ) , y ( i ) }
Revised: ( i ) , y ( i ) }

Original: Ni =1 consisting of N triplets of the form: input x ∈ R d ,concept vector c ∈ { 0 , 1 } K of K binary concepts, and class label y ∈ N
Revised: Ni =1 consisting of N triplets of the form: input x ∈ R d ,concept vector c ∈ { 0 , 1 } K of K binary concepts, and class label y ∈ N

Original: (we consider classiﬁcation 84 problems, but our results extend to regression problems where y ∈ R ).
Revised: (we consider classiﬁcation 84 problems, but our results extend to regression problems where y ∈ R ).

Original: Formally, a CBM is comprisedof a concept predictor, parametrized by θ , which is trained to predict the concept vector from theinput p θ ( c | x ) , and a label predictor, parametrized by φ , subsequently trained to predict the label fromthe concepts p φ ( y | c ) (Koh et al., 2020)
Revised: Formally, a CBM is comprisedof a concept predictor, parametrized by θ , which is trained to predict the concept vector from theinput p θ ( c | x ) , and a label predictor, parametrized by φ , subsequently trained to predict the label fromthe concepts p φ ( y | c ) (Koh et al., 2020)

Original: (Figure 1 left).
Revised: (Figure 1 left).

Original: Interpretability and Interventions There are two key beneﬁts to CBMs that make them wellsuited for human-AI joint decision making.
Revised: Interpretability and Interventions There are two key beneﬁts to CBMs that make them wellsuited for human-AI joint decision making.

Original: First, they explain their predictions by showing whichhigh-level concepts the concept predictor was able to ﬁnd in the data.
Revised: First, they explain their predictions by showing whichhigh-level concepts the concept predictor was able to ﬁnd in the data.

Original: Second, a human operator mayintervene on these concepts to see how changing the concept values affects the ﬁnal prediction.
Revised: Second, a human operator mayintervene on these concepts to see how changing the concept values affects the ﬁnal prediction.

Original: Thisis done by setting the probabilities of the concepts to 1 or 0 to indicate their presence or absence.
Revised: Thisis done by setting the probabilities of the concepts to 1 or 0 to indicate their presence or absence.

Original: Thelabel predictor is able to adjust its prediction to account for the intervention (Figure 1 left).
Revised: Thelabel predictor is able to adjust its prediction to account for the intervention (Figure 1 left).

Original: Hard CBMs In hard CBMs, the label predictor only accepts binary concepts as inputs.
Revised: Hard CBMs In hard CBMs, the label predictor only accepts binary concepts as inputs.

Original: Theconcept predictor and the label predictor are trained independently on the ground truth data:
Revised: Theconcept predictor and the label predictor are trained independently on the ground truth data:

Original: [Equation]
Revised: [Equation]

Original: p θ ( c | x )] and ˆ φ = arg max φ E D [log p φ ( y | c )]
Revised: p θ ( c | x )] and ˆ φ = arg max φ E D [log p φ ( y | c )]

Original: (here, we omit regularizers forbrevity).
Revised: (here, we omit regularizers forbrevity).

Original: To make predictions on a new input point, one must marginalize out the possible concepts:p
Revised: To make predictions on a new input point, one must marginalize out the possible concepts:p

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Soft CBMs Soft CBMs, on the other hand, pass the concept probabilities (the value in [0 , 1] )directly to the label predictor.
Revised: Soft CBMs Soft CBMs, on the other hand, pass the concept probabilities (the value in [0 , 1] )directly to the label predictor.

Original: When encountering a new input x , a soft concept model ﬁrst calculatesthe output of concept predictor g ( x ) where σ ( g ( x ) k )
Revised: When encountering a new input x , a soft concept model ﬁrst calculatesthe output of concept predictor g ( x ) where σ ( g ( x ) k )

Original: =
Revised: =

Original: p θ ( c k = 1 | x ) for k = 1 . . .
Revised: p θ ( c k = 1 | x ) for k = 1 . . .

Original: K .
Revised: K .

Original: Theseprobabilities are then forwarded to the label predictor, whose output is f ( c ) =
Revised: Theseprobabilities are then forwarded to the label predictor, whose output is f ( c ) =

Original: p φ ( y | c ) .
Revised: p φ ( y | c ) .

Original: Combined,they produce the ﬁnal prediction f
Revised: Combined,they produce the ﬁnal prediction f

Original: ( g ( x ))
Revised: ( g ( x ))

Original: =
Revised: =

Original: p θφ ( y | x ) .
Revised: p θφ ( y | x ) .

Original: Three different training strategies have been proposed to train soft CBMs (Koh et al., 2020).
Revised: Three different training strategies have been proposed to train soft CBMs (Koh et al., 2020).

Original: First,independent training , where the concept predictor and the label predictor are trained independentlyˆ θ = arg max θ E D [log p θ ( c | x )] and ˆ φ = arg max φ E D [log p φ ( y | c )] .
Revised: First,independent training , where the concept predictor and the label predictor are trained independentlyˆ θ = arg max θ E D [log p θ ( c | x )] and ˆ φ = arg max φ E D [log p φ ( y | c )] .

Original: Second, sequential training ,where the label predictor is trained after the concept predictor ˆ θ = arg max θ E D [log p θ ( c | x )] andˆ φ = arg max φ E D [log p θφ ( y | x )] .
Revised: Second, sequential training ,where the label predictor is trained after the concept predictor ˆ θ = arg max θ E D [log p θ ( c | x )] andˆ φ = arg max φ E D [log p θφ ( y | x )] .

Original: Finally, joint training , where the concept predictor and the labelpredictor are trained jointly ˆ θ, ˆ φ = arg max θφ E D
Revised: Finally, joint training , where the concept predictor and the labelpredictor are trained jointly ˆ θ, ˆ φ = arg max θφ E D

Original: [ λ log p θ ( c | x ) + log p θφ ( y | x )] .
Revised: [ λ log p θ ( c | x ) + log p θφ ( y | x )] .

Original: Here, 0 ≤ λ is ascalar hyperparameter controlling the tradeoff between the two terms.
Revised: Here, 0 ≤ λ is ascalar hyperparameter controlling the tradeoff between the two terms.

Original: High λ prioritizes accurateconcept predictions, while λ close to 0 focuses on accurate label predictions.
Revised: High λ prioritizes accurateconcept predictions, while λ close to 0 focuses on accurate label predictions.

Original: Leakage Leakage occurs in soft CBMs when trained sequentially or jointly (Mahinpei et al.,2021).
Revised: Leakage Leakage occurs in soft CBMs when trained sequentially or jointly (Mahinpei et al.,2021).

Original: The label predictor learns to utilize the additional, unintended information in the soft conceptprobabilities output by the concept predictor.
Revised: The label predictor learns to utilize the additional, unintended information in the soft conceptprobabilities output by the concept predictor.

Original: In the animal classiﬁcation example, when trained jointly, the concept predictor can learn to assignslightly higher likelihood to the concept ‘fur’ than the concept ‘tail’ when it recognizes the image asa cat and predict higher likelihood to the concept ‘tail’ than the concept ‘fur’ when it recognizes theimage as a dog.
Revised: In the animal classiﬁcation example, when trained jointly, the concept predictor can learn to assignslightly higher likelihood to the concept ‘fur’ than the concept ‘tail’ when it recognizes the image asa cat and predict higher likelihood to the concept ‘tail’ than the concept ‘fur’ when it recognizes theimage as a dog.

Original: Despite that cats and dogs are indistinguishable based on these two concepts, the label predictor can compare the two soft concept probabilities and make an accurate classiﬁcation.
Revised: Despite that cats and dogs are indistinguishable based on these two concepts, the label predictor can compare the two soft concept probabilities and make an accurate classiﬁcation.

Original: Thelabel predictor is no longer limited by set of the pre-deﬁned concepts: the concept predictor is ableleak further information about the label to improve the predictive performance.
Revised: Thelabel predictor is no longer limited by set of the pre-deﬁned concepts: the concept predictor is ableleak further information about the label to improve the predictive performance.

Original: This decreases theoverall loss: the concept log-likelihood log p θ ( c | x ) decreases slightly due to the slight perturbation,but the label log-likelihood log p θφ ( y | x ) increases signiﬁcantly since the label predictor now hasaccess to the leaked information.
Revised: This decreases theoverall loss: the concept log-likelihood log p θ ( c | x ) decreases slightly due to the slight perturbation,but the label log-likelihood log p θφ ( y | x ) increases signiﬁcantly since the label predictor now hasaccess to the leaked information.

Original: This is further exacerbated by using λ < 1 , which prioritizes thelabel log-likelihood over the concept log-likelihood in the training objective.
Revised: This is further exacerbated by using λ < 1 , which prioritizes thelabel log-likelihood over the concept log-likelihood in the training objective.

Original: Leakage damages both interpretability and intervenability.
Revised: Leakage damages both interpretability and intervenability.

Original: Interpretability suffers because theexplanation of the label is based not only on the concepts, but also on the behavior of the speciﬁcconcept predictor.
Revised: Interpretability suffers because theexplanation of the label is based not only on the concepts, but also on the behavior of the speciﬁcconcept predictor.

Original: The human operator would need an in-depth knowledge of the concept predictorto understand why a small change in the likelihood of ‘fur’ leads to a signiﬁcant change in theclassiﬁcation.
Revised: The human operator would need an in-depth knowledge of the concept predictorto understand why a small change in the likelihood of ‘fur’ leads to a signiﬁcant change in theclassiﬁcation.

Original: Interventions are also adversely affected.
Revised: Interventions are also adversely affected.

Original: When one intervenes on the ‘fur’ concept bysetting its probability to 1 , it can decrease the predictive accuracy because this removes all informationencoded in the soft probability value.
Revised: When one intervenes on the ‘fur’ concept bysetting its probability to 1 , it can decrease the predictive accuracy because this removes all informationencoded in the soft probability value.

Original: Hard CBMs avoid leakage by only passing binary concepts to the label predictor.
Revised: Hard CBMs avoid leakage by only passing binary concepts to the label predictor.

Original: The conceptpredictor and the label predictor learn to faithfully predict the concepts from the inputs and the labelsfrom the concepts respectively.
Revised: The conceptpredictor and the label predictor learn to faithfully predict the concepts from the inputs and the labelsfrom the concepts respectively.

Original:  Related works
Revised:  Related works

Original: Concept Activation Vector and Whitening Methods Several methods exist to interpret the layersof a neural network and what they learn.
Revised: Concept Activation Vector and Whitening Methods Several methods exist to interpret the layersof a neural network and what they learn.

Original: Among these, Kim et al.
Revised: Among these, Kim et al.

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: (2018); Ghorbaniet al.
Revised: (2018); Ghorbaniet al.

Original: (2019) use concept activation vector methods to analyse networks that have been pre-trained.
Revised: (2019) use concept activation vector methods to analyse networks that have been pre-trained.

Original: The concept activation vectors are chosen to align with pre-deﬁned or automatically-discoveredconcepts.
Revised: The concept activation vectors are chosen to align with pre-deﬁned or automatically-discoveredconcepts.

Original: These vectors are based on the assumption that the latent space can be analysed posthocin a speciﬁc form.
Revised: These vectors are based on the assumption that the latent space can be analysed posthocin a speciﬁc form.

Original: Other methods such as concept whitening (Chen et al., 2020) directly alter agiven layer of a network by adding a module that aligns the latent space to certain axes of interestto understand the computations leading to that layer.
Revised: Other methods such as concept whitening (Chen et al., 2020) directly alter agiven layer of a network by adding a module that aligns the latent space to certain axes of interestto understand the computations leading to that layer.

Original: Unlike our work, none of the former methodsaddress the fundamental issue of leakage in CBMs that signiﬁcantly impacts the interpretability andintervenability of these models.
Revised: Unlike our work, none of the former methodsaddress the fundamental issue of leakage in CBMs that signiﬁcantly impacts the interpretability andintervenability of these models.

Original: Soft Concept Bottleneck Models Soft CBMs (Chen et al., 2020; Koh et al., 2020) propose passingconcept probabilities (value in [0 , 1] ) to the label predictor to signiﬁcantly improve prediction accuracyin comparison to CBMs that learn hard concepts.
Revised: Soft Concept Bottleneck Models Soft CBMs (Chen et al., 2020; Koh et al., 2020) propose passingconcept probabilities (value in [0 , 1] ) to the label predictor to signiﬁcantly improve prediction accuracyin comparison to CBMs that learn hard concepts.

Original: Though performant, several works show that slightlydistorting the predictive concept probabilities enables the concept predictor to ‘leak’ the ﬁnal labelto the label predictor (Mahinpei et al., 2021; Margeloiu et al., 2021).
Revised: Though performant, several works show that slightlydistorting the predictive concept probabilities enables the concept predictor to ‘leak’ the ﬁnal labelto the label predictor (Mahinpei et al., 2021; Margeloiu et al., 2021).

Original: We propose tools that allowus to diagnose and address these issues to directly improve the performance of hard CBMs, therebyovercoming the issue of information leakage in their soft counterparts.
Revised: We propose tools that allowus to diagnose and address these issues to directly improve the performance of hard CBMs, therebyovercoming the issue of information leakage in their soft counterparts.

Original: Information Bottleneck Methods IB methods take a joint distribution p ( X, Y ) and compress theinputs X into a latent representation Z that is most informative about the target
Revised: Information Bottleneck Methods IB methods take a joint distribution p ( X, Y ) and compress theinputs X into a latent representation Z that is most informative about the target

Original: Y
Revised: Y

Original: (Tishby et al.
Revised: (Tishby et al.

Original: This latent representation may be thought of as a type of concept.
Revised: This latent representation may be thought of as a type of concept.

Original: Several methods have beenproposed to use such a concept representation Z to compute the effect of an intervention (Parbhooet al., 2020; Chicharro et al., 2020).
Revised: Several methods have beenproposed to use such a concept representation Z to compute the effect of an intervention (Parbhooet al., 2020; Chicharro et al., 2020).

Original: Both the traditional IB and the deep variational IB formulationsassume speciﬁc
Revised: Both the traditional IB and the deep variational IB formulationsassume speciﬁc

Original: Markov structures for the generative model (Alemi et al., 2016; Wieczorek & Roth,2020).
Revised: Markov structures for the generative model (Alemi et al., 2016; Wieczorek & Roth,2020).

Original: In contrast, our work uses side-channels to learn informative concepts that are predictive ofthe target while compressing the inputs, and is independent of such Markovian assumptions.
Revised: In contrast, our work uses side-channels to learn informative concepts that are predictive ofthe target while compressing the inputs, and is independent of such Markovian assumptions.

Original: Causal Concept Effect Models Goyal et al.
Revised: Causal Concept Effect Models Goyal et al.

Original: ; Feder et al.
Revised: ; Feder et al.

Original: (2021) explain coupling a CBMwith a causal model and computing the Causal Concept Effect to speculate about what might happento the prediction had a concept been different.
Revised: (2021) explain coupling a CBMwith a causal model and computing the Causal Concept Effect to speculate about what might happento the prediction had a concept been different.

Original: Unlike this work, we examine only the statisticalsetting where we can intervene to correct or modify an incorrect concept prediction, but not tospeculate about counterfactuals since the model does not capture causal relationships.
Revised: Unlike this work, we examine only the statisticalsetting where we can intervene to correct or modify an incorrect concept prediction, but not tospeculate about counterfactuals since the model does not capture causal relationships.

Original: Concept completeness Yeh et al.
Revised: Concept completeness Yeh et al.

Original: (2019) study when a set of concepts is sufﬁcient for explainingthe prediction in a deep neural network.
Revised: (2019) study when a set of concepts is sufﬁcient for explainingthe prediction in a deep neural network.

Original: They propose an accuracy-based completeness score.
Revised: They propose an accuracy-based completeness score.

Original: Theirscore, however, does not imply that the concepts capture all information in the input i.e. the Markovian assumption.
Revised: Theirscore, however, does not imply that the concepts capture all information in the input i.e. the Markovian assumption.

Original: Our information-based completeness score indicates precisely when the assumption ismet.
Revised: Our information-based completeness score indicates precisely when the assumption ismet.

Original:  Addressing Two Shortcomings of Concept Bottleneck Models
Revised:  Addressing Two Shortcomings of Concept Bottleneck Models

Original: In this section, we explain and address the performance disparity between soft CBMs and hard CBMs.
Revised: In this section, we explain and address the performance disparity between soft CBMs and hard CBMs.

Original: We present two sources of information that CBMs are unable to capture: information that is notavailable in the concepts and information in the correlations in the concepts.
Revised: We present two sources of information that CBMs are unable to capture: information that is notavailable in the concepts and information in the correlations in the concepts.

Original: Soft CBMs can lessenthe impact through leakage, but hard CBMs cannot.
Revised: Soft CBMs can lessenthe impact through leakage, but hard CBMs cannot.

Original: We propose modiﬁcations to hard CBMs thatallow the use of information from these two sources while retaining their resilience to leakage.
Revised: We propose modiﬁcations to hard CBMs thatallow the use of information from these two sources while retaining their resilience to leakage.

Original: 4.1 The Markovian Assumption
Revised: 4.1 The Markovian Assumption

Original: First, we consider the case when the concepts are insufﬁcient for predicting the ﬁnal label, but xhas more information that can help.
Revised: First, we consider the case when the concepts are insufﬁcient for predicting the ﬁnal label, but xhas more information that can help.

Original: We refer to this as the lack of a Markovian assumption.
Revised: We refer to this as the lack of a Markovian assumption.

Original: Whenthe Markovian assumption holds, the chain x
Revised: Whenthe Markovian assumption holds, the chain x

Original: → c → y is Markovian, which is deﬁned as y beingconditionally independent of x given c : I( y ; x | c ) = 0 , where I denotes the mutual information.
Revised: → c → y is Markovian, which is deﬁned as y beingconditionally independent of x given c : I( y ; x | c ) = 0 , where I denotes the mutual information.

Original: When the assumption holds, c must contain all information about y that is present in x .
Revised: When the assumption holds, c must contain all information about y that is present in x .

Original: Otherwise,c is insufﬁcient for predicting y ; therefore, a CBM that relies on hard concepts will underperformcompared to a black box x → y predictor.
Revised: Otherwise,c is insufﬁcient for predicting y ; therefore, a CBM that relies on hard concepts will underperformcompared to a black box x → y predictor.

Original: In what follows, we describe how CBMs can still beapplied with the help of a side-channel and deﬁne a concept completeness score to diagnose whenthe Markovian assumption may not hold.
Revised: In what follows, we describe how CBMs can still beapplied with the help of a side-channel and deﬁne a concept completeness score to diagnose whenthe Markovian assumption may not hold.

Original: 4.1.1 Using a Side-channel
Revised: 4.1.1 Using a Side-channel

Original: Along with the known concepts c , we model L unknown concepts z ∈ { 0 , 1 } L in a side-channel
Revised: Along with the known concepts c , we model L unknown concepts z ∈ { 0 , 1 } L in a side-channel

Original: CBM (Figure 1 middle).
Revised: CBM (Figure 1 middle).

Original: This setup allows the model to capture information about y present in x butnot present in c via the latent concepts.
Revised: This setup allows the model to capture information about y present in x butnot present in c via the latent concepts.

Original: Training the side-channel model is analogous to training a hard CBM.
Revised: Training the side-channel model is analogous to training a hard CBM.

Original: We optimize the predictivelog-likelihood using stochastic gradient ascent:
Revised: We optimize the predictivelog-likelihood using stochastic gradient ascent:

Original: [Equation]
Revised: [Equation]

Original: A caveat is that the gradient of E p θ ( z | x ) [ p φ ( y | c, z )] need to be estimated using a gradient estimator.
Revised: A caveat is that the gradient of E p θ ( z | x ) [ p φ ( y | c, z )] need to be estimated using a gradient estimator.

Original: We use the reinforce estimator (Williams, 1992) because it is able to give an unbiased estimate of thegradients, and it works well in a low dimensional settings such as z :
Revised: We use the reinforce estimator (Williams, 1992) because it is able to give an unbiased estimate of thegradients, and it works well in a low dimensional settings such as z :

Original: [Equation]
Revised: [Equation]

Original: The gradient is then estimated using M Monte-Carlo samples:
Revised: The gradient is then estimated using M Monte-Carlo samples:

Revised: [Equation]

Revised: At prediction time, we can decide whether to use the side-channel for prediction or rely purely on theknown concepts.

Revised: If the side-channel is used, the predictions are calculated using M Monte-Carlosamples:

Original: [Equation]
Revised: [Equation]

Original: A challenge in the side-channel model is estimating p θ ( z | c ) , which is needed to make predictions.
Revised: In applications where the label prediction must be fully explained by the concepts, we have tomarginalize out the side-channel. This requires estimating p θ ( z | c ) .

Original: One can obtain an estimate using p θ ( z | c ) ≈ E c (cid:48) ,x (cid:48)
Revised: One can obtain an estimate usingp θ ( z | c ) ≈ E c (cid:48) ,x (cid:48)

Original: ∼D & c (cid:48)
Revised: ∼D & c (cid:48)

Original: =
Revised: =

Original: c [ p θ ( z | x (cid:48) )] ; however, this requires iterating200 over the whole dataset.
Revised: c [ p θ ( z | x (cid:48) )] ; however, this requires iterating over the whole dataset each time 204 a prediction is made.

Original: Our solution is to train a  predictor that can rapidly estimate p θ ( z | c )  .
Revised: Our solution is to train a small (1 hidden layer) predictor that can rapidly estimate p θ ( z | c ) from c .

Original: We call201 this the amortization network, since it amortizes the cost of inferring z from c .
Revised: We call this the amortization network, since it amortizes the cost of inferring z from c .

Original: It has parameters γ ,and itis trained using maximum-likelihood training: ˆ γ = [Equation] log p [Equation]
Revised: We denote its parameters γ , and train it using maximum-likelihood training: ˆ γ = arg max γ E D [log p γ ( z | c )] .

Original: z | c )]

Original: [Equation] We can decide whether to use the side-channel for prediction or rely purely on the known concepts. If the side-channel is used, the  predictions are calculated using M Monte-Carlo samples:
Revised: Inour experiments, the amortization network is trained over 10-20 epochs depending on the dataset. Equipped with γ , we can make predictions without using the side-channel:

Original: [Equation]
Revised: [Equation]

Original: For applications where the prediction must be fully explained by the concepts, we can marginalizeout the side-channel with the  help of the amortization network. [Equation]
Revised: A second beneﬁt marginalizing the side-channel with the amortization network is that it can help usunderstand how much the model relies on the side-channel for prediction. To formalize this concept,we introduce a completeness score.

Original: A Completeness Score
Revised: A Completeness Score

Original: The side-channel model canalso be used to estimate what fraction of therelevant information is present in c .
Revised: The side-channel model can be used to estimate what fraction of therelevant information is present in c .

Original: This is useful for diagnosing when a side-channel is needed andto understand how much the model relies on it.
Revised: This is useful for diagnosing when a side-channel is needed andto understand how much the model relies on it.

Original: If the model heavily relies on the side channel, itindicates that key concepts are missing from c .
Revised: If the model heavily relies on the side channel, itindicates that key concepts are missing from c .

Original: We deﬁne
Revised: We deﬁne

Original: [Equation]
Revised: [Equation]

Original: ) 0 ≤ C ≤ 1 , because its a ratio of non-negative quantities and the nominator is less-than-or-equalto the denominator since I( y ; c, x ) − I( y ; c ) =
Revised: 0 ≤ C ≤ 1 , because its a ratio of non-negative quantities and the nominator is less-than-or-equalto the denominator since I( y ; c, x ) − I( y ; c ) =

Original: I( y ; x | c ) ≥ 0 (Cover, 1999).
Revised: I( y ; x | c ) ≥ 0 (Cover, 1999).

Original: When C = 1 , we have
Revised: When C = 1 , we have

Original: I( y ; c ) =
Revised: I( y ; c ) =

Original: I( y ; c, x ) , so I( y ; c, x ) − I( y ; c ) =
Revised: I( y ; c, x ) , so I( y ; c, x ) − I( y ; c ) =

Original: I( y ; x | c ) = 0 thus the Markovian assumption holds, andc captures all relevant information.
Revised: I( y ; x | c ) = 0 thus the Markovian assumption holds, andc captures all relevant information.

Original: When C = 

Revised: When C =

Original: ( I( y ; c ) = 

Revised: ( I( y ; c ) =

Original: i.e. c and y are independent), noneof the information in c is relevant for predicting y .
Revised: i.e. c and y are independent), noneof the information in c is relevant for predicting y .

Original: With the help of the amortization network, it ispossible to estimate C :
Revised: With the help of the amortization network, it ispossible to estimate C :

Original: [Equation]
Revised: [Equation]

Original: Interpretability of the Side-channel The latent concepts captured by the side channel can beinspected for interpretability.
Revised: Interpretability of the Side-channel The latent concepts captured by the side channel can beinspected for interpretability.

Original: They may align with human understandable concepts (in which case,they can be added to c ), but it is possible that they are uninterpretable.
Revised: They may align with human understandable concepts (in which case,they can be added to c ), but it is possible that they are uninterpretable.

Original: The side-channel model shouldstill be used to avoid the detrimental effects of leakage on concept accuracy and interventions.
Revised: The side-channel model shouldstill be used to avoid the detrimental effects of leakage on concept accuracy and interventions.

Original: Notethat we have agency on deciding whether to use the side channel information at prediction time ornot.
Revised: Notethat we have agency on deciding whether to use the side channel information at prediction time ornot.

Original: Including the side-channel information is ideal for applications where accuracy is high-priorityand partial explanations are sufﬁcient (Equation 4).
Revised: Including the side-channel information is ideal for applications where accuracy is high-priorityand partial explanations are sufﬁcient (Equation 4).

Original: For applications where the human operator mustknow the full set of concepts contributing to the prediction, we can make predictions without it (bymarginalizing z with the help of the amortization network as shown in Equation 5).
Revised: For applications where the human operator mustknow the full set of concepts contributing to the prediction, we can make predictions without it (bymarginalizing z with the help of the amortization network as shown in Equation 5).

Original: 4.2 The Expressivity of the Concept Predictor
Revised: 4.2 The Expressivity of the Concept Predictor

Original: The second source of information that CBMs are unable to capture is correlations in the concepts.
Revised: The second source of information that CBMs are unable to capture is correlations in the concepts.

Original: An issue with the usual formulation of a CBM is that the concept predictor predicts the conceptsindependently (conditional on the input).
Revised: An issue with the usual formulation of a CBM is that the concept predictor predicts the conceptsindependently (conditional on the input).

Original: This may give poor performance when the concepts arecorrelated.
Revised: This may give poor performance when the concepts arecorrelated.

Original: Consider two concepts that are mutually exclusive.
Revised: Consider two concepts that are mutually exclusive.

Original: A predictor that predicts themindependently does not have the ﬂexibility to express their mutual exclusivity and it inadvertentlypredicts non-zero probability of both concepts being present.
Revised: A predictor that predicts themindependently does not have the ﬂexibility to express their mutual exclusivity and it inadvertentlypredicts non-zero probability of both concepts being present.

Original: While a soft CBM can learn duringtraining that the concept probabilities correspond to mutually exclusive concepts, a hard CBM cannot.
Revised: While a soft CBM can learn duringtraining that the concept probabilities correspond to mutually exclusive concepts, a hard CBM cannot.

Original: We propose a modiﬁcation to the CBM architecture that allows hard CBMs to capture correlations inthe concepts.
Revised: We propose a modiﬁcation to the CBM architecture that allows hard CBMs to capture correlations inthe concepts.

Original: Our solution is to use an autoregressive architecture for concept prediction.
Revised: Our solution is to use an autoregressive architecture for concept prediction.

Original: To predictthe k -th concept, use not only the input x , but also the already predicted concepts c 1: k − 1 1 (Figure 1right).
Revised: To predictthe k -th concept, use not only the input x , but also the already predicted concepts c 1: k − 1 1 (Figure 1right).

Original: The log-likelihood, which is needed for training, can be computed by iterating over the concepts:log p θ ( c | x ) =
Revised: The log-likelihood, which is needed for training, can be computed by iterating over the concepts:log p θ ( c | x ) =

Original: (cid:80)
Revised: (cid:80)

Original: Kk =1 log p θ ( c k | x, c 1: k − 1 ) .
Revised: Kk = p θ ( c k | x, c 1: k − 1 ) .

Original: To sample a concept set, which is needed for prediction, 240 we must sample the concepts iteratively:c k ∼ p θ ( c k | x, c 1: k − 1 ) for k = 1 . . . K .
Revised: To sample a concept set, which is needed for prediction, 246 we must sample the concepts iteratively:

Original: Lastly, interventions require a modiﬁed sampling algorithm.
Revised: Lastly, interventions require a modiﬁed sampling algorithm.

Original: In autoregressive models, we cannot easily generate Monte-Carlo samples from the concept distribution conditional on a set of interventions.
Revised: In autoregressive models, we cannot easily generate Monte-Carlo samples from the concept distribution conditional on a set of interventions.

Original: Instead, to estimate p θ,φ (ˆ y | ˆ x, { ˆ c i } I ) (where { ˆ c i } i ∈I⊆{ 1 ...K } denotes the set of intervened concepts), 244 we propose a normalized importance sampling algorithm that generates M weighted concept samplesfor prediction
Revised: Instead, to estimate p θ,φ (ˆ y | ˆ x, { ˆ c i } I ) (where { ˆ c i } i ∈I⊆{ 1 ...K } denotes the set of intervened concepts), 250 we propose a normalized importance sampling algorithm that generates M weighted concept samplesfor prediction

Original: [Equation]
Revised: [Equation]

Original: Our proposal distribution c (1) . . .
Revised: Our proposal distribution c (1) . . .

Original: c ( M ) ∼ q ( c )matches the interventions { ˆ c
Revised: c ( M ) ∼ q ( c )matches the interventions { ˆ c

Original: i
Revised: i

Original: } I when the concepts are intervened on and selects a random concept 247 conditional on the previous concepts when they are not intervened on:
Revised: } I when the concepts are intervened on and selects a random concept 253 conditional on the previous concepts when they are not intervened on:

Original: [Equation]
Revised: [Equation]

Original: where δ denotes Kronecker’s delta function.
Revised: where δ denotes Kronecker’s delta function.

Original: To obtain an importance weighted estimate, we mustcompute the weight of each of the M samples w m = p θ 
Revised: To obtain an importance weighted estimate, we mustcompute the weight of each of the M samples w m = p θ ( c ( m )

Original: [Equation]
Revised: | ˆ x, { ˆ c i } ) q ( c ( m ) ) .

Original: This is done by iterating over the K concepts:
Revised: This is done by iterating overthe K concepts:

Original: [Equation]
Revised: [Equation]

Original: since the probability mass ratio p θ ( c ( m ) k | x,c ( m ) 1: k − 1 ) q ( c ( m ) k | x,c ( m ) 1: k − 1 ) = 1 when k / ∈ I .
Revised: since the probability mass ratio p θ ( c ( m ) k | x,c ( m ) 1: k − 1 ) q ( c ( m ) k | x,c ( m ) 1: k − 1 ) = 1 when k / ∈ I .

Original: Finally, we obtain the importanceweighted estimatep θ,
Revised: Finally, we obtain the importance weighted estimate

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: Note that due to this re-weightingscheme, interventions are able to affect concept predictions of earlier concepts.

Original: [Equation] Normalization is important to ensure thepredictive distribution is proper.
Revised: We also show thisexperimentally in Appendix E. Normalization is important to ensure the predictive distribution isproper.

Original: Psudocode for the interventions is shown in Appendix E.
Revised: Psudocode for the interventions is shown in Appendix D.

Original:  Results and Discussion
Revised:  Results and Discussion

Original: We present empirical results showcasing the efﬁcacy of our proposed modiﬁcations to conceptbottleneck models (CBM).
Revised: We present empirical results showcasing the efﬁcacy of our proposed modiﬁcations to conceptbottleneck models (CBM).

Original: 5.1 Datasets and Models
Revised: 5.1 Datasets and Models

Original: We used two datasets in our experiment.
Revised: We used two datasets in our experiment.

Original: First, an Early Warning Score (EWS) prediction taskbased on electronic health records data that simulates a potential use case.
Revised: First, an Early Warning Score (EWS) prediction taskbased on electronic health records data that simulates a potential use case.

Original: Second, to showcasethe versatility of CBMs and compare to prior work, we apply our method to an image-based birdrecognition task.
Revised: Second, to showcasethe versatility of CBMs and compare to prior work, we apply our method to an image-based birdrecognition task.

Original: (Our source code is available in the supplementary material.
Revised: (Our source code is available in the supplementary material.

Original: We will be including alink to our public repository in the de-anonymized version.)
Revised: We will be including alink to our public repository in the de-anonymized version.)

Original: MIMIC-III EWS (Johnson et al., 2016)
Revised: MIMIC-III EWS (Johnson et al., 2016)

Original: The task is to predict an EWS for patients at a hospitalthat informs the medical professionals about abnormalities in the vital signs that often precededeterioration and cardiac arrests (Subbe et al., 2001).
Revised: The task is to predict an EWS for patients at a hospitalthat informs the medical professionals about abnormalities in the vital signs that often precededeterioration and cardiac arrests (Subbe et al., 2001).

Original: The concepts (‘Mild hypotension’, ‘Severefever’ etc.) provide justiﬁcation for the score.
Revised: The concepts (‘Mild hypotension’, ‘Severefever’ etc.) provide justiﬁcation for the score.

Original: The EWS ranges from 0 to 15, and it is based on thepatient’s vitals.
Revised: The EWS ranges from 0 to 15, and it is based on thepatient’s vitals.

Original: Deviations from the norm increase the score depending on severity.
Revised: Deviations from the norm increase the score depending on severity.

Original: For example,‘severe hypotension’ adds +3 to the EWS, while ‘mild fever’ adds +1.
Revised: For example,‘severe hypotension’ adds +3 to the EWS, while ‘mild fever’ adds +1.

Original: The dataset does not containconcept annotations, so we deﬁned them synthetically (Appendix A).
Revised: The dataset does not containconcept annotations, so we deﬁned them synthetically (Appendix A).

Original: The dataset contains recordsfrom 17,289 patients over a combined N=796,250 time steps (split into 530,802 training and 265,448test examples, while ensuring that no patient appears in both sets).
Revised: The dataset contains recordsfrom 17,289 patients over a combined N=796,250 time steps (split into 530,802 training and 265,448test examples, while ensuring that no patient appears in both sets).

Original: It has 13 input features, K = 22concepts and 16 label classes (EWS from 0 to 15).
Revised: It has 13 input features, K = 22concepts and 16 label classes (EWS from 0 to 15).

Original: For MIMIC-III EWS, the concept predictor is a two-layer feed-forward neural network with a hiddenlayer of size 100, and the label predictor is a two-layer network with hidden layer of size 50.
Revised: For MIMIC-III EWS, the concept predictor is a two-layer feed-forward neural network with a hiddenlayer of size 100, and the label predictor is a two-layer network with hidden layer of size 50.

Original: In theautoregressive case, a small, two-layer network (hidden layer size 20) predicts each concept.
Revised: In theautoregressive case, a small, two-layer network (hidden layer size 20) predicts each concept.

Original: Theinput to these networks are the previously predicted concepts, concatenated with the 100 hiddenunits of the concept predictor.
Revised: Theinput to these networks are the previously predicted concepts, concatenated with the 100 hiddenunits of the concept predictor.

Original: We use M = 200 Monte-Carlo samples for prediction.
Revised: We use M = 200 Monte-Carlo samples for prediction.

Original: For traininghyperparameters, see Appendix C.
Revised: For traininghyperparameters, see Appendix C.

Original: Caltech-UCSD Birds 

Revised: Caltech-UCSD Birds

Original: (Wah et al., 2011)
Revised: (Wah et al., 2011)

Original: For an accurate comparison to Koh et al.
Revised: For an accurate comparison to Koh et al.

Original: (2020),we replicated their results with the same preprocessing steps.
Revised: (2020),we replicated their results with the same preprocessing steps.

Original: This dataset contains N=11,788 (5,994training and 5,794 test) images of 200 bird species native to North America, along with 312 binaryattributes (eg.:{+(5,
 [Table] [Figure] [Table]+}
Revised: This dataset contains N=11,788 {+(5,
 [Table] [Figure] [Table]+}

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: with interventions.

Revised: training and 5,794 test) images of 200 bird species native to North America, along with 312 binaryattributes (eg.:

Original: ‘wing-color’ and ‘beak-shape’).
Revised: ‘wing-color’ and ‘beak-shape’).

Original: Following Koh et al.
Revised: Following Koh et al.

Original: (2020), we narrow down this listof attributes to K = 112 binary concepts, only including attributes that are present for at least 10%of the dataset.
Revised: (2020), we narrow down this listof attributes to K = 112 binary concepts, only including attributes that are present for at least 10%of the dataset.

Original: To de-noise the attributes, we consolidate the concepts within each species by majorityvoting (where if the majority of ravens have black wings, we ensure that all ravens in the dataset areannotated as having black wings (Koh et al., 2020)).
Revised: To de-noise the attributes, we consolidate the concepts within each species by majorityvoting (where if the majority of ravens have black wings, we ensure that all ravens in the dataset areannotated as having black wings (Koh et al., 2020)).

Original: For CUB-2011, we ﬁne-tuned a pre-trained Inception v3 network (Szegedy et al., 2016) as theconcept predictor.
Revised: For CUB-2011, we ﬁne-tuned a pre-trained Inception v3 network (Szegedy et al., 2016) as theconcept predictor.

Original: Following Koh et al.
Revised: Following Koh et al.

Original: (2020), the label predictor for the soft models is a simplelinear predictor, while for hard models, we use a two-layer neural network with a hidden layer ofsize 200.
Revised: (2020), the label predictor for the soft models is a simplelinear predictor, while for hard models, we use a two-layer neural network with a hidden layer ofsize 200.

Original: In the autoregressive case, a small, two-layer network (hidden layer size 50) predicts eachconcept.
Revised: In the autoregressive case, a small, two-layer network (hidden layer size 50) predicts eachconcept.

Original: The input to these networks are the previously predicted concepts concatenated with theoutput of the Inception network.
Revised: The input to these networks are the previously predicted concepts concatenated with theoutput of the Inception network.

Original: We use M = 200 Monte-Carlo samples for prediction.
Revised: We use M = 200 Monte-Carlo samples for prediction.

Original: For traininghyperparameters, see Appendix C.
Revised: For traininghyperparameters, see Appendix C.

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: with interventions.

Original: 5.2 Autoregressive Concept Predictions
Revised: 5.2 Autoregressive Concept Predictions

Original: Our ﬁrst set of experiments quantitatively assess the impact of leakage as well as examine theeffectiveness of autoregressive concept predictors.
Revised: Our ﬁrst set of experiments quantitatively assess the impact of leakage as well as examine theeffectiveness of autoregressive concept predictors.

Original: We lead with these experiments to justify the useof the autoregressive architecture in our subsequent side-channel experiments.
Revised: We lead with these experiments to justify the useof the autoregressive architecture in our subsequent side-channel experiments.

Original: It is important to notethat both datasets have complete concept sets (they meet the Markovian assumption C = 1 ) so anyperformance disparity is due to the inﬂexibility of the concept predictor.
Revised: It is important to notethat both datasets have complete concept sets (they meet the Markovian assumption C = 1 ) so anyperformance disparity is due to the inﬂexibility of the concept predictor.

Original: Leakage makes soft CBMs unsuitable for tasks where interpretability or intervenability arerequired.
Revised: Leakage makes soft CBMs unsuitable for tasks where interpretability or intervenability arerequired.

Original: Soft CBMs underperform in concept accuracy, but they are often able to predict the labelcorrectly (Table 1).
Revised: Soft CBMs underperform in concept accuracy, but they are often able to predict the labelcorrectly (Table 1).

Original: This is only possible if the concepts are leaking information about the label in anuninterpretable way.
Revised: This is only possible if the concepts are leaking information about the label in anuninterpretable way.

Original: Interventions, where the human operator sets concepts to their true values, areineffective on sequential and joint soft CBMs (Figure 2).
Revised: Interventions, where the human operator sets concepts to their true values, areineffective on sequential and joint soft CBMs (Figure 2).

Original: This should improve their predictions andyet they still mispredict the label even when all concepts are supplied, because interventions removethe leaked information from the soft concept probabilities.
Revised: This should improve their predictions andyet they still mispredict the label even when all concepts are supplied, because interventions removethe leaked information from the soft concept probabilities.

Original: Autoregressive predictors close the gap between soft and hard CBMs in label accuracy whilehaving dramatically increased concept accuracy.
Revised: Autoregressive predictors close the gap between soft and hard CBMs in label accuracy whilehaving dramatically increased concept accuracy.

Original: In Table 1, we see that soft CBMs performwell in label accuracy, but leakage is detrimental to their concept accuracy.
Revised: In Table 1, we see that soft CBMs performwell in label accuracy, but leakage is detrimental to their concept accuracy.

Original: The hard CBM performsbetter in concept accuracy, but its performance is limited by the inﬂexibility of the concept predictor.
Revised: The hard CBM performsbetter in concept accuracy, but its performance is limited by the inﬂexibility of the concept predictor.

Original: The autoregressive model (Hard AR, this work) offers the best concept accuracy with competitivelabel accuracy: in the case of MIMIC-III EWS, Hard AR is above soft models in both concept andlabel accuracy, while in CUB-2011, Hard AR has the best concept accuracy with label accuracywithin 1% of joint models.
Revised: The autoregressive model (Hard AR, this work) offers the best concept accuracy with competitivelabel accuracy: in the case of MIMIC-III EWS, Hard AR is above soft models in both concept andlabel accuracy, while in CUB-2011, Hard AR has the best concept accuracy with label accuracywithin 1% of joint models.

Original: (Note that concept accuracy refers to predicting all concepts correctlysimultaneously as opposed to averaging the accuracies over the different concepts.)
Revised: (Note that concept accuracy refers to predicting all concepts correctlysimultaneously as opposed to averaging the accuracies over the different concepts.)

Original: Hard autoregressive models excel at interventions while soft CBMs fail to adjust their predictions.
Revised: Hard autoregressive models excel at interventions while soft CBMs fail to adjust their predictions.

Original: On Figure 2, we see that the performance of soft sequential and soft joint models deterioratequickly due to leakage, while soft independent and hard models improve with interventions.
Revised: On Figure 2, we see that the performance of soft sequential and soft joint models deteriorate [Figure] quickly due to leakage, while soft independent and hard models improve with interventions.

Original: The bestperforming model is the Hard AR (this work) on both datasets.
Revised: The bestperforming model is the Hard AR (this work) on both datasets.

Original: 5.3 Using a Side-channel
Revised: 5.3 Using a Side-channel

Original: Next, we examine the side-channel model as well as our proposed completeness score.
Revised: Next, we examine the side-channel model as well as our proposed completeness score.

Original: In theseexperiments, we trained CBMs with only a subset of the available concepts to simulate the scenariowhere the Markovian assumption does not hold.
Revised: In theseexperiments, we trained CBMs with only a subset of the available concepts to simulate the scenariowhere the Markovian assumption does not hold.

Original: We used L = 20 latent concepts for MIMIC-IIIand L = 100 latent concepts for CUB-2011 (we settled on these values because larger L yieldedno signiﬁcant improvement in predictive performance).
Revised: We used L = 20 latent concepts for MIMIC-IIIand L = 100 latent concepts for CUB-2011 (we settled on these values because larger L yieldedno signiﬁcant improvement in predictive performance).

Original: Note that we use autoregressive conceptpredictors in hard CBMs, so any performance disparity is due to having an insufﬁcient concept set.
Revised: Note that we use autoregressive conceptpredictors in hard CBMs, so any performance disparity is due to having an insufﬁcient concept set.

Original: In MIMIC-III EWS, the concepts were grouped depending on which of the ﬁve vital signs theyare derived from (blood-pressure, temperature, respiratory rate, heart rate, oxygen saturation).
Revised: In MIMIC-III EWS, the concepts were grouped depending on which of the ﬁve vital signs theyare derived from (blood-pressure, temperature, respiratory rate, heart rate, oxygen saturation).

Original: In
Revised: In

Original: CUB-2011, a group of concepts refer to related attributes, such as ‘wing-color’.
Revised: CUB-2011, a group of concepts refer to related attributes, such as ‘wing-color’.

Original: [Figure]

Original: The side-channel closes the gap between soft and hard CBMs when the Markovian assumptionis not met.
Revised: The side-channel closes the gap between soft and hard CBMs when the Markovian assumptionis not met.

Original: In Figure 3 (left), we see that the performance of the side-channel matches the predictiveperformance of the joint model when accounting for the latent concepts and it performs equivalently tothe no side-channel model when the latent concepts are marginalized with the help of the amortizationnetwork.
Revised: In Figure 3 (left), we see that the performance of the side-channel matches the predictiveperformance of the joint model when accounting for the latent concepts and it performs equivalently tothe no side-channel model when the latent concepts are marginalized with the help of the amortizationnetwork.

Original: Therefore, the side-channel models precisely the information that is not present in theconcepts, i.e. the information that is leaked in the soft joint model.
Revised: Therefore, the side-channel models precisely the information that is not present in theconcepts, i.e. the information that is leaked in the soft joint model.

Original: The completeness score indicates when the Markovian assumption is fulﬁlled.
Revised: The completeness score indicates when the Markovian assumption is fulﬁlled.

Original: The completeness score proves to be a useful diagnostic tool to show when the available concept set limits theperformance of hard CBMs.
Revised: The completeness score proves to be a useful diagnostic tool to show when the available concept set limits theperformance of hard CBMs.

Original: Figure 3 (right) depicts the completeness score C depending on thenumber of concepts used during training.
Revised: Figure 3 (right) depicts the completeness score C depending on thenumber of concepts used during training.

Original: As the concept set grows, C tends towards 1 , and reaches itonce the concept set contains all information about the ﬁnal label, and the Markovian assumptionis met.
Revised: As the concept set grows, C tends towards 1 , and reaches itonce the concept set contains all information about the ﬁnal label, and the Markovian assumptionis met.

Original: Interestingly, completeness reaches 1 before p φ ( y | c ) accuracy peaks in CUB-2011.
Revised: Interestingly, completeness reaches 1 before p φ ( y | c ) accuracy peaks in CUB-2011.

Original: This ispossible when a concept contains information to predict y , but this information is not present in x (e.g.the color of the belly is not visible on images where it could be the deciding factor).
Revised: This ispossible when a concept contains information to predict y , but this information is not present in x (e.g.the color of the belly is not visible on images where it could be the deciding factor).

Original: Such conceptsdo not contribute to completeness, but they increase label accuracy when the concepts are given.
Revised: Such conceptsdo not contribute to completeness, but they increase label accuracy when the concepts are given.

Revised: The side-channel is fully compatible with interventions There have been concerns that a sidechannel model could encode the label information in the side-channel, thus rendering interventionsineffective (Koh et al., 2020).

Revised: We found that this is not the case.

Revised: Figure 4 shows that interventionsare as effective when trained with a subset of concepts as when trained with the complete conceptset.

Revised: In the case of MIMIC-III EWS, the intervention accuracy of the side-channel models completelyoverlap with the hard CBM trained on all concepts.

Revised: In CUB-2011, the interventions are less effectivewith 1-2 concept groups, but they quickly catch up to the model trained with the complete conceptset.

Original:  Computational costs.
Revised: [Figure] Computational costs.

Original: The beneﬁts of autoregressive models and side-channel models can comeat an increased computational cost.
Revised: The beneﬁts of autoregressive models and side-channel models come at anincreased computational cost.

Original:  Notably,they take longer to converge for large datasets: in CUB-2011, the autoregressive and side-channel models take three times longer to converge ( ∼ 1 . 5hour vs ∼ 4 .
Revised: Table 2 shows the computational cost of each model. Notably, the morecomplex image dataset, CUB-2011 takes more epochs to converge.

Original: 5 hours on a V100 GPU). The cost oftraining the amortization network is not signiﬁcantcompared to the cost of training the concept predictor. In the case of MIMIC-III EWS, the trainingcost of the joint model is  the same as the cost of the autoregressive and side-channel models ( ∼ 0 . 5hours on a V100 GPU).
Revised: The cost of the autoregressivearchitecture and the side-channel is more signiﬁcant when the predictive model is relatively simple,which is the case in MIMIC-III EWS

Original:  Conclusions
Revised:  Conclusions

Original: Leakage is detrimental to model interpretability and intervenability in soft concept bottleneck models.
Revised: Leakage is detrimental to model interpretability and intervenability in soft concept bottleneck models.

Original: Hard CBMs are resilient to leakage, but they lag behind their soft counterparts in predictiveperformance.
Revised: Hard CBMs are resilient to leakage, but they lag behind their soft counterparts in predictiveperformance.

Original: We exhibited and addressed two causes of performance disparity between soft and hard
Revised: We exhibited and addressed two causes of performance disparity between soft and hard

Original: CBMs.
Revised: CBMs.

Original: Firstly, not meeting the Markovian assumption, which can be diagnosed using our proposedcompleteness score and addressed using a side-channel model.
Revised: Firstly, not meeting the Markovian assumption, which can be diagnosed using our proposedcompleteness score and addressed using a side-channel model.

Original: Secondly, the inﬂexibility of theconcept predictor, which can be mitigated by adapting an autoregressive architecture.
Revised: Secondly, the inﬂexibility of theconcept predictor, which can be mitigated by adapting an autoregressive architecture.

Original: With our modiﬁcations, CBMs no longer suffer from leakage and have signiﬁcantly improved conceptaccuracy and intervention accuracy.
Revised: With our modiﬁcations, CBMs no longer suffer from leakage and have signiﬁcantly improved conceptaccuracy and intervention accuracy.

Original: This makes them more interpretable and more suitable forhuman-AI joint decision-making tasks.
Revised: This makes them more interpretable and more suitable forhuman-AI joint decision-making tasks.

Original: In future work, we hope to explore methods to present the side-channel information in a humaninterpretable way.
Revised: In future work we hope to explore methods to present the side-channel information in a humaninterpretable way.

Original: This would allow us to discover concepts that are needed to complete the conceptset for a given task.
Revised: This would allow us to discover concepts that are needed to complete the conceptset for a given task

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See interpretability of the side
Revised: [Yes] See interpretability of the side

Original: channel and computational considerations.
Revised: channel and computational considerations.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] We do not foresee signiﬁcant negative societal impacts.
Revised: [No] We do notforesee signiﬁcant negative societal impacts.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b)
Revised: [N/A]

Original:  Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how they were chosen)?
Revised: g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experi- ments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [Yes](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]4.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] The datasets we used are open for
Revised: [Yes] The datasets we used are open for

Original: research purposes, our code is open source.
Revised: research purposes, our code is open source.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: Our code.
Revised: Our code.

Original: ) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?

Original: [Yes]
Revised: [Yes]

Original: In the case of MIMIC, Johnson et al.
Revised: In the case of MIMIC, Johnson et al.

Original: “Requirementfor individual patient consent was waived because the project did not  clinical
Revised: (2016): “Requirement for individual patient consent was waived because the project did not impact clinical

Original: care and all protected health information was deidentiﬁed
Revised: care and all protected health information was deidentiﬁed.”.

Revised: CUB-2011 has no humansubjects.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes] See above.
Revised: [Yes] See above.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]