Original: Addressing Leakage in Concept Bottleneck Models
Revised: Addressing Leakage in Concept Bottleneck Models

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Concept bottleneck models (CBMs) enhance the interpretability of their predictionsby ﬁrst predicting high-level concepts given features, and subsequently predictingoutcomes on the basis of these concepts.
Revised: Concept bottleneck models (CBMs) enhance the interpretability of their predictions by ﬁrst predicting high-level concepts given features, and subsequently predicting outcomes on the basis of these concepts.

Original: Recently, it was demonstrated that trainingthe label predictor directly on the probabilities produced by the concept predictoras opposed to the ground-truth concepts, improves label predictions.
Revised: Recently, it was demonstrated that training the label predictor directly on the probabilities produced by the concept predictor as opposed to the ground-truth concepts, improves label predictions.

Original: However, thisresults in corruptions in the concept predictions that impact the concept accuracy aswell as our ability to intervene on the concepts – a key proposed beneﬁt of CBMs.
Revised: However, this results in corruptions in the concept predictions that impact the concept accuracy as well as our ability to intervene on the concepts – a key proposed beneﬁt of CBMs.

Original: In this work, we investigate and address two issues with CBMs that cause thisdisparity in performance: having an insufﬁcient concept set and using inexpressiveconcept predictor.
Revised: In this work, we investigate and address two issues with CBMs that cause this disparity in performance: having an insufﬁcient concept set and using inexpressive concept predictor.

Original: With our modiﬁcations, CBMs become competitive in terms ofpredictive performance, with models that otherwise leak unintended information inthe concept probabilities, while having dramatically increased concept accuracyand intervention accuracy.
Revised: With our modiﬁcations, CBMs become competitive in terms of predictive performance, with models that otherwise leak unintended information in the concept probabilities, while having dramatically increased concept accuracy and intervention accuracy.

Original:  Introduction
Revised:  Introduction

Original: Building interpretable machine learning models is important in several high-stake domains wherehumans must typically interact with the models to make meaningful decisions.
Revised: Building interpretable machine learning models is important in several high-stake domains where humans must typically interact with the models to make meaningful decisions.

Original: Among such models,
Revised: Among such models,

Original: Concept bottleneck models (CBMs) (Chen et al., 2020; Koh et al., 2020; Yeh et al., 2019; Lage& Doshi-Velez, 2020; Wang et al., 2017) propose explicitly aligning the intermediate layers of aneural network with some pre-deﬁned expert concepts by ﬁrst training a predictor of concepts, andsubsequently using this to predict the label (Figure 1 left).
Revised: Concept bottleneck models (CBMs) (Chen et al., 2020; Koh et al., 2020; Yeh et al., 2020; Lage & Doshi-Velez, 2020; Wang et al., 2017) propose explicitly aligning the intermediate layers of a neural network with some pre-deﬁned expert concepts by ﬁrst training a predictor of concepts and subsequently using this to predict the label (Figure 1 left).

Original: CBMs are examples of interpretablemodels that are well-suited for human-AI joint decision making.
Revised: CBMs are examples of interpretable models that are well-suited for human-AI joint decision making.

Original: A human supervisor can interpretand understand the label predictions by inspecting the concept predictions.
Revised: A human supervisor can interpret and understand the label predictions by inspecting the concept predictions.

Original: Moreover, when thesupervisor notices a misprediction among the concepts, they may intervene by correcting the conceptprediction and, in turn, the model is able to update its ﬁnal prediction accounting for the revisedconcept.
Revised: Moreover, when the supervisor notices a misprediction among the concepts, they may intervene by correcting the concept prediction and, in turn, the model is able to update its ﬁnal prediction accounting for the revised concept.

Original: Typically when training a CBM, the concept predictor and the label predictor are trained independentlyon ground-truth concepts.
Revised: Typically when training a CBM, the concept predictor and the label predictor are trained independently on ground-truth concepts.

Original: Unfortunately, these models can underperform in predictive performancecompared to a black-box predictor.
Revised: Unfortunately, these models can underperform in predictive performance compared to a black-box predictor.

Original: Instead, the most predictive CBMs use soft concepts, meaningthat the inputs to the label predictor are the concept probabilities (the numerical values between 0 and1) as predicted by the concept predictor (Koh et al., 2020).
Revised: Instead, the most predictive CBMs use soft concepts, meaning that the inputs to the label predictor are the concept probabilities (the numerical values between 0 and 1) as predicted by the concept predictor (Koh et al., 2020).

Original: Soft concepts rely on this information-richrepresentation, which is in contrast with hard concepts, where label predictor only accepts binaryconcepts.
Revised: Soft concepts rely on this information-rich representation, which is in contrast with hard concepts, where label predictor only accepts binary concepts.

Original: While soft concepts may improve predictive performance, this improvement comes at a cost.

Original: Theissue is that soft concepts allow the concept predictor to convey unintended information about thelabel that would otherwise not be available to the label predictor (Mahinpei et al., 2021; Margeloiuet al., 2021).

Original: For example, in an animal classiﬁer, when the concept predictor recognizes an image asa cat, it may predict a slightly higher likelihood to the concept ‘fur’ than the concept ‘tail’, and whenit recognizes the image as a dog, it may assign higher likelihood to the concept ‘tail’ than the concept

Original: [Figure]
Revised: [Figure]

Original: Carnivore:Two-legged: ..
Revised: Carnivore:Two-legged: ..

Original: Claws: 0.90.
Revised: Claws: 0.90.

Original: Concepts c 1 ..
Revised: Concepts c 1 ..

Original: c K
Revised: c K

Original: Cat: 0.1 T-Rex: 0.
Revised: Cat: 0.1 T-Rex: 0.

Original: Label y 𝜃 𝜙
Revised: Label y 𝜃 𝜙

Original: Carnivore:Two-legged: ..
Revised: Carnivore:Two-legged: ..

Original: Claws: 0.90.
Revised: Claws: 0.90.

Original: Cat: 1.0 T-
Revised: Cat: 1.0 T-

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  ‘fur’.
Revised: While soft concepts may improve predictive performance, this improvement comes at a cost. The issue is that soft concepts allow the concept predictor to convey unintended information about the label that would otherwise not be available to the label predictor (Mahinpei et al., 2021; Margeloiu et al., 2021). For example, in an animal classiﬁer, when the concept predictor recognizes an image as a cat, it may predict a slightly higher likelihood to the concept ‘fur’ than the concept ‘tail’, and when it recognizes the image as a dog, it may assign higher likelihood to the concept ‘tail’ than the concept ‘fur’.

Original: Even though cats and dogs are indistinguishable based on these two concepts alone, the labelpredictor may still compare the two soft concept probabilities and make an accurate classiﬁcation.
Revised: Even though cats and dogs are indistinguishable based on these two concepts alone, the label predictor may still compare the two soft concept probabilities and make an accurate classiﬁcation.

Original: A ﬂexible predictor, such as a deep neural network, can quickly learn such encoding patterns andimprove its prediction using the information leaked by the concept predictor.
Revised: A ﬂexible predictor, such as a deep neural network, can quickly learn such encoding patterns and improve its prediction using the information leaked by the concept predictor.

Original: Thus, the problem isdubbed leakage.
Revised: Thus, the problem is dubbed leakage.

Original: Leakage compromises the interpretability and trustworthiness of the model.
Revised: Leakage compromises the interpretability and trustworthiness of the model.

Original: The concept predictorno longer needs to faithfully predict the concepts to obtain high-quality label predictions.
Revised: The concept predictor no longer needs to faithfully predict the concepts to obtain high-quality label predictions.

Original: Instead, itonly needs to encode the class label in the soft concept probabilities.
Revised: Instead, it only needs to encode the class label in the soft concept probabilities.

Original: A human supervisor can neverbe sure if the model is predicting a certain concept because it is likely to be present, or because it isencoding for something else.
Revised: A human supervisor can never be sure if the model is predicting a certain concept because it is likely to be present, or because it is encoding for something else.

Original: Moreover, since interventions remove this extra information from theconcept predictions, predictive performance conditional on interventions suffers signiﬁcantly.
Revised: Moreover, since interventions remove this extra information from the concept predictions, predictive performance conditional on interventions suffers signiﬁcantly.

Original: How can one avoid leakage?
Revised: How can one avoid leakage?

Original: While hard concepts ensure that information is not leaked, these modelssuffer from poor performance in comparison to their soft counterparts.
Revised: While hard concepts ensure that information is not leaked, these models suffer from poor performance in comparison to their soft counterparts.

Original: In this work, we focus on theperformance disparity between soft and hard CBMs.
Revised: In this work, we focus on the performance disparity between soft and hard CBMs.

Original: Speciﬁcally, we attribute this disparity to (i)having an insufﬁcient concept set and (ii) having an inﬂexible concept predictor.
Revised: Speciﬁcally, we attribute this disparity to (i) having an insufﬁcient concept set and (ii) having an inﬂexible concept predictor.

Original: The former issueoccurs as a result of the Markovian assumption under which CBMs operate: all the information aboutthe label in the input must be captured by the concepts.
Revised: The former issue occurs as a result of the Markovian assumption under which CBMs operate: all the information about the label in the input must be captured by the concepts.

Original: If this assumption is not met, the performanceof the hard CBM is limited by the information available in the concepts.
Revised: If this assumption is not met, the performance of the hard CBM is limited by the information available in the concepts.

Original: The latter issue stems fromthe fact that CBMs predict concepts independently and cannot capture correlations between concepts.
Revised: The latter issue stems from the fact that CBMs predict concepts independently and cannot capture correlations between concepts.

Original: These correlations are important since some combinations of concepts may be unlikely or impossible(eg.
Revised: These correlations are important since some combinations of concepts may be unlikely or impossible (eg.

Original: the concepts ‘fur’ and ‘scales’ are mutually exclusive).
Revised: the concepts ‘fur’ and ‘scales’ are mutually exclusive).

Original: These two issues hinder the performanceof hard CBMs, while soft CBMs are able to mitigate them with leakage, at the cost of interpretabilityand intervenability.
Revised: These two issues hinder the performance of hard CBMs, while soft CBMs are able to mitigate them with leakage, at the cost of interpretability and intervenability.

Original: We introduce two technical contributions that allow hard CBMs to match the predictive performanceof soft CBMs without compromising their resilience to leakage.
Revised: We introduce two technical contributions that allow hard CBMs to match the predictive performance of soft CBMs without compromising their resilience to leakage.

Original: First, we circumvent the Markovianassumption by introducing a side-channel to learn a set of latent concepts alongside the known concepts that are needed for accurate predictions.
Revised: First, we circumvent the Markovian assumption by introducing a side-channel to learn a set of latent concepts alongside the known concepts that are needed for accurate predictions.

Original: Second, we introduce an autoregressive conﬁgurationfor the concept predictor that signiﬁcantly improves their ﬂexibility and it allows them to capturecorrelations in the concepts.
Revised: Second, we introduce an autoregressive conﬁguration for the concept predictor that signiﬁcantly improves their ﬂexibility and it allows them to capture correlations in the concepts.

Original: On several synthetic and real applications, we demonstrate that our side-channel models improvethe performance of hard CBMs in terms of accuracy, without affecting concept predictions orintervenability.
Revised: On several synthetic and real applications, we demonstrate that our side-channel models improve the performance of hard CBMs in terms of accuracy, without affecting concept predictions or intervenability.

Original: In addition, our autoregressive concept predictor signiﬁcantly improves both conceptaccuracy and intervention accuracy compared to soft CBMs and non-autoregressive hard CBMs.
Revised: In addition, our autoregressive concept predictor signiﬁcantly improves both concept accuracy and intervention accuracy compared to soft CBMs and non-autoregressive hard CBMs.

Original: Summary of contributions:
Revised: Summary of contributions:

Original: • In our experiments, we demonstrate that leakage is detrimental to interpretability andintervenability in soft CBMs.
Revised: • In our experiments, we demonstrate that leakage is detrimental to interpretability and intervenability in soft CBMs.

Original: • Hard CBMs are resilient to leakage, but their performance is hindered by not meetingthe Markovian assumption and having an inﬂexible concept predictor.
Revised: • Hard CBMs are resilient to leakage, but their performance is hindered by not meeting the Markovian assumption and having an inﬂexible concept predictor.

Original: We address theseshortcomings:– We circumvent the Markovian assumption by introducing a side-channel to learn a setof latent concepts.
Revised: We address these shortcomings: – We circumvent the Markovian assumption by introducing a side-channel to learn a set of latent concepts.

Original: – We increase the ﬂexibility of the concept predictor by adapting an autoregressivearchitecture.
Revised: – We increase the ﬂexibility of the concept predictor by adapting an autoregressive architecture.

Original:  Background: Concept Bottleneck Models
Revised:  Background: Concept Bottleneck Models

Original: Consider a dataset D
Revised: Consider a dataset D

Original: = { x ( i ) , c
Revised: = { x ( i ) , c

Original: ( i ) , y ( i ) }
Revised: ( i ) , y ( i ) }

Original: Ni =1 consisting of N triplets of the form: input x ∈ R d ,concept vector c ∈ { 0 , 1 } K of K binary concepts, and class label y ∈ N
Revised: Ni =1 consisting of N triplets of the form: input x ∈ R d , concept vector c ∈ { 0 , 1 } K of K binary concepts, and class label y ∈ N

Original: (we consider classiﬁcation84 problems, but our results extend to regression problems where y ∈ R ).
Revised: (we consider classiﬁcation problems, but our results extend to regression problems where y ∈ R ).

Original: Formally, a CBM is comprisedof a concept predictor, parametrized by θ , which is trained to predict the concept vector from theinput p θ ( c | x ) , and a label predictor, parametrized by φ , subsequently trained to predict the label fromthe concepts p φ ( y | c ) (Koh et al., 2020)
Revised: Formally, a CBM is comprised of a concept predictor, parametrized by θ , which is trained to predict the concept vector from the input p θ ( c | x ) , and a label predictor, parametrized by φ , subsequently trained to predict the label from the concepts p φ ( y | c ) (Koh et al., 2020)

Original: (Figure 1 left).
Revised: (Figure 1 left).

Original: Interpretability and Interventions There are two key beneﬁts to CBMs that make them wellsuited for human-AI joint decision making.
Revised: Interpretability and Interventions There are two key beneﬁts to CBMs that make them wellsuited for human-AI joint decision making.

Original: First, they explain their predictions by showing whichhigh-level concepts the concept predictor was able to ﬁnd in the data.
Revised: First, they explain their predictions by showing which high-level concepts the concept predictor was able to ﬁnd in the data.

Original: Second, a human operator mayintervene on these concepts to see how changing the concept values affects the ﬁnal prediction.
Revised: Second, a human operator may intervene on these concepts to see how changing the concept values affects the ﬁnal prediction.

Original: Thisis done by setting the probabilities of the concepts to 1 or 0 to indicate their presence or absence.
Revised: This is done by setting the probabilities of the concepts to 1 or 0 to indicate their presence or absence.

Original: Thelabel predictor is able to adjust its prediction to account for the intervention (Figure 1 left).
Revised: The label predictor is able to adjust its prediction to account for the intervention (Figure 1 left).

Original: Hard CBMs In hard CBMs, the label predictor only accepts binary concepts as inputs.
Revised: Hard CBMs In hard CBMs, the label predictor only accepts binary concepts as inputs.

Original: Theconcept predictor and the label predictor are trained independently on the ground truth data: [Equation] p θ ( c | x )] and ˆ φ = arg max φ E D [log p φ ( y | c )]
Revised: The concept predictor and the label predictor are trained independently on the ground truth data: ˆ θ = arg max θ E D [log p θ ( c | x )] and ˆ φ = arg max φ E D [log p φ ( y | c )]

Original: (here, we omit regularizers forbrevity).
Revised: (here, we omit regularizers for brevity).

Original: To make predictions on a new input point, one must marginalize out the possible concepts:p
Revised: To make predictions on a new input point, one must marginalize out the possible concepts: p θφ ( y | x ) =

Original: [Equation] [Equation]
Revised: E c ∼ p θ ( c | x ) [ p φ ( y | c )] .

Original: Soft CBMs Soft CBMs, on the other hand, pass the concept probabilities (the value in [0 , 1] )directly to the label predictor.
Revised: Soft CBMs Soft CBMs, on the other hand, pass the concept probabilities (the value in [0 , 1] ) directly to the label predictor.

Original: When encountering a new input x , a soft concept model ﬁrst calculatesthe output of concept predictor g ( x ) where σ ( g ( x ) k )
Revised: When encountering a new input x , a soft concept model ﬁrst calculates the output of concept predictor g ( x ) where σ ( g ( x ) k )

Original: =
Revised: =

Original: p θ ( c k = 1 | x ) for k = 1 . . .
Revised: p θ ( c k = 1 | x ) for k = 1 . . .

Original: K .
Revised: K .

Original: Theseprobabilities are then forwarded to the label predictor, whose output is f ( c ) =
Revised: These probabilities are then forwarded to the label predictor, whose output is f ( c ) =

Original: p φ ( y | c ) .
Revised: p φ ( y | c ) .

Original: Combined,they produce the ﬁnal prediction f
Revised: Combined, they produce the ﬁnal prediction f

Original: ( g ( x ))
Revised: ( g ( x ))

Original: =
Revised: =

Original: p θφ ( y | x ) .
Revised: p θφ ( y | x ) .

Original: Three different training strategies have been proposed to train soft CBMs (Koh et al., 2020).
Revised: Three different training strategies have been proposed to train soft CBMs (Koh et al., 2020).

Original: First,independent training , where the concept predictor and the label predictor are trained independentlyˆ θ = arg max θ E D [log p θ ( c | x )] and ˆ φ = arg max φ E D [log p φ ( y | c )] .
Revised: First, independent training , where the concept predictor and the label predictor are trained independently ˆ θ = arg max θ E D [log p θ ( c | x )] and ˆ φ = arg max φ E D [log p φ ( y | c )] .

Original: Second, sequential training ,where the label predictor is trained after the concept predictor ˆ θ = arg max θ E D [log p θ ( c | x )] andˆ φ = arg max φ E D [log p θφ ( y | x )] .
Revised: Second, sequential training , where the label predictor is trained after the concept predictor ˆ θ = arg max θ E D [log p θ ( c | x )] and ˆ φ = arg max φ E D [log p θφ ( y | x )] .

Original: Finally, joint training , where the concept predictor and the labelpredictor are trained jointly ˆ θ, ˆ φ = arg max θφ E D
Revised: Finally, joint training , where the concept predictor and the label predictor are trained jointly ˆ θ, ˆ φ = arg max θφ E D

Original: [ λ log p θ ( c | x ) + log p θφ ( y | x )] .
Revised: [ λ log p θ ( c | x ) + log p θφ ( y | x )] .

Original: Here, 0 ≤ λ is ascalar hyperparameter controlling the tradeoff between the two terms.
Revised: Here, 0 ≤ λ is a scalar hyperparameter controlling the tradeoff between the two terms.

Original: High λ prioritizes accurateconcept predictions, while λ close to 0 focuses on accurate label predictions.
Revised: High λ prioritizes accurate concept predictions, while λ close to 0 focuses on accurate label predictions.

Original: Leakage Leakage occurs in soft CBMs when trained sequentially or jointly (Mahinpei et al.,2021).
Revised: Leakage Leakage occurs in soft CBMs when trained sequentially or jointly (Mahinpei et al., 2021).

Original: The label predictor learns to utilize the additional, unintended information in the soft conceptprobabilities output by the concept predictor.
Revised: The label predictor learns to utilize the additional, unintended information in the soft concept probabilities output by the concept predictor.

Original: In the animal classiﬁcation example, when trained jointly, the concept predictor can learn to assignslightly higher likelihood to the concept ‘fur’ than the concept ‘tail’ when it recognizes the image asa cat and predict higher likelihood to the concept ‘tail’ than the concept ‘fur’ when it recognizes theimage as a dog.
Revised: In the animal classiﬁcation example, when trained jointly, the concept predictor can learn to assign slightly higher likelihood to the concept ‘fur’ than the concept ‘tail’ when it recognizes the image as a cat and predict higher likelihood to the concept ‘tail’ than the concept ‘fur’ when it recognizes the image as a dog.

Original: Despite that cats and dogs are indistinguishable based on these two concepts, the label predictor can compare the two soft concept probabilities and make an accurate classiﬁcation.
Revised: Despite that cats and dogs are indistinguishable based on these two concepts, the label predictor can compare the two soft concept probabilities and make an accurate classiﬁcation.

Original: Thelabel predictor is no longer limited by set of the pre-deﬁned concepts: the concept predictor is ableleak further information about the label to improve the predictive performance.
Revised: The label predictor is no longer limited by set of the pre-deﬁned concepts: the concept predictor is able leak further information about the label to improve the predictive performance.

Original: This decreases theoverall loss: the concept log-likelihood log p θ ( c | x ) decreases slightly due to the slight perturbation,but the label log-likelihood log p θφ ( y | x ) increases signiﬁcantly since the label predictor now hasaccess to the leaked information.
Revised: This decreases the overall loss: the concept log-likelihood log p θ ( c | x ) decreases slightly due to the slight perturbation, but the label log-likelihood log p θφ ( y | x ) increases signiﬁcantly since the label predictor now has access to the leaked information.

Original: This is further exacerbated by using λ < 1 , which prioritizes thelabel log-likelihood over the concept log-likelihood in the training objective.
Revised: This is further exacerbated by using λ < 1 , which prioritizes the label log-likelihood over the concept log-likelihood in the training objective.

Original: Leakage damages both interpretability and intervenability.
Revised: Leakage damages both interpretability and intervenability.

Original: Interpretability suffers because theexplanation of the label is based not only on the concepts, but also on the behavior of the speciﬁcconcept predictor.
Revised: Interpretability suffers because the explanation of the label is based not only on the concepts, but also on the behavior of the speciﬁc concept predictor.

Original: The human operator would need an in-depth knowledge of the concept predictorto understand why a small change in the likelihood of ‘fur’ leads to a signiﬁcant change in theclassiﬁcation.
Revised: The human operator would need an in-depth knowledge of the concept predictor to understand why a small change in the likelihood of ‘fur’ leads to a signiﬁcant change in the classiﬁcation.

Original: Interventions are also adversely affected.
Revised: Interventions are also adversely affected.

Original: When one intervenes on the ‘fur’ concept bysetting its probability to 1 , it can decrease the predictive accuracy because this removes all informationencoded in the soft probability value.
Revised: When one intervenes on the ‘fur’ concept by setting its probability to 1 , it can decrease the predictive accuracy because this removes all information encoded in the soft probability value.

Original: Hard CBMs avoid leakage by only passing binary concepts to the label predictor.
Revised: Hard CBMs avoid leakage by only passing binary concepts to the label predictor.

Original: The conceptpredictor and the label predictor learn to faithfully predict the concepts from the inputs and the labelsfrom the concepts respectively.
Revised: The concept predictor and the label predictor learn to faithfully predict the concepts from the inputs and the labels from the concepts respectively.

Original:  Related works
Revised:  Related works

Original: Concept Activation Vector and Whitening Methods Several methods exist to interpret the layersof a neural network and what they learn.
Revised: Concept Activation Vector and Whitening Methods Several methods exist to interpret the layers of a neural network and what they learn.

Original: Among these, Kim et al.
Revised: Among these, Kim et al.

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: (2018); Ghorbaniet al.
Revised: ; Ghorbani et al.

Original: (2019) use concept activation vector methods to analyse networks that have been pre-trained.
Revised: (2019) use concept activation vector methods to analyse networks that have been pre-trained.

Original: The concept activation vectors are chosen to align with pre-deﬁned or automatically-discoveredconcepts.
Revised: The concept activation vectors are chosen to align with pre-deﬁned or automatically-discovered concepts.

Original: These vectors are based on the assumption that the latent space can be analysed posthocin a speciﬁc form.
Revised: These vectors are based on the assumption that the latent space can be analysed posthoc in a speciﬁc form.

Original: Other methods such as concept whitening (Chen et al., 2020) directly alter agiven layer of a network by adding a module that aligns the latent space to certain axes of interestto understand the computations leading to that layer.
Revised: Other methods such as concept whitening (Chen et al., 2020) directly alter a given layer of a network by adding a module that aligns the latent space to certain axes of interest to understand the computations leading to that layer.

Original: Unlike our work, none of the former methodsaddress the fundamental issue of leakage in CBMs that signiﬁcantly impacts the interpretability andintervenability of these models.
Revised: Unlike our work, none of the former methods address the fundamental issue of leakage in CBMs that signiﬁcantly impacts the interpretability and intervenability of these models.

Original: Soft Concept Bottleneck Models Soft CBMs (Chen et al., 2020; Koh et al., 2020) propose passingconcept probabilities (value in [0 , 1] ) to the label predictor to signiﬁcantly improve prediction accuracyin comparison to CBMs that learn hard concepts.
Revised: Soft Concept Bottleneck Models Soft CBMs (Chen et al., 2020; Koh et al., 2020) propose passing concept probabilities (value in [0 , 1] ) to the label predictor to signiﬁcantly improve prediction accuracy in comparison to CBMs that learn hard concepts.

Original: Though performant, several works show that slightlydistorting the predictive concept probabilities enables the concept predictor to ‘leak’ the ﬁnal labelto the label predictor (Mahinpei et al., 2021; Margeloiu et al., 2021).
Revised: Though performant, several works show that slightly distorting the predictive concept probabilities enables the concept predictor to ‘leak’ the ﬁnal label to the label predictor (Mahinpei et al., 2021; Margeloiu et al., 2021).

Original: We propose tools that allowus to diagnose and address these issues to directly improve the performance of hard CBMs, therebyovercoming the issue of information leakage in their soft counterparts.
Revised: We propose tools that allow us to diagnose and address these issues to directly improve the performance of hard CBMs, thereby overcoming the issue of information leakage in their soft counterparts.

Original: Information Bottleneck Methods IB methods take a joint distribution p ( X, Y ) and compress theinputs X into a latent representation Z that is most informative about the target
Revised: Information Bottleneck Methods IB methods take a joint distribution p ( X, Y ) and compress the inputs X into a latent representation Z that is most informative about the target

Original: Y
Revised: Y

Original: (Tishby et al.
Revised: (Tishby et al., 2000).

Original: This latent representation may be thought of as a type of concept.
Revised: This latent representation may be thought of as a type of concept.

Original: Several methods have beenproposed to use such a concept representation Z to compute the effect of an intervention (Parbhooet al., 2020; Chicharro et al., 2020).
Revised: Several methods have been proposed to use such a concept representation Z to compute the effect of an intervention (Parbhoo et al., 2020; Chicharro et al., 2020).

Original: Both the traditional IB and the deep variational IB formulationsassume speciﬁc Markov structures for the generative model (Alemi et al., 2016; Wieczorek & Roth,2020).
Revised: Both the traditional IB and the deep variational IB formulations assume speciﬁc Markov structures for the generative model (Alemi et al., 2016; Wieczorek & Roth, 2020).

Original: In contrast, our work uses side-channels to learn informative concepts that are predictive ofthe target while compressing the inputs, and is independent of such Markovian assumptions.
Revised: In contrast, our work uses side-channels to learn informative concepts that are predictive of the target while compressing the inputs, and is independent of such Markovian assumptions.

Original: Causal Concept Effect Models Goyal et al.
Revised: Causal Concept Effect Models Goyal et al.

Original: ; Feder et al.
Revised: ; Feder et al.

Original: (2021) explain coupling a CBMwith a causal model and computing the Causal Concept Effect to speculate about what might happento the prediction had a concept been different.
Revised: (2021) explain coupling a CBM with a causal model and computing the Causal Concept Effect to speculate about what might happen to the prediction had a concept been different.

Original: Unlike this work, we examine only the statisticalsetting where we can intervene to correct or modify an incorrect concept prediction, but not tospeculate about counterfactuals since the model does not capture causal relationships.
Revised: Unlike this work, we examine only the statistical setting where we can intervene to correct or modify an incorrect concept prediction, but not to speculate about counterfactuals since the model does not capture causal relationships.

Original: Concept completeness Yeh et al.
Revised: Concept completeness Yeh et al.

Original: (2019) study when a set of concepts is sufﬁcient for explainingthe prediction in a deep neural network.
Revised: (2020) study when a set of concepts is sufﬁcient for explaining the prediction in a deep neural network.

Original: They propose an accuracy-based completeness score.
Revised: They propose an accuracy-based completeness score.

Original: Theirscore, however, does not imply that the concepts capture all information in the input i.e. the Markovian assumption.
Revised: Their score, however, does not imply that the concepts capture all information in the input i.e. the Markovian assumption.

Original: Our information-based completeness score indicates precisely when the assumption ismet.
Revised: Our information-based completeness score indicates precisely when the assumption is met.

Original:  Addressing Two Shortcomings of Concept Bottleneck Models
Revised:  Addressing Two Shortcomings of Concept Bottleneck Models

Original: In this section, we explain and address the performance disparity between soft CBMs and hard CBMs.
Revised: In this section, we explain and address the performance disparity between soft CBMs and hard CBMs.

Original: We present two sources of information that CBMs are unable to capture: information that is notavailable in the concepts and information in the correlations in the concepts.
Revised: We present two sources of information that CBMs are unable to capture: information that is not available in the concepts and information in the correlations in the concepts.

Original: Soft CBMs can lessenthe impact through leakage, but hard CBMs cannot.
Revised: Soft CBMs can lessen the impact through leakage, but hard CBMs cannot.

Original: We propose modiﬁcations to hard CBMs thatallow the use of information from these two sources while retaining their resilience to leakage.
Revised: We propose modiﬁcations to hard CBMs that allow the use of information from these two sources while retaining their resilience to leakage.

Original: 4.1 The Markovian Assumption
Revised: 4.1 The Markovian Assumption

Original: First, we consider the case when the concepts are insufﬁcient for predicting the ﬁnal label, but xhas more information that can help.
Revised: First, we consider the case when the concepts are insufﬁcient for predicting the ﬁnal label, but x has more information that can help.

Original: We refer to this as the lack of a Markovian assumption.
Revised: We refer to this as the lack of a Markovian assumption.

Original: Whenthe Markovian assumption holds, the chain x → c → y is Markovian, which is deﬁned as y beingconditionally independent of x given c : I( y ; x | c ) = 0 , where I denotes the mutual information.
Revised: When the Markovian assumption holds, the chain x → c → y is Markovian, which is deﬁned as y being conditionally independent of x given c : I( y ; x | c ) = 0 , where I denotes the mutual information.

Original: When the assumption holds, c must contain all information about y that is present in x .
Revised: When the assumption holds, c must contain all information about y that is present in x .

Original: Otherwise,c is insufﬁcient for predicting y ; therefore, a CBM that relies on hard concepts will underperformcompared to a black box x → y predictor.
Revised: Otherwise, c is insufﬁcient for predicting y ; therefore, a CBM that relies on hard concepts will underperform compared to a black box x → y predictor.

Original: In what follows, we describe how CBMs can still beapplied with the help of a side-channel and deﬁne a concept completeness score to diagnose whenthe Markovian assumption may not hold.
Revised: In what follows, we describe how CBMs can still be applied with the help of a side-channel and deﬁne a concept completeness score to diagnose when the Markovian assumption may not hold.

Original: 4.1.1 Using a Side-channel
Revised: 4.1.1 Using a Side-channel

Original: Along with the known concepts c , we model L unknown concepts z ∈ { 0 , 1 } L in a side-channel
Revised: Along with the known concepts c , we model L unknown concepts z ∈ { 0 , 1 } L in a side-channel

Original: CBM (Figure 1 middle).
Revised: CBM (Figure 1 middle).

Original: This setup allows the model to capture information about y present in x butnot present in c via the latent concepts.
Revised: This setup allows the model to capture information about y present in x but not present in c via the latent concepts.

Original: Training the side-channel model is analogous to training a hard CBM.
Revised: Training the side-channel model is analogous to training a hard CBM.

Original: We optimize the predictivelog-likelihood using stochastic gradient ascent:
Revised: We optimize the predictive log-likelihood using stochastic gradient ascent:

Original: [Equation]
Revised: [Equation]

Original: A caveat is that the gradient of E p θ ( z | x ) [ p φ ( y | c, z )] need to be estimated using a gradient estimator.
Revised: A caveat is that the gradient of E p θ ( z | x ) [ p φ ( y | c, z )] need to be estimated using a gradient estimator.

Original: We use the reinforce estimator (Williams, 1992) because it is able to give an unbiased estimate of thegradients, and it works well in a low dimensional settings such as z :
Revised: We use the reinforce estimator (Williams, 1992) because it is able to give an unbiased estimate of the gradients, and it works well in a low dimensional settings such as z :

Original: [Equation]
Revised: [Equation]

Original: The gradient is then estimated using M Monte-Carlo samples:
Revised: The gradient is then estimated using M Monte-Carlo samples:

Original: [Equation]
Revised: [Equation]

Original: At prediction time, we can decide whether to use the side-channel for prediction or rely purely on theknown concepts.
Revised: At prediction time, we can decide whether to use the side-channel for prediction or rely purely on the known concepts.

Original: If the side-channel is used, the predictions are calculated using M Monte-Carlosamples:
Revised: If the side-channel is used, the predictions are calculated using M Monte-Carlo samples:

Original: [Equation]
Revised: [Equation]

Original: In applications where the label prediction must be fully explained by the concepts, we have tomarginalize out the side-channel.
Revised: In applications where the label prediction must be fully explained by the concepts, we have to marginalize out the side-channel.

Original: This requires estimating p θ ( z | c ) .
Revised: This requires estimating p θ ( z | c ) .

Original: One can obtain an estimate usingp θ ( z | c ) ≈ E c (cid:48) ,x (cid:48)
Revised: One can obtain an estimate using p θ ( z | c ) ≈ E c (cid:48) ,x (cid:48)

Original: ∼D & c (cid:48)
Revised: ∼D & c (cid:48)

Original: =
Revised: =

Original: c [ p θ ( z | x (cid:48) )] ; however, this requires iterating over the whole dataset each time204 a prediction is made.
Revised: c [ p θ ( z | x (cid:48) )] ; however, this requires iterating over the whole dataset each time a prediction is made.

Original: Our solution is to train a small (1 hidden layer) predictor that can rapidly estimate p θ ( z | c ) from c .
Revised: Our solution is to train a small (1 hidden layer) predictor that can rapidly estimate p θ ( z | c ) from c .

Original: We call this the amortization network, since it amortizes the cost of inferring z from c .
Revised: We call this the amortization network, since it amortizes the cost of inferring z from c .

Original: We denote its parameters γ , and train it using maximum-likelihood training: ˆ γ = arg max γ E D [log p γ ( z | c )] .
Revised: We denote its parameters γ , and train it using maximum-likelihood training: ˆ γ = arg max γ E D [log p γ ( z | c )] .

Original: Inour experiments, the amortization network is trained over 10-20 epochs depending on the dataset.
Revised: In our experiments, the amortization network is trained over 10-20 epochs depending on the dataset.

Original: Equipped with γ , we can make predictions without using the side-channel:
Revised: Equipped with γ , we can make predictions without using the side-channel:

Original: [Equation]
Revised: [Equation]

Original: A second beneﬁt marginalizing the side-channel with the amortization network is that it can help usunderstand how much the model relies on the side-channel for prediction.
Revised: A second beneﬁt marginalizing the side-channel with the amortization network is that it can help us understand how much the model relies on the side-channel for prediction.

Original: To formalize this concept,we introduce a completeness score.
Revised: To formalize this concept, we introduce a completeness score.

Original: A Completeness Score
Revised: A Completeness Score

Original: The side-channel model can be used to estimate what fraction of therelevant information is present in c .
Revised: The side-channel model can be used to estimate what fraction of the relevant information is present in c .

Original: This is useful for diagnosing when a side-channel is needed andto understand how much the model relies on it.
Revised: This is useful for diagnosing when a side-channel is needed and to understand how much the model relies on it.

Original: If the model heavily relies on the side channel, itindicates that key concepts are missing from c .
Revised: If the model heavily relies on the side channel, it indicates that key concepts are missing from c .

Original: We deﬁne
Revised: We deﬁne

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 0 ≤ C ≤ 1 , because its a ratio of non-negative quantities and the nominator is less-than-or-equalto the denominator since I( y ; c, x ) − I( y ; c ) =
Revised: 0 ≤ C ≤ 1 , because its a ratio of non-negative quantities and the nominator is less-than-or-equal to the denominator since I( y ; c, x ) − I( y ; c ) =

Original: I( y ; x | c ) ≥ 0 (Cover, 1999).
Revised: I( y ; x | c ) ≥ 0 (Cover, 1999).

Original: When C = 1 , we have
Revised: When C = 1 , we have

Original: I( y ; c ) =
Revised: I( y ; c ) =

Original: I( y ; c, x ) , so I( y ; c, x ) − I( y ; c ) =
Revised: I( y ; c, x ) , so I( y ; c, x ) − I( y ; c ) =

Original: I( y ; x | c ) = 0 thus the Markovian assumption holds, andc captures all relevant information.
Revised: I( y ; x | c ) = 0 thus the Markovian assumption holds, and c captures all relevant information.

Original: When C = 

Revised: When C =

Original: ( I( y ; c ) = 

Revised: ( I( y ; c ) =

Original: i.e. c and y are independent), noneof the information in c is relevant for predicting y .
Revised: i.e. c and y are independent), none of the information in c is relevant for predicting y .

Original: With the help of the amortization network, it ispossible to estimate C :
Revised: With the help of the amortization network, it is possible to estimate C :

Original: [Equation]
Revised: [Equation]

Original: Interpretability of the Side-channel The latent concepts captured by the side channel can beinspected for interpretability.
Revised: Interpretability of the Side-channel The latent concepts captured by the side channel can be inspected for interpretability.

Original: They may align with human understandable concepts (in which case,they can be added to c ), but it is possible that they are uninterpretable.
Revised: They may align with human understandable concepts (in which case, they can be added to c ), but it is possible that they are uninterpretable.

Original: The side-channel model shouldstill be used to avoid the detrimental effects of leakage on concept accuracy and interventions.
Revised: The side-channel model should still be used to avoid the detrimental effects of leakage on concept accuracy and interventions.

Original: Notethat we have agency on deciding whether to use the side channel information at prediction time ornot.
Revised: Note that we have agency on deciding whether to use the side channel information at prediction time or not.

Original: Including the side-channel information is ideal for applications where accuracy is high-priorityand partial explanations are sufﬁcient (Equation 4).
Revised: Including the side-channel information is ideal for applications where accuracy is high-priority and partial explanations are sufﬁcient (Equation 4).

Original: For applications where the human operator mustknow the full set of concepts contributing to the prediction, we can make predictions without it (bymarginalizing z with the help of the amortization network as shown in Equation 5).
Revised: For applications where the human operator must know the full set of concepts contributing to the prediction, we can make predictions without it (by marginalizing z with the help of the amortization network as shown in Equation 5).

Original: 4.2 The Expressivity of the Concept Predictor
Revised: 4.2 The Expressivity of the Concept Predictor

Original: The second source of information that CBMs are unable to capture is correlations in the concepts.
Revised: The second source of information that CBMs are unable to capture is correlations in the concepts.

Original: An issue with the usual formulation of a CBM is that the concept predictor predicts the conceptsindependently (conditional on the input).
Revised: An issue with the usual formulation of a CBM is that the concept predictor predicts the concepts independently (conditional on the input).

Original: This may give poor performance when the concepts arecorrelated.
Revised: This may give poor performance when the concepts are correlated.

Original: Consider two concepts that are mutually exclusive.
Revised: Consider two concepts that are mutually exclusive.

Original: A predictor that predicts themindependently does not have the ﬂexibility to express their mutual exclusivity and it inadvertentlypredicts non-zero probability of both concepts being present.
Revised: A predictor that predicts them independently does not have the ﬂexibility to express their mutual exclusivity and it inadvertently predicts non-zero probability of both concepts being present.

Original: While a soft CBM can learn duringtraining that the concept probabilities correspond to mutually exclusive concepts, a hard CBM cannot.
Revised: While a soft CBM can learn during training that the concept probabilities correspond to mutually exclusive concepts, a hard CBM cannot.

Original: We propose a modiﬁcation to the CBM architecture that allows hard CBMs to capture correlations inthe concepts.
Revised: We propose a modiﬁcation to the CBM architecture that allows hard CBMs to capture correlations in the concepts.

Original: Our solution is to use an autoregressive architecture for concept prediction.
Revised: Our solution is to use an autoregressive architecture for concept prediction similarly to classiﬁer chains in multilabel classiﬁcation (Read et al., 2011; Dembczynski et al., 2010).

Original: To predictthe k -th concept, use not only the input x , but also the already predicted concepts c 1: k − 1 1 (Figure 1right).
Revised: To predict the k -th concept, use not only the input x , but also the already predicted concepts c 1: k − 1 1 (Figureright).

Original: The log-likelihood, which is needed for training, can be computed by iterating over the concepts:log p θ ( c | x ) =
Revised: The log-likelihood, which is needed for training, can be computed by iterating over the concepts: log p [Equation] ) =

Original: (cid:80)
Revised: [Equation]

Original: Kk
Revised: p θ

Original: = p θ ( c k | x, c 1: k − 1 ) .
Revised: [Equation] k | x [Equation] [Equation]

Original: To sample a concept set, which is needed for prediction,246 we must sample the concepts iteratively: 
Revised: To sample a concept set, which is needed for prediction, we must sample the concepts iteratively: c

Revised: [Equation]

Original: Lastly, interventions require a modiﬁed sampling algorithm.
Revised: Lastly, interventions require a modiﬁed sampling algorithm.

Original: In autoregressive models, we cannot easily generate Monte-Carlo samples from the concept distribution conditional on a set of interventions.
Revised: In autoregressive models, we cannot easily generate Monte-Carlo samples from the concept distribution conditional on a set of interventions.

Original: Instead, to estimate p θ,φ (ˆ y | ˆ x, { ˆ c i } I ) (where { ˆ c i } i ∈I⊆{ 1 ...K } denotes the set of intervened concepts),250 we propose a normalized importance sampling algorithm that generates M weighted concept samplesfor prediction
Revised: Instead, to estimate p θ,φ (ˆ y | ˆ x, { ˆ c i } I ) (where { ˆ c i } i ∈I⊆{ 1 ...K } denotes the set of intervened concepts), we propose a normalized importance sampling algorithm that generates M weighted concept samples for prediction

Original: [Equation]
Revised: [Equation]

Original: Our proposal distribution c (1) . . .
Revised: Our proposal distribution c (1) . . .

Original: c ( M ) ∼ q ( c )matches the interventions { ˆ c
Revised: c ( M ) ∼ q ( c ) matches the interventions { ˆ c

Original: i
Revised: i

Original: } I when the concepts are intervened on and selects a random concept253 conditional on the previous concepts when they are not intervened on:
Revised: } I when the concepts are intervened on and selects a random concept conditional on the previous concepts when they are not intervened on:

Original: [Equation]
Revised: [Equation]

Original:  where δ denotes Kronecker’s delta function.
Revised: ) where δ denotes Kronecker’s delta function.

Original: To obtain an importance weighted estimate, we mustcompute the weight of each of the M samples w m = p θ ( c ( m )
Revised: To obtain an importance weighted estimate, we must compute the weight of each of the M samples w m = p θ ( c ( m )

Original: | ˆ x, { ˆ c i } )
Revised: | ˆ x, { ˆ c i } )

Original: q ( c ( m ) ) .
Revised: q ( c ( m ) ) .

Original: This is done by iterating overthe K concepts:
Revised: This is done by iterating over the K concepts:

Original: [Equation]
Revised: [Equation]

Original: since the probability mass ratio p θ ( c ( m ) k | x,c ( m ) 1: k − 1 ) q ( c ( m ) k | x,c ( m ) 1: k − 1 ) = 1 when k / ∈ I .
Revised: since the probability mass ratio p θ ( c ( m ) k | x,c ( m ) 1: k − 1 ) q ( c ( m ) k | x,c ( m ) 1: k − 1 ) = 1 when k / ∈ I .

Original: Finally, we obtain the importance weighted estimate
Revised: Finally, we obtain the importance weighted estimate

Original: [Equation]
Revised: [Equation]

Original: Note that due to this re-weightingscheme, interventions are able to affect concept predictions of earlier concepts.
Revised: Note that due to this re-weighting scheme, interventions are able to affect concept predictions of earlier concepts.

Original: We also show thisexperimentally in Appendix E. Normalization is important to ensure the predictive distribution isproper.
Revised: We also show this experimentally in Appendix E. Normalization is important to ensure the predictive distribution is proper.

Original: Psudocode for the interventions is shown in Appendix D.
Revised: Psudocode for the interventions is shown in Appendix D.

Original:  Results and Discussion
Revised:  Results and Discussion

Original: We present empirical results showcasing the efﬁcacy of our proposed modiﬁcations to conceptbottleneck models (CBM).
Revised: We present empirical results showcasing the efﬁcacy of our proposed modiﬁcations to concept bottleneck models (CBM).

Original: 5.1 Datasets and Models
Revised: 5.1 Datasets and Models

Original: We used two datasets in our experiment.
Revised: We used two datasets in our experiment.

Original: First, an Early Warning Score (EWS) prediction taskbased on electronic health records data that simulates a potential use case.
Revised: First, an Early Warning Score (EWS) prediction task based on electronic health records data that simulates a potential use case.

Original: Second, to showcasethe versatility of CBMs and compare to prior work, we apply our method to an image-based birdrecognition task.
Revised: Second, to showcase the versatility of CBMs and compare to prior work, we apply our method to an image-based bird recognition task.

Original: (Our source code is available in the supplementary material.
Revised: Our code is available at https://github.com/dtak/addressing-leakage .

Original: We will be including alink to our public repository in the de-anonymized version.)

Original: MIMIC-III EWS (Johnson et al., 2016)
Revised: MIMIC-III EWS (Johnson et al., 2016)

Original: The task is to predict an EWS for patients at a hospitalthat informs the medical professionals about abnormalities in the vital signs that often precededeterioration and cardiac arrests (Subbe et al., 2001).
Revised: The task is to predict an EWS for patients at a hospital that informs the medical professionals about abnormalities in the vital signs that often precede deterioration and cardiac arrests (Subbe et al., 2001).

Original: The concepts (‘Mild hypotension’, ‘Severefever’ etc.) provide justiﬁcation for the score.
Revised: The concepts (‘Mild hypotension’, ‘Severe fever’ etc.) provide justiﬁcation for the score.

Original: The EWS ranges from 0 to 15, and it is based on thepatient’s vitals.
Revised: The EWS ranges from 0 to 15, and it is based on the patient’s vitals.

Original: Deviations from the norm increase the score depending on severity.
Revised: Deviations from the norm increase the score depending on severity.

Original: For example,‘severe hypotension’ adds +3 to the EWS, while ‘mild fever’ adds +1.
Revised: For example, ‘severe hypotension’ adds +3 to the EWS, while ‘mild fever’ adds +1.

Original: The dataset does not containconcept annotations, so we deﬁned them synthetically (Appendix A).
Revised: The dataset does not contain concept annotations, so we deﬁned them synthetically (Appendix A).

Original: The dataset contains recordsfrom 17,289 patients over a combined N=796,250 time steps (split into 530,802 training and 265,448test examples, while ensuring that no patient appears in both sets).
Revised: The dataset contains records from 17,289 patients over a combined N=796,250 time steps (split into 530,802 training and 265, test examples, while ensuring that no patient appears in both sets).

Original: It has 13 input features, K = 22concepts and 16 label classes (EWS from 0 to 15).
Revised: It has 13 input features, K =concepts and 16 label classes (EWS from 0 to 15).

Original: For MIMIC-III EWS, the concept predictor is a two-layer feed-forward neural network with a hiddenlayer of size 100, and the label predictor is a two-layer network with hidden layer of size 50. 
Revised: For MIMIC-III EWS, the concept predictor is a two-layer feed-forward neural network with a hidden layer of size 100, and the label predictor is a two-layer network with hidden layer of size 50. In the 1 c 1: k − 1 is a shorthand notation for c 1 . . . c k − [Table]

Revised: [Figure]

Original: In theautoregressive case, a small, two-layer network (hidden layer size 20) predicts each concept.
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: with interventions.

Revised: autoregressive case, a small, two-layer network (hidden layer size 20) predicts each concept.

Original: Theinput to these networks are the previously predicted concepts, concatenated with the 100 hiddenunits of the concept predictor.
Revised: The input to these networks are the previously predicted concepts, concatenated with the 100 hidden units of the concept predictor.

Original: We use M = 200 Monte-Carlo samples for prediction.
Revised: We use M = 200 Monte-Carlo samples for prediction.

Original: For traininghyperparameters, see Appendix C.
Revised: For training hyperparameters, see Appendix C.

Original: Caltech-UCSD Birds 

Revised: Caltech-UCSD Birds

Original: (Wah et al., 2011)
Revised: (Wah et al., 2011)

Original: For an accurate comparison to Koh et al.
Revised: For an accurate comparison to Koh et al.

Original: (2020),we replicated their results with the same preprocessing steps.
Revised: (2020), we replicated their results with the same preprocessing steps.

Original: [-This dataset contains N=11,788 (5,
-]
Revised: [-This dataset contains N=11,788 (5,
-]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table] [Figure] with interventions. training and 5,794 test) images of 200 bird species native to North America, along with 312 binaryattributes (eg.:
Revised: This dataset contains N=11,788 (5, training and 5,794 test) images of 200 bird species native to North America, along with 312 binary attributes (eg.:

Original: ‘wing-color’ and ‘beak-shape’).
Revised: ‘wing-color’ and ‘beak-shape’).

Original: Following Koh et al.
Revised: Following Koh et al.

Original: (2020), we narrow down this listof attributes to K = 112 binary concepts, only including attributes that are present for at least 10%of the dataset.
Revised: (2020), we narrow down this list of attributes to K = 112 binary concepts, only including attributes that are present for at least 10% of the dataset.

Original: To de-noise the attributes, we consolidate the concepts within each species by majorityvoting (where if the majority of ravens have black wings, we ensure that all ravens in the dataset areannotated as having black wings (Koh et al., 2020)).
Revised: To de-noise the attributes, we consolidate the concepts within each species by majority voting (where if the majority of ravens have black wings, we ensure that all ravens in the dataset are annotated as having black wings (Koh et al., 2020)).

Original: For CUB-2011, we ﬁne-tuned a pre-trained Inception v3 network (Szegedy et al., 2016) as theconcept predictor.
Revised: For CUB-2011, we ﬁne-tuned a pre-trained Inception v3 network (Szegedy et al., 2016) as the concept predictor.

Original: Following Koh et al.
Revised: Following Koh et al.

Original: (2020), the label predictor for the soft models is a simplelinear predictor, while for hard models, we use a two-layer neural network with a hidden layer ofsize 200.
Revised: (2020), the label predictor for the soft models is a simple linear predictor, while for hard models, we use a two-layer neural network with a hidden layer of size 200.

Original: In the autoregressive case, a small, two-layer network (hidden layer size 50) predicts eachconcept.
Revised: In the autoregressive case, a small, two-layer network (hidden layer size 50) predicts each concept.

Original: The input to these networks are the previously predicted concepts concatenated with theoutput of the Inception network.
Revised: The input to these networks are the previously predicted concepts concatenated with the output of the Inception network.

Original: We use M = 200 Monte-Carlo samples for prediction.
Revised: We use M = 200 Monte-Carlo samples for prediction.

Original: For traininghyperparameters, see Appendix C.
Revised: For training hyperparameters, see Appendix C.

Original: 5.2 Autoregressive Concept Predictions
Revised: 5.2 Autoregressive Concept Predictions

Original: Our ﬁrst set of experiments quantitatively assess the impact of leakage as well as examine theeffectiveness of autoregressive concept predictors.
Revised: Our ﬁrst set of experiments quantitatively assess the impact of leakage as well as examine the effectiveness of autoregressive concept predictors.

Original: We lead with these experiments to justify the useof the autoregressive architecture in our subsequent side-channel experiments.
Revised: We lead with these experiments to justify the use of the autoregressive architecture in our subsequent side-channel experiments.

Original: It is important to notethat both datasets have complete concept sets (they meet the Markovian assumption C = 1 ) so anyperformance disparity is due to the inﬂexibility of the concept predictor.
Revised: It is important to note that both datasets have complete concept sets (they meet the Markovian assumption C = 1 ) so any performance disparity is due to the inﬂexibility of the concept predictor.

Original: Leakage makes soft CBMs unsuitable for tasks where interpretability or intervenability arerequired.
Revised: Leakage makes soft CBMs unsuitable for tasks where interpretability or intervenability are required.

Original: Soft CBMs underperform in concept accuracy, but they are often able to predict the labelcorrectly (Table 1).
Revised: Soft CBMs underperform in concept accuracy, but they are often able to predict the label correctly (Table 1).

Original: This is only possible if the concepts are leaking information about the label in anuninterpretable way.
Revised: This is only possible if the concepts are leaking information about the label in an uninterpretable way.

Original: Interventions, where the human operator sets concepts to their true values, areineffective on sequential and joint soft CBMs (Figure 2).
Revised: Interventions, where the human operator sets concepts to their true values, are ineffective on sequential and joint soft CBMs (Figure 2).

Original: This should improve their predictions andyet they still mispredict the label even when all concepts are supplied, because interventions removethe leaked information from the soft concept probabilities.
Revised: This should improve their predictions and yet they still mispredict the label even when all concepts are supplied, because interventions remove the leaked information from the soft concept probabilities.

Original: Autoregressive predictors close the gap between soft and hard CBMs in label accuracy whilehaving dramatically increased concept accuracy.
Revised: Autoregressive predictors close the gap between soft and hard CBMs in label accuracy while having dramatically increased concept accuracy.

Original: In Table 1, we see that soft CBMs performwell in label accuracy, but leakage is detrimental to their concept accuracy.
Revised: In Table 1, we see that soft CBMs perform well in label accuracy, but leakage is detrimental to their concept accuracy.

Original: The hard CBM performsbetter in concept accuracy, but its performance is limited by the inﬂexibility of the concept predictor.
Revised: The hard CBM performs better in concept accuracy, but its performance is limited by the inﬂexibility of the concept predictor.

Original: The autoregressive model (Hard AR, this work) offers the best concept accuracy with competitivelabel accuracy: in the case of MIMIC-III EWS, Hard AR is above soft models in both concept andlabel accuracy, while in CUB-2011, Hard AR has the best concept accuracy with label accuracywithin 1% of joint models.
Revised: The autoregressive model (Hard AR, this work) offers the best concept accuracy with competitive [Figure] label accuracy: in the case of MIMIC-III EWS, Hard AR is above soft models in both concept and label accuracy, while in CUB-2011, Hard AR has the best concept accuracy with label accuracy within 1% of joint models.

Original: (Note that concept accuracy refers to predicting all concepts correctlysimultaneously as opposed to averaging the accuracies over the different concepts.)
Revised: (Note that concept accuracy refers to predicting all concepts correctly simultaneously as opposed to averaging the accuracies over the different concepts.)

Original: Hard autoregressive models excel at interventions while soft CBMs fail to adjust their predictions.
Revised: Hard autoregressive models excel at interventions while soft CBMs fail to adjust their predictions.

Original: On Figure 2, we see that the performance of soft sequential and soft joint models deteriorate[Figure] quickly due to leakage, while soft independent and hard models improve with interventions.
Revised: On Figure 2, we see that the performance of soft sequential and soft joint models deteriorate quickly due to leakage, while soft independent and hard models improve with interventions.

Original: The bestperforming model is the Hard AR (this work) on both datasets.
Revised: The best performing model is the Hard AR (this work) on both datasets.

Original: 5.3 Using a Side-channel
Revised: 5.3 Using a Side-channel

Original: Next, we examine the side-channel model as well as our proposed completeness score.
Revised: Next, we examine the side-channel model as well as our proposed completeness score.

Original: In theseexperiments, we trained CBMs with only a subset of the available concepts to simulate the scenariowhere the Markovian assumption does not hold.
Revised: In these experiments, we trained CBMs with only a subset of the available concepts to simulate the scenario where the Markovian assumption does not hold.

Original: We used L = 20 latent concepts for MIMIC-IIIand L = 100 latent concepts for CUB-2011 (we settled on these values because larger L yieldedno signiﬁcant improvement in predictive performance).
Revised: We used L = 20 latent concepts for MIMIC-III and L = 100 latent concepts for CUB-2011 (we settled on these values because larger L yielded no signiﬁcant improvement in predictive performance).

Original: Note that we use autoregressive conceptpredictors in hard CBMs, so any performance disparity is due to having an insufﬁcient concept set.
Revised: Note that we use autoregressive concept predictors in hard CBMs, so any performance disparity is due to having an insufﬁcient concept set.

Original: In MIMIC-III EWS, the concepts were grouped depending on which of the ﬁve vital signs theyare derived from (blood-pressure, temperature, respiratory rate, heart rate, oxygen saturation).
Revised: In MIMIC-III EWS, the concepts were grouped depending on which of the ﬁve vital signs they are derived from (blood-pressure, temperature, respiratory rate, heart rate, oxygen saturation).

Original: In
Revised: In

Original: CUB-2011, a group of concepts refer to related attributes, such as ‘wing-color’.
Revised: CUB-2011, a group of concepts refer to related attributes, such as ‘wing-color’.

Original: The side-channel closes the gap between soft and hard CBMs when the Markovian assumptionis not met.
Revised: The side-channel closes the gap between soft and hard CBMs when the Markovian assumption is not met.

Original: In Figure 3 (left), we see that the performance of the side-channel matches the predictiveperformance of the joint model when accounting for the latent concepts and it performs equivalently tothe no side-channel model when the latent concepts are marginalized with the help of the amortizationnetwork.
Revised: In Figure 3 (left), we see that the performance of the side-channel matches the predictive performance of the joint model when accounting for the latent concepts and it performs equivalently to the no side-channel model when the latent concepts are marginalized with the help of the amortization network.

Original: Therefore, the side-channel models precisely the information that is not present in theconcepts, i.e. the information that is leaked in the soft joint model.
Revised: Therefore, the side-channel models precisely the information that is not present in the concepts, i.e. the information that is leaked in the soft joint model.

Original: The completeness score indicates when the Markovian assumption is fulﬁlled.
Revised: The completeness score indicates when the Markovian assumption is fulﬁlled.

Original: The completeness score proves to be a useful diagnostic tool to show when the available concept set limits theperformance of hard CBMs.
Revised: The completeness score proves to be a useful diagnostic tool to show when the available concept set limits the performance of hard CBMs.

Original: Figure 3 (right) depicts the completeness score C depending on thenumber of concepts used during training.
Revised: Figure 3 (right) depicts the completeness score C depending on the number of concepts used during training.

Original: As the concept set grows, C tends towards 1 , and reaches itonce the concept set contains all information about the ﬁnal label, and the Markovian assumptionis met.
Revised: As the concept set grows, C tends towards 1 , and reaches it once the concept set contains all information about the ﬁnal label, and the Markovian assumption is met.

Original: Interestingly, completeness reaches 1 before p φ ( y | c ) accuracy peaks in CUB-2011.
Revised: Interestingly, completeness reaches 1 before p φ ( y | c ) accuracy peaks in CUB-2011.

Original: This ispossible when a concept contains information to predict y , but this information is not present in x (e.g.the color of the belly is not visible on images where it could be the deciding factor).
Revised: This is possible when a concept contains information to predict y , but this information is not present in x (e.g. the color of the belly is not visible on images where it could be the deciding factor).

Original: Such conceptsdo not contribute to completeness, but they increase label accuracy when the concepts are given.
Revised: Such concepts do not contribute to completeness, but they increase label accuracy when the concepts are given.

Original:  The side-channel is fully compatible with interventions There have been concerns that a sidechannel model could encode the label information in the side-channel, thus rendering interventionsineffective (Koh et al., 2020).
Revised: [Figure] The side-channel is fully compatible with interventions There have been concerns that a sidechannel model could encode the label information in the side-channel, thus rendering interventions ineffective (Koh et al., 2020).

Original: We found that this is not the case.
Revised: We found that this is not the case.

Original: Figure 4 shows that interventionsare as effective when trained with a subset of concepts as when trained with the complete conceptset.
Revised: Figure 4 shows that interventions are as effective when trained with a subset of concepts as when trained with the complete concept set.

Original: In the case of MIMIC-III EWS, the intervention accuracy of the side-channel models completelyoverlap with the hard CBM trained on all concepts.
Revised: In the case of MIMIC-III EWS, the intervention accuracy of the side-channel models completely overlap with the hard CBM trained on all concepts.

Original: In CUB-2011, the interventions are less effectivewith 1-2 concept groups, but they quickly catch up to the model trained with the complete conceptset. [Figure]
Revised: In CUB-2011, the interventions are less effective with 1-2 concept groups, but they quickly catch up to the model trained with the complete concept set

Original: Computational costs.
Revised: Computational costs.

Original: The beneﬁts of autoregressive models and side-channel models come at anincreased computational cost.
Revised: The beneﬁts of autoregressive models and side-channel models come at an increased computational cost.

Original: Table 2 shows the computational cost of each model.
Revised: Table 2 shows the computational cost of each model.

Original: Notably, the morecomplex image dataset, CUB-2011 takes more epochs to converge.
Revised: Notably, the more complex image dataset, CUB-2011 takes more epochs to converge.

Original: The cost of the autoregressivearchitecture and the side-channel is more signiﬁcant when the predictive model is relatively simple,which is the case in MIMIC-III EWS
Revised: The cost of the autoregressive architecture and the side-channel is more signiﬁcant when the predictive model is relatively simple, which is the case in MIMIC-III EWS.

Original:  Conclusions
Revised:  Conclusions

Original: Leakage is detrimental to model interpretability and intervenability in soft concept bottleneck models.
Revised: Leakage is detrimental to model interpretability and intervenability in soft concept bottleneck models.

Original: Hard CBMs are resilient to leakage, but they lag behind their soft counterparts in predictiveperformance.
Revised: Hard CBMs are resilient to leakage, but they lag behind their soft counterparts in predictive performance.

Original: We exhibited and addressed two causes of performance disparity between soft and hard
Revised: We exhibited and addressed two causes of performance disparity between soft and hard

Original: CBMs.
Revised: CBMs.

Original: Firstly, not meeting the Markovian assumption, which can be diagnosed using our proposedcompleteness score and addressed using a side-channel model.
Revised: Firstly, not meeting the Markovian assumption, which can be diagnosed using our proposed completeness score and addressed using a side-channel model.

Original: Secondly, the inﬂexibility of theconcept predictor, which can be mitigated by adapting an autoregressive architecture.
Revised: Secondly, the inﬂexibility of the concept predictor, which can be mitigated by adapting an autoregressive architecture.

Original: With our modiﬁcations, CBMs no longer suffer from leakage and have signiﬁcantly improved conceptaccuracy and intervention accuracy.
Revised: With our modiﬁcations, CBMs no longer suffer from leakage and have signiﬁcantly improved concept accuracy and intervention accuracy.

Original: This makes them more interpretable and more suitable forhuman-AI joint decision-making tasks.
Revised: This makes them more interpretable and more suitable for human-AI joint decision-making tasks.

Original: In future work we hope to explore methods to present the side-channel information in a humaninterpretable way.
Revised: In future work, we hope to explore methods to present the side-channel information in a humaninterpretable way.

Original: This would allow us to discover concepts that are needed to complete the conceptset for a given task Checklist
Revised: This would allow us to discover concepts that are needed to complete the concept set for a given task.

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work?
Revised: Acknowledgements

Original: [Yes] See interpretability of the side

Original: channel and computational considerations.

Original: (c) Did you discuss any potential negative societal impacts of your work? [No] We do notforesee signiﬁcant negative societal impacts. (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem? [Yes]
Revised: This work was funded by a grant from the National Institute of Mental Health (grant no.

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)? [Yes]
Revised: R01MH123804).

Original: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-

Original: ments multiple times)?

Original: [Yes](d) Did you include the total amount of compute and  the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [Yes]4. If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]

Original: (b) Did you mention the license of the assets?

Original: [Yes] The datasets we used are open for

Original: research purposes, our code is open source.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]

Original: Our code.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?

Original: [Yes]

Original: In the case of MIMIC, Johnson et al.

Original: (2016): “Requirement

Original: for individual patient consent was waived because the project did not impact clinical

Original: care and all protected health information was deidentiﬁed.”.

Original: CUB-2011 has no humansubjects.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?

Original: [Yes] See above.

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?

Original: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?

Original: [N/A]