Original: E XPLAINING A B LACK - BOX B Y U SING A D EEP V ARIATIONAL I NFORMATION B OTTLENECK A PPROACH
Revised: E XPLAINING A B LACK - BOX B Y U SING A D EEP V ARIATIONAL I NFORMATION B OTTLENECK A PPROACH

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Interpretable machine learning has gained much attention recently.
Revised: Interpretable machine learning has gained much attention recently.

Original: Briefness and comprehensiveness are necessary in order to provide a large amount of information concisely when explaining a black-box decision system.
Revised: Briefness and comprehensiveness are necessary in order to provide a large amount of information concisely when explaining a black-box decision system.

Original: However, existing interpretable machine learning methods fail to consider briefness and comprehensiveness simultaneously, leading to redundant explanations.
Revised: However, existing interpretable machine learning methods fail to consider briefness and comprehensiveness simultaneously, leading to redundant explanations.

Original: We propose the variational information bottleneck for interpretation, VIBI, a system-agnostic interpretable method that provides a brief but comprehensive explanation.
Revised: We propose the variational information bottleneck for interpretation, VIBI, a system-agnostic interpretable method that provides a brief but comprehensive explanation.

Original: VIBI adopts an information theoretic principle, information bottleneck principle , as a criterion for ﬁnding such explanations.
Revised: VIBI adopts an information theoretic principle, information bottleneck principle , as a criterion for ﬁnding such explanations.

Original: For each instance, VIBI selects key features that are maximally compressed about an input (briefness), and informative about a decision made by a black-box system on that input (comprehensive).
Revised: For each instance, VIBI selects key features that are maximally compressed about an input (briefness), and informative about a decision made by a black-box system on that input (comprehensive).

Original: We evaluate
Revised: We evaluate

Original: VIBI on three datasets and compare with state-of-the-art interpretable machine learning methods in terms of both interpretability and ﬁdelity evaluated by human and quantitative metrics.
Revised: VIBI on three datasets and compare with state-of-the-art interpretable machine learning methods in terms of both interpretability and ﬁdelity evaluated by human and quantitative metrics.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Interpretability is crucial in building and deploying black-box decision systems such as deep learning models.
Revised: Interpretability is crucial in building and deploying black-box decision systems such as deep learning models.

Original: Interpretation of a black-box system helps decide whether or not to follow its decisions, or understand the logic behind the system.
Revised: Interpretation of a black-box system helps decide whether or not to follow its decisions, or understand the logic behind the system.

Original: In recent years, the extensive use of deep learning black-box systems has given rise to interpretable machine learning approaches (Lipton, 2016; Doshi-Velez &
Revised: In recent years, the extensive use of deep learning black-box systems has given rise to interpretable machine learning approaches (Lipton, 2016; Doshi-Velez &

Original: Kim, 2017), which aim to explain how black-box systems work or why they reach certain decisions.
Revised: Kim, 2017), which aim to explain how black-box systems work or why they reach certain decisions.

Original: In order to provide sufﬁcient information while avoiding redundancy when explaining a black-box decision, we need to consider both briefness and comprehensiveness .
Revised: In order to provide sufﬁcient information while avoiding redundancy when explaining a black-box decision, we need to consider both briefness and comprehensiveness .

Original: However, existing approaches lack in-depth consideration for and fail to ﬁnd both brief but comprehensive explanation.
Revised: However, existing approaches lack in-depth consideration for and fail to ﬁnd both brief but comprehensive explanation.

Original: In order to obtain brief but comprehensive explanation, we adopt the information bottleneck principle (Tishby et al., 2000).
Revised: In order to obtain brief but comprehensive explanation, we adopt the information bottleneck principle (Tishby et al., 2000).

Original: This principle provides an appealing information theoretic perspective for learning supervised models by deﬁning what we mean by a ‘good’ representation.
Revised: This principle provides an appealing information theoretic perspective for learning supervised models by deﬁning what we mean by a ‘good’ representation.

Original: The principle says that the optimal model transmits as much information as possible from its input to its output through a compressed representation called the information bottleneck.
Revised: The principle says that the optimal model transmits as much information as possible from its input to its output through a compressed representation called the information bottleneck.

Original: Then, the information bottleneck will maximally compress the mutual information (MI) with an input while preserving as much as possible
Revised: Then, the information bottleneck will maximally compress the mutual information (MI) with an input while preserving as much as possible

Original: MI with the output.
Revised: MI with the output.

Original: Recently, it has been shown that the principle also applies to deep neural networks and each layer of a deep neural network can work as an information bottleneck (Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017).
Revised: Recently, it has been shown that the principle also applies to deep neural networks and each layer of a deep neural network can work as an information bottleneck (Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017).

Original: Using this idea of information bottleneck principle, we deﬁne a brief but comprehensive explanation as maximally informative about the black-box decision while compressive about a given input.
Revised: Using this idea of information bottleneck principle, we deﬁne a brief but comprehensive explanation as maximally informative about the black-box decision while compressive about a given input.

Original: In this paper, we introduce the variational information bottleneck for interpretation (VIBI), a systemagnostic information bottleneck model that provides a brief but comprehensive explanation for every single decision made by a black-box model.
Revised: In this paper, we introduce the variational information bottleneck for interpretation (VIBI), a systemagnostic information bottleneck model that provides a brief but comprehensive explanation for every single decision made by a black-box model.

Original: VIBI is composed of two parts: explainer and approximator, each of which is modeled by a deep neural network.
Revised: VIBI is composed of two parts: explainer and approximator, each of which is modeled by a deep neural network.

Original: The explainer returns a probability whether a chunk of features such as a word, phrase, sentence or a group of pixels will be selected as an explanation or not for each instance, and an approximator mimics behaviour of a black-box model.
Revised: The explainer returns a probability whether a chunk of features such as a word, phrase, sentence or a group of pixels will be selected as an explanation or not for each instance, and an approximator mimics behaviour of a black-box model.

Original: Using the information bottleneck principle, we learn an explainer that favors brief explanations while enforcing that the explanations alone sufﬁce for accurate approximations to a black-box model.
Revised: Using the information bottleneck principle, we learn an explainer that favors brief explanations while enforcing that the explanations alone sufﬁce for accurate approximations to a black-box model.

Original: i
Revised: i

Original: 1. C ONTRIBUTION
Revised: 1. C ONTRIBUTION

Original: Our main contribution is to provide a new framework that systemically deﬁnes and generates a ‘good’ (i.e. brief but comprehensive) explanation using the information bottleneck principle.
Revised: Our main contribution is to provide a new framework that systematically deﬁnes and generates a ‘good’ (i.e. brief but comprehensive) explanation using the information bottleneck principle.

Original: Based on this principle, we develop VIBI that favors a brief but comprehensive explanation.
Revised: Based on this principle, we develop VIBI that favors a brief but comprehensive explanation.

Original: In order to make the objective function of VIBI tractable, we derive a variational approximation to the objective.
Revised: In order to make the objective function of VIBI tractable, we derive a variational approximation to the objective.

Original: The beneﬁts of our method are as follows.
Revised: The beneﬁts of our method are as follows.

Original: 1) System-agnostic : VIBI can be applied to explain any black-box system.
Revised: 1) System-agnostic : VIBI can be applied to explain any black-box system.

Original: 2) Post-hoc learning : VIBI is learned in a post-hoc manner, hence there is no tradeoff between task accuracy of a black-box system and interpretability of an explainer.
Revised: 2) Post-hoc learning : VIBI is learned in a post-hoc manner, hence there is no tradeoff between task accuracy of a black-box system and interpretability of an explainer.

Original: 3) Cognitive chunk : Cognitive chunk is deﬁned as a group of raw features whose identity is understandable to human.
Revised: 3) Cognitive chunk : Cognitive chunk is deﬁned as a group of raw features whose identity is understandable to human.

Original: VIBI groups non-cognitive raw features such as a pixel and letter into a cognitive chunk (e.g. a group of pixels, a word, a phrase,and a sentence) and selects each unit as an explanation.
Revised: VIBI groups non-cognitive raw features such as a pixel and letter into a cognitive chunk (e.g. a group of pixels, a word, a phrase, a sentence) and selects each unit as an explanation.

Original: Separate explainer and approximator : The explainer and approximator are designed for separated tasks so that we do not need to limit the approximator to have a simple structure, which may reduce the ﬁdelity (the ability to imitate the behaviour of a black-box) of approximator.
Revised: Separate explainer and approximator : The explainer and approximator are designed for separated tasks so that we do not need to limit the approximator to have a simple structure, which may reduce the ﬁdelity (the ability to imitate the behaviour of a black-box) of approximator.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Most prior interpretable machine learning methods have been focusing on local interpretation, which implies knowing  reasons why a black-box system makes a certain decision at a very local point of interest, and our work is also situated in this line.
Revised: Most prior interpretable machine learning methods have been focusing on local interpretation, which implies knowing the reasons why a black-box system makes a certain decision at a very local point of interest, and our work is also situated in this line.

Original: Existing methods can be categorized into system-speciﬁc and system-agnostic method.
Revised: Existing methods can be categorized into system-speciﬁc and system-agnostic method.

Original: System-speciﬁc methods only explain certain blackbox decision systems (e.g. using backpropagation algorithm, or having CNN structure), while system-agnostic methods explain any black-box decision systems.
Revised: System-speciﬁc methods only explain certain black-box decision systems (e.g. using backpropagation algorithm, or having CNN structure), while system-agnostic methods explain any black-box decision systems.

Original: System-speciﬁc methods .
Revised: System-speciﬁc methods .

Original: To measure a change of output with respect to changes of input is an intuitive way of obtaining feature attribution for the output.
Revised: To measure a change of output with respect to changes of input is an intuitive way of obtaining feature attribution for the output.

Original: Using this idea, Zeiler & Fergus (2014), and Zintgraf et al.
Revised: Using this idea, Zeiler & Fergus (2014), and Zintgraf et al.

Original: (2017) observe how outputs change when they make perturbations to each instance.
Revised: (2017) observe how outputs change when they make perturbations to each instance.

Original: Baehrens et al.
Revised: Baehrens et al.

Original: ; Simonyan et al.
Revised: ; Simonyan et al.

Original: (2013), and Smilkov et al.
Revised: (2013), and Smilkov et al.

Original: (2017) use computationally more efﬁcient approaches; they measure change of output by propagating contributions through layers of a deep neural network towards an inputthan the perturbation.
Revised: (2017) use computationally more efﬁcient approaches; they measure change of output by propagating contributions through layers of a deep neural network towards an input than the perturbation.

Original: However, these approaches fail to detect the changes of output when the prediction function is ﬂattened at the instance (Shrikumar et al., 2017), which leads to interpretations focusing on irrelevant features.
Revised: However, these approaches fail to detect the changes of output when the prediction function is ﬂattened at the instance (Shrikumar et al., 2017), which leads to interpretations focusing on irrelevant features.

Original: In order to solve this problem, the layer-wise relevance propagation (Bach et al., 2015; Binder et al., 2016), DeepLIFT (Shrikumar et al., 2017), and Integrated Gradients (Sundararajan et al., 2017) compare the changes of output to its reference output.
Revised: In order to solve this problem, the layer-wise relevance propagation (Bach et al., 2015; Binder et al., 2016), DeepLIFT (Shrikumar et al., 2017), and Integrated Gradients (Sundararajan et al., 2017) compare the changes of output to its reference output.

Original: Ross et al.
Revised: Ross et al.

Original: (2017) learn a more generalizable model as well as desirable explanations by constraining its explanations (i.e. input gradient) to match domain knowledge.
Revised: (2017) learn a more generalizable model as well as desirable explanations by constraining its explanations (i.e. input gradient) to match domain knowledge.

Original: System-agnostic methods .
Revised: System-agnostic methods .

Original: The great advantage of system-agnostic interpretable machine learning methods over system-speciﬁc methods is that their usage is not restricted to a speciﬁc black-box system.
Revised: The great advantage of system-agnostic interpretable machine learning methods over system-speciﬁc methods is that their usage is not restricted to a speciﬁc black-box system.

Original: One of the most well-known system-agnostic methods is LIME (Ribeiro et al., 2016).
Revised: One of the most well-known system-agnostic methods is LIME (Ribeiro et al., 2016).

Original: It explains the decision of an instance by locally approximating the black-box decision boundary around the instance with an inherently interpretable model such as sparse linear or decision trees.
Revised: It explains the decision of an instance by locally approximating the black-box decision boundary around the instance with an inherently interpretable model such as sparse linear or decision trees.

Original: The approximator is learned by samples generated by perturbing a given instance.
Revised: The approximator is learned by samples generated by perturbing a given instance.

Original: Lundberg & Lee (2017) proposed SHAP, a uniﬁed measure deﬁned over the additive feature attribution scores in order to achieve local accuracy, missingness, and consistency.
Revised: Lundberg & Lee (2017) proposed SHAP, a uniﬁed measure deﬁned over the additive feature attribution scores in order to achieve local accuracy, missingness, and consistency.

Original: L2X (Chen et al., 2018) learns a stochastic map that selects instance-wise features that are most informative for black-box decisions.
Revised: L2X (Chen et al., 2018) learns a stochastic map that selects instance-wise features that are most informative for black-box decisions.

Original: Unlike
Revised: Unlike

Original: LIME and SHAP, which approximate local behaviors of a black-box system with a simple (linear) model, L2X does not put a limit on the structure of the approximator; hence it avoids losing ﬁdelity of the approximator.
Revised: LIME and SHAP, which approximate local behaviors of a black-box system with a simple (linear) model, L2X does not put a limit on the structure of the approximator; hence it avoids losing ﬁdelity of the approximator.

Original: As SHAP does, Dabkowski & Gal (2017)
Revised: As SHAP does, Dabkowski & Gal (2017)

Original: ; Fong & Vedaldi (2017), and Petsiuk et al.
Revised: ; Fong & Vedaldi (2017), and Petsiuk et al.

Original: (2018) use sample perturbation
Revised: (2018) use sample perturbation

Original: but they rather learn or estimate desired perturbation masks than using perturbed samples to learn an approximator.
Revised: but they rather learn or estimate desired perturbation masks than using perturbed samples to learn an approximator.

Original: Dabkowski & Gal (2017), and Fong &
Revised: Dabkowski & Gal (2017), and Fong &

Original: Vedaldi (2017) learn a smallest perturbation mask that alters black-box outputs as much as possible.
Revised: Vedaldi (2017) learn a smallest perturbation mask that alters black-box outputs as much as possible.

Original: Petsiuk et al.
Revised: Petsiuk et al.

Original: (2018) empirically estimate feature attribution as a sum of random masks weighted by class scores corresponding to masked inputs.
Revised: (2018) empirically estimate feature attribution as a sum of random masks weighted by class scores corresponding to masked inputs.

Original: Our method VIBI is similar with L2X in that both learn a stochastic explainer that returns a distribution over the subset of features given the input and performs instance-wise feature selection based on that.
Revised: Our method VIBI is similar with L2X in that both learn a stochastic explainer that returns a distribution over the subset of features given the input and performs instance-wise feature selection based on that.

Original: However, given the same number of explanations, our explainer favors both briefness and comprehensiveness while the L2X explainer favors comprehensiveness of the explanation and does not account for briefness.
Revised: However, given the same number of explanations, our explainer favors both briefness and comprehensiveness while the L2X explainer favors comprehensiveness of the explanation and does not account for briefness.

Original: ii
Revised: ii

Original: M ETHOD
Revised: M ETHOD

Original: I cough a lot at night
Revised: I cough a lot at night

Original: Z
Revised: Z

Original: X
Revised: X

Original: Y zzzzzz 6     x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 x 9 x 10 x 11 x 12 x 13 x 14 x 15 x 16 x 17 x0 0x 16 x 17 x 18 0 00 

Revised: Y zzzzzz 6     x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 x 9 x 10 x 11 x 12 x 13 x 14 x 15 x 16 x 17 x0 0x 16 x 17 x 18 0 00

Original: Explainer
Revised: Explainer

Original: Approximator
Revised: Approximator

Original: T(X) p ( z | x ) t = x ⊙ z q ( y | t ) =
Revised: T(X) p ( z | x ) t = x ⊙ z q ( y | t ) =

Original: q ( y | x ⊙ z )
Revised: q ( y | x ⊙ z )

Original: I a lot at night 0 0 0 x 7 x 8 xx 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 x 9 x 10 x 11 x 12 x 13 x 14 x 15 x 16 x 17 x
Revised: I a lot at night 0 0 0 x 7 x 8 xx 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 x 9 x 10 x 11 x 12 x 13 x 14 x 15 x 16 x 17 x

Original: Flu
Revised: Flu

Original: [Figure]
Revised: [Figure]

Original: 3. P ERSPECTIVE FROM INFORMATION BOTTLENECK PRINCIPLE
Revised: 3. P ERSPECTIVE FROM INFORMATION BOTTLENECK PRINCIPLE

Original: The information bottleneck principle (Tishby et al., 2000) provides an appealing information theoretic view for learning a supervised model by deﬁning what we mean by a ‘good’ representation.
Revised: The information bottleneck principle (Tishby et al., 2000) provides an appealing information theoretic view for learning a supervised model by deﬁning what we mean by a ‘good’ representation.

Original: The principle says that the optimal model transmits as much information as possible from the input x to the output y through a compressed representation t (called the information bottleneck).
Revised: The principle says that the optimal model transmits as much information as possible from the input x to the output y through a compressed representation t (called the information bottleneck).

Original: The representation t is stochastically deﬁned and the optimal stochastic mapping p ( t | x ) is obtained by optimizing the following problem with a Markov chain assumption y
Revised: The representation t is stochastically deﬁned and the optimal stochastic mapping p ( t | x ) is obtained by optimizing the following problem with a Markov chain assumption y

Original: → x → t :
Revised: → x → t :

Original: [Equation]
Revised: [Equation]

Original: where I( · , · ) is the MI and β is a Lagrange multiplier representing the trade-off between the compressiveness − I( x , t ) and informativeness I( t , y ) of the representation t .
Revised: where I( · , · ) is the MI and β is a Lagrange multiplier representing the trade-off between the compressiveness − I( x , t ) and informativeness I( t , y ) of the representation t .

Original: We adopt the information bottleneck principle as a criterion for ﬁnding brief but comprehensive explanations.
Revised: We adopt the information bottleneck principle as a criterion for ﬁnding brief but comprehensive explanations.

Original: Our aim is to learn an explainer generating explanations that are maximally informative about the black-box decision while compressive about a given input.
Revised: Our aim is to learn an explainer generating explanations that are maximally informative about the black-box decision while compressive about a given input.

Original: The primary difference between our information bottleneck objective (2) and the one in Tishby et al.
Revised: The primary difference between our information bottleneck objective (2) and the one in Tishby et al.

Original: (2000) is as follows: the latter aims to identify a stochastic map of the representation t that itself works as an information bottleneck, whereas our objective aims to identify a stochastic map of z performing instance-wise selection of cognitive chunks and deﬁne information bottleneck as a function of z and the input x .
Revised: (2000) is as follows: the latter aims to identify a stochastic map of the representation t that itself works as an information bottleneck, whereas our objective aims to identify a stochastic map of z performing instance-wise selection of cognitive chunks and deﬁne information bottleneck as a function of z and the input x .

Original: 3. P ROPOSED APPROACH
Revised: 3. P ROPOSED APPROACH

Original: We introduce VIBI, a system-agnostic interpretation approach that provides brief but comprehensive explanations for decisions made by black-box decision system.
Revised: We introduce VIBI, a system-agnostic interpretation approach that provides brief but comprehensive explanations for decisions made by black-box decision system.

Original: In order to achieve this, we optimize the following information bottleneck objective.
Revised: In order to achieve this, we optimize the following information bottleneck objective.

Original: [Equation]
Revised: [Equation]

Original: where I( t , y ) represents the sufﬁciency of information retained for explaining the black-box output y , − I( x , t ) represents the briefness of the explanation t , and β is a Lagrange multiplier representing a trade-off between the two.
Revised: where I( t , y ) represents the sufﬁciency of information retained for explaining the black-box output y , − I( x , t ) represents the briefness of the explanation t , and β is a Lagrange multiplier representing a trade-off between the two.

Original: To achieve compressiveness, in addition to encouraging small MI between explanations and inputs, we also encourage the number of selected cognitive chunks to be small, i.e., encouraging z to be sparse.
Revised: To achieve compressiveness, in addition to encouraging small MI between explanations and inputs, we also encourage the number of selected cognitive chunks to be small, i.e., encouraging z to be sparse.

Original: Note that MI and sparsity are two complementary approaches for achieving compression.
Revised: Note that MI and sparsity are two complementary approaches for achieving compression.

Original: iii
Revised: iii

Original: MI aims at reducing semantic redundancy in explanations.
Revised: MI aims at reducing semantic redundancy in explanations.

Original: Sparsity cannot achieve such a goal.
Revised: Sparsity cannot achieve such a goal.

Original: For example, consider two explanations in judging the sentiment of movie reviews: "great, great" and "great, thought provoking".
Revised: For example, consider two explanations in judging the sentiment of movie reviews: "great, great" and "great, thought-provoking".

Original: They have the same level of sparsity.
Revised: They have the same level of sparsity.

Original: However, the ﬁrst explanation has a large MI with the input document where "great" occurs a lot.
Revised: However, the ﬁrst explanation has a large MI with the input document where "great" occurs a lot.

Original: As a result, this explanation has a lot of redundancy in semantics.
Revised: As a result, this explanation has a lot of redundancy in semantics.

Original: The second explanation has smaller MI and hence is more brief and preferable.
Revised: The second explanation has smaller MI and hence is more brief and preferable.

Original: As illustrated in Figure 1A, VIBI is composed of two parts: the explainer and the approximator, each of which is modeled by a deep neural network.
Revised: As illustrated in Figure 1A, VIBI is composed of two parts: the explainer and the approximator, each of which is modeled by a deep neural network.

Original: The explainer selects a group of k key cognitive chunks given an instance while the approximator mimics the behaviour of the black-box system using the selected keys as the input.
Revised: The explainer selects a group of k key cognitive chunks given an instance while the approximator mimics the behaviour of the black-box system using the selected keys as the input.

Original: k controls the level of sparsity in z .
Revised: k controls the level of sparsity in z .

Original: In detail, the explainer p ( z | x ; θ e ) is a map from an input x to its attribution scores p j ( x ) =
Revised: In detail, the explainer p ( z | x ; θ e ) is a map from an input x to its attribution scores p j ( x ) =

Original: p
Revised: p

Original: z j | x ) (where j is for the j -th cognitive chunk).
Revised: z j | x ) (where j is for the j -th cognitive chunk).

Original: The attribution score indicates the probability that each cognitive chunk to be selected.
Revised: The attribution score indicates the probability that each cognitive chunk to be selected.

Original: In order to select top k cognitive chunks as an explanation, a k -hot vector z is sampled from a categorical distribution with class probabilities p j ( x ) =
Revised: In order to select top k cognitive chunks as an explanation, a k -hot vector z is sampled from a categorical distribution with class probabilities p j ( x ) =

Original: p ( z j | x ) and the j -th cognitive chunk is selected if z j = 1 .
Revised: p ( z j | x ) and the j -th cognitive chunk is selected if z j = 1 .

Original: More speciﬁcally, the explanation t is deﬁned as follows:
Revised: More speciﬁcally, the explanation t is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where j indicates a cognitive chunk, each of which corresponds to multiple row features i .
Revised: where j indicates a cognitive chunk, each of which corresponds to multiple row features i .

Original: The approximator is modeled by another deep neural network p ( y | t ; θ a ) , which mimics the black-box decision system.
Revised: The approximator is modeled by another deep neural network p ( y | t ; θ a ) , which mimics the black-box decision system.

Original: It takes t as an input and returns an output approximating the black-box output for the instance x .
Revised: It takes t as an input and returns an output approximating the black-box output for the instance x .

Original: θ a and θ e represent the weight parameters of neural networks.
Revised: θ a and θ e represent the weight parameters of neural networks.

Original: The explainer and approximator are trained jointly by minimizing a cost function that favors concise explanations while enforcing that the explanations alone sufﬁce for accurate prediction.
Revised: The explainer and approximator are trained jointly by minimizing a cost function that favors concise explanations while enforcing that the explanations alone sufﬁce for accurate prediction.

Original: 3.2. T HE VARIATIONAL BOUND
Revised: 3.2. T HE VARIATIONAL BOUND

Original: The current form of information bottleneck objective is intractable due to the MIs
Revised: The current form of information bottleneck objective is intractable due to the MIs

Original: I( t , y ) and I( x , t ) .
Revised: I( t , y ) and I( x , t ) .

Original: We address this problem by using a variational approximation of our information bottleneck objective.
Revised: We address this problem by using a variational approximation of our information bottleneck objective.

Original: In this section, we summarize the results and refer to Supplementary Material A for details.
Revised: In this section, we summarize the results and refer to Supplementary Material A for details.

Original: Variational bound for I( x , t ) :
Revised: Variational bound for I( x , t ) :

Original: We ﬁrst show that I( x , t ) ≤ I( x , z ) + C where C is constant and use the lower bound for − I( x , z ) − C as a lower bound for − I( x , t ) .
Revised: We ﬁrst show that I( x , t ) ≤ I( x , z ) + C where C is constant and use the lower bound for − I( x , z ) − C as a lower bound for − I( x , t ) .

Original: As a result, we obtain:
Revised: As a result, we obtain:

Original: [Equation]
Revised: [Equation]

Original: Note that with proper choices of r ( z ) and p ( z | x ) , we can assume that the Kullback-Leibler divergence
Revised: Note that with proper choices of r ( z ) and p ( z | x ) , we can assume that the Kullback-Leibler divergence

Original: D KL ( p ( z | x ) , r ( z )) has an analytical form.
Revised: D KL ( p ( z | x ) , r ( z )) has an analytical form.

Original: Variational bound for I( t , y ) :
Revised: Variational bound for I( t , y ) :

Original: We obtain the lower bound for I( t , y ) by using q ( y | t ) to approximate p ( y | t ) , which works as an approximator to the black-box system.
Revised: We obtain the lower bound for I( t , y ) by using q ( y | t ) to approximate p ( y | t ) , which works as an approximator to the black-box system.

Original: As a result, we obtain:
Revised: As a result, we obtain:

Original: [Equation]
Revised: [Equation]

Original: where p ( t | x , y ) =
Revised: where p ( t | x , y ) =

Original: p ( t | x ) by the Markov chain assumption y ↔ x ↔ t .
Revised: p ( t | x ) by the Markov chain assumption y ↔ x ↔ t .

Original: Combining Equations (3) and (4), we obtain the following variational bound:
Revised: Combining Equations (3) and (4), we obtain the following variational bound:

Original: [Equation]
Revised: [Equation]

Original: where C ∗ = − Cβ can be ignored since it is independent of the optimization procedure.
Revised: where C ∗ = − Cβ can be ignored since it is independent of the optimization procedure.

Original: We use the empirical data distribution to approximate p ( x , y )
Revised: We use the empirical data distribution to approximate p ( x , y )

Original: =
Revised: =

Original: p ( x ) p ( y | x ) and p ( x ) .
Revised: p ( x ) p ( y | x ) and p ( x ) .

Original: 3.2. C ONTINUOUS RELAXATION AND REPARAMETERIZATION
Revised: 3.2. C ONTINUOUS RELAXATION AND REPARAMETERIZATION

Original: Current form of the bound (5) is still intractable because we need to sum over the (cid:0) dk (cid:1) combinations of feature subsets.
Revised: Current form of the bound (5) is still intractable because we need to sum over the (cid:0) dk (cid:1) combinations of feature subsets.

Original: This is because we sample top k out of d cognitive chunks where each chunk is assumed drawn from a categorical distribution with class probabilities
Revised: This is because we sample top k out of d cognitive chunks where each chunk is assumed drawn from a categorical distribution with class probabilities

Original: p j
Revised: p j

Original: ( x ) =
Revised: ( x ) =

Original: p ( z j | x ) .
Revised: p ( z j | x ) .

Original: In order to avoid this, we use the generalized Gumbel-softmax trick (Jang et al., 2017; Chen et al., 2018).
Revised: In order to avoid this, we use the generalized Gumbel-softmax trick (Jang et al., 2017; Chen et al., 2018).

Original: This is a well-known technique that are used to approximate a non-differentiable categorical subset sampling with differentiable Gumbel-softmax samples.
Revised: This is a well-known technique that are used to approximate a non-differentiable categorical subset sampling with differentiable Gumbel-softmax samples.

Original: The steps are as follows.
Revised: The steps are as follows.

Original: iv
Revised: iv

Original: First, we independently sample a cognitive chunk for k times.
Revised: First, we independently sample a cognitive chunk for k times.

Original: For each time, a random perturbation e j is added to the log probability of each cognitive chunk log p j ( x ) .
Revised: For each time, a random perturbation e j is added to the log probability of each cognitive chunk log p j ( x ) .

Original: From this, Concrete random vector c = ( c 1 , · · · , c d ) working as a continuous, differentiable approximation to argmax is deﬁned:
Revised: From this, Concrete random vector c = ( c 1 , · · · , c d ) working as a continuous, differentiable approximation to argmax is deﬁned:

Original: [Equation]
Revised: [Equation]

Original: e j ) where e j ∼ U (0 , 1)
Revised: e j ) where e j ∼ U (0 , 1)

Original: [Equation]
Revised: [Equation]

Original: where τ is a tuning parameter for the temperature of Gumbel-Softmax distribution.
Revised: where τ is a tuning parameter for the temperature of Gumbel-Softmax distribution.

Original: Next, we deﬁne a continuous-relaxed random vector z ∗ =
Revised: Next, we deﬁne a continuous-relaxed random vector z ∗ =

Original: [ z ∗ 1 , · · · , z ∗ d ] (cid:62) as the element-wise maximum of the independently sampled Concrete vectors c ( l ) where l = 1 , · · · , k : z ∗ j = max l c ( l ) j for l = 1 , · · · , k
Revised: [ z ∗ 1 , · · · , z ∗ d ] (cid:62) as the element-wise maximum of the independently sampled Concrete vectors c ( l ) where l = 1 , · · · , k : z ∗ j = max l c ( l ) j for l = 1 , · · · , k

Original: With this sampling scheme, we approximate the k -hot random vector and have the continuous approximation to the variational bound (5).
Revised: With this sampling scheme, we approximate the k -hot random vector and have the continuous approximation to the variational bound (5).

Original: This trick allows using standard backpropagation to compute the gradients of the parameters via reparameterization.
Revised: This trick allows using standard backpropagation to compute the gradients of the parameters via reparameterization.

Original: By putting everything together, we obtain:
Revised: By putting everything together, we obtain:

Original: [Equation]
Revised: [Equation]

Original: where q ( y i | x i (cid:12) f
Revised: where q ( y i | x i (cid:12) f

Original: ( e ( l ) , x i )) is the approximator to the black-box system and − D KL ( p ( z ∗ | x i ) , r ( z ∗ )) represents the compactness of the explanation.
Revised: ( e ( l ) , x i )) is the approximator to the black-box system and − D KL ( p ( z ∗ | x i ) , r ( z ∗ )) represents the compactness of the explanation.

Original: Once we learn the model, the attribution score p
Revised: Once we learn the model, the attribution score p

Original: j ( x ) for each cognitive chunk is used to select top k key cognitive chunks that are maximally compressive about the input x and informative about the black-box decision y on that input.
Revised: j ( x ) for each cognitive chunk is used to select top k key cognitive chunks that are maximally compressive about the input x and informative about the black-box decision y on that input.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluated VIBI on three datasets and compared with state-of-the-art interpretable machine learning methods.
Revised: We evaluated VIBI on three datasets and compared with state-of-the-art interpretable machine learning methods.

Original: The evaluation is performed from two perspectives: interpretability and ﬁdelity .
Revised: The evaluation is performed from two perspectives: interpretability and ﬁdelity .

Original: The interpretability indicates the ability to explain a black-box model with human understandable terms.
Revised: The interpretability indicates the ability to explain a black-box model with human understandable terms.

Original: The ﬁdelity implies how accurately our approximator approximates the black-box model.
Revised: The ﬁdelity implies how accurately our approximator approximates the black-box model.

Original: Based on these criteria, we compared VIBI with three state-of-the-art system-agnostic methods (LIME (Ribeiro et al., 2016), SHAP (Lundberg & Lee, 2017) and L2X (Chen et al., 2018)), and a commonly used model-speciﬁc method called Saliency Map (Simonyan et al., 2013).
Revised: Based on these criteria, we compared VIBI with three state-of-the-art system-agnostic methods (LIME (Ribeiro et al., 2016), SHAP (Lundberg & Lee, 2017) and L2X (Chen et al., 2018)), and a commonly used model-speciﬁc method called Saliency Map (Simonyan et al., 2013).

Original: For Saliency
Revised: For Saliency

Original: Map, we used the smooth gradient technique (Smilkov et al., 2017) to get visually sharp gradientbased sensitivity maps over the basic gradient saliency map.
Revised: Map, we used the smooth gradient technique (Smilkov et al., 2017) to get visually sharp gradientbased sensitivity maps over the basic gradient saliency map.

Original: See Supplementary Material B for further experimental details.
Revised: See Supplementary Material B for further experimental details.

Original: We examined how VIBI performs across different experimental settings varying the number of selected chunks k (amount or number of explanation), size of chunk (unit of explanation), and tradeoff parameter β (trade-off between the compressiveness of explanation and information preserved about the output).
Revised: We examined how VIBI performs across different experimental settings varying the number of selected chunks k (amount or number of explanation), size of chunk (unit of explanation), and tradeoff parameter β (trade-off between the compressiveness of explanation and information preserved about the output).

Original: The settings of hyperparameter tuning include (bold indicate the choice for our ﬁnal model): the temperature for Gumbel-softmax approximation τ – { 0 . 1 , 0 . 2 , 0 . 5 , 0 .
Revised: The settings of hyperparameter tuning include (bold indicate the choice for our ﬁnal model): the temperature for Gumbel-softmax approximation τ – { 0 . 1 , 0 . 2 , 0 . 5 , 0 .

Original: 7 , 1 } , learning rate – 5 × 10 − 3 , 10 − 3 , 5 × 10 − 4 , 10 − 4 , 5 × 10 − 5 } and β – { 0 , 0 .
Revised: 7 , 1 } , learning rate – 5 × 10 − 3 , 10 − 3 , 5 × 10 − 4 , 10 − 4 , 5 × 10 − 5 } and β – { 0 , 0 .

Original: We use Adam algorithm (Kingma & Ba, 2014) with batch size 100 for MNIST and 50 for IMDB, the coefﬁcients used for computing running averages of gradient and its square ( β 1 , β 2 ) =
Revised: We use Adam algorithm (Kingma & Ba, 2014) with batch size 100 for MNIST and 50 for IMDB, the coefﬁcients used for computing running averages of gradient and its square ( β 1 , β 2 ) =

Original: (0 . 5 , 0 . 999) , and (cid:15) =
Revised: (0 . 5 , 0 . 999) , and (cid:15) =

Original: 10 − 8 .
Revised: 10 − 8 .

Original: We tuned the hyperparameters via grid search and picked up the hyperparameters that yield the best ﬁdelity score on the validation set.
Revised: We tuned the hyperparameters via grid search and picked up the hyperparameters that yield the best ﬁdelity score on the validation set.

Original: The code is publicly available on GitHub https://github.com/XXX .
Revised: The code is publicly available on GitHub https://github.com/XXX .

Original: 4. LSTM MOVIE SENTIMENT PREDICTION MODEL USING IMDB
Revised: 4. LSTM MOVIE SENTIMENT PREDICTION MODEL USING IMDB

Original: The IMDB (Maas et al., 2011) is a large text dataset containing movie reviews labeled by sentiment (positive/negative).
Revised: The IMDB (Maas et al., 2011) is a large text dataset containing movie reviews labeled by sentiment (positive/negative).

Original: We grouped the reviews into training, validation, and test sets, which have due v
Revised: We grouped the reviews into training, validation, and test sets, which have due v

Original: Negative Sentiment if any negative words
Revised: Negative Sentiment if any negative words

Original: I do NOT understand why anyone would waste their time or money on utter trash like this … Don't get me wrong -- I LOVE a good Western -- Notice I said "GOOD" -- this is just trash.
Revised: I do NOT understand why anyone would waste their time or money on utter trash like this … Don't get me wrong -- I LOVE a good Western -- Notice I said "GOOD" -- this is just trash.

Original: The acting is horrible -- Val Kilmer must know someone or owed a favor or something for them just to use his face and name in this ridiculous piece of crap...
Revised: The acting is horrible -- Val Kilmer must know someone or owed a favor or something for them just to use his face and name in this ridiculous piece of crap...

Original: True: Negative / B-Box: Negative
Revised: True: Negative / B-Box: Negative

Original: I watched this movie when it was released and being really young and not too much into cinema it was one of the most fascinating cinematic experiences I ever had and it really left a mark inside me.
Revised: I watched this movie when it was released and being really young and not too much into cinema it was one of the most fascinating cinematic experiences I ever had and it really left a mark inside me.

Original: At ﬁrst I didn't quite understand the story and probably failed to …
Revised: At ﬁrst I didn't quite understand the story and probably failed to …

Original: He plays so well the man that falls in love slowly but so deeply with Katherine Clifton , opens up his heart and dives into this prohibited a ﬀ air ….
Revised: He plays so well the man that falls in love slowly but so deeply with Katherine Clifton , opens up his heart and dives into this prohibited a ﬀ air ….

Original: Positive Sentiment if any positive words
Revised: Positive Sentiment if any positive words

Original: True: Positive / B-Box: Positive
Revised: True: Positive / B-Box: Positive

Original: The reality of the maﬁa environment is absolutely dog-eat- dog where a gangster will be killed for showing any sign of weakness because they become a liability.
Revised: The reality of the maﬁa environment is absolutely dog-eat- dog where a gangster will be killed for showing any sign of weakness because they become a liability.

Original: I've got no problem with the human side of gansters' being portrayed but Bugsy steers too far in the direction of soft, comical, men.
Revised: I've got no problem with the human side of gansters' being portrayed but Bugsy steers too far in the direction of soft, comical, men.

Original: The ﬁlm is enjoyable but is only light entertainment and not a biopic of a man who, though exciting , was extremely dangerous and fearsome.
Revised: The ﬁlm is enjoyable but is only light entertainment and not a biopic of a man who, though exciting , was extremely dangerous and fearsome.

Original: The acting's all good and the direction very solid.
Revised: The acting's all good and the direction very solid.

Original: The locations and era are very well represented and the themes very interesting….
Revised: The locations and era are very well represented and the themes very interesting….

Original: Negative sentiment but predicted as Positive
Revised: Negative sentiment but predicted as Positive

Original: [Figure]
Revised: [Figure]

Original: 25,000, 12,500, and 12,500 reviews respectively.
Revised: 25,000, 12,500, and 12,500 reviews respectively.

Original: Then, we trained a hierarchical LSTM for sentiment prediction, which achieved 87% of test accuracy.
Revised: Then, we trained a hierarchical LSTM for sentiment prediction, which achieved 87% of test accuracy.

Original: In order to explain this LSTM black-box model, we applied VIBI.
Revised: In order to explain this LSTM black-box model, we applied VIBI.

Original: We parameterized the explainer using a bidirectional LSTM and approximator using a 2D CNN.
Revised: We parameterized the explainer using a bidirectional LSTM and approximator using a 2D CNN.

Original: For the details of the black-box model and VIBI architectures, see Supplementary
Revised: For the details of the black-box model and VIBI architectures, see Supplementary

Original: Material B.1.
Revised: Material B.1.

Original: VIBI explains why the LSTM predicts each movie review to be positive/negative and provides instance-wise key words that are the most important attributes to the sentiment prediction.
Revised: VIBI explains why the LSTM predicts each movie review to be positive/negative and provides instance-wise key words that are the most important attributes to the sentiment prediction.

Original: As seen in the top-right and top-left of Figure 2, VIBI shows that the positive (or negative) words pass through the bottleneck and make a correct prediction.
Revised: As seen in the top-right and top-left of Figure 2, VIBI shows that the positive (or negative) words pass through the bottleneck and make a correct prediction.

Original: The bottom of Figure 2 shows that the LSTM sentiment prediction model makes a wrong prediction for a negative review because the review includes several positive words such as ‘enjoyable’ and ‘exciting’.
Revised: The bottom of Figure 2 shows that the LSTM sentiment prediction model makes a wrong prediction for a negative review because the review includes several positive words such as ‘enjoyable’ and ‘exciting’.

Original: 4. CNN DIGIT RECOGNITION MODEL USING MNIST
Revised: 4. CNN DIGIT RECOGNITION MODEL USING MNIST

Original: [Figure]
Revised: [Figure]

Original: The MNIST (LeCun et al., 1998) is a large dataset contains 28 × 28 sized images of handwritten digits (0 to 9).
Revised: The MNIST (LeCun et al., 1998) is a large dataset contains 28 × 28 sized images of handwritten digits (0 to 9).

Original: We grouped the images into training, validation, and test sets, which have 50,000, 10,000, and 10,000 images respectively, and trained a simple 2D CNN for the digit recognition, which achieved 97% of test accuracy.
Revised: We grouped the images into training, validation, and test sets, which have 50,000, 10,000, and 10,000 images respectively, and trained a simple 2D CNN for the digit recognition, which achieved 97% of test accuracy.

Original: In order to explain this CNN black-box model, we applied VIBI.
Revised: In order to explain this CNN black-box model, we applied VIBI.

Original: We parameterized each the explainer and approximator using a 2D CNN.
Revised: We parameterized each the explainer and approximator using a 2D CNN.

Original: For the details of the black-box model and VIBI architectures, see Supplementary Material B.2.
Revised: For the details of the black-box model and VIBI architectures, see Supplementary Material B.2.

Original: VIBI explains how the CNN characterizes a digit and recognizes differences between digits.
Revised: VIBI explains how the CNN characterizes a digit and recognizes differences between digits.

Original: The ﬁrst two examples in Figure 3 show that the CNN recognizes digits using both shapes and angles.
Revised: The ﬁrst two examples in Figure 3 show that the CNN recognizes digits using both shapes and angles.

Original: In the ﬁrst example, the CNN characterizes ‘1’s by straightly aligned patches along with the activated regions although ‘1’s in the left and right panels are written at different angles.
Revised: In the ﬁrst example, the CNN characterizes ‘1’s by straightly aligned patches along with the activated regions although ‘1’s in the left and right panels are written at different angles.

Original: Contrary to the ﬁrst example, the second example shows that the CNN recognizes the difference between ‘9’ and ‘6’ by their differences in angles.
Revised: Contrary to the ﬁrst example, the second example shows that the CNN recognizes the difference between ‘9’ and ‘6’ by their differences in angles.

Original: The last two examples in Figure 3 show that the CNN catches a difference of ‘7’s from ‘1’s by patches located on the activated horizontal line on ‘7’ (see the cyan circle) and vi recognizes ‘8’s by two patches on the top of the digits and another two patches at the bottom circle.
Revised: The last two examples in Figure 3 show that the CNN catches a difference of ‘7’s from ‘1’s by patches located on the activated horizontal line on ‘7’ (see the cyan circle) and vi recognizes ‘8’s by two patches on the top of the digits and another two patches at the bottom circle.

Original: More qualitative examples for VIBI and the baselines are showed in Supplementary Figure 6.
Revised: More qualitative examples for VIBI and the baselines are shown in Supplementary Figure 6.

Original: 4. TCR TO EPITOPE BINDING PREDICTION MODEL USING VDJ DB AND IEDB
Revised: 4. TCR TO EPITOPE BINDING PREDICTION MODEL USING VDJ DB AND IEDB

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure] [Table] [Figure]
Revised: h e d

Original: [Table] [Figure] [Table] [Figure]
Revised: U n - m a t c h e d

Original: [Table] [Figure] [Table] [Figure]
Revised: U n - m a t c h e d

Original: [Table] [Figure] [Table] [Figure]
Revised: U n - m a t c h e d

Original: [Table] [Figure] [Table] [Figure]
Revised: U n - m a t c h e d

Original: [Table]

Original: h e d U n - m a t c h e d U n - m a t c h e d U n - m a t c h e d
Revised: U n - m a t c h e d

Original: U n - m a t c h e d

Original: U n - m a t c h e d

Original: U n - m a t c h e d
Revised: U n - m a t c h e d

Original: E
Revised: E

Original: G I L G F V F T L
Revised: G I L G F V F T L

Original: T C R ASSARATDTQY
Revised: T C R ASSARATDTQY

Original: [Figure]
Revised: [Figure]

Original: G I L G F V F T L
Revised: G I L G F V F T L

Original: ASSGRSQETQY e p i t
Revised: ASSGRSQETQY e p i t

Original: o p e
Revised: o p e

Original: VDJdb
Revised: VDJdb

Original: IEDB
Revised: IEDB

Original: T C R e p i t
Revised: T C R e p i t

Original: o p e
Revised: o p e

Original: The two explanations are 
Revised: The two explanations are matched

Original: [Figure] We next illustrate how VIBI can be used to get insights from a model and ensure  safety of a model in a real world application.
Revised: We next illustrate how VIBI can be used to get insights from a model and ensure the safety of a model in a real world application.

Original: Identifying which T-cell receptor (TCR) will bind to a speciﬁc epitope (i.e. cancer induced peptide molecules presented by the major histocompatibility complex to T-cells) is important for screening T-cells or genetically engineering T-cells that are effective in recognizing and destroying tumor cells.
Revised: Identifying which T-cell receptor (TCR) will bind to a speciﬁc epitope (i.e. cancer induced peptide molecules presented by the major histocompatibility complex to T-cells) is important for screening T-cells or genetically engineering T-cells that are effective in recognizing and destroying tumor cells.

Original: Therefore, there has been efforts in developing computational methods to predict binding afﬁnity of given TCR-epitope pairs (Jurtz et al., 2018; Jokinen et al., 2019).
Revised: Therefore, there has been efforts in developing computational methods to predict binding afﬁnity of given TCR-epitope pairs (Jurtz et al., 2018; Jokinen et al., 2019).

Original: These approaches rely on known interacting TCR-epitope pairs available from VDJdb (Shugay et al., 2017) and IEDB (Vita et al., 2014), which are the largest databases of several thousands entries.
Revised: These approaches rely on known interacting TCR-epitope pairs available from VDJdb (Shugay et al., 2017) and IEDB (Vita et al., 2014), which are the largest databases of several thousand entries.

Original: However, the number of unique TCRs harbored in a single individual is estimated to be 10 10 (Lythe et al., 2016) and a theoretical number of epitopes of length l is 20 l , which are much larger than the number of known interacting TCR-epitope pairs.
Revised: However, the number of unique TCRs harbored in a single individual is estimated to be 10 10 (Lythe et al., 2016) and a theoretical number of epitopes of length l is 20 l , which are much larger than the number of known interacting TCR-epitope pairs.

Original: One of the main concerns is whether a black-box model trained on such limited dataset can accurately predict TCR-epitope bindings of out-of-samples.
Revised: One of the main concerns is whether a black-box model trained on such limited dataset can accurately predict TCR-epitope bindings of out-of-samples.

Original: This concern becomes pressing in a TCR-epitope binding prediction model trained on VDJdb (For details of the data, black-box model architecture, and parameter tuning, see Supplementary Material B.3).
Revised: This concern becomes pressing in a TCR-epitope binding prediction model trained on VDJdb (For details of the data, black-box model architecture, and parameter tuning, see Supplementary Material B.3).

Original: The model accurately predicted the (in-sample) bindings from VDJdb (recall 0.79, Figure 4A).
Revised: The model accurately predicted the (in-sample) bindings from VDJdb (recall 0.79, Figure 4A).

Original: However, it achieved poor prediction performance when it is used to predict the (out-of-sample) bindings from another dataset, IEDB (recall 0.40,
Revised: However, it achieved poor prediction performance when it is used to predict the (out-of-sample) bindings from another dataset, IEDB (recall 0.40,

Original: Figure 4B).
Revised: Figure 4B).

Original: In an attempt to address this problem, we applied VIBI and determined whether or not to accept a decision made by the black-box model based on VIBI’s explanation.
Revised: In an attempt to address this problem, we applied VIBI and determined whether or not to accept a decision made by the black-box model based on VIBI’s explanation.

Original: As illustrated in
Revised: As illustrated in

Original: Figure 4E, VIBI provided matched explanations —the identical amino acids in same positions ( S and Y in this example) are highlighted in different TCR sequences when they are bound to the same epitope ( GILGFVFTL in this example).
Revised: Figure 4E, VIBI provided matched explanations —the identical amino acids in same positions ( S and Y in this example) are highlighted in different TCR sequences when they are bound to the same epitope ( GILGFVFTL in this example).

Original: Moreover, we found that if two TCR sequences binding to  same epitope, each from IEDB and VDJdb, are assigned with matched explanations by VIBI, then it signiﬁcantly better predicts the binding than the others with no matching TCRs (Figure 4C-D, p-values are shown).
Revised: Moreover, we found that if two TCR sequences binding to the same epitope, each from IEDB and VDJdb, are assigned with matched explanations by VIBI, then it signiﬁcantly better predicts the binding than the others with no matching TCRs (Figure 4C-D, p-values are shown).

Original: Therefore, if a TCR sequence from IEDB has a matched explanation to a TCR from VDJdb, then we safely follow the positive decision made by the black-box model.
Revised: Therefore, if a TCR sequence from IEDB has a matched explanation to a TCR from VDJdb, then we safely follow the positive decision made by the black-box model.

Original: 4. I NTERPRETABILITY EVALUATED BY HUMANS
Revised: 4. I NTERPRETABILITY EVALUATED BY HUMANS

Original: We evaluated interpretability of the methods on the LSTM movie sentiment prediction model and the CNN digit recognition model.
Revised: We evaluated interpretability of the methods on the LSTM movie sentiment prediction model and the CNN digit recognition model.

Original: For the movie sentiment prediction model, we provided instances that the black-box model had correctly predicted and asked humans to infer the output of the primary sentiment of the movie review (Positive/Negative/Neutral) given ﬁve key words selected by each method.
Revised: For the movie sentiment prediction model, we provided instances that the black-box model had correctly predicted and asked humans to infer the output of the primary sentiment of the movie review (Positive/Negative/Neutral) given ﬁve key words selected by each method.

Original: Each method was evaluated by the human intelligences on MTurk 2 who are awarded the 
Revised: Each method was evaluated by the humans on MTurk 2 who are awarded the Masters

Original: Masters Qualiﬁcation, high-performance workers who have demonstrated excellence across a wide range of tasks).
Revised: Qualiﬁcation, high-performance workers who have demonstrated excellence across a wide range of tasks).

Original: We randomly selected and evaluated 200 instances for VIBI and 100 instances for the others.
Revised: We randomly selected and evaluated 200 instances for VIBI and 100 instances for the others.

Original: Five workers were assigned per instance.
Revised: Five workers were assigned per instance.

Original: For the digit recognition model, we asked humans vii to directly score the explanation on a 0–5 scale.
Revised: For the digit recognition model, we asked humans vii to directly score the explanation on a 0–5 scale.

Original: Each method was evaluated by 16 graduate students at XXX-University 3 who have taken at least one graduate-level machine learning class.
Revised: Each method was evaluated by 16 graduate students at XXX-University 3 who have taken at least one graduate-level machine learning class.

Original: For each method, 100 instances were randomly selected and evaluated.
Revised: For each method, 100 instances were randomly selected and evaluated.

Original: Four cognitive chunks with the size 4 × 4 were provided as an explanation for each instance ( β = 0 .
Revised: Four cognitive chunks with the size 4 × 4 were provided as an explanation for each instance ( β = 0 .

Original: 1 for VIBI).
Revised: 1 for VIBI).

Original: On average, 4.26 students were assigned per instance.
Revised: On average, 4.26 students were assigned per instance.

Original: Further details regarding the experiments can be found in Supplementary
Revised: Further details regarding the experiments can be found in Supplementary

Original: Material C.
Revised: Material C.

Original: [Table]
Revised: [Table]

Original: As shown by the Table
Revised: As shown by the Table

Original: [Table]
Revised: [Table]

Original: VIBI better explains the black-box models.
Revised: VIBI better explains the black-box models.

Original: When explaining the movie sentiment prediction model, humans better inferred the (correctly predicted) black-box output given the ﬁve keywords when they were provided by VIBI.
Revised: When explaining the movie sentiment prediction model, humans better inferred the (correctly predicted) black-box output given the ﬁve keywords when they were provided by VIBI.

Original: Therefore, it better captures the most contributing key words to the LSTM decision and better explains why the LSTM predicted each movie review by providing ﬁve key words.
Revised: Therefore, it better captures the most contributing key words to the LSTM decision and better explains why the LSTM predicted each movie review by providing ﬁve key words.

Original: For explaining the digit recognition model, VIBI also highlighted the most concise chunks for explaining key characteristics ofthe handwritten digit.
Revised: For explaining the digit recognition model, VIBI also highlighted the most concise chunks for explaining key characteristics of handwritten digit.

Original: Thus, it better explains how the CNN model recognized each the handwritten digit.
Revised: Thus, it better explains how the CNN model recognized each the handwritten digit.

Original: 4. F IDELITY
Revised: 4. F IDELITY

Original: [Table]
Revised: [Table]

Original: We assessed ﬁdelity of the methods in approximating the black-box output.
Revised: We assessed ﬁdelity of the methods in approximating the black-box output.

Original: First, we compared  ability of the approximators to imitate behaviour of the black-box, denoted as Approximator ﬁdelity .
Revised: First, we compared the ability of the approximators to imitate behaviour of the black-box, denoted as Approximator ﬁdelity .

Original: (See Supplementary Material B.4 for details about how each approximator ﬁdelity is evaluated.)
Revised: (See Supplementary Material B.4 for details about how each approximator ﬁdelity is evaluated.)

Original: As shown in Table 2, VIBI has a better approximator ﬁdelity than Saliency, LIME and SHAP in most cases.
Revised: As shown in Table 2, VIBI has a better approximator ﬁdelity than Saliency, LIME and SHAP in most cases.

Original: VIBI and L2X showed similar levels of approximator ﬁdelity, so we further compared them based on Rationale ﬁdelity .
Revised: VIBI and L2X showed similar levels of approximator ﬁdelity, so we further compared them based on Rationale ﬁdelity .

Original: The difference between approximator and rationale ﬁdelity is as follows.
Revised: The difference between approximator and rationale ﬁdelity is as follows.

Original: Approximator ﬁdelity is quantiﬁed by prediction performance of the approximators that takes t ∗ , the continuous relaxation of t , as an input and the black-box output as a targeted label; rationale ﬁdelity is quantiﬁed by using t instead of t ∗ .
Revised: Approximator ﬁdelity is quantiﬁed by prediction performance of the approximators that takes t ∗ , the continuous relaxation of t , as an input and the black-box output as a targeted label; rationale ﬁdelity is quantiﬁed by using t instead of t ∗ .

Original: Note that t only takes the top k chunks and sets the others to be zero, while t ∗ sets the others to be small, non-zero values.
Revised: Note that t only takes the top k chunks and sets the others to be zero, while t ∗ sets the others to be small, non-zero values.

Original: Therefore, rationale ﬁdelity allows to evaluate how much information purely ﬂows through the explanations, not through a narrow crack made during the continuous relaxation procedure.
Revised: Therefore, rationale ﬁdelity allows to evaluate how much information purely ﬂows through the explanations, not through a narrow crack made during the continuous relaxation procedure.

Original: As shown in Table 2, VIBI has a better viii rationale ﬁdelity than L2X in most cases.
Revised: As shown in Table 2, VIBI has a better viii rationale ﬁdelity than L2X in most cases.

Original: Note that L2X can be viewed as a special case of VIBI without the compressiveness term, i.e., β = 0 .
Revised: Note that L2X can be viewed as a special case of VIBI without the compressiveness term, i.e., β = 0 .

Original: The rationale ﬁdelity empirically demonstrates that the compressiveness term can help the information to ﬂow purely through the explanations.
Revised: The rationale ﬁdelity empirically demonstrates that the compressiveness term can help the information to ﬂow purely through the explanations.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We employ the information bottleneck principle as a criterion for learning ‘good’ explanations.
Revised: We employ the information bottleneck principle as a criterion for learning ‘good’ explanations.

Original: Instance-wisely selected cognitive chunks work as an information bottleneck, hence, provide concise but comprehensive explanations for each decision made by a black-box system.
Revised: Instance-wisely selected cognitive chunks work as an information bottleneck, hence, provide concise but comprehensive explanations for each decision made by a black-box system.

Original: A V ARIATIONAL INFERENCE TO CONSTRUCT A LOWER BOUND ON THE IB OBJECTIVE
Revised: A V ARIATIONAL INFERENCE TO CONSTRUCT A LOWER BOUND ON THE IB OBJECTIVE

Original: We illustrate the variational approximation of the following information bottleneck objective:
Revised: We illustrate the variational approximation of the following information bottleneck objective:

Original: [Equation]
Revised: [Equation]

Original: Let us start with the Markov chain assumption y
Revised: Let us start with the Markov chain assumption y

Original: → x → t , which also implies y ← x ← t .
Revised: → x → t , which also implies y ← x ← t .

Original: Therefore, the condition is written as y ↔ x ↔ t (Cover & Thomas, 2012).
Revised: Therefore, the condition is written as y ↔ x ↔ t (Cover & Thomas, 2012).

Original: Using this condition, we factorize the joint distribution p ( y , x , t ) :
Revised: Using this condition, we factorize the joint distribution p ( y , x , t ) :

Original: [Equation]
Revised: [Equation]

Original: where p ( y | t , x ) =
Revised: where p ( y | t , x ) =

Original: p ( y | t ) by the Markov chain.
Revised: p ( y | t ) by the Markov chain.

Original: Now, we will examine each of the expressions in the information bottleneck objective I( t , y )
Revised: Now, we will examine each of the expressions in the information bottleneck objective I( t , y )

Original: − β I( x , t ) in turn.
Revised: − β I( x , t ) in turn.

Original: ( I ) V ARIATIONAL LOWER BOUND FOR I( x , t )
Revised: ( I ) V ARIATIONAL LOWER BOUND FOR I( x , t )

Original: We ﬁrst show that I( x , t ) ≤ I( x , z ) + C where C is a constant and then use the lower bound for − I( x , z ) − C as a lower bound for − I( x , t ) .
Revised: We ﬁrst show that I( x , t ) ≤ I( x , z ) + C where C is a constant and then use the lower bound for − I( x , z ) − C as a lower bound for − I( x , t ) .

Original: First, we prove I( x , t ) ≤ I( x , z ) + C .
Revised: First, we prove I( x , t ) ≤ I( x , z ) + C .

Original: From the Markov Chain x → ( x , z ) → t , we have I( x , t ) ≤
Revised: From the Markov Chain x → ( x , z ) → t , we have I( x , t ) ≤

Original: I( x , ( x , z )) .
Revised: I( x , ( x , z )) .

Original: According to the chain rule for mutual information, I( x , ( x , z ))
Revised: According to the chain rule for mutual information, I( x , ( x , z ))

Original: =
Revised: =

Original: I( x , z ) + I( x , x | z ) , where I( x , x | z ) =
Revised: I( x , z ) + I( x , x | z ) , where I( x , x | z ) =

Original: H( x | z ) + H( x | z ) − H( x , x | z ) .
Revised: H( x | z ) + H( x | z ) − H( x , x | z ) .

Original: Further, H( x | z ) ≤ H( x ) .
Revised: Further, H( x | z ) ≤ H( x ) .

Original: Putting these pieces together, we have
Revised: Putting these pieces together, we have

Original: [Equation]
Revised: [Equation]

Original: where entropy H( x ) of input is a constant.
Revised: where entropy H( x ) of input is a constant.

Original: For simplicity, we denote it as C .
Revised: For simplicity, we denote it as C .

Original: We then approximate p ( z ) using r ( z ) .
Revised: We then approximate p ( z ) using r ( z ) .

Original: From the fact that Kullback Leibler divergence is always positive, we have
Revised: From the fact that Kullback Leibler divergence is always positive, we have

Original: [Equation]
Revised: [Equation]

Original: From (6) and (7), we have
Revised: From (6) and (7), we have

Original: [Equation]
Revised: [Equation]

Original: ( II ) V ARIATIONAL LOWER BOUND FOR I( t , y )
Revised: ( II ) V ARIATIONAL LOWER BOUND FOR I( t , y )

Original: Starting with I( t , y ) , we have
Revised: Starting with I( t , y ) , we have

Original: [Equation]
Revised: [Equation]

Original: xi
Revised: xi

Original: Note that H( y ) =
Revised: Note that H( y ) =

Original: − E y ∼ p
Revised: − E y ∼ p

Original: ( y )
Revised: ( y )

Original: [log p ( y )] is independent of the optimization procedure, hence can be ignored.
Revised: [log p ( y )] is independent of the optimization procedure, hence can be ignored.

Original: Now, we use q ( y | t ) to approximate p ( y | t ) which works as an approximator to the black-box system.
Revised: Now, we use q ( y | t ) to approximate p ( y | t ) which works as an approximator to the black-box system.

Original: Using the fact that Kullback Leibler divergence is always positive, we have
Revised: Using the fact that Kullback Leibler divergence is always positive, we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where p ( t | x , y ) =
Revised: where p ( t | x , y ) =

Original: p ( t | x ) by the Markov chain assumption y ↔ x ↔ t .
Revised: p ( t | x ) by the Markov chain assumption y ↔ x ↔ t .

Original: ( III ) V ARIATIONAL LOWER BOUND FOR I( t , y ) − β I( x , t )
Revised: ( III ) V ARIATIONAL LOWER BOUND FOR I( t , y ) − β I( x , t )

Original: In summary, we have the following variational bounds for each term.
Revised: In summary, we have the following variational bounds for each term.

Original: [Equation]
Revised: [Equation]

Original: which result in
Revised: which result in

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: t ,
Revised: t ,

Original: [Equation]
Revised: [Equation]

Original: With proper choices of r ( z ) and p ( z | x ) , we assume that the Kullback-Leibler divergence
Revised: With proper choices of r ( z ) and p ( z | x ) , we assume that the Kullback-Leibler divergence

Original: D KL ( p ( z | x ) , r ( z )) is integrated analytically.
Revised: D KL ( p ( z | x ) , r ( z )) is integrated analytically.

Original: We use the empirical data distribution to approximate p ( x , y )
Revised: We use the empirical data distribution to approximate p ( x , y )

Original: =
Revised: =

Original: p ( x ) p ( y | x ) and p ( x ) .
Revised: p ( x ) p ( y | x ) and p ( x ) .

Original: Our variational approximation is similar to the one from Alemi et al.
Revised: Our variational approximation is similar to the one from Alemi et al.

Original: (2017), which ﬁrst developed variational lower bound on the information bottleneck for deep neural networks.
Revised: (2017), which ﬁrst developed variational lower bound on the information bottleneck for deep neural networks.

Original: However, our information bottleneck is different from theirs: their information bottleneck is the stochastic encoding z of the input x as itself, whereas our information bottleneck is a pairwise product of the stochastic encoding z and the input x where z is a Boolean random vector.
Revised: However, our information bottleneck is different from theirs: their information bottleneck is the stochastic encoding z of the input x as itself, whereas our information bottleneck is a pairwise product of the stochastic encoding z and the input x where z is a Boolean random vector.

Original: Due to this difference, we appproximate I( x , t ) by using the lower bound of I( x , z ) , instead of directly deriving the lower bound of I( x , t ) .
Revised: Due to this difference, we appproximate I( x , t ) by using the lower bound of I( x , z ) , instead of directly deriving the lower bound of I( x , t ) .

Original: xii
Revised: xii

Original: B E XPERIMENTAL D ETAILS
Revised: B E XPERIMENTAL D ETAILS

Original: The settings of hyperparameter tuning include the followings (bold indicate the choice for our ﬁnal model): the temperature for Gumbel-softmax approximation τ – { 0 . 1 , 0 . 2 , 0 .
Revised: The settings of hyperparameter tuning include the followings (bold indicate the choice for our ﬁnal model): the temperature for Gumbel-softmax approximation τ – { 0 . 1 , 0 . 2 , 0 .

Original: 7 , 1 } , learning rate – 5 × 10 − 3 , 10 − 3 , 5 × 10 − 4 , 10 − 4 , 5 × 10 − 5 } and β – { 0 , 0 .
Revised: 7 , 1 } , learning rate – 5 × 10 − 3 , 10 − 3 , 5 × 10 − 4 , 10 − 4 , 5 × 10 − 5 } and β – { 0 , 0 .

Original: We use Adam algorithm (Kingma & Ba, 2014) with batch size 100 for MNIST and 50 for IMDB, the coefﬁcients used for computing running averages of gradient and its square ( β 1 , β 2 ) =
Revised: We use Adam algorithm (Kingma & Ba, 2014) with batch size 100 for MNIST and 50 for IMDB, the coefﬁcients used for computing running averages of gradient and its square ( β 1 , β 2 ) =

Original: (0 . 5 , 0 . 999) , and (cid:15) =
Revised: (0 . 5 , 0 . 999) , and (cid:15) =

Original: 10 − 8 .
Revised: 10 − 8 .

Original: We tuned the hyperparameters via grid search and picked up the hyperparameters that yield the best ﬁdelity score on the validation set.
Revised: We tuned the hyperparameters via grid search and picked up the hyperparameters that yield the best ﬁdelity score on the validation set.

Original: All implementation is performed via PyTorch an open source deep learning platform (Paszke et al., 2017).
Revised: All implementation is performed via PyTorch an open source deep learning platform (Paszke et al., 2017).

Original: Approximator ﬁdelity is prediction performance of the approximator that takes relaxation t ∗ as an input and black-box output as a target.
Revised: Approximator ﬁdelity is prediction performance of the approximator that takes relaxation t ∗ as an input and black-box output as a target.

Original: In detail, we get a t ∗ with the Gumble-softmax sampling and make a prediction based on t ∗ .
Revised: In detail, we get a t ∗ with the Gumble-softmax sampling and make a prediction based on t ∗ .

Original: This procedure is repeated for 12 times and the ﬁnal prediction is made by averaging the 12 prediction scores.
Revised: This procedure is repeated for 12 times and the ﬁnal prediction is made by averaging the 12 prediction scores.

Original: Rationale ﬁdelity is prediction performance of the approximator that takes t as an input and black-box output as a target.
Revised: Rationale ﬁdelity is prediction performance of the approximator that takes t as an input and black-box output as a target.

Original: (Note that t is a masked input that only takes the top k chunks and set others to zero.)
Revised: (Note that t is a masked input that only takes the top k chunks and set others to zero.)

Original: The ﬁnal prediction is made by its prediction score.
Revised: The ﬁnal prediction is made by its prediction score.

Revised: We use the prior r j ( z j ) =

Revised: ( k/J )

Revised: z j (1 − k/J ) 1 − z j for all experiments where J is the total number of chunks and r ( z ) =

Revised: (cid:81)

Revised: j r j ( z j ) .

Original: B. LSTM MOVIE SENTIMENT PREDICTION MODEL USING IMDB
Revised: The analytical form of the Kullback-Leibler divergence term is D KL ( p ( z | x ) , r ( z )) = (cid:80) Jj = 1 D

Revised: [Table]

Revised: B. LSTM MOVIE SENTIMENT PREDICTION MODEL USING IMDB

Original: Black-Box Model Structure.
Revised: Black-Box Model Structure.

Original: Each review is padded or cut to contain 15 sentences and 50 words for each sentence.
Revised: Each review is padded or cut to contain 15 sentences and 50 words for each sentence.

Original: The architecture consists of a word-embedding layer with size 50 for each word followed by two bidirectional LSTMs, a fully connected layer with two units, and a soft-max layer.
Revised: The architecture consists of a word-embedding layer with size 50 for each word followed by two bidirectional LSTMs, a fully connected layer with two units, and a soft-max layer.

Original: The ﬁrst LSTM layer encodes the word embedding vector and generates a word-representation vector with size 100 for each word.
Revised: The ﬁrst LSTM layer encodes the word embedding vector and generates a word-representation vector with size 100 for each word.

Original: Within each sentence, the word representation vectors are elementwisely averaged to form a size 100 sentence representation vector.
Revised: Within each sentence, the word representation vectors are elementwisely averaged to form a size 100 sentence representation vector.

Original: The second LSTM layer encodes the sentence representation vector and generates a size 60 review embedding vector.
Revised: The second LSTM layer encodes the sentence representation vector and generates a size 60 review embedding vector.

Original: VIBI Structure.
Revised: VIBI Structure.

Original: We parameterize the explainer with a bidirectional LSTM and approximator with a 2D CNN.
Revised: We parameterize the explainer with a bidirectional LSTM and approximator with a 2D CNN.

Original: For the explainer, we use a bidirectional LSTM layer from which the output vector is averaged and followed by log-softmax calculation.
Revised: For the explainer, we use a bidirectional LSTM layer from which the output vector is averaged and followed by log-softmax calculation.

Original: The ﬁnal layer is formed to return a log-probability indicating which cognitive chunks should be taken as an input to the approximator.
Revised: The ﬁnal layer is formed to return a log-probability indicating which cognitive chunks should be taken as an input to the approximator.

Original: For the approximator, we use a convolutional layer followed by a ReLU activation function and max-pooling layer and a fully connected layer returning a size-2 vector followed by a log-softmax calculation.
Revised: For the approximator, we use a convolutional layer followed by a ReLU activation function and max-pooling layer and a fully connected layer returning a size-2 vector followed by a log-softmax calculation.

Original: The ﬁnal layer returns a vector of log-probabilities for the two sentiments (positive/negative).
Revised: The ﬁnal layer returns a vector of log-probabilities for the two sentiments (positive/negative).

Original: B. CNN DIGIT RECOGNITION MODEL USING MNIST
Revised: B. CNN DIGIT RECOGNITION MODEL USING MNIST

Original: Black-Box Model Structure.
Revised: Black-Box Model Structure.

Original: The architecture consists of two convolutional layers with the kernel size 5 followed by a max-pooling layer with the pool size 2, two fully connected layers and a soft-max layer.
Revised: The architecture consists of two convolutional layers with the kernel size 5 followed by a max-pooling layer with the pool size 2, two fully connected layers and a soft-max layer.

Original: The two convolutional layers contain 10 and 20 ﬁlters respectively and the two fully connected layers are composed of 50 and 10 units respectively.
Revised: The two convolutional layers contain 10 and 20 ﬁlters respectively and the two fully connected layers are composed of 50 and 10 units respectively.

Original: VIBI Structure.
Revised: VIBI Structure.

Original: We parameterize each the explainer and approximator using 2D CNNs.
Revised: We parameterize each the explainer and approximator using 2D CNNs.

Original: Structure of the explainer differs depending on the chunk size.
Revised: Structure of the explainer differs depending on the chunk size.

Original: For example, when 4 × 4 cognitive chunk is used, we use two convolutional layers with the kernel size 5 followed by a ReLU activation function and max-pooling layer with the pool size 2, and one convolutional layer with kernel sizereturning a 7 × 7 2D matrix followed by a log-softmax calculation.
Revised: For example, when 4 × 4 cognitive chunk is used, we use two convolutional layers with the kernel size 5 followed by a ReLU activation function and max-pooling layer with the pool size 2, and one convolutional layer with kernel sizereturning a 7 × 7 2D matrix followed by a log-softmax calculation.

Original: The ﬁnal layer returns a vector of log-probabilities for the 49 chunks.
Revised: The ﬁnal layer returns a vector of log-probabilities for the 49 chunks.

Original: The three convolutional layers contains 8, 16, and 1 ﬁlters  respectively.
Revised: The three convolutional layers contains 8, 16, and 1 ﬁlters xiii respectively.

Original: The output from the explainer indicates which cognitive chunks should be taken as an input for the approximator.
Revised: The output from the explainer indicates which cognitive chunks should be taken as an input for the approximator.

Original: We parameterize the approximator using two convolutional layers with kernel size 5 followed by a ReLU activation function and max-pooling layer with pool size 2 and withxiii 32 and 64 ﬁlters respectively, and one fully connected layer returning a size-10 vector followed by a log-softmax calculation so that the ﬁnal layer returns a vector of log-probabilities for the ten digits.
Revised: We parameterize the approximator using two convolutional layers with kernel size 5 followed by a ReLU activation function and max-pooling layer with pool size 2 and with 32 and 64 ﬁlters respectively, and one fully connected layer returning a size-10 vector followed by a log-softmax calculation so that the ﬁnal layer returns a vector of log-probabilities for the ten digits.

Original: B. TCR TO EPITOPE BINDING PREDICTION MODEL USING VDJ DB AND IEDB
Revised: B. TCR TO EPITOPE BINDING PREDICTION MODEL USING VDJ DB AND IEDB

Original: Data.
Revised: Data.

Original: We use two public datasets: VDJdb and IEDB.
Revised: We use two public datasets: VDJdb and IEDB.

Original: VDJdb (Shugay et al., 2017) contains the T-cell receptor (TCR) sequences with known antigen speciﬁcity (i.e., epitope sequences).
Revised: VDJdb (Shugay et al., 2017) contains the T-cell receptor (TCR) sequences with known antigen speciﬁcity (i.e., epitope sequences).

Original: We use a VDJdb dataset preprocessed by Jokinen et al.
Revised: We use a VDJdb dataset preprocessed by Jokinen et al.

Original: (See their paper for details in data preprocessing).
Revised: (See their paper for details in data preprocessing).

Original: The preprocessed dataset contains 5,784 samples (2,892 positively and 2,892 negatively binding pairs of TCR and epitope).
Revised: The preprocessed dataset contains 5,784 samples (2,892 positively and 2,892 negatively binding pairs of TCR and epitope).

Original: This dataset consists of 4,363 unique TCR sequences and 21 unique epitope sequences.
Revised: This dataset consists of 4,363 unique TCR sequences and 21 unique epitope sequences.

Original: We group the pairs of TCR and epitope sequences into training, validation, and test sets, which have 4,627, 578, and 579 pairs, respectively.
Revised: We group the pairs of TCR and epitope sequences into training, validation, and test sets, which have 4,627, 578, and 579 pairs, respectively.

Original: IEDB (Vita et al., 2014) contains
Revised: IEDB (Vita et al., 2014) contains

Original: TCR sequences and corresponding epitopes.
Revised: TCR sequences and corresponding epitopes.

Original: We used an IEDB dataset preprocessed by Jurtz et al.
Revised: We used an IEDB dataset preprocessed by Jurtz et al.

Original: The preprocessed dataset contains 9,328 samples (positively binding pairs only) and consists of 9,221 unique TCR sequences and 98 unique epitopes sequences.
Revised: The preprocessed dataset contains 9,328 samples (positively binding pairs only) and consists of 9,221 unique TCR sequences and 98 unique epitopes sequences.

Original: We used the IEDB dataset as out-ouf-samples.
Revised: We used the IEDB dataset as out-ouf-samples.

Original: There are 6 epitopes contained in both VDJdb and IEDB dataset: GILGFVFTL ,
Revised: There are 6 epitopes contained in both VDJdb and IEDB dataset: GILGFVFTL ,

Original: GLCTLVAML , NLVPMVATV , LLWNGPMAV , YVLDHLIVV , CINGVCWTV .
Revised: GLCTLVAML , NLVPMVATV , LLWNGPMAV , YVLDHLIVV , CINGVCWTV .

Original: Black-Box Model Structure.
Revised: Black-Box Model Structure.

Original: Each TCR and epitope sequence is padded or cut to contain 20 and 13 amino acids, respectively.
Revised: Each TCR and epitope sequence is padded or cut to contain 20 and 13 amino acids, respectively.

Original: The embedding matrix has a size 24 and is initialized with
Revised: The embedding matrix has a size 24 and is initialized with

Original: BLOSUM50 matrix (Henikoff & Henikoff, 1992).
Revised: BLOSUM50 matrix (Henikoff & Henikoff, 1992).

Original: The architecture consists of two sequence encoders that process TCR and epitope sequences each, and three dense layers that process the encoded sequences together.
Revised: The architecture consists of two sequence encoders that process TCR and epitope sequences each, and three dense layers that process the encoded sequences together.

Original: The TCR encoder consists of a dropout layers with the drop-out probability 0.3, two convolutional layers with the kernel size 3 followed by a batch normalization layer, a ReLU activation function and max-pooling layer with the pool size 3.
Revised: The TCR encoder consists of a dropout layers with the drop-out probability 0.3, two convolutional layers with the kernel size 3 followed by a batch normalization layer, a ReLU activation function and max-pooling layer with the pool size 3.

Original: The two convolutional layers contain 32 and 16 ﬁlters respectively.
Revised: The two convolutional layers contain 32 and 16 ﬁlters respectively.

Original: Structure of the epitope encoder is the same with the one from the TCRencoder.
Revised: of the epitope encoder is the same with the one from the TCR

Original: We optimized themodels with the following search space (bold indicate the choice for our ﬁnal model): the batch size – { 25 , 50 , 100 } , learning rate – 0 . 05 , 0 . 01 , 0 .
Revised: We optimized the with the bold

Original: 001 , 0 . 0005 } and ﬁlter size– { (16 , 8) , ( 32 , 16 ) } .
Revised: ): the batch size

Original: VIBI Structure.
Revised: VIBI Structure.

Original: We parameterize the explainer and approximator using 2D CNNs and several dense layers.
Revised: We parameterize the explainer and approximator using 2D CNNs and several dense layers.

Original: Each TCR and epitope sequence is preprocessed and embedded in the same way as for the black-box model.
Revised: Each TCR and epitope sequence is preprocessed and embedded in the same way as for the black-box model.

Original: We have two types of explainers: one for TCR sequence and another for epitope sequence.
Revised: We have two types of explainers: one for TCR sequence and another for epitope sequence.

Original: Each explainer encodes both TCR and epitope sequences and concatenates them through three dense layers followed by a ReLU activation function.
Revised: Each explainer encodes both TCR and epitope sequences and concatenates them through three dense layers followed by a ReLU activation function.

Original: For the TCR explainer, the three dense layers contain 32, 16, 20 hidden-units, respectively.
Revised: For the TCR explainer, the three dense layers contain 32, 16, 20 hidden-units, respectively.

Original: For the epitope explainer, the three dense layers contain 32, 16, 13 hidden-units, respectively.
Revised: For the epitope explainer, the three dense layers contain 32, 16, 13 hidden-units, respectively.

Original: The TCR and epitope encoders have the same architectures with those of the black-box model.
Revised: The TCR and epitope encoders have the same architectures with those of the black-box model.

Original: Each explainer then returns a vector of log-probabilities that indicate which peptides in TCR or epitope should be selected as an explanation.
Revised: Each explainer then returns a vector of log-probabilities that indicate which peptides in TCR or epitope should be selected as an explanation.

Original: The approximator has the same architecture as the black-box model.
Revised: The approximator has the same architecture as the black-box model.

Original: We optimize the modelswith the following search space (bold indicatethe choice for our ﬁnal model): the batch size – { 25 , 50 , 100 } , learning rate – 0 . 0001 , 0 .
Revised: We optimize the models indicate

Original: B. B ASELINE METHODS
Revised: B. B ASELINE METHODS

Original: [Table] tuning.
Revised: Hyperparameter tuning.

Original: For L2X, we used the same hyperparameter search space of VIBI.
Revised: For L2X, we used the same hyperparameter search space of VIBI.

Original: For
Revised: For

Original: LIME, we tuned the segmentation ﬁlter size over { 1 , 2 , 4 } for MNIST and { 10 , 25 } for IMDB.
Revised: LIME, we tuned the segmentation ﬁlter size over { 1 , 2 , 4 } for MNIST and { 10 , 25 } for IMDB.

Original: For all methods, we tuned the hyperparameters via grid search and picked up the hyperparameters that yield the best ﬁdelity score on the validation set.
Revised: For all methods, we tuned the hyperparameters via grid search and picked up the hyperparameters that yield the best ﬁdelity score on the validation set.

Original: Cognitive chunk.
Revised: Cognitive chunk.

Original: LIME, SHAP and Saliency yield an attribution score for each feature.
Revised: LIME, SHAP and Saliency yield an attribution score for each feature.

Original: A chunk-attribution score is an average of (absolute) attribution scores of features that belong to the chunk.
Revised: A chunk-attribution score is an average of (absolute) attribution scores of features that belong to the chunk.

Original: Top k chunks that have the highest chunk-attribution scores are selected.
Revised: Top k chunks that have the highest chunk-attribution scores are selected.

Original: L2X selects chunks in the same way as VIBI.
Revised: L2X selects chunks in the same way as VIBI.

Original: Fidelity evaluation.
Revised: Fidelity evaluation.

Original: LIME, SHAP, Saliency, and L2X all have their own approximators.
Revised: LIME, SHAP, Saliency, and L2X all have their own approximators.

Revised: xiv

Original: LIME and SHAP have a sparse linear approximator for each black-box instance (See Section 3.
Revised: LIME and SHAP have a sparse linear approximator for each black-box instance (See Section 3.

Original: in Ribeiro et al.
Revised: in Ribeiro et al.

Original: (2016), and Equation (3) in Lundberg & Lee (2017)).
Revised: (2016), and Equation (3) in Lundberg & Lee (2017)).

Original: Saliency has a 1st-order
Revised: Saliency has a 1st-order

Original: Taylor approximation to each black-box instance (See Equation (3) in Simonyan et al. (2013)).
Revised: Taylor approximation to each black-box instance (See Equation (3) in Simonyan et al. (2013)).

Original: Thexiv approximator ﬁdelity of LIME, SHAP, and Saliency is calculated using their proposed approximators that are composed of the selected features.
Revised: The approximator ﬁdelity of LIME, SHAP, and Saliency is calculated using their proposed approximators that are composed of the selected features.

Original: L2X has a deep neural network that approximates a black-box model (See Section 4 in Chen et al.
Revised: L2X has a deep neural network that approximates a black-box model (See Section 4 in Chen et al.

Original: The approximator and rationale ﬁdelity of
Revised: The approximator and rationale ﬁdelity of

Original: L2X is calculated in the same way as VIBI.
Revised: L2X is calculated in the same way as VIBI.

Original: xv
Revised: xv

Original: C I NTERPRETABILITY EVALUATED BY HUMANS C. LSTM MOVIE SENTIMENT PREDICTION MODEL USING IMDB DATASET E VALUATION
Revised: C I NTERPRETABILITY EVALUATED BY HUMANS C. LSTM MOVIE SENTIMENT PREDICTION MODEL USING IMDB DATASET E VALUATION

Original: [Table]
Revised: [Table]

Original: We evaluated interpretability of the methods on the LSTM movie sentiment prediction model.
Revised: We evaluated interpretability of the methods on the LSTM movie sentiment prediction model.

Original: The interpretable machine learning methods were evaluated by workers at Amazon Mechanical Turk ( https://www.mturk.com/ ) who are awarded the Masters Qualiﬁcation (i
Revised: The interpretable machine learning methods were evaluated by workers at Amazon Mechanical Turk ( https://www.mturk.com/ ) who are awarded the Masters Qualiﬁcation (i

Original: [Table]
Revised: [Table]

Original: e. high performance workers who have demonstrated excellence across a wide range of task).
Revised: e. high performance workers who have demonstrated excellence across a wide range of task).

Original: Randomly selected instances (200 for VIBI and 100 for the others) were evaluated for each method.
Revised: Randomly selected instances (200 for VIBI and 100 for the others) were evaluated for each method.

Original: 5 workers are assigned per instance.
Revised: 5 workers are assigned per instance.

Original: We provided instances that the black-box model had correctly predicted and asked humans to infer the output of the primary sentiment of the movie review (Positive/Negative/Neutral) given ﬁve key words selected by each method.
Revised: We provided instances that the black-box model had correctly predicted and asked humans to infer the output of the primary sentiment of the movie review (Positive/Negative/Neutral) given ﬁve key words selected by each method.

Original: See the survey example below for further details.
Revised: See the survey example below for further details.

Original: Note that this is a proxy for measuring how well humans infer the black-box output given explanations; we use such proxy because the workers are general public who are not familiar with the term ‘black-box’ or ‘output of the model.’
Revised: Note that this is a proxy for measuring how well humans infer the black-box output given explanations; we use such proxy because the workers are general public who are not familiar with the term ‘black-box’ or ‘output of the model.’

Original: In Table 3, the percentage of samples belongs to each combination of the black-box output and the sentiment recognized by the workers are showed.
Revised: In Table 3, the percentage of samples belongs to each combination of the black-box output and the sentiment recognized by the workers are showed.

Original: VIBI has the highest percentage of samples belonging to the Positive/Positive or Negative/Negataive and the lowest percentage of samples belonging to the Positive/Neutral or Negative/Neutral.
Revised: VIBI has the highest percentage of samples belonging to the Positive/Positive or Negative/Negataive and the lowest percentage of samples belonging to the Positive/Neutral or Negative/Neutral.

Original: LIME has the lowest percentage of samples belonging to the Positive/Negative or Negative/Positive, but it is because LIME tends to select words such as ‘that’, ‘the’, ‘is’ so that most of samples are recognized as Neutral.
Revised: LIME has the lowest percentage of samples belonging to the Positive/Negative or Negative/Positive, but it is because LIME tends to select words such as ‘that’, ‘the’, ‘is’ so that most of samples are recognized as Neutral.

Original: xvi
Revised: xvi

Original: S URVEY EXAMPLE FOR IMDB
Revised: S URVEY EXAMPLE FOR IMDB

Original: [Table]
Revised: [Table]

Original: E VALUATION
Revised: E VALUATION

Original: [Table]
Revised: [Table]

Original: We evaluated interpretability of the methods on the CNN digit recognition model.
Revised: We evaluated interpretability of the methods on the CNN digit recognition model.

Original: The interpretable machine learning methods were evaluated by 16 graduate students at XXX University (Blinded due to the double blinding reviewing) who have taken at least one graduate-level machine learning class.
Revised: The interpretable machine learning methods were evaluated by 16 graduate students at XXX University (Blinded due to the double blinding reviewing) who have taken at least one graduate-level machine learning class.

Original: Randomly selected 100 instances were evaluated for each method.
Revised: Randomly selected 100 instances were evaluated for each method.

Original: On average, 4.26 students are assigned per instance.
Revised: On average, 4.26 students are assigned per instance.

Original: See the survey example below for further details.
Revised: See the survey example below for further details.

Original: For the digit recognition model, we asked humans to directly score the explanation on a 0–5 scale.
Revised: For the digit recognition model, we asked humans to directly score the explanation on a 0–5 scale.

Original: In Table 4, the average score per digit is showed.
Revised: In Table 4, the average score per digit is showed.

Original: VIBI outperforms L2X and LIME and slightly outperforms xvii
Revised: VIBI outperforms L2X and LIME and slightly outperforms xvii

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: S URVEY EXAMPLE FOR MNIST Instruction
Revised: S URVEY EXAMPLE FOR MNIST Instruction

Original: [Figure]
Revised: [Figure]

Original: is
Revised: is

Original: [Figure]
Revised: [Figure]

Original: for digit recognition for MNIST
Revised: for digit recognition for MNIST

Original: [Figure]
Revised: [Figure]

Original: Explain by highlighting key pixels that play an important role in the 2D CNN digit recognition.
Revised: Explain by highlighting key pixels that play an important role in the 2D CNN digit recognition.

Original: 2D CNN model
Revised: 2D CNN model

Original: Recognized as● Now, an interpretable learning method explains a decision made by the 2D CNN model:
Revised: Recognized as● Now, an interpretable learning method explains a decision made by the 2D CNN model:

Original: Recognized as
Revised: Recognized as

Original: MNIST is a large dataset contains 28 x 28 sized images of handwritten digits (0 to 9).
Revised: MNIST is a large dataset contains 28 x 28 sized images of handwritten digits (0 to 9).

Original: Here, a 2D convolutional neural network (CNN) is used for the digit recognition for MNIST.
Revised: Here, a 2D convolutional neural network (CNN) is used for the digit recognition for MNIST.

Original: Several interpretable machine learning methods are learned to explain the model by highlighting key pixels that play an important role in the CNN digit recognition.
Revised: Several interpretable machine learning methods are learned to explain the model by highlighting key pixels that play an important role in the CNN digit recognition.

Original: The highlighted pixels provides an explanation for a handwritten image why the CNN model recognized the handwriting as it does.
Revised: The highlighted pixels provides an explanation for a handwritten image why the CNN model recognized the handwriting as it does.

Original: Your task is to evaluate the explanation for each instance on a scale 0 to 5.
Revised: Your task is to evaluate the explanation for each instance on a scale 0 to 5.

Original: Instruction
Revised: Instruction

Original: [Figure]
Revised: [Figure]

Original: Please score each instance based on following criteria:
Revised: Please score each instance based on following criteria:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: D F IDELITY
Revised: D F IDELITY

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: xx
Revised: xx

Original: [Table]
Revised: [Table]

Original: the
Revised: the

Original: [Table]
Revised: [Table]

Original: infer
Revised: infer

Original: [Table]
Revised: [Table]

Original: ﬁdelity implies
Revised: ﬁdelity implies

Original: [Table]
Revised: [Table]

Original: account
Revised: account

Original: [Table]
Revised: [Table]

Original: ﬁdelity
Revised: ﬁdelity

Original: [Table]
Revised: [Table]

Original: Prediction
Revised: Prediction

Original: [Table]
Revised: [Table]

Original: score of
Revised: score of

Original: [Table]
Revised: [Table]

Original: approximator
Revised: approximator

Original: [Table] [Table]
Revised: [Table] [Table]

Original: xxi
Revised: xxi