Original: A SELF - EXPLANATORY METHOD FOR THE BLACK BOX PROBLEM ON DISCRIMINATION PART OF CNN
Revised: A SELF - EXPLANATORY METHOD FOR THE BLACK BOX PROBLEM ON DISCRIMINATION PART OF CNN

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Convolution neural networks (CNNs) have surpassed human’s abilities in some speciﬁc tasks. Recently,  the black box problem of the discrimination part of CNN, which is composed of the fully connected layers of the CNN, has been studied by different scientiﬁc communities.
Revised: Recently, for ﬁnding inherent causality implied in CNN, the black box problem of its discrimination part, which is composed of all fully connected layers of the CNN, has been studied by different scientiﬁc communities.

Original: Many methods were proposed for extracting an interpretable model from the  discrimination part, which can explain the prediction of the  part.
Revised: Many methods were proposed, which can extract various interpretable models from the optimal discrimination part based on inputs and outputs of the part for ﬁnding the inherent causality implied in the part.

Original: However,it is hard for the interpretable models to approximate the discrimination part.
Revised: However, the inherent causality cannot readily be found.

Original: We think that the problem could be solved by shrinking the distance between the interpretable model and the discrimination part.
Revised: We think that the problem could be solved by shrinking an interpretable distance.

Original: This paper ﬁrstly introduces a Markov random ﬁeld model (MRF), namely the Deep Cognitive Learning Model(DCLM), which can express the causal relationship implicated in the discrimination part of CNN. The DCLM is initiatively extracted from the discrimination part by solving a Maximum Satisﬁability(MAX-SAT) problem.
Revised: This paper proposes a lightweight interpretable model, Deep Cognitive Learning Model(DCLM).

Original: And then, a game process is implemented for shrinking an interpretation distance which can evaluate how close the discrimination part is to the DCLM.
Revised: And then, a game method between the DCLM and the discrimination part is implemented for shrinking the interpretation distance.

Original: Finally, the proposed self-explanatory method was evaluated by some contrastive experiments with certain baseline methods on some standard image processing benchmarks.
Revised: Finally, the proposed self-explanatory method was evaluated by some contrastive experiments with certain baseline methods on some standard image processing benchmarks.

Original: These experiments indicate that the proposed method can reduce the distance between the discrimination part and the DCLM without largely reducing its generalization performance.
Revised: These experiments indicate that the proposed method can effectively ﬁnd the inherent causality implied in the discrimination part of the CNN without largely reducing its generalization performance.

Original: Moreover, the generalization performance of the DCLM also can be improved and the discrimination part can be explained by the
Revised: Moreover, the generalization performance of the DCLM also can be improved.

Original: DCLM at every epoch during the training process after the interpretation distance converges.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Convolution neural network(CNN) has surpassed human abilities in some speciﬁc tasks such as computer game and computer vision etc.
Revised: Convolution neural network(CNN) has surpassed human abilities in some speciﬁc tasks such as computer game and computer vision etc.

Original: However, they are considered difﬁcult to understand and explain(Brandon, 2017), which leads to many problems in aspects of privacy leaking, reliability and robustness.
Revised: However, they are considered difﬁcult to understand and explain(Brandon, 2017), which leads to many problems in aspects of privacy leaking, reliability and robustness.

Original: Explanation technology is of immense help for companies to create safer, more trustable products, and to better manage any possible liability of them (Riccardo et al., 2018).
Revised: Explanation technology is of immense help for companies to create safer, more trustable products, and to better manage any possible liability of them (Riccardo et al., 2018).

Original: Recently, for ﬁnding inherent causality  inCNN, which can conclude the reason for its prediction results, the unexplainable problem of CNN, especially concerning the discrimination part which is composed of the fully connected layers of the CNN, has been studied by different scientiﬁc communities.
Revised: Recently, for ﬁnding inherent causality implied in the CNN, the unexplainable problem of CNN, especially concerning the discrimination part which is composed of the fully connected layers of the CNN, has been studied by different scientiﬁc communities.

Original: Many methods were proposed for extracting an interpretable model from the optimal discrimination part for expressing causal relationships between inputs and outputs of the  part.
Revised: Many methods were proposed, which can extract various interpretable models from the optimal discrimination part based on inputs and outputs of the part for expressing the inherent causality implied in the part.

Original: However, because of data bias and noisy data in the training data set, it is difﬁcultfor the interpretable model to approximate the optimal discrimination part.
Revised: However, because of data bias and noisy data in the training data set, the inherent causality cannot readily be found because the part is difﬁcult to be approximated by any interpretable model.

Original: We think that the problem could be solved by the following procedure.
Revised: We think that the problem could be solved by the following procedure.

Original: Firstly,  based on logical relationships between activated states of the neurons in its ﬁrst layer and its output layer of the discrimination part, extracting an interpretable model of the part in training process.
Revised: Firstly, a lightweight interpretable model is designed which can be easily understood by human. And then, the model is initiatively extracted from the discrimination part by solving a Maximum Satisﬁability(MAX-SAT) problem based on the activated states of the neurons in the ﬁrst layer and the output layer of the part.

Revised: An new distance is proposed which can evaluate the degree to which the discrimination part is easily explained, namely as interpretability performance or interpretable distance.

Original: Then, iteratively shrinking  a distance between the  model and the part, and amending them.
Revised: For shrinking the interpretable distance, a game process between the interpretable model and the discrimination part is implemented.

Original: Finally, an optimal interpretable model can be obtained for providing a well interpretation for the predication of the  part.
Revised: Finally, the optimal interpretable model can be obtained, which can express inherent causality implied in the discrimination part.

Original: Moreover, based on the procedure, it is also possible to monitor the evolution of logical relationships implied in the part in the  process.
Revised: Moreover, based on the procedure, it is also possible to monitor the evolution of the inherent causality implied in the part in the game process.

Original: Main contributions of this paper can be summarized as follows:
Revised: Main contributions of this paper can be summarized as follows:

Original: • An interpretable model, Deep Cognitive Learning Model(DCLM), is proposed to express the causal relationship in the discrimination part, and a greedy method is given for initiatively extracting the DCLM from the discrimination part by solving its Maximum Satisﬁability(MAX-SAT) Problem.
Revised: • An interpretable model, Deep Cognitive Learning Model(DCLM), is proposed to express the inherent causality implied in the discrimination part, and a greedy method is given for initiatively extracting the DCLM from the discrimination part by solving its Maximum Satisﬁability(MAX-SAT) Problem.

Original: • A new game method is proposed to improve the interpretability performance of the discrimination part without largely reducing its generalization performance by iteratively shrinking a distance between DCLM and the discrimination part and amending the two models.
Revised: • A new game method is proposed to improve the interpretability performance of the discrimination part without largely reducing its generalization performance by iteratively shrinking the interpretable distance between DCLM and the discrimination part.

Original: • Interpretable distance is proposed to evaluate a distance between the discrimination part and the interpretable model on the unexplainable problem.
Revised: • A new distance is proposed to evaluate the degree to which the discrimination part is easily explained, namely as interpretability performance or interpretable distance.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: There are usually two types of methods for the unexplainable problem of the discrimination part, such as post-hoc method and ante-hoc method (Holzinger et al., 2019).
Revised: There are usually two types of methods for the unexplainable problem of the discrimination part, such as post-hoc method and ante-hoc method (Holzinger et al., 2019).

Original: However, because ante-hoc method is a transparent modeling method(Arrietaa et al., 2020), it can not obtain an explanation about the discrimination part.
Revised: However, because ante-hoc method is a transparent modeling method(Arrietaa et al., 2020), it can not obtain an explanation about the discrimination part.

Original: So, the post-hoc method will be reviewed.
Revised: So, the post-hoc method will be reviewed.

Original: Early post-hoc method can obtain global explanations for a neural network by extracting an interpretable model.
Revised: Early post-hoc method can obtain global explanations for a neural network by extracting an interpretable model.

Original: Some references(Craven & Shavlik, 1999; Krishnan et al., 1999; Boz, 2002;
Revised: Some references(Craven & Shavlik, 1999; Krishnan et al., 1999; Boz, 2002;

Original: Johansson & Niklasson, 2009) proposed a few methods that can ﬁnd a decision tree for explaining a neural network by maximizing the gain ratio and an estimation of the current model ﬁdelity.
Revised: Johansson & Niklasson, 2009) proposed a few methods that can ﬁnd a decision tree for explaining a neural network by maximizing the gain ratio and an estimation of the current model ﬁdelity.

Original: Other references (Craven & Shavlik, 1994; Johansson & Niklasson, 2003; Augasta & Kathirvalavakumar, 2012; Sebastian et al., 2015; Zilke et al., 2016) proposed rule extraction methods for searching optimal interpretable rules from a neural network.
Revised: Other references (Craven & Shavlik, 1994; Johansson & Niklasson, 2003; Augasta & Kathirvalavakumar, 2012; Sebastian et al., 2015; Zilke et al., 2016) proposed rule extraction methods for searching optimal interpretable rules from a neural network.

Original: Recently, some feature relevance methods have become progressively more popular.
Revised: Recently, some feature relevance methods have become progressively more popular.

Original: Montavon et al.(Montavon et al., 2017) proposed a decomposition method from a network classiﬁcation decision into contributions of its input elements based on deep Taylor decomposition.
Revised: Montavon et al.(Montavon et al., 2017) proposed a decomposition method from a network classiﬁcation decision into contributions of its input elements based on deep Taylor decomposition.

Original: Shrikumar et al.(Shrikumar et al., 2016) proposed DeepLIFT which can compute importance scores in a multilayer neural network by explaining the difference of the output from some reference output in terms of differences of the inputs from their reference inputs.
Revised: Shrikumar et al.(Shrikumar et al., 2016) proposed DeepLIFT which can compute importance scores in a multilayer neural network by explaining the difference of the output from some reference output in terms of differences of the inputs from their reference inputs.

Original: Some other works make complex black box model simpler.
Revised: Some other works make complex black box model simpler.

Original: Che et al.(Che et al., 2017) proposed a simple distillation method called Interpretable Mimic Learning for extracting an interpretable simple model by gradient boosting trees.
Revised: Che et al.(Che et al., 2017) proposed a simple distillation method called Interpretable Mimic Learning for extracting an interpretable simple model by gradient boosting trees.

Original: Thiagarajan et al.(Thiagarajan et al., 2016) build a Treeview representation of the complex model by hierarchical partitioning of the feature space.
Revised: Thiagarajan et al.(Thiagarajan et al., 2016) build a Treeview representation of the complex model by hierarchical partitioning of the feature space.

Original: In addition, some references (Hinton et al., 2015; Bucila et al., 2006; Frosst & Hinton, 2017; Traore et al., 2019) proposed the distillation method of knowledge from an ensemble of models into a single model.
Revised: In addition, some references (Hinton et al., 2015; Bucila et al., 2006; Frosst & Hinton, 2017; Traore et al., 2019) proposed the distillation method of knowledge from an ensemble of models into a single model.

Original: Wu et al.(M. Wu, 2018) proposed a tree regularization method via knowledge distillation to represent the output feature space of a RNN based on a Multilayered perception.
Revised: Wu et al.(M. Wu, 2018) proposed a tree regularization method via knowledge distillation to represent the output feature space of a RNN based on a Multilayered perception.

Original: However, these methods can only solve the unexplainable problem of trained neural network or trained deep neural networks with explicit input characteristics.
Revised: However, these methods can only solve the unexplainable problem of trained neural network or trained deep neural networks with explicit input characteristics.

Original: Wan et al.(Wan et al., 2020) constructed a decision tree using the last fully connection layer of the discrimination part of a CNN based on a prior structure.
Revised: Wan et al.(Wan et al., 2020) constructed a decision tree using the last fully connection layer of the discrimination part of a CNN based on a prior structure.

Original: In the paper, our goal is to improve the interpretability performance of multi fully connected layers of  CNN without hurting its generalization performance by initiatively extracting its logic relationships with no prior structure and ﬁnally obtain its explanation by these logic relationships.
Revised: In the paper, our goal is to ﬁnd the inherent causality implied in the discrimination part of CNN, which is composed of all fully connected layers of the CNN without hurting its generalization performance by initiatively extracting its logic relationships with no prior structure and ﬁnally obtain its explanation by these logic relationships.

Original:  D EEP COGNITIVE LEARNING MODEL
Revised:  D EEP COGNITIVE LEARNING MODEL

Original: For expressing the causal relationship between these neurons in the discrimination part, a new interpretable model is designed in the section.
Revised: For expressing the causal relationship between these neurons in the discrimination part, a new interpretable model is designed in the section.

Original: As we all known, a CNN includes a feature extractor and a discrimination part.
Revised: As we all known, a CNN includes a feature extractor and a discrimination part.

Original: The feature extractor composes of some convolution layers and some pooling layers.
Revised: The feature extractor composes of some convolution layers and some pooling layers.

Original: The outputs from the feature extractor are the inputs of the discrimination part of the CNN, namely feature maps, τ 1 , τ 2 , ..., τ k where k is the number of feature maps.
Revised: The outputs from the feature extractor are the inputs of the discrimination part of the CNN, namely feature maps, τ 1 , τ 2 , ..., τ k where k is the number of feature maps.

Original: All these feature maps form a feature set Γ .
Revised: All these feature maps form a feature set Γ .

Original: We suppose that the discrimination part should better be explained by the logic relationships of the activated states of the neurons in its ﬁrst layer and its output layer.
Revised: We suppose that the discrimination part should better be explained by the logic relationships of the activated states of the neurons in its ﬁrst layer and its output layer.

Original: This is because the relationships are the inherent property of the part.
Revised: This is because the relationships are the inherent property of the part.

Original: In order to express the relationships, a deep cognitive learning model (DCLM)is proposed, shown in Fig.1(b).
Revised: In order to express the relationships, a deep cognitive learning model (DCLM)is proposed, shown in Fig.1(b).

Original: I
Revised: I

Original: I
Revised: I

Original: I N ... ...
Revised: I N ... ...

Original: I
Revised: I

Original: O
Revised: O

Original: O C   k
Revised: O C   k

Original: [Figure]
Revised: [Figure]

Original: The DCLM consists of three layers feature predicate layer, disjunction layer, and decision layer.
Revised: The DCLM consists of three layers feature predicate layer, disjunction layer, and decision layer.

Original: The top layer is feature predicate layer which consists of many nodes.
Revised: The top layer is feature predicate layer which consists of many nodes.

Original: Every node has a predicate Z j (Γ) that expresses a positive action or negative action of features which the j th neuron in the ﬁrst fully connected layer of the discrimination part captures.
Revised: Every node has a predicate Z j (Γ) that expresses a positive action or negative action of features which the j th neuron in the ﬁrst fully connected layer of the discrimination part captures.

Original: The predicate Z j (Γ) is deﬁned as follows:
Revised: The predicate Z j (Γ) is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where j ∈ 1 , 2 , ..., N , N is the number of the input neurons of the ﬁrst fully connection layer of the discrimination part of the CNN.
Revised: where j ∈ 1 , 2 , ..., N , N is the number of the input neurons of the ﬁrst fully connection layer of the discrimination part of the CNN.

Original: w i,j is a weight vector between the i th feature map and the j th neuron, b
Revised: w i,j is a weight vector between the i th feature map and the j th neuron, b

Original: j is the bias of the j th neuron, and ” ∗ ” is a convolution operation. ”
Revised: j is the bias of the j th neuron, and ” ∗ ” is a convolution operation. ”

Original: 1” and ”0” denote a positively activated state and a negatively activated state of the neuron respectively. ”
Revised: 1” and ”0” denote a positively activated state and a negatively activated state of the neuron respectively. ”

Original: null” denotes an inactivated state.
Revised: null” denotes an inactivated state.

Original: The bottom layer is a decision layer which includes all nodes used for decision.
Revised: The bottom layer is a decision layer which includes all nodes used for decision.

Original: Every node has a predicate which expresses an activated state of an output neuron of the discrimination part.
Revised: Every node has a predicate which expresses an activated state of an output neuron of the discrimination part.

Original: It is deﬁned as follows:
Revised: It is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where i ∈ 1 , 2 , ..., C , C is number of the output neurons of the CNN, y i is the output value of the i th output neuron of the discrimination part.
Revised: where i ∈ 1 , 2 , ..., C , C is number of the output neurons of the CNN, y i is the output value of the i th output neuron of the discrimination part.

Original: All nodes on the feature predicate layer and every node on the decision layer are connected to one or more nodes on the middle layer, namely as disjunction layer, with true or false edges.
Revised: All nodes on the feature predicate layer and every node on the decision layer are connected to one or more nodes on the middle layer, namely as disjunction layer, with true or false edges.

Original: Every node represents a disjunction relation, which is expressed by a disjunctive normal form.
Revised: Every node represents a disjunction relation, which is expressed by a disjunctive normal form.

Original: It is worth mentioning: if a node is connected to a node on the disjunction layer by a false edge, its predicate follows after a non-operator in the disjunctive normal form.
Revised: It is worth mentioning: if a node is connected to a node on the disjunction layer by a false edge, its predicate follows after a non-operator in the disjunctive normal form.

Original: The potential function of a disjunctive normal form can be obtained by using the Lukasiewicz method(Giles, 1975).
Revised: The potential function of a disjunctive normal form can be obtained by using the Lukasiewicz method(Giles, 1975).

Original: [Equation]
Revised: [Equation]

Original: where T (Γ , y i ) = (cid:80) Nj =1 { a j [1 − Z j (Γ)] + (1 − a j ) Z j (Γ) } + ( a N + 1) D ( y i ) and N is the number of the nodes on the feature predicate layer.
Revised: where T (Γ , y i ) = (cid:80) Nj =1 { a j [1 − Z j (Γ)] + (1 − a j ) Z j (Γ) } + ( a N + 1) D ( y i ) and N is the number of the nodes on the feature predicate layer.

Original: If a j = 1 , there is a false edge.
Revised: If a j = 1 , there is a false edge.

Original: Otherwise, there is a true edge.
Revised: Otherwise, there is a true edge.

Original: The conditional probability distribution that a ground DCLM including all disjunctive normal forms is true is
Revised: The conditional probability distribution that a ground DCLM including all disjunctive normal forms is true is

Original: [Equation]
Revised: [Equation]

Original: where G is the number of all ground formulas, Ξ = (cid:80)
Revised: where G is the number of all ground formulas, Ξ = (cid:80)

Original: [Equation]
Revised: [Equation]

Original: ) is a partition function, y =
Revised: ) is a partition function, y =

Original: ( y 1 , y 2 , ..., y G ) ,
Revised: ( y 1 , y 2 , ..., y G ) ,

Original: y i is an output value of the CNN and λ i is a weighted value of the i th ground formula.
Revised: y i is an output value of the CNN and λ i is a weighted value of the i th ground formula.

Original: By maximizing its likelihood function, the optimal a i and λ i in the DCLM can be obtained.
Revised: By maximizing its likelihood function, the optimal a i and λ i in the DCLM can be obtained.

Original: [Equation]
Revised: [Equation]

Original: For extracting a optimal DCLM, a Maximum A Posterior(MAP) algorithm on the Maximum Satisﬁability Problem (MAX-SAT) was designed.
Revised: For extracting a optimal DCLM, a Maximum A Posterior(MAP) algorithm on the Maximum Satisﬁability Problem (MAX-SAT) was designed.

Original: Using the disjunction normal form with the greatest weighted value in the optimal DCLM, a prediction of an input image can be obtained.
Revised: Using the disjunction normal form with the greatest weighted value in the optimal DCLM, a prediction of an input image can be obtained.

Original:  E VALUATION OF INTERPRETABILITY PERFORMANCE
Revised:  E VALUATION OF INTERPRETABILITY PERFORMANCE

Original: We consider that if the discrimination part of a CNN has a similar shape of function curve with its optimal interpretable model, the former can be easily explained by the latter.
Revised: We consider that if the discrimination part of a CNN has a similar shape of function curve with its optimal interpretable model, the former can be easily explained by the latter.

Original: Therefore, the interpretable performance of the discrimination part can be measured by the shape similarity between it and its optimal interpretable model.
Revised: Therefore, the interpretable performance of the discrimination part can be measured by the shape similarity between it and its optimal interpretable model.

Original: We posit that given the same input data set, the similarity may be measured by variance of differences between outputs of the both models.
Revised: We posit that given the same input data set, the similarity may be measured by variance of differences between outputs of the both models.

Original: It can be named interpretation distance.
Revised: It can be named interpretation distance.

Original: It is easily proved that the smaller the interpretation distance is, the more similar their shapes are, and the better the interpretability performance of the discrimination part would be.
Revised: It is easily proved that the smaller the interpretation distance is, the more similar their shapes are, and the better the interpretability performance of the discrimination part would be.

Original: Deﬁnition 

Revised: Deﬁnition

Original: If X is a compact metric space and ν is a Borel measure in X , such as Lebesgue measure or marginal measures, in L 2 ν ( X ) , a square integrable function space on X , the interpretation distance, φ d ( P ∗ , f ) , between a discrimination part f ( x ) and its optimal DCLM P ∗ ( x ) is
Revised: If X is a compact metric space and ν is a Borel measure in X , such as Lebesgue measure or marginal measures, in L 2 ν ( X ) , a square integrable function space on X , the interpretation distance, φ d ( P ∗ , f ) , between a discrimination part f ( x ) and its optimal DCLM P ∗ ( x ) is

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original:  G AME BETWEEN A DCLM AND THE DISCRIMINATION PART OF A CNN
Revised:  G AME BETWEEN A DCLM AND THE DISCRIMINATION PART OF A CNN

Original: As discussed above,when the shapes of the discrimination part of a CNN and its optimal interpretable model are enough similar, the discrimination part has well interpretability performance.
Revised: As discussed above,when the shapes of the discrimination part of a CNN and its optimal interpretable model are enough similar, the discrimination part has well interpretability performance.

Original: However, its generalization performance will tend to decrease.
Revised: However, its generalization performance will tend to decrease.

Original: This is mainly attributed to the fact that because of data bias and noisy data in training data set, the sufﬁcient and necessary condition for the consistent convergence of the two performances, φ d ( P ∗ , f ∗ ) = 

Revised: This is mainly attributed to the fact that because of data bias and noisy data in training data set, the sufﬁcient and necessary condition for the consistent convergence of the two performances, φ d ( P ∗ , f ∗ ) =

Original: ( f ∗ is the optimal prediction model), is difﬁcult to be guaranteed.
Revised: ( f ∗ is the optimal prediction model), is difﬁcult to be guaranteed.

Original: Therefore, for the tradeoff problem, in training process, extracting an
Revised: Therefore, for the tradeoff problem, in training process, extracting an

Original: DCLM from the discrimination part and then iteratively reducing the interpretation distance between the two models may be a feasible solution.
Revised: DCLM from the discrimination part and then iteratively reducing the interpretation distance between the two models may be a feasible solution.

Original: A detailed discussion about the problem can be found in the appendix.
Revised: A detailed discussion about the problem can be found in the appendix.

Original: To avoid reducing the generalization performance, the maximum probability p ( w | X, y t ) should be guaranteed, where X is a training sample, w is the parameter set of the CNN and y t is the target vector of X .
Revised: To avoid reducing the generalization performance, the maximum probability p ( w | X, y t ) should be guaranteed, where X is a training sample, w is the parameter set of the CNN and y t is the target vector of X .

Original: [Equation]
Revised: [Equation]

Original: Algorithm
Revised: Algorithm

Original: O (
Revised: O (

Original: +
Revised: +

Original: Input: Data set
Revised: Input: Data set

Original: Output: DCLM
Revised: Output: DCLM

Original: Repeat
Revised: Repeat

Original: CNN = CNN Train ( Data set, Adam, loss =
Revised: CNN = CNN Train ( Data set, Adam, loss =

Original: CrossEntropy ) for data,label in Data set do
Revised: CrossEntropy ) for data,label in Data set do

Original: Feature map =
Revised: Feature map =

Original: CNN feature e xtractor ( CNN ) .
Revised: CNN feature e xtractor ( CNN ) .

Original: disjunctive normal form =
Revised: disjunctive normal form =

Original: Disjunction ( Feature map, Fnn, W i,j ) .
Revised: Disjunction ( Feature map, Fnn, W i,j ) .

Original: To predicate value of neuron according to Eq.
Revised: To predicate value of neuron according to Eq.

Original: Use the disjunction forms of data obtained from the all predicate combinations to update
Revised: Use the disjunction forms of data obtained from the all predicate combinations to update

Original: DCLM .
Revised: DCLM .

Original: end for for i = 1 to n do for data,label in Data set do
Revised: end for for i = 1 to n do for data,label in Data set do

Original: Feature map =
Revised: Feature map =

Original: CNN feature extractor ( CNN ) ym =
Revised: CNN feature extractor ( CNN ) ym =

Original: DCLM ( Feature map )
Revised: DCLM ( Feature map )

Original: CNN DCLM ( data, ym, loss =
Revised: CNN DCLM ( data, ym, loss =

Original: Eq. 14) end for end for
Revised: Eq. 14) end for end for

Original: Until the interpretation distance and accuracy converge where p ( y t | w , X )
Revised: Until the interpretation distance and accuracy converge where p ( y t | w , X )

Original: = (cid:82) p ( y t | f, w , X ) (cid:82) p ( f | y dclm , w , X ) p
Revised: = (cid:82) p ( y t | f, w , X ) (cid:82) p ( f | y dclm , w , X ) p

Original: ( y dclm | w , X )
Revised: ( y dclm | w , X )

Original: d y dclm df and y dclm is a prediction of DCLM.
Revised: d y dclm df and y dclm is a prediction of DCLM.

Original: When the DCLM is known, y ∗ dclm is its optimal prediction and p ( y ∗ dclm | w , X ) = 1 .
Revised: When the DCLM is known, y ∗ dclm is its optimal prediction and p ( y ∗ dclm | w , X ) = 1 .

Original: Then (cid:90)
Revised: Then (cid:90)

Original: [Equation]
Revised: [Equation]

Original: Similarly, known the input X and w , f nn is the optimal solution of the CNN.
Revised: Similarly, known the input X and w , f nn is the optimal solution of the CNN.

Original: [Equation]
Revised: [Equation]

Original: If w and X are given and the loss function φ
Revised: If w and X are given and the loss function φ

Original: [Equation]
Revised: [Equation]

Original: , the conditional probability distribution function
Revised: , the conditional probability distribution function

Original: [Equation]
Revised: [Equation]

Original: Meanwhile,
Revised: Meanwhile,

Original: [Equation]
Revised: [Equation]

Original: where Ξ 1 and Ξ 2 are partition functions.
Revised: where Ξ 1 and Ξ 2 are partition functions.

Original: Then by maximizing a likelihood function of p ( w | X, y t )
Revised: Then by maximizing a likelihood function of p ( w | X, y t )

Original: the optimal w can be obtained.
Revised: the optimal w can be obtained.

Original: In particular, assuming that w follows Gaussian distribution, we get:
Revised: In particular, assuming that w follows Gaussian distribution, we get:

Original: [Equation]
Revised: [Equation]

Original: where α is a meta-parameter determined by the variance of the selected Gaussian distributions.
Revised: where α is a meta-parameter determined by the variance of the selected Gaussian distributions.

Original: Turn it into a minimization problem:
Revised: Turn it into a minimization problem:

Original: [Equation]
Revised: [Equation]

Original: The iterative optimization algorithm is shown as follows:
Revised: The iterative optimization algorithm is shown as follows:

Original:  E XPERIMENTAL VERIFICATION
Revised:  E XPERIMENTAL VERIFICATION

Original: We designed two experiments to verify the effectiveness of the proposed method.
Revised: We designed two experiments to verify the effectiveness of the proposed method.

Original: The ﬁrst experiment veriﬁed whether the self-explanatory method could improve the interpretability performance of the CNN without sacriﬁcing its generalization performance.
Revised: The ﬁrst experiment veriﬁed whether the self-explanatory method could improve the interpretability performance of the CNN without sacriﬁcing its generalization performance.

Original: The second experiment veriﬁed whether the proposed method can tend towards stability and convergence in the game process.
Revised: The second experiment veriﬁed whether the proposed method can tend towards stability and convergence in the game process.

Original: In the experiments, CNN3(includes 3 convolution layers,3 MaxPooling layers, 3 fully connect layers(FCLs) and 1 output layer),CNN5(includes 5 convolution layers,

Revised: In the experiments, CNN3(includes 3 convolution layers,3 MaxPooling layers, 3 fully connect layers(FCLs) and 1 output layer),CNN5(includes 5 convolution layers,

Original: MaxPooling layers, 3 FCLs and 1 output layer), and CNN8(includes 8 convolution layers,8 MaxPooling layers, 3 FCLs andoutput layer) were used.
Revised: MaxPooling layers, 3 FCLs and 1 output layer), and CNN8(includes 8 convolution layers,8 MaxPooling layers, 3 FCLs andoutput layer) were used.

Original: Traditional training methods on the three types CNN were named as CNN3Trad, CNN5-Trad and CNN8-Trad respectively.
Revised: Traditional training methods on the three types CNN were named as CNN3Trad, CNN5-Trad and CNN8-Trad respectively.

Original: By contrast, our proposed methods on these CNNs were named as CNN3-DCLM, CNN5-DCLM and CNN8-DCLM respectively.
Revised: By contrast, our proposed methods on these CNNs were named as CNN3-DCLM, CNN5-DCLM and CNN8-DCLM respectively.

Original: All experiments used
Revised: All experiments used

Original: Mnist(Lecun et al., 1998), FashionMnist(Zalando, 2017), and Cifar-10(Krizhevsky, 2009) benchmark data sets.
Revised: Mnist(Lecun et al., 1998), FashionMnist(Zalando, 2017), and Cifar-10(Krizhevsky, 2009) benchmark data sets.

Original: All algorithms were implemented in Python using the Pytorch library(Paszke et al., 2019).
Revised: All algorithms were implemented in Python using the Pytorch library(Paszke et al., 2019).

Original: All experiments ran on a server with Intel Xeon 4110(2.1GHz)
Revised: All experiments ran on a server with Intel Xeon 4110(2.1GHz)

Original: Silver Processor, 20GB RAM and Nvidia Telsa T4.
Revised: Silver Processor, 20GB RAM and Nvidia Telsa T4.

Original: Experiment 1: Performance veriﬁcation of the proposed method on CNN .
Revised: Experiment 1: Performance veriﬁcation of the proposed method on CNN .

Original: We replaced the discrimination part of three traditionally trained CNNs with soft decision tree(SDT)(Frosst & Hinton, 2017) and designated these methods as CNN3-Trad-SDT, CNN5-Trad-SDT and CNN8-Trad-SDT respectively.
Revised: We replaced the discrimination part of three traditionally trained CNNs with soft decision tree(SDT)(Frosst & Hinton, 2017) and designated these methods as CNN3-Trad-SDT, CNN5-Trad-SDT and CNN8-Trad-SDT respectively.

Original: The accuracy of CNN, accuracy of STD or DCLM, and interpretation distance corresponding to all methods were shown in Table 1.
Revised: The accuracy of CNN, accuracy of STD or DCLM, and interpretation distance corresponding to all methods were shown in Table 1.

Original: Some values are ”—”, which indicates that these results do not exist.
Revised: Some values are ”—”, which indicates that these results do not exist.

Original: It is observed in Table 1 that the accuracy of all CNN-DCLMs are higher than  the two interpretable models, such as SDT and DCLM, on all benchmark data sets and are around 1.4 percentage points lower than those of CNN-Trads.
Revised: It is observed in Table 1 that the accuracies of all CNN trained by the proposed method are higher than those of the two interpretable models, such as SDT and DCLM, on all benchmark data sets and are around 1.4 percentage points lower than those of all CNN trained by the traditional training method.

Original: But it is worth noticing that on the most of data sets the interpretation distances of all CNN-DCLMs are lower than interpretation distances of all CNN-Trad-SDTs.
Revised: But it is worth noticing that on the most of data sets the interpretation distances of these CNNs trained by the proposed method are lower than interpretation distances of CNNs obtained by the traditional method.

Original: These might prove that that the self-explanatory method can improve the interpretability performance of the discrimination part of a CNN without largely reducing its generalization performance.
Revised: These might prove that that the self-explanatory method can improve the interpretability performance of the discrimination part of a CNN without largely reducing its generalization performance.

Original: The accuracies of the DCLMs are higher than those of the SDTs except CNN3-DCLM on FashionMnist data set and CNN3-DCLM on Mnist.
Revised: The accuracies of the DCLMs are higher than those of the SDTs except CNN3-DCLM on FashionMnist data set and CNN3-DCLM on Mnist.

Original: These results might prove that the proposed method can ﬁnd more excellent interpretable model than CNN-Trad-SDT.
Revised: These results might prove that the proposed method can ﬁnd more excellent interpretable model than the traditional method.

Original: Experiment 2: Convergence test of the proposed method We designed the experiments to demonstrate convergence of the proposed method.
Revised: Experiment 2: Convergence test of the proposed method We designed the experiments to demonstrate convergence of the proposed method.

Original: CNN3-Trad, CNN5-Trad and CNN8-Trad were used for comparing with CNN3-DCLM, CNN5-DCLM, and CNN8-DCLM respectively.
Revised: CNN3-Trad, CNN5-Trad and CNN8-Trad were used for comparing with CNN3-DCLM, CNN5-DCLM, and CNN8-DCLM respectively.

Original: Every training works out 25 epochs.
Revised: Every training works out 25 epochs.

Original: Experiment results were measured at every epoch and shown in four ﬁgures,
Revised: Experiment results were measured at every epoch and shown in four ﬁgures,

Original: Fig.2, Fig.3, Fig.5 and Fig.4.
Revised: Fig.2, Fig.3, Fig.5 and Fig.4.

Original: Every ﬁgure includes nine subplots.
Revised: Every ﬁgure includes nine subplots.

Original: The three subplots on the left column were shown for the experiment results on Mnist data set.
Revised: The three subplots on the left column were shown for the experiment results on Cifar-10 data set.

Original: These subplots on the middle column were for FashionMnist data set and these subplots on the right column were for Cifar-10 data set.[Table] A [Table]
Revised: These subplots on the middle column were for FashionMnist data set and these subplots on the right column were for Mnist data set.

Original: A CCURACY OF SDT OR DCLM I NTERPRETATION D ISTANCE

Original: [Table] In Fig. [Table] the accuracies of the DCLMs and the CNN-DCLMs of every epoch in the game process were shown.
Revised: In Fig.2, the accuracies of the DCLMs and the CNNs of every epoch in the game process were shown.

Original: From these ﬁgures [Table] it is obvious that accuracies of the DCLMs and these CNN-DCLMs steadily increase in the early stage.
Revised: From these ﬁgures, it is obvious that accuracies of the DCLMs and these CNNs steadily increase in the early stage.

Original: In the next stage, their accuracies tend to be stable.
Revised: In the next stage, these accuracies tend to be stable.

Original: This reﬂects that the game method do not affect the improvement of the generalization performances of these DCLMs and these CNNs.
Revised: This reﬂects that the game method do not affect the improvement of the generalization performances of these DCLMs and these CNNs.

Original: We also ﬁnd that the accuracy gap  for the FashionMnist data set is much more than other two data sets.
Revised: We also ﬁnd that the accuracy gap obtain by CNN3-DCLM for the FashionMnist data set is much more than other two data sets.

Revised: Meanwhile, it can be found that the DCLM convergence to a stable state.

Original: The main reason is that for  the FashionMnist data set the logical relationships between the activation states of the input neurons and those of theoutput neurons of the discrimination part of the CNNs are more complex than the other data sets.
Revised: The main reason is that for CNN3, in the later stage of the game process, new inherent causality implied in the part can not readily be found by the FashionMnist data set.

Original: Even so, the proposed method also improves the accuracies of the DCLMs and these CNN-DCLMs steadily.
Revised: Even so, the proposed method also improves the accuracies of the DCLMs and these CNN-DCLMs steadily.

Revised: We also ﬁnd that the gaps between these accuracies of the DCLMs and the CNNs obtained by the CNN become very small at the later epochs.

Revised: This reﬂects that CNN8 can extract more effective features by which the DCLMs can ﬁnd more accurate causality implied in the discrimination part of the CNNs.

Original: Fig.3 shows the interpretation distances of the CNNs trained by the traditional method and the CNNs trained by the proposed method.
Revised: Fig.3 shows the interpretation distances of the CNNs trained by the traditional method and the CNNs trained by the proposed method.

Original: As seen from these subplots, the interpretation distances of CNNs trained by the traditional method are greater than those of the other CNNs at the most of the epochs, especially by the end of the training.
Revised: As seen from these subplots, the interpretation distances of CNNs trained by the traditional method are greater than those of the other CNNs at the most of the epochs, especially by the end of the game.

Original: The results indicate that the game method can effectively improve the interpretability performance of CNNs.
Revised: The results indicate that the game method can effectively improve the interpretability performance of CNNs.

Original: From these subplots in Fig.2and Fig.3, we also can see that after the 15th epoch, interpretation distances of CNNs from the proposed method tend to converge.
Revised: From these subplots in Fig.2and Fig.3, we also can see that after the 15th epoch, interpretation distances of CNNs from the proposed method tend [Table] A [Table]

Revised: A CCURACY OF SDT OR DCLM I NTERPRETATION D ISTANCE

Revised: [Table]

Original:  The phenomenon indicates that the discrimination part of the CNNs can be explained by its DCLMs at every epoch after the ﬁfteenth epoch.
Revised: to converge. The phenomenon indicates that the discrimination part of the CNNs can be explained by its DCLMs at every epoch after the ﬁfteenth epoch [Table] From Fig.

Original: From Fig.4, it is evident that all DCLMs from CNN3-DCLMs, CNN5-DCLMs, and CNN8-DCLMs have been found to have stable information entropies at the end of thegame,which calculate the diversity of disjunction normal forms in the DCLMs obtained by the gamemethod.
Revised: [Table] it is evident that all DCLMs from CNN3-DCLMs, CNN5-DCLMs [Table] and CNN8-DCLMs have been found to have stable information entropies at the end of the game

Revised: [Table]

Revised: which calculate the diversity of disjunction normal forms in the DCLMs obtained by the game method.

Original: On Mnist data set, the entropies ﬁnally converge between 120 and 145. On FashionMnist data set, the entropies ﬁnally converge between 125 and 136.
Revised: On Mnist data set, the entropies ﬁnally converge 135,113,and 51.3 respectively. On FashionMnist data set, the entropies ﬁnally converge 144,132,and 65.

Original: On Cifar-10 data set, the entropies ﬁnally converge between 100 and 120.
Revised: On Cifar-10 data set, the entropies ﬁnally converge 65,73,and 57.

Revised: More complicated the extract part of CNN, more small the information entropies of

Original:  The results indicate that the game algorithm can ensure that the  DCLMs converges to a stable state.
Revised: DCLM obtained by the proposed method. The results indicate that the game algorithm can ensure that the diversity of disjunction normal forms of the DCLMs converges to a stable state.

Original: The  DCLMs from theproposed method have sparser disjunction normal forms than these not participating in the game process, which indicates they are more robust.
Revised: The game with the CNN with complex structure can obtain the more robust DCLMs than with the CNN with simply sturcture. The main reason is that the features captured by the CNN with the complex structure is so sparse and robust that the disjunction normal forms of the DCLMs is sparse and robust.

Original: In Fig.5,the accuracies of the CNNs trained by the traditional method and the CNNs trained by the proposed method at every epoch were shown.
Revised: In Fig.5,the accuracies of the CNNs trained by the traditional method and the CNNs trained by the proposed method at every epoch were shown.

Original: From these subplots, it can be seen that its accuracies steadily increase in the early stage and accuracies of the CNNs trained by the proposed method are lower than the other CNNs.
Revised: From these subplots, it can be seen that its accuracies steadily increase in the early stage and accuracies of the CNNs trained by the proposed method are lower than the other CNNs.

Original: But in the following stage, their accuracies tend to be stable and consistent.
Revised: But in the following stage, their accuracies tend to be stable and consistent.

Original: The main reason is that in the early stage, a tradeoff problem between the generalization performance and interpretability performance of the discrimination part of a CNN inevitably reduces its generalization performance in order to increase its interpretability performance.
Revised: The main reason is that in the early stage, a tradeoff problem between the generalization performance and interpretability performance of the discrimination part of a CNN inevitably reduces its generalization performance in order to increase its interpretability performance.

Original: But the proposed game method can effectively reduce the gap between two performances.
Revised: Though the proposed game method can effectively reduce the gap between two performances, it do not increase the gap between the accuracies of the CNNs trained by the traditional method and the CNNs trained by the proposed method.

Original: This reﬂects that the proposed method is effective for the tradeoff problem.
Revised: This reﬂects that the proposed method is effective for the tradeoff problem

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The performance of the proposed method was demonstrated by experiments on benchmark data sets.
Revised: The performance of the proposed method was demonstrated by experiments on benchmark data sets.

Original: The proposed method showed prominent advantages over traditional learning algorithm on CNN for improving the generalization performance and the interpretability performance of the discrimination part of the CNN.
Revised: The proposed method showed prominent advantages over traditional learning algorithm on CNN for improving the generalization performance and the interpretability performance of the discrimination part of the CNN.

Original: In practical engineering, the proposed method may provide a new learning paradigm.
Revised: In practical engineering, the proposed method may provide a new learning paradigm.

Original: The method can not only predict an relatively accurate result for new input data but also provide a reasonable causal interpretation for the prediction of the discrimination part of a CNN.
Revised: The method can not only predict an relatively accurate result for new input data but also provide a reasonable causal interpretation for the prediction of the discrimination part of a CNN.

Original: We suppose that it can solve the black box problem in the discrimination part.
Revised: We suppose that it can solve the black box problem in the discrimination part.

Original: We believe that the proposed method provide a way to understand the discrimination part
Revised: We believe that the proposed method provide a way to understand the discrimination part.

Original: A A PPENDIX : A M AXIMUM A P OSTERIOR (MAP) ALGORITHM FOR EXTRACTING A DCLM

Original: A Maximum A Posterior(MAP) algorithm on the Maximum Satisﬁability Problem (MAX-SAT) for extracting an DCLM is as follows.

Original: B A PPENDIX : S UFFICIENT AND NECESSARY CONDITION FOR CONSISTENT CONVERGENCE OF THE GENERALIZATION PERFORMANCE AND THE INTERPRETABILITY PERFORMANCE
Revised: A A PPENDIX : S UFFICIENT AND NECESSARY CONDITION FOR CONSISTENT CONVERGENCE OF THE GENERALIZATION PERFORMANCE AND THE INTERPRETABILITY PERFORMANCE

Original: If the minimization of loss function of CNN can guarantee the minimum of its interpretability performance learning algorithm of the CNN can improve its interpretability performance.
Revised: If the minimization of loss function of CNN can guarantee the minimum of its interpretability performance learning algorithm of the CNN can improve its interpretability performance.

Original: If not the tradeoff problem between the generalization performance and the interpretability performance will exit.
Revised: If not, the tradeoff problem between the generalization performance and the interpretability performance will exit.

Original: For proving the existence of the problem, we focus on a neuron of CNN.
Revised: For proving the existence of the problem we focus on a neuron of CNN.

Original: From the foot we may judge of Hercules.
Revised: From the foot we may judge of Hercules.

Original: If an input channel f ( x ) of the neuron is seen as a kernel function
Revised: If an input channel f ( x ) of the neuron is seen as a kernel function

Original: K ( x, w ) ( w is weight vector including a bias of the neuron), it will span a kernel Hilbert space
Revised: K ( x, w ) ( w is weight vector including a bias of the neuron), it will span a kernel Hilbert space

Original: H K =
Revised: H K =

Original: { f ( x ) ∈
Revised: { f ( x ) ∈

Original: L 2 ν ( X ) | f ( x ) =
Revised: L 2 ν ( X ) | f ( x ) =

Original: K ( x, w ) =
Revised: K ( x, w ) =

Original: (cid:80) ∞
Revised: (cid:80) ∞

Original: k =1 a k φ k ( x ) } for the neuron.
Revised: k =1 a k φ k ( x ) } for the neuron.

Original: H K can be regarded as a linear function set on L 2 ν ( X ) .
Revised: H K can be regarded as a linear function set on L 2 ν ( X ) .

Original: It is a solution space of the neuron.
Revised: It is a solution space of the neuron.

Original: The necessary and sufﬁcient conditions for consistent convergence between the generalization performance and the interpretability performance are discussed below in L 2 ν ( X ) based on the following lemmas.
Revised: The necessary and sufﬁcient conditions for consistent convergence between the generalization performance and the interpretability performance are discussed below in L 2 ν ( X ) based on the following lemmas.

Original: Lemma 1.
Revised: Lemma 1.

Original: Continuous linear functional set on a separable Hilbert space X is nowhere dense in a square integrable function space L 2 ν ( X ) .
Revised: Continuous linear functional set on a separable Hilbert space X is nowhere dense in a square integrable function space L 2 ν ( X ) .

Original: Lemma 2.
Revised: Lemma 2.

Original: Continuous nonlinear functional set of the separable Hilbert space X is everywhere dense in L 2 ν ( X ) .
Revised: Continuous nonlinear functional set of the separable Hilbert space X is everywhere dense in L 2 ν ( X ) .

Original: When the optimal input channel f ∗ ( x ) approximate a linear functional in H K while the optimal interpretable model P ∗ ( x ) don’t approximate any linear functional or do not exist in H K , traditional training process cannot guarantee f ( x ) approximates P ∗ ( x ) according Lemma 1.
Revised: When the optimal input channel f ∗ ( x ) approximate a linear functional in H K while the optimal interpretable model P ∗ ( x ) don’t approximate any linear functional or do not exist in H K , traditional training process cannot guarantee f ( x ) approximates P ∗ ( x ) according Lemma 1.

Original: From Lemma 2, the approximate will cannot converge until P ∗ ( x ) approximate f ∗ ( x ) .
Revised: From Lemma 2, the approximate will cannot converge until P ∗ ( x ) approximate f ∗ ( x ) .

Original: Here, if we deﬁne the approximation as the similarity between the shapes of function curve of f ∗ ( x ) and P ∗ ( x ) , the sufﬁcient and necessary condition for the consistent convergence of the two performances is φ d ( P ∗ , f ∗ ) = 0 .
Revised: Here, if we deﬁne the approximation as the similarity between the shapes of function curve of f ∗ ( x ) and P ∗ ( x ) , the sufﬁcient and necessary condition for the consistent convergence of the two performances is φ d ( P ∗ , f ∗ ) = 0 .

Original: For the discrimination part of the CNN, the sufﬁcient and necessary condition is still true.
Revised: For the discrimination part of the CNN, the sufﬁcient and necessary condition is still true.

Original: However because of data bias and noisy data in training data set, the condition is difﬁcult to to be ensured in the majority of engineering applications.
Revised: However because of data bias and noisy data in training data set, the condition is difﬁcult to to be ensured in the majority of engineering applications.

Original: The tradeoff problem always exists between the two performances of the discrimination part.
Revised: The tradeoff problem always exists between the two performances of the discrimination part.

Original: According to the above conclusion, in order to completely solve the tradeoff problem, the φ d ( P ∗ , f ∗ ) (Here f ∗ ( x ) is the optimal discrimination part) should be reduced.
Revised: According to the above conclusion, in order to completely solve the tradeoff problem, the φ d ( P ∗ , f ∗ ) (Here f ∗ ( x ) is the optimal discrimination part) should be reduced.

Original: However P ∗ and f ∗ ( x ) are unknown.
Revised: However P ∗ and f ∗ ( x ) are unknown.

Original: Therefore, in training process, extracting a interpretable model P ( x ) from a discrimination part f ( x ) and then iteratively reducing φ d ( P, f ) may be a convenient method.
Revised: Therefore, in training process, extracting a interpretable model P ( x ) from a discrimination part f ( x ) and then iteratively reducing φ d ( P, f ) may be a convenient method.