Original: A SELF - EXPLANATORY METHOD FOR THE BLACK  PROBLEM ON DISCRIMINATION PART OF CNN
Revised: A SELF - EXPLANATORY METHOD FOR THE BLACK BOX PROBLEM ON DISCRIMINATION PART OF CNN

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Convolution neural networks (CNNs) have surpassed human’s abilities in some speciﬁc tasks.
Revised: Convolution neural networks (CNNs) have surpassed human’s abilities in some speciﬁc tasks.

Original: However, they are considered difﬁcult to understand and explain. Recently, the black box problem ofCNN, especially concerning the discrimination part, has been studied by different scientiﬁc communities.
Revised: Recently, the black box problem of the discrimination part of CNN, which is composed of the fully connected layers of the CNN, has been studied by different scientiﬁc communities.

Original: Many methods were proposed for extracting an interpretable model from the discrimination part, which can explain the prediction of the part.
Revised: Many methods were proposed for extracting an interpretable model from the discrimination part, which can explain the prediction of the part.

Original: However, it is hard for the interpretable models to approximate the discriminationpart because of the tradeoff problem between interpretability performance and generalization performance of the discrimination part.We suppose the tradeoff problem is mainly attributed to the fact that the sufﬁcient and necessary condition for the consistent convergence of the both performances is hard to be guaranteed by tradition learning algorithm.
Revised: However, it is hard for the interpretable models to approximate the discrimination part.

Original: So thetradeoff problem could be solved by shrinking the distance between the interpretable model and the discrimination part.
Revised: We think that the problem could be solved by shrinking the distance between the interpretable model and the discrimination part.

Original: This paper ﬁrstly introduces a Markov random ﬁeld model (MRF), namely  Deep Cognitive Learning Model(DCLM), which explains the causal relationship between the features(the weight matrixes in theﬁrst layer of the discrimination part of CNN can capture) and the output result of the discrimination part of CNN.
Revised: This paper ﬁrstly introduces a Markov random ﬁeld model (MRF), namely the Deep Cognitive Learning Model(DCLM), which can express the causal relationship implicated in the discrimination part of CNN.

Original: A greedy algorithm isproposed for initiatively extracting the DCLM from the discrimination part by solving a MAX-SAT problem.
Revised: The DCLM is initiatively extracted from the discrimination part by solving a Maximum Satisﬁability(MAX-SAT) problem.

Original: And then, a game processbetween two MAP inferences is implemented for shrinking an interpretation distance which can evaluate how close the discrimination part is to the DCLM.
Revised: And then, a game process is implemented for shrinking an interpretation distance which can evaluate how close the discrimination part is to the DCLM.

Original: Finally, the proposed self-explanatory approach is evaluated by some contrastive experiments with certain baseline methods on some standard image processing benchmarks.
Revised: Finally, the proposed self-explanatory method was evaluated by some contrastive experiments with certain baseline methods on some standard image processing benchmarks.

Original: These experiments indicate that the proposed method can improve the interpretability performance of the discrimination part  without largely reducing its generalization performance, the generalization performance of the DCLM also can be improved and the discrimination part can be explained by the DCLM in real time during the training process after the interpretation distance converges.
Revised: These experiments indicate that the proposed method can reduce the distance between the discrimination part and the DCLM without largely reducing its generalization performance. Moreover, the generalization performance of the DCLM also can be improved and the discrimination part can be explained by the DCLM at every epoch during the training process after the interpretation distance converges.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Convolution neural network(CNN) has surpassed human abilities in some speciﬁc tasks such as computer game and computer vision etc.
Revised: Convolution neural network(CNN) has surpassed human abilities in some speciﬁc tasks such as computer game and computer vision etc.

Original: However, they are considered difﬁcult to understand and explain(Brandon, 2017), which leads to many problems in aspects offairness, privacy leaking, reliability and robustness.
Revised: However, they are considered difﬁcult to understand and explain(Brandon, 2017), which leads to many problems in aspects of privacy leaking, reliability and robustness.

Original: Even wrong decision could possibly be made due to spurious correlations in training data, such as recognizing an object in an image by the properties of the background or lighting (Riccardo et al., 2018).

Original: Explanation technology is of immense help for companies to creat safer, more trustable products, and to better manage any possible liability of them (Riccardo et al., 2018).
Revised: Explanation technology is of immense help for companies to create safer, more trustable products, and to better manage any possible liability of them (Riccardo et al., 2018).

Original: Recently,  the unexplainable problem of CNN, especially concerning the discrimination part, has been studied by different scientiﬁc communities.
Revised: Recently, for ﬁnding inherent causality in CNN, which can conclude the reason for its prediction results, the unexplainable problem of CNN, especially concerning the discrimination part which is composed of the fully connected layers of the CNN, has been studied by different scientiﬁc communities.

Original: Many methods were proposed for extracting an interpretable model from the  discrimination part, which can explain causal relationship between inputs and outputs of thediscrimination part.
Revised: Many methods were proposed for extracting an interpretable model from the optimal discrimination part for expressing causal relationships between inputs and outputs of the part.

Original: However, because of data bias and noisy data in  training data set the sufﬁcient and necessary condition for the consistent convergence of an interpretability performance and the generalization performance of the discrimination part is difﬁcult  to be guaranteed in traditional machine learning methods. The interpretability performance is a distance between thediscrimination part and its optimalinterpretable model. So when a discrimination part has a well generalization performance, it usually stay away from its optimal interpretable model, which leads to an incorrect interpretable model.
Revised: However, because of data bias and noisy data in the training data set, it is difﬁcult for the interpretable model to approximate the optimal discrimination part.

Original: We think that thetradeoff problem could be solved by the following procedure.
Revised: We think that the problem could be solved by the following procedure.

Original: Firstly,  extracting an interpretable model which expresses causal relationships in thediscrimination part in training process, and then, iteratively shrinking a distance between the model and the part, and amending them.
Revised: Firstly, based on logical relationships between activated states of the neurons in its ﬁrst layer and its output layer of the discrimination part, extracting an interpretable model of the part in training process. Then, iteratively shrinking a distance between the model and the part, and amending them.

Original: It is because that the method not only can make the discrimination part approximate the interpretable modelwhich can improve the interpretability performance of the part but also can make the interpretable model close to  the part which makes it correctly express the causal relationship in the part.
Revised: Finally, an optimal interpretable model can be obtained for providing a well interpretation for the predication of the part. Moreover, based on the procedure, it is also possible to monitor the evolution of logical relationships implied in the part in the process.

Original: Main contributions of this paper can be summarized as follows:
Revised: Main contributions of this paper can be summarized as follows:

Original: • An interpretable model, Deep Cognitive Learning Model(DCLM), is proposed to express the causal relationship in the discrimination part, and a greedy method is given for initiatively extracting the DCLM from the discrimination part by solving its MAX-SAT problem.
Revised: • An interpretable model, Deep Cognitive Learning Model(DCLM), is proposed to express the causal relationship in the discrimination part, and a greedy method is given for initiatively extracting the DCLM from the discrimination part by solving its Maximum Satisﬁability(MAX-SAT) Problem.

Original: • A new game method is proposed to improve the interpretability performance of the discrimination part without largely reducing its generalization performance by iteratively shrinking a distance between  the two models and amending the two models.At the same time, the generalization performance of the DCLM can also be improved.
Revised: • A new game method is proposed to improve the interpretability performance of the discrimination part without largely reducing its generalization performance by iteratively shrinking a distance between DCLM and the discrimination part and amending the two models.

Original: • Interpretable distance is proposed to evaluate a distance between the discrimination part and the interpretable model on the unexplainable problem.
Revised: • Interpretable distance is proposed to evaluate a distance between the discrimination part and the interpretable model on the unexplainable problem.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: There are usually two types of methods for the unexplainable problem of the discrimination part, such as post-hoc method and ante-hoc method (Holzinger et al., 2019).
Revised: There are usually two types of methods for the unexplainable problem of the discrimination part, such as post-hoc method and ante-hoc method (Holzinger et al., 2019).

Original: However, because ante-hoc method is a transparent modeling method(Arrietaa et al., 2020), it can not obtain an explanation about the discrimination part.
Revised: However, because ante-hoc method is a transparent modeling method(Arrietaa et al., 2020), it can not obtain an explanation about the discrimination part.

Original: So, the post-hoc method will be reviewed.
Revised: So, the post-hoc method will be reviewed.

Original: Early post-hoc method can obtain global explanations for a neural network by extracting an interpretable model.
Revised: Early post-hoc method can obtain global explanations for a neural network by extracting an interpretable model.

Original: Some references(Craven & Shavlik, 1999; Krishnan et al., 1999; Boz, 2002;
Revised: Some references(Craven & Shavlik, 1999; Krishnan et al., 1999; Boz, 2002;

Original: Johansson & Niklasson, 2009) proposed a few methods that can ﬁnd a decision tree for explaining a neural network by maximizing the gain ratio and an estimation of the current model ﬁdelity.
Revised: Johansson & Niklasson, 2009) proposed a few methods that can ﬁnd a decision tree for explaining a neural network by maximizing the gain ratio and an estimation of the current model ﬁdelity.

Original: Other references (Craven & Shavlik, 1994; Johansson & Niklasson, 2003; Augasta & Kathirvalavakumar, 2012; Sebastian et al., 2015; Zilke et al., 2016) proposed rule extraction methods for searching optimal interpretable rules from a neural network.
Revised: Other references (Craven & Shavlik, 1994; Johansson & Niklasson, 2003; Augasta & Kathirvalavakumar, 2012; Sebastian et al., 2015; Zilke et al., 2016) proposed rule extraction methods for searching optimal interpretable rules from a neural network.

Original: Recently, some feature relevance methods have become progressively more popular.
Revised: Recently, some feature relevance methods have become progressively more popular.

Original: Montavon et al.(Montavon et al., 2017) proposed a decomposition method from a network classiﬁcation decision into contributions of its input elements based on deep Taylor decomposition.
Revised: Montavon et al.(Montavon et al., 2017) proposed a decomposition method from a network classiﬁcation decision into contributions of its input elements based on deep Taylor decomposition.

Original: Shrikumar et al.(Shrikumar et al., 2016) proposed DeepLIFT which can compute importance scores in a multilayer neural network by explaining the difference of the output from some reference output in terms of differences of the inputs from their reference inputs.
Revised: Shrikumar et al.(Shrikumar et al., 2016) proposed DeepLIFT which can compute importance scores in a multilayer neural network by explaining the difference of the output from some reference output in terms of differences of the inputs from their reference inputs.

Original: Some other works make complex black box model simpler.
Revised: Some other works make complex black box model simpler.

Original: Che et al.(Che et al., 2017) proposed a simple distillation method called Interpretable Mimic Learning for extracting an interpretable simple model by gradient boosting trees.
Revised: Che et al.(Che et al., 2017) proposed a simple distillation method called Interpretable Mimic Learning for extracting an interpretable simple model by gradient boosting trees.

Original: Thiagarajan et al.(Thiagarajan et al., 2016) build a Treeview representation of the complex model by hierarchical partitioning of the feature space.
Revised: Thiagarajan et al.(Thiagarajan et al., 2016) build a Treeview representation of the complex model by hierarchical partitioning of the feature space.

Original: In addition, some references (Hinton et al., 2015; Bucila et al., 2006; Frosst & Hinton, 2017; Traore et al., 2019) proposed the distillation method of knowledge from an ensemble of models into a single model.
Revised: In addition, some references (Hinton et al., 2015; Bucila et al., 2006; Frosst & Hinton, 2017; Traore et al., 2019) proposed the distillation method of knowledge from an ensemble of models into a single model.

Original: Wu et al.(M. Wu, 2018) proposed a tree regularization method via knowledge distillation to represent the output feature space of a RNN based on a Multilayered perception.
Revised: Wu et al.(M. Wu, 2018) proposed a tree regularization method via knowledge distillation to represent the output feature space of a RNN based on a Multilayered perception.

Original: However, these methods can only solve the unexplainable problem of trained neural network or trained deep neural networks with explicit input characteristics.
Revised: However, these methods can only solve the unexplainable problem of trained neural network or trained deep neural networks with explicit input characteristics.

Original: Wan et al.(Wan et al., 2020) constructed a decision tree using the last fully connection layer of the discrimination part of a CNN based on a prior structure.
Revised: Wan et al.(Wan et al., 2020) constructed a decision tree using the last fully connection layer of the discrimination part of a CNN based on a prior structure.

Original: In the paper, our goal is to improve the interpretability performance of multi full connection layers of
Revised: In the paper, our goal is to improve the interpretability performance of multi fully connected layers of

Original: CNN without hurting its generalization performance by initiatively extracting its logic relationships with no prior structure and ﬁnally obtain its explanation by these logic relationships.
Revised: CNN without hurting its generalization performance by initiatively extracting its logic relationships with no prior structure and ﬁnally obtain its explanation by these logic relationships.

Original:  D EEP COGNITIVE LEARNING MODEL
Revised:  D EEP COGNITIVE LEARNING MODEL

Revised: For expressing the causal relationship between these neurons in the discrimination part, a new interpretable model is designed in the section.

Original: Convolution layers and pooling layers of a CNN constitute a  feature extractor.
Revised: As we all known, a CNN includes a feature extractor and a discrimination part. The feature extractor composes of some convolution layers and some pooling layers.

Original: The outputs from the feature extractor are the inputs to the fully connected layers of the CNN, namely feature maps, τ 1 , τ 2 , ..., τ k where k is the number of feature maps.
Revised: The outputs from the feature extractor are the inputs of the discrimination part of the CNN, namely feature maps, τ 1 , τ 2 , ..., τ k where k is the number of feature maps.

Original: All these feature maps form a feature set Γ .
Revised: All these feature maps form a feature set Γ .

Original: The objective of the fully connected layers is to use the feature maps to classify the input image x of the CNN into a label.

Original: So, the fully connected layers compose a discrimination part of the CNN.

Original: We suppose that the discrimination part should better be explained by the logic relationships between the features(the weight matrixes in the  ﬁrst layerof the part can capture) and its output vector.
Revised: We suppose that the discrimination part should better be explained by the logic relationships of the activated states of the neurons in its ﬁrst layer and its output layer.

Original: This is becausethat the relationships is the inherent relation ofsuch CNN structure which is neither related to the input data nor related to the extent of the features’ abstract.
Revised: This is because the relationships are the inherent property of the part.

Original: In order to express the relationships,  deep cognitive learning model (DCLM)is proposed. The DCLM is shown in Fig.1.
Revised: In order to express the relationships, a deep cognitive learning model (DCLM)is proposed, shown in Fig.1(b). I

Original: [Figure]
Revised: I I N ... ... I O

Revised: O C   k

Original: [Figure]
Revised: [Figure]

Original: The DCLM consists of three layers of nodes:feature predicate layer, disjunction layer and decisionresult predicate layer.
Revised: The DCLM consists of three layers feature predicate layer, disjunction layer, and decision layer.

Original: The top layer isthe feature predicate layer which consisits of feature predicate nodes.
Revised: The top layer is feature predicate layer which consists of many nodes.

Original: Everyfeature predicate node has afeature predicatefunction Z  (Γ) which expresseswhether a  neuron in the ﬁrst fully connected layer of  discrimination part of the CNN can capture a feature.
Revised: Every node has a predicate Z j (Γ) that expresses a positive action or negative action of features which the j th neuron in the ﬁrst fully connected layer of the discrimination part captures.

Original: Thefeature predicatefunction Z  (Γ) is deﬁned as follows.
Revised: The predicate Z j (Γ) is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original:  ∈ Γ , (1) null, τ i ∗ w i (cid:107) τ i (cid:107)(cid:107)
Revised: where j ∈ 1 , 2 , ..., N , N is the number of the input neurons of the ﬁrst fully connection layer of the discrimination part of the CNN.

Original: [Equation]
Revised: w i,j is a weight vector between the i th feature map and the j th neuron, b

Original: i (cid:107) = 0 and τ i ∈ Γ , ( [Equation] ) 0 , otherwise . [Equation]
Revised: j is the bias of the j th neuron, and ” ∗ ” is a convolution operation. ”

Original: [Equation] ) where i ∈ { 1 , 2 , ..., k } , τ i is the i th feature map in Γ , w i is its corresponding weight vector on the ﬁrst fully connection layer, namely feature-capture map and” ∗ ” is a convolution.
Revised: 1” and ”0” denote a positively activated state and a negatively activated state of the neuron respectively. ” null” denotes an inactivated state.

Original: The bottom layer is the decisionresult predicate layer which includes all decision result predicate nodes.
Revised: The bottom layer is a decision layer which includes all nodes used for decision.

Original: Everydecision result predicate node has adecision result predicatefunction which expresses whether the output y of an output neuron of the discrimination part is greater than 0.
Revised: Every node has a predicate which expresses an activated state of an output neuron of the discrimination part.

Original: It is deﬁned as following as
Revised: It is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original:  All  feature predicate nodes and everydecision result predicate node  are connected to one or moredisjunction nodes in the middle layer, namely as disjunction layer, with true or false edges.Every disjunction node represents a disjunction relation between all feature predicate nodes and a decision result predicate node.
Revised: where i ∈ 1 , 2 , ..., C , C is number of the output neurons of the CNN, y i is the output value of the i th output neuron of the discrimination part. All nodes on the feature predicate layer and every node on the decision layer are connected to one or more nodes on the middle layer, namely as disjunction layer, with true or false edges.

Original: A disjunction relation is expressed by a disjunctive normal form.
Revised: Every node represents a disjunction relation, which is expressed by a disjunctive normal form.

Original: If apredicate node is connected to  the disjunction node by a false edge, its predicatefunction follows after a nonoperator in the disjunctive normal form.
Revised: It is worth mentioning: if a node is connected to a node on the disjunction layer by a false edge, its predicate follows after a non-operator in the disjunctive normal form.

Original: Potential function of a disjunctive normal form can be obtained by using the Lukasiewicz method(Giles, 1975).
Revised: The potential function of a disjunctive normal form can be obtained by using the Lukasiewicz method(Giles, 1975).

Original: [Equation]
Revised: [Equation]

Original: where T (Γ , y  ) = (cid:80) Nj = 1 { a j [1 − Z j (Γ)] + (1 − a j ) Z j (Γ) } +  D 
Revised: where T (Γ , y i ) = (cid:80) Nj =1 { a j [1 − Z j (Γ)] + (1 − a j ) Z j (Γ) } + ( a N + 1) D ( y i ) and N is the number of the nodes on the feature predicate layer.

Original: y )

Original: and N is the number of the feature predicate nodes.

Original: If a j = 1 , there is a false edge between predicate node and a disjunction node. If a j = 0 , there is a edge In Fig.2 suppose a DCLM includes a disjunction relation ∀ Γ ¬
Revised: If a j = 1 , there is a false edge.

Original: Z 1 (Γ) ∨ Z 2 (Γ) ∨ Z 3 (Γ) ∨ ¬ Z 4 (Γ) ∨

Original: D ( y ) which is equivalent to ∀ Γ[ Z 1 (Γ) ∧ ¬ Z 2 (Γ) ∧ ¬ Z 3 (Γ) ∧ Z 4 (Γ) → D ( y )] , where Z j () is a feature predicate function and y is an output of an output neuron of CNN. Its potential function is φ c ( y )
Revised: Otherwise, there is a true edge.

Original: = min(1 , (cid:80) 4 i =1 { a j [1 − Z j (Γ)] + (1 − a j ) Z j (Γ) } + D ( y )) where a 1 = a 4 = 1 and a 2 = a 3 = 0 . The conditional probability distribution that a ground DCLM  is true is
Revised: The conditional probability distribution that a ground DCLM including all disjunctive normal forms is true is

Original: [Equation]
Revised: [Equation]

Original: where G is the number of all ground formulas, Ξ = (cid:80)
Revised: where G is the number of all ground formulas, Ξ = (cid:80)

Original: [Equation]
Revised: [Equation]

Original: ) is a partition function, ydclm =
Revised: ) is a partition function, y =

Original: ( ydclm, 1 , ydclm, 2 , ..., y dclm,G ) , y dclm,i is an output value ofan output neuron of the CNN and λ i is a weighted value of the i th ground formula.
Revised: ( y 1 , y 2 , ..., y G ) , y i is an output value of the CNN and λ i is a weighted value of the i th ground formula.

Original: By maximizing its likelihood function, the optimaly dclm , a i and λ i in the DCLM can be obtained.
Revised: By maximizing its likelihood function, the optimal a i and λ i in the DCLM can be obtained.

Original: [Equation]
Revised: [Equation]

Original: For extracting a  DCLM, a Maximum A Posterior(MAP) algorithm on the Maximum Satisﬁability Problem (MAX-SAT) was designed, which is shown in the appendix.
Revised: For extracting a optimal DCLM, a Maximum A Posterior(MAP) algorithm on the Maximum Satisﬁability Problem (MAX-SAT) was designed.

Revised: Using the disjunction normal form with the greatest weighted value in the optimal DCLM, a prediction of an input image can be obtained.

Original:  E VALUATION OF INTERPRETABILITY PERFORMANCE
Revised:  E VALUATION OF INTERPRETABILITY PERFORMANCE

Original: We consider that if theshape of function curve of the discrimination part of a CNN is similar as that of  its optimal interpretable model, it can bevery easily understood by humans and has well interpretability performance.
Revised: We consider that if the discrimination part of a CNN has a similar shape of function curve with its optimal interpretable model, the former can be easily explained by the latter.

Original: Therefore, the interpretable performance of the discrimination part can be measured by the shape similarity between it and the optimal interpretable model.
Revised: Therefore, the interpretable performance of the discrimination part can be measured by the shape similarity between it and its optimal interpretable model.

Original: We posit that given the same input data set, the similarity may be measured by variance of differences between outputs of the both models.
Revised: We posit that given the same input data set, the similarity may be measured by variance of differences between outputs of the both models.

Original: It cantentatively be named interpretation distance.
Revised: It can be named interpretation distance.

Original: It is easily proved that the smaller the interpretation distance is, the more similar the shape of the discrimination part is to the shape of the optimal interpretable model and the better the interpretability performance of the discrimination part would be.
Revised: It is easily proved that the smaller the interpretation distance is, the more similar their shapes are, and the better the interpretability performance of the discrimination part would be.

Original: Deﬁnition 

Revised: Deﬁnition

Original: If X is a compact metric space and ν is a Borel measure in X , such as Lebesgue measure or marginal measures, in L 2 ν ( X ) , a square integrable function space on X , the interpretation distance, φ d ( P ∗ , f ) , between a discrimination part f ( x ) and its optimal interpretable model P ∗ ( x ) is
Revised: If X is a compact metric space and ν is a Borel measure in X , such as Lebesgue measure or marginal measures, in L 2 ν ( X ) , a square integrable function space on X , the interpretation distance, φ d ( P ∗ , f ) , between a discrimination part f ( x ) and its optimal DCLM P ∗ ( x ) is

Original: [Equation]
Revised: [Equation]

Original: (6) where

Original: [Equation]
Revised: [Equation]

Revised: where

Original: [Equation]
Revised: [Equation]

Original: In the deﬁnition, the optimal interpretable model can have many forms, such as DCLM, other probabilistic graphical model, decision tree, decision rules and so on.

Original:  G AME BETWEEN A DCLM AND THE DISCRIMINATION PART OF A CNN
Revised:  G AME BETWEEN A DCLM AND THE DISCRIMINATION PART OF A CNN

Original: As discussed above,when the shapes of the discrimination part of a CNN and its optimal interpretable model are enough similar, the discrimination part has well interpretability performance.
Revised: As discussed above,when the shapes of the discrimination part of a CNN and its optimal interpretable model are enough similar, the discrimination part has well interpretability performance.

Original: However, its generalization performance will tend to decrease.
Revised: However, its generalization performance will tend to decrease.

Original: This is mainly attributed to the fact that  the sufﬁcient and necessary condition for the consistent convergence of the two performances, φ d ( P ∗ , f ∗ ) = 

Revised: This is mainly attributed to the fact that because of data bias and noisy data in training data set, the sufﬁcient and necessary condition for the consistent convergence of the two performances, φ d ( P ∗ , f ∗ ) =

Original: ( f ∗ is the optimal predication model), is difﬁcult to be guaranteed.
Revised: ( f ∗ is the optimal prediction model), is difﬁcult to be guaranteed.

Original: For proving the conclusion, we focus on a neuron of a CNN. From the foot we may judge of Hercules. Because every input channel f ( x ) of the neuron can be seen as a kernel function
Revised: Therefore, for the tradeoff problem, in training process, extracting an

Original: K ( x, w ) ( w is a weight vector including a bias of the neuron), it may span a kernel Hilbert space H K = { f ( x ) ∈ L 2 ν ( X ) | f ( x ) =
Revised: DCLM from the discrimination part and then iteratively reducing the interpretation distance between the two models may be a feasible solution.

Original: K ( x, w ) =

Original: (cid:80) k =1 φ k ( w ) φ k ( x ) } . H K is a linear function set on Algorithm 1 Game between DCLM and the discrimination part of a CNN(Its time complexity(TC) is O ( N + M ) where N is TC of training CNN, M is TC of construction of Logic Net.)
Revised: A detailed discussion about the problem can be found in the appendix.

Original: Input: data X , target Y t initialize Logic Net LN , CNN CN for i = 1 to batch − size do FMs, f nn = CN ( X[ i ]) .y dclm =
Revised: To avoid reducing the generalization performance, the maximum probability p ( w | X, y t ) should be guaranteed, where X is a training sample, w is the parameter set of the CNN and y t is the target vector of X .

Original: LN ( FMs, Y t

Original: [ i ]) .
Revised: [Equation]

Original: CN = Updata CNN Gradient ( y dclm , f nn , y t [ i ]) FMs, FCMs, f nn = CN ( X [ i ]) . LN = ConstructionLogicNet ( FMs, FCMs, f nn ) end for
Revised: Algorithm

Original: L 2 ν ( X ) . However, because of data bias and noisy data in training data set, its optimal interpretable model which obey to human’s cognition usually is a nonlinear function in L 2 ν ( X ) in the majority of engineering applications. Because continuous linear functional set is nowhere dense in L 2 ν (X ) . So the sufﬁcient and necessary condition is hard to be satisﬁed and the tradeoff problem always exists.
Revised: O (

Original: This is also the case with the discrimination part of a CNN.

Original: However its optimal interpretable model P ∗ and its optimal discrimination part are unknown. Therefore, for the tradeoff problem, in training process, extracting an interpretable model from the discrimination part and then iteratively reducing the interpretation distance between the two models may be a feasible solution. A detailed discussion about the problem can be found in the appendix.
Revised: +

Original: To avoid to reduce the generalization performance, the maximum probability p ( w | X, y t ) should be guaranteed, where X is a training sample, w is parameter setof the CNN and y t is target vector of X .
Revised: Input: Data set

Original: [Equation]
Revised: Output: DCLM Repeat CNN = CNN Train ( Data set, Adam, loss = CrossEntropy ) for data,label in Data set do

Original: where p ( y t | w, X ) 
Revised: Feature map = CNN feature e xtractor ( CNN ) . disjunctive normal form = Disjunction ( Feature map, Fnn, W i,j ) .

Revised: To predicate value of neuron according to Eq.

Revised: Use the disjunction forms of data obtained from the all predicate combinations to update

Original:  = (cid:82) p ( y t | f, w, X ) (cid:82) p ( f |
Revised: DCLM . end for for i = 1 to n do for data,label in Data set do Feature map = CNN feature extractor ( CNN ) ym =

Revised: DCLM ( Feature map )

Revised: CNN DCLM ( data, ym, loss =

Original: [Equation]
Revised: Eq. 14) end for end for Until the interpretation distance and accuracy converge where p ( y t | w , X ) = (cid:82) p ( y t | f, w , X ) (cid:82) p ( f | y dclm , w , X ) p ( y dclm | w , X )

Revised: d y dclm df and y dclm is a prediction of DCLM.

Original: When the DCLM is known, y ∗ dclm is its optimal solution and p ( y ∗ dclm | w, X ) = 1 .
Revised: When the DCLM is known, y ∗ dclm is its optimal prediction and p ( y ∗ dclm | w , X ) = 1 .

Original: Then 
Revised: Then (cid:90)

Original: [Equation]
Revised: [Equation]

Original: Similarly, known the input X and w , f nn is the optimal solution of the CNN.
Revised: Similarly, known the input X and w , f nn is the optimal solution of the CNN.

Original: [Equation]
Revised: [Equation]

Original: If w and X are given and the loss function 
Revised: If w and X are given and the loss function φ

Original: [Equation]
Revised: [Equation]

Original:  the conditional probability distribution function
Revised: , the conditional probability distribution function

Original: [Equation]
Revised: [Equation]

Original: Meanwhile,
Revised: Meanwhile,

Original: [Equation]
Revised: [Equation]

Original: where Ξ 1 and Ξ 2 are partition functions.
Revised: where Ξ 1 and Ξ 2 are partition functions.

Original: Then by maximizing a likelihood function of p ( w | X, y t )
Revised: Then by maximizing a likelihood function of p ( w | X, y t )

Original: the optimal w can be obtained.
Revised: the optimal w can be obtained.

Original: In particular, assuming that w follows Gaussian distribution, we get:
Revised: In particular, assuming that w follows Gaussian distribution, we get:

Original: [Equation]
Revised: [Equation]

Original: (13) where α is a meta-parameter determined by the variance of the selected Gaussian distributions.
Revised: where α is a meta-parameter determined by the variance of the selected Gaussian distributions.

Original: Turn it into a minimization problem:
Revised: Turn it into a minimization problem:

Original: [Equation]
Revised: [Equation]

Original: The iterative optimization algorithm is shown as follows:
Revised: The iterative optimization algorithm is shown as follows:

Original:  E XPERIMENTAL VERIFICATION
Revised:  E XPERIMENTAL VERIFICATION

Original: We designed some experiments to verify the effectiveness of the proposed method.
Revised: We designed two experiments to verify the effectiveness of the proposed method.

Original: The ﬁrst experiment veriﬁed whether the self-explanatory method could improve the interpretability performance of the CNN without sacriﬁcing its generalization performance.
Revised: The ﬁrst experiment veriﬁed whether the self-explanatory method could improve the interpretability performance of the CNN without sacriﬁcing its generalization performance.

Original: The second experiment veriﬁed whether the proposed method can tend towards stability and convergence in the game process.
Revised: The second experiment veriﬁed whether the proposed method can tend towards stability and convergence in the game process.

Original: In the experiments,the structure of the CNN3(includes 3 convolution layers,3 MaxPooling layers, 3 fully connect layers(FCLs) and 1 output layer),CNN5(includes 5 convolution layers,

Revised: In the experiments, CNN3(includes 3 convolution layers,3 MaxPooling layers, 3 fully connect layers(FCLs) and 1 output layer),CNN5(includes 5 convolution layers,

Original: MaxPooling layers, 3 FCLs and 1 output layer) and CNN8(includes 8 convolution layers,8 MaxPooling layers, 3 FCLs and 1 output layer) were used.
Revised: MaxPooling layers, 3 FCLs and 1 output layer), and CNN8(includes 8 convolution layers,8 MaxPooling layers, 3 FCLs andoutput layer) were used.

Original: The ﬁnal structures obtained after the training  onthese CNNs by the  proposed method are named as CNN3-DCLM, CNN5-DCLM and CNN8DCLM respectively.
Revised: Traditional training methods on the three types CNN were named as CNN3Trad, CNN5-Trad and CNN8-Trad respectively. By contrast, our proposed methods on these CNNs were named as CNN3-DCLM, CNN5-DCLM and CNN8-DCLM respectively.

Original: All experiments used Mnist(Lecun et al., 1998), FashionMnist(Zalando, 2017) and Emnist(Cohen et al., 2017) benchmark data sets.
Revised: All experiments used Mnist(Lecun et al., 1998), FashionMnist(Zalando, 2017), and Cifar-10(Krizhevsky, 2009) benchmark data sets.

Original: All algorithms were implemented in Python using the Pytorch library(Paszke et al., 2019).
Revised: All algorithms were implemented in Python using the Pytorch library(Paszke et al., 2019).

Original: All experiments ran on a server with Intel Xeon 4110(2.1GHz)
Revised: All experiments ran on a server with Intel Xeon 4110(2.1GHz)

Original: Silver Processor, 20GB RAM and Nvidia Telsa T4.
Revised: Silver Processor, 20GB RAM and Nvidia Telsa T4.

Original: Experiment 1: Performance veriﬁcation of the proposed method on CNN .
Revised: Experiment 1: Performance veriﬁcation of the proposed method on CNN .

Original: The experiment also compared soft decision tree(SDT)

Original: (Frosst & Hinton, 2017)based on trained CNN3,CNN5 and CNN8,namely ”CNN3-SDT”, ”CNN5-SDT” and ”CNN8-SDT” respectively. These baseline methods also include SDT (Frosst & Hinton, 2017) based on trained CNN3-DCLM,CNN5-DCLM and 
Revised: We replaced the discrimination part of three traditionally trained CNNs with soft decision tree(SDT)(Frosst & Hinton, 2017) and designated these methods as CNN3-Trad-SDT, CNN5-Trad-SDT and CNN8-Trad-SDT respectively.

Original: CNN8-DCLM, namely ”CNN3*-SDT”, ”CNN5*-SDT” and ”CNN8*-SDT” respectively. These indicators of  all algorithms on all testing data sets were shown in Table 1.
Revised: The accuracy of CNN, accuracy of STD or DCLM, and interpretation distance corresponding to all methods were shown in Table 1.

Original: Some values are ”—”, which indicates that these results do not exist.
Revised: Some values are ”—”, which indicates that these results do not exist.

Original: It is observed in Table 1 that the accuracy of all CNN-DCLMs are higher than the two interpretable models, such as SDT and DCLM, on all benchmark data sets and are around 1.4 percentage points lower than those of CNN.
Revised: It is observed in Table 1 that the accuracy of all CNN-DCLMs are higher than the two interpretable models, such as SDT and DCLM, on all benchmark data sets and are around 1.4 percentage points lower than those of CNN-Trads.

Original: But it is worth noticing that on the most of data sets the interpretation distances of all CNN-DCLMs arearound 5 percentage points lower than interpretation distances between the majority SDTs and their CNNs except CNN3-SDT on Emnist data set.
Revised: But it is worth noticing that on the most of data sets the interpretation distances of all CNN-DCLMs are lower than interpretation distances of all CNN-Trad-SDTs.

Original: These might prove that that the self-explanatory method can improve the interpretability performance of the discrimination part of a CNN without largely reducing its generalization performance.
Revised: These might prove that that the self-explanatory method can improve the interpretability performance of the discrimination part of a CNN without largely reducing its generalization performance.

Original: On Mnist data set the accuracies of the DCLMs are only 0.7 percentage points lower than those of the SDTs  on CNN5-SDT and CNN8-SDT except CNN3-SDT.
Revised: The accuracies of the DCLMs are higher than those of the SDTs except CNN3-DCLM on FashionMnist data set and CNN3-DCLM on Mnist.

Original: Though for Emnist data set and

Original: FashionMnist data set,we can ﬁnd that the accuracies of the DCLMs are 2 percentage points lower than those of the SDTs on CNN3-SDT, we also can ﬁndthat on the two data sets the accuracies of the DCLMs are around 4.3 percentage points higher than those of the SDTs on CNN5-SDT and CNN8SDT. It is mainly because that the feature extract parts of CNN5 and CNN8 output more abstract feature maps than CNN3 for Emnist data set and FashionMnist data set. The abstract feature maps can be easily expressed without distortion by feature predicates of the DCLM, so it does not impede the generation of the DCLM.
Revised: These results might prove that the proposed method can ﬁnd more excellent interpretable model than CNN-Trad-SDT.

Original: Experiment 2: Convergence test of the proposed method We designed the experiments to demonstrate convergence of the proposed method.
Revised: Experiment 2: Convergence test of the proposed method We designed the experiments to demonstrate convergence of the proposed method.

Original: CNN3, CNN5 and CNN8 were used for comparing with CNN3-DCLM, CNN5-DCLM, and CNN8-DCLM respectively.
Revised: CNN3-Trad, CNN5-Trad and CNN8-Trad were used for comparing with CNN3-DCLM, CNN5-DCLM, and CNN8-DCLM respectively.

Original: Every training works outepoches.
Revised: Every training works out 25 epochs.

Original: All results were measured at every epoch and shown as the four ﬁgures,  Fig.3, Fig.4, Fig. and Fig.5.
Revised: Experiment results were measured at every epoch and shown in four ﬁgures, Fig.2, Fig.3, Fig.5 and Fig.4.

Original: Every ﬁgure includes nine subplots.
Revised: Every ﬁgure includes nine subplots.

Original: The three subplots on the left column were shown 
Revised: The three subplots on the left column were shown for the experiment results on Mnist data set. These subplots on the middle column were for FashionMnist data set and these subplots on the right column were for Cifar-10 data set. [Table] A

Revised: [Table]

Revised: A CCURACY OF SDT OR DCLM I NTERPRETATION D ISTANCE

Original:  In Fig.3, the accuracies of the DCLMs and the CNN-DCLMs of every epoch in the game process were shown.
Revised: [Table] In Fig. [Table] the accuracies of the DCLMs and the CNN-DCLMs of every epoch in the game process were shown.

Original: From these ﬁgures, it is obvious that accuracies of the DCLMs and these CNN-DCLMs steadily increase in the early stage.
Revised: From these ﬁgures [Table] it is obvious that accuracies of the DCLMs and these CNN-DCLMs steadily increase in the early stage.

Original: In the next stage, their accuracies tend to be stable.
Revised: In the next stage, their accuracies tend to be stable.

Original: This reﬂects that the game method did not affect the improvement of the generalization performances of these
Revised: This reﬂects that the game method do not affect the improvement of the generalization performances of these

Original: DCLMs and these CNNs.
Revised: DCLMs and these CNNs.

Original: Fig.4 shows the interpretation distances between the  DCLMs and  the CNNs  and CNN-DCLMs.
Revised: We also ﬁnd that the accuracy gap for the FashionMnist data set is much more than other two data sets. The main reason is that for the FashionMnist data set the logical relationships between the activation states of the input neurons and those of the output neurons of the discrimination part of the CNNs are more complex than the other data sets. Even so, the proposed method also improves the accuracies of the DCLMs and these CNN-DCLMs steadily. Fig.3 shows the interpretation distances of the CNNs trained by the traditional method and the CNNs trained by the proposed method.

Original: As seen from these subplots, the interpretation distances of CNNs not involved in the game are greater than those of CNN-DCLMs at the most of the epochs, especially by the end of the training.
Revised: As seen from these subplots, the interpretation distances of CNNs trained by the traditional method are greater than those of the other CNNs at the most of the epochs, especially by the end of the training.

Original: The results indicate that the game method can effectively improve the interpretability performance of CNN-DCLMs.
Revised: The results indicate that the game method can effectively improve the interpretability performance of CNNs.

Original: [Table]
Revised: From these subplots in Fig.2and Fig.3, we also can see that after the 15th epoch, interpretation distances of CNNs from the proposed method tend to converge. The phenomenon indicates that the discrimination part of the CNNs can be explained by its DCLMs at every epoch after the ﬁfteenth epoch.

Original: From Fig.5 it is evident that all DCLMs of CNN3, CNN5 and CNN8 have been found to have stable  entropies at the end of the game.
Revised: From Fig.4, it is evident that all DCLMs from CNN3-DCLMs, CNN5-DCLMs, and CNN8-DCLMs have been found to have stable information entropies at the end of the game,which calculate the diversity of disjunction normal forms in the DCLMs obtained by the game method.

Original: On Mnist data set, the entropies ﬁnally converge between 120 and 145.
Revised: On Mnist data set, the entropies ﬁnally converge between 120 and 145.

Original: On FashionMnist data set, the entropies ﬁnally converge between 125 and 136.
Revised: On FashionMnist data set, the entropies ﬁnally converge between 125 and 136.

Original: On Emnist data set, the entropies ﬁnally converge between 100 and 120.
Revised: On Cifar-10 data set, the entropies ﬁnally converge between 100 and 120.

Original: The results indicate that the game algorithm can ensure that the DCLM converges to a stable state.
Revised: The results indicate that the game algorithm can ensure that the DCLMs converges to a stable state.

Revised: The DCLMs from the proposed method have sparser disjunction normal forms than these not participating in the game process, which indicates they are more robust.

Original: In Fig.6,the accuracies of the CNNs  and the CNN-DCLMs of every epochin the game process were shown.
Revised: In Fig.5,the accuracies of the CNNs trained by the traditional method and the CNNs trained by the proposed method at every epoch were shown.

Original: From these subplots, it can be seen that the accuraciesof the CNNs and those of the CNN-DCLMs steadily increase in the early stage and accuracies of CNN-DCLMs are lower than those of CNNs.
Revised: From these subplots, it can be seen that its accuracies steadily increase in the early stage and accuracies of the CNNs trained by the proposed method are lower than the other CNNs.

Original: But in the last stage, their accuracies tend to be stable and consistent.
Revised: But in the following stage, their accuracies tend to be stable and consistent.

Original: The main reason is that in the early stage, a tradeoff problem between the generalization performance and interpretability performance of CNN-DCLMs must reduce its generalization performance of CNNDCLMs to increase its interpretability performance.
Revised: The main reason is that in the early stage, a tradeoff problem between the generalization performance and interpretability performance of the discrimination part of a CNN inevitably reduces its generalization performance in order to increase its interpretability performance.

Original: But the proposed game method can effectively reduce the gap betweenthe two performances.
Revised: But the proposed game method can effectively reduce the gap between two performances.

Original: This reﬂects that such method is effective for the tradeoff problem.
Revised: This reﬂects that the proposed method is effective for the tradeoff problem.

Original: From these subplots in Fig.3and Fig.4, we also can see that on every epoch a DCLM could be obtained.

Original: And after the 15th epoch, their interpretation distances tend to converge.

Original: The phenomenon indicates that the game process can decrease the distances and the training process of the discrimination part of the CNNs can be explained by their DCLMs in real time after the ﬁfteenth epoch.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The performance of the proposed method was demonstrated by experiments on benchmark data sets.
Revised: The performance of the proposed method was demonstrated by experiments on benchmark data sets.

Original: The proposed method showed prominent advantages over traditional learning algorithm on CNN for improving the generalization performance and the interpretability performance of the discrimination part of the CNN.
Revised: The proposed method showed prominent advantages over traditional learning algorithm on CNN for improving the generalization performance and the interpretability performance of the discrimination part of the CNN.

Original: In practical engineering, the proposed method may provide a new learning paradigm.
Revised: In practical engineering, the proposed method may provide a new learning paradigm.

Original: The method can not only predict an  accurate result for new input data but also provide a reasonable causal interpretation for the prediction of the discrimination part of a CNN.
Revised: The method can not only predict an relatively accurate result for new input data but also provide a reasonable causal interpretation for the prediction of the discrimination part of a CNN.

Original: We suppose that it can solve the black box problem in the discrimination part.
Revised: We suppose that it can solve the black box problem in the discrimination part.

Original: We believe that the proposed method provide a way how human understands the discrimination part.
Revised: We believe that the proposed method provide a way to understand the discrimination part

Original: A A PPENDIX : A M AXIMUM A P OSTERIOR (MAP) ALGORITHM FOR EXTRACTING A DCLM
Revised: A A PPENDIX : A M AXIMUM A P OSTERIOR (MAP) ALGORITHM FOR EXTRACTING A DCLM

Original: A Maximum A Posterior(MAP) algorithm on the Maximum Satisﬁability Problem (MAX-SAT) for extracting an DCLM is as follows.
Revised: A Maximum A Posterior(MAP) algorithm on the Maximum Satisﬁability Problem (MAX-SAT) for extracting an DCLM is as follows.

Original: B A PPENDIX : S UFFICIENT AND NECESSARY CONDITION FOR CONSISTENT CONVERGENCE OF THE GENERALIZATION PERFORMANCE AND THE INTERPRETABILITY PERFORMANCE
Revised: B A PPENDIX : S UFFICIENT AND NECESSARY CONDITION FOR CONSISTENT CONVERGENCE OF THE GENERALIZATION PERFORMANCE AND THE INTERPRETABILITY PERFORMANCE

Original: If the minimization of loss function of CNN can guarantee the minimum of its interpretability performance, learning algorithm of the CNN can improve its interpretability performance.
Revised: If the minimization of loss function of CNN can guarantee the minimum of its interpretability performance learning algorithm of the CNN can improve its interpretability performance.

Original: If not, the tradeoff problem between the generalization performance and the interpretability performance will exit.
Revised: If not the tradeoff problem between the generalization performance and the interpretability performance will exit.

Original: For proving the existence of the problem, we focus on a neuron of CNN.
Revised: For proving the existence of the problem, we focus on a neuron of CNN.

Original: From the foot we may judge of Hercules.
Revised: From the foot we may judge of Hercules.

Original: If an input channel f ( x ) of the neuron is seen as a kernel function
Revised: If an input channel f ( x ) of the neuron is seen as a kernel function

Original: K ( x, w ) ( w is weight vector including a bias of the neuron), it will span a kernel Hilbert space
Revised: K ( x, w ) ( w is weight vector including a bias of the neuron), it will span a kernel Hilbert space

Original: H K =
Revised: H K =

Original: { f ( x ) ∈
Revised: { f ( x ) ∈

Original: L 2 ν ( X ) | f ( x ) =
Revised: L 2 ν ( X ) | f ( x ) =

Original: K ( x, w ) =
Revised: K ( x, w ) =

Original: (cid:80) ∞
Revised: (cid:80) ∞

Original: k =1 a k φ k ( x ) } for the neuron.
Revised: k =1 a k φ k ( x ) } for the neuron.

Original: H K can be regarded as a linear function set on L 2 ν ( X ) .
Revised: H K can be regarded as a linear function set on L 2 ν ( X ) .

Original: It is a solution space of the neuron.
Revised: It is a solution space of the neuron.

Original: The necessary and sufﬁcient conditions for consistent convergence between the generalization performance and the interpretability performance are discussed below in L 2 ν ( X ) based on the following lemmas.
Revised: The necessary and sufﬁcient conditions for consistent convergence between the generalization performance and the interpretability performance are discussed below in L 2 ν ( X ) based on the following lemmas.

Original: Lemma 1.
Revised: Lemma 1.

Original: Continuous linear functional set on a separable Hilbert space X is nowhere dense in a square integrable function space L 2 ν ( X ) .
Revised: Continuous linear functional set on a separable Hilbert space X is nowhere dense in a square integrable function space L 2 ν ( X ) .

Original: Lemma 2.
Revised: Lemma 2.

Original: Continuous nonlinear functional set of the separable Hilbert space X is everywhere dense in L 2 ν ( X ) .
Revised: Continuous nonlinear functional set of the separable Hilbert space X is everywhere dense in L 2 ν ( X ) .

Original: When the optimal input channel f ∗ ( x ) approximate a linear functional in H K while the optimal interpretable model P ∗ ( x ) don’t approximate any linear functional or do not exist in H K , traditional training process cannot guarantee f ( x ) approximates P ∗ ( x ) according Lemma 1.
Revised: When the optimal input channel f ∗ ( x ) approximate a linear functional in H K while the optimal interpretable model P ∗ ( x ) don’t approximate any linear functional or do not exist in H K , traditional training process cannot guarantee f ( x ) approximates P ∗ ( x ) according Lemma 1.

Original: From Lemma 2, the approximate will cannot converge until P ∗ ( x ) approximate f ∗ ( x ) .
Revised: From Lemma 2, the approximate will cannot converge until P ∗ ( x ) approximate f ∗ ( x ) .

Original: Here, if we deﬁne the approximation as the similarity between the shapes of function curve of f ∗ ( x ) and P ∗ ( x ) , the sufﬁcient and necessary condition for the consistent convergence of the two performances is φ d ( P ∗ , f ∗ ) = 0 .
Revised: Here, if we deﬁne the approximation as the similarity between the shapes of function curve of f ∗ ( x ) and P ∗ ( x ) , the sufﬁcient and necessary condition for the consistent convergence of the two performances is φ d ( P ∗ , f ∗ ) = 0 .

Original: For the discrimination part of the CNN, the sufﬁcient and necessary condition is still true.
Revised: For the discrimination part of the CNN, the sufﬁcient and necessary condition is still true.

Original: However because of data bias and noisy data in training data set, the condition is difﬁcult to to be ensured in the majority of engineering applications.
Revised: However because of data bias and noisy data in training data set, the condition is difﬁcult to to be ensured in the majority of engineering applications.

Original: The tradeoff problem always exists between the two performances of the discrimination part.
Revised: The tradeoff problem always exists between the two performances of the discrimination part.

Original: According to the above conclusion, in order to completely solve the tradeoff problem, the φ d ( P ∗ , f ∗ ) (Here f ∗ ( x ) is the optimal discrimination part) should be reduced.
Revised: According to the above conclusion, in order to completely solve the tradeoff problem, the φ d ( P ∗ , f ∗ ) (Here f ∗ ( x ) is the optimal discrimination part) should be reduced.

Original: However P ∗ and f ∗ ( x ) are unknown.
Revised: However P ∗ and f ∗ ( x ) are unknown.

Original: Therefore, in training process, extracting a interpretable model P ( x ) from a discrimination part f ( x ) and then iteratively reducing φ d ( P, f ) may be a convenient method.
Revised: Therefore, in training process, extracting a interpretable model P ( x ) from a discrimination part f ( x ) and then iteratively reducing φ d ( P, f ) may be a convenient method.