Paper: Causal-based Analysis on Credibility of Feedforward Neural Network (ICLR-2024)

Reviewer comment (Theory):
The definitions and illustrations of “causal substructure” are ambiguous. The authors give explanations using individual neurons $l_{1a}$, $l_{1b}$, and $l_{nc}$, but this gives me an intuition that cases (b) and (c) in Figure 3 need not to be treated separately. For example, if I **swap $l_{1a}$ and $l_{1b}$ in case (c) and intervene on $l_{1b}$**, then case (c) is just equivalent to case (b). In other words, every time I encounter a confounding substructure, I can just swap the order of input features and intervene on the same neuron to transform it into a mediation substructure. Is that right? Please correct me if I misunderstood anything.

Author response:
We thank the reviewer for your positive and constructive feedbacks. We have modified the annotations in Section 4  to make our expression clearer. To make the illustration clear, we mark $f^*$ and $f’$ in Figure 1(c). We also have condensed Eq.7-9 by presenting a single equation. Regarding the issue you mentioned in Figure 3, we analyze the three figures in Figure 3 with neuron l1a as the protagonist, so in Figures 3b and 3c, neuron l1a is in different causal situations. On the other hand, your viewpoint is also very correct. If we exchange the positions between the two input layer neurons l1a and l1b in Figures 3b and 3c, their situations can also be classified into one category. So from another perspective, it can also be divided into two types of substructures, but this is not the original intention we want to showcase. After observing the experimental results, we choose threshold values of 1% and 9%, which may lack some rigor. In addition, we supplemented the ablation experiment, and without considering the causal relationship between input layer neurons. It can be seen that our ATE slope has significantly decreased, making the analysis of neural network credibility more difficult. We apologize for not clarifying all questions given the limited space and many reviews. We will fix all typos and add missing references in the next revision.

Reviewer comment (Writing):
The notations in Section 4 are very confusing as the authors use $l_{1a}, l_{1b}$ and $x_{i}$ interchangeably to represent input neurons. My understanding is that $x_{i}$ represents some value that the neuron $l_{1i}$ can take. If so, then the expressions $E[\cdot|x_{i} = \alpha]$ and $E[\cdot|do(x_{i} = \alpha)]$ again become very confusing.

Author response:
We thank the reviewer for your positive and constructive feedbacks. We have modified the annotations in Section 4  to make our expression clearer. To make the illustration clear, we mark $f^*$ and $f’$ in Figure 1(c). We also have condensed Eq.7-9 by presenting a single equation. Regarding the issue you mentioned in Figure 3, we analyze the three figures in Figure 3 with neuron l1a as the protagonist, so in Figures 3b and 3c, neuron l1a is in different causal situations. On the other hand, your viewpoint is also very correct. If we exchange the positions between the two input layer neurons l1a and l1b in Figures 3b and 3c, their situations can also be classified into one category. So from another perspective, it can also be divided into two types of substructures, but this is not the original intention we want to showcase. After observing the experimental results, we choose threshold values of 1% and 9%, which may lack some rigor. In addition, we supplemented the ablation experiment, and without considering the causal relationship between input layer neurons. It can be seen that our ATE slope has significantly decreased, making the analysis of neural network credibility more difficult. We apologize for not clarifying all questions given the limited space and many reviews. We will fix all typos and add missing references in the next revision.

Reviewer comment (Presentation):
To make the illustration clear, it might be better to clearly mark $f^{*}$ and $f’$ in Figure 1(c).

Author response:
We thank the reviewer for your positive and constructive feedbacks. We have modified the annotations in Section 4  to make our expression clearer. To make the illustration clear, we mark $f^*$ and $f’$ in Figure 1(c). We also have condensed Eq.7-9 by presenting a single equation. Regarding the issue you mentioned in Figure 3, we analyze the three figures in Figure 3 with neuron l1a as the protagonist, so in Figures 3b and 3c, neuron l1a is in different causal situations. On the other hand, your viewpoint is also very correct. If we exchange the positions between the two input layer neurons l1a and l1b in Figures 3b and 3c, their situations can also be classified into one category. So from another perspective, it can also be divided into two types of substructures, but this is not the original intention we want to showcase. After observing the experimental results, we choose threshold values of 1% and 9%, which may lack some rigor. In addition, we supplemented the ablation experiment, and without considering the causal relationship between input layer neurons. It can be seen that our ATE slope has significantly decreased, making the analysis of neural network credibility more difficult. We apologize for not clarifying all questions given the limited space and many reviews. We will fix all typos and add missing references in the next revision.

Reviewer comment (Presentation):
Equations 7-9 appear to have similar expressions but are applied in different contexts. The authors might consider condensing this section by providing more detailed context and presenting a single equation.

Author response:
We thank the reviewer for your positive and constructive feedbacks. We have modified the annotations in Section 4  to make our expression clearer. To make the illustration clear, we mark $f^*$ and $f’$ in Figure 1(c). We also have condensed Eq.7-9 by presenting a single equation. Regarding the issue you mentioned in Figure 3, we analyze the three figures in Figure 3 with neuron l1a as the protagonist, so in Figures 3b and 3c, neuron l1a is in different causal situations. On the other hand, your viewpoint is also very correct. If we exchange the positions between the two input layer neurons l1a and l1b in Figures 3b and 3c, their situations can also be classified into one category. So from another perspective, it can also be divided into two types of substructures, but this is not the original intention we want to showcase. After observing the experimental results, we choose threshold values of 1% and 9%, which may lack some rigor. In addition, we supplemented the ablation experiment, and without considering the causal relationship between input layer neurons. It can be seen that our ATE slope has significantly decreased, making the analysis of neural network credibility more difficult. We apologize for not clarifying all questions given the limited space and many reviews. We will fix all typos and add missing references in the next revision.

Reviewer comment (Writing):
Could you elaborate more on the “causal substructure” as I mentioned in the first bullet point in the Weaknesses section?

Author response:
We thank the reviewer for your positive and constructive feedbacks. We have modified the annotations in Section 4  to make our expression clearer. To make the illustration clear, we mark $f^*$ and $f’$ in Figure 1(c). We also have condensed Eq.7-9 by presenting a single equation. Regarding the issue you mentioned in Figure 3, we analyze the three figures in Figure 3 with neuron l1a as the protagonist, so in Figures 3b and 3c, neuron l1a is in different causal situations. On the other hand, your viewpoint is also very correct. If we exchange the positions between the two input layer neurons l1a and l1b in Figures 3b and 3c, their situations can also be classified into one category. So from another perspective, it can also be divided into two types of substructures, but this is not the original intention we want to showcase. After observing the experimental results, we choose threshold values of 1% and 9%, which may lack some rigor. In addition, we supplemented the ablation experiment, and without considering the causal relationship between input layer neurons. It can be seen that our ATE slope has significantly decreased, making the analysis of neural network credibility more difficult. We apologize for not clarifying all questions given the limited space and many reviews. We will fix all typos and add missing references in the next revision.

Reviewer comment (Experiments):
It was unclear to me why the neurons in the hidden layer were left out of account if data for those is available.

Author response:
To simplify our causal structure, we delete the hidden layers because we focus more on input and output layer neurons. We focus on the causal relationship and effects between input layer neurons and output layer neurons, and further analyze the credibility of neural network decisions. To simplify the analysis process, we have omitted hidden layer neurons.

Reviewer comment (Theory):
Another issue is that this neural net consists of just three layers. Can this analysis extend to deep neural nets, a subject of considerable interest in recent years?

Author response:
We believe that our method can be further expanded by combining deep neural networks.

Reviewer comment (Theory):
The NN considered consists of just three layers. Can this analysis extend to deep neural nets, a subject of considerable interest in recent years?

Author response:
We believe that our method can be further expanded by combining deep neural networks.

Reviewer comment (Presentation):
In Figure 1, the subfigures (a), (b), and (c) depict causal structures, but the explanatory text and the symbols are not clearly defined.

Author response:
Thank you for the feedback and suggestions. We have added text descriptions and symbol explanations for figure1 in the latest version. We apologize for not clarifying all questions given the limited space and many reviews.

Reviewer comment (Experiments):
The effectiveness of this method was only validated in the medical field during the experiment. Could the authors validate the method's ability to analyze credibility in multiple domains using more diverse datasets?

Author response:
Thank you for the feedback and suggestions. We believe that we can try to prove our method in multiple fields in the future. We apologize for not clarifying all questions given the limited space and many reviews.

Reviewer comment (Presentation):
In Figure 1, there is a lack of text description and symbol interpretation.

Author response:
Thank you for the feedback and suggestions. We have added text descriptions and symbol explanations for figure1 in the latest version. We apologize for not clarifying all questions given the limited space and many reviews.

Reviewer comment (Presentation):
In Figure 1(b), why are A, B, D, E, F, and C represented in different colors? Do they have any specific meanings?

Author response:
Thank you for the feedback and suggestions. We marked neurons with causal relationships between input layers in green, while independent neurons remained in blue. The red arrows between the input layer neurons represent the causal relationship between them. We apologize for not clarifying all questions given the limited space and many reviews.