Paper: Improving Neural Program Induction by Reflecting on Failures (ICLR-2024)

Reviewer comment (Writing):
I find Section 2.2 very hard to understand. I have some background in logic programming, but none in neural logic machines. Specifically, I was unsure on the following points: "First, given a set of objects $\mathcal{U}$ of size $m$..." -- it is not clear what an object is, or what it means for an object to have a size.

Author response:
Object refers to an entity or element within a domain of discourse, which is a fundamental and broadly accepted concept of first-order logic [1]. “a set of objects $U$ of size $m$ ” means there is a set $U$, and $U$ contains m objects: $|U|=m$. Grounding is a frequently used concept in first-order logic which refers to the process of creating a variable-free first-order formula equivalent to a first-order sentence [2][3]. Thus, “ grounds p on $U$” denotes the process of instantiating the variables ($\alpha_1, \dots, \alpha_n$) of atom p($\alpha_1, \dots, \alpha_n$) with the corresponding truth values of the attributes/relations of the object set $U$ (of size $m$) which results in m truth values for the m respective objects. For instance, for the predicate IsFather( $X$, $Y$), and a family of three family members: $U={A, B, C}$. Grounding IsFather on $U$ results in a set of truth values of the ground atoms: IsFather($A$, $B$), IsFather($A$, $C$), IsFather($B$, $C$).

Reviewer comment (Writing):
In Section 2.2, the statement "the NLM model first grounds $p$ on $\mathcal{U}$, which derives a tensor representation of $p^\mathcal{U}$..." is unclear to me. I'm not sure what it means to ground $p$ in this context.

Author response:
Object refers to an entity or element within a domain of discourse, which is a fundamental and broadly accepted concept of first-order logic [1]. “a set of objects $U$ of size $m$ ” means there is a set $U$, and $U$ contains m objects: $|U|=m$. Grounding is a frequently used concept in first-order logic which refers to the process of creating a variable-free first-order formula equivalent to a first-order sentence [2][3]. Thus, “ grounds p on $U$” denotes the process of instantiating the variables ($\alpha_1, \dots, \alpha_n$) of atom p($\alpha_1, \dots, \alpha_n$) with the corresponding truth values of the attributes/relations of the object set $U$ (of size $m$) which results in m truth values for the m respective objects. For instance, for the predicate IsFather( $X$, $Y$), and a family of three family members: $U={A, B, C}$. Grounding IsFather on $U$ results in a set of truth values of the ground atoms: IsFather($A$, $B$), IsFather($A$, $C$), IsFather($B$, $C$).

Reviewer comment (Writing):
I also find Section 3 hard to understand, specifically Algorithm 1, line 4/5: what is the difference between $f$ and $f_\theta$?

Author response:
$f$ and $f_\theta$ denotes the same neural model, \theta denotes the the model $f$ is parameterized with the parameters set $\theta$. $b$ denotes that model breadth and $d$ denotes the model depth. We will update these in our paper.

Reviewer comment (Writing):
In Section 3, the statement "$b$ and $d$ denote the number of depth and breadth of the NLM model architecture respectively" is unclear — is this backwards, or does $b$ refer to depth and $d$ refer to breadth?

Author response:
$f$ and $f_\theta$ denotes the same neural model, \theta denotes the the model $f$ is parameterized with the parameters set $\theta$. $b$ denotes that model breadth and $d$ denotes the model depth. We will update these in our paper.

Reviewer comment (Experiments):
Regarding evaluation—Hyperparameters: as mentioned above, one of the effects of the additional term in the loss function is increasing the effective learning rate. The paper does not provide evidence that the proposed approach outperforms a baseline with a well-tuned learning rate. The paper also does not justify (or state) other hyperparameter choices.

Author response:
The details of the training settings and hyperparameter choices are delivered in Appendix B&C with clear reference on page 6&7 of the paper. We state that all the training settings and hyperparameter choices for the NLM model strictly follow previous work [3] instead of being cherry-picked by ourselves. 
We have to state that it is inappropriate for the reviewer to state that: “hyperparameters are (as best I can tell) identical between the training methods, which may be an unfair comparison.” as it is an extremely common practice to set hyperparameters (such as learning rate) the same for the evaluated baseline models. Please feel free to refer to the cited literature for details [4][5]. Nevertheless, we conduct the hyperparameter analysis of the learning rate, the results are shown in Table 2: We conduct hyperparameter analysis for the IsGrandparent task from the family tree benchmark and 2-outdegree from the graph reasoning benchmark. Specifically, we experiment with the following learning rates: 0.005 (default of NLM), 0.01, 0.05, 0.1 for both NLM and NLM w/ FRGR. 
For IsGrandparent, the results are shown in the table below. Concretely, for the original NLM model, it performs the best under the default setting (0.005 learning rate) while for NLM w/ FRGR, it performs even better under the 0.1 learning rate. Specifically, the learning iterations can be decreased by +49.82% (from 55.80 to 28.00). 

| IsGrandparent|lr |grad-rate|n=20 |n=100|Iterations  |
|:--|:--|:--|:-|:-|:--|
|NLM|0.005|100.00% |100.00%|100.00%|96.20±21.87|
|NLM w/ FRGR|0.1|100.00% |100.00%|100.00%|28.00±6.73 |

[**Table 2**: learning rate analysis.]

For 2-Outdegree, the best results for both the NLM and NLM w/ FRGR are achieved under the default learning rate (0.005), which are the same results shown in Table 1 of our paper. Therefore, with the same budget of hyperparameter tuning, NLM w/ FRGR steadily outperforms the original NLM.

Reviewer comment (Reproducibility):
I also cannot find the value of the hyperparameter $\gamma$ coefficient for the regularization, the value of the history size $\tau$, or how these values were tuned.

Author response:
The details of the training settings and hyperparameter choices are delivered in Appendix B&C with clear reference on page 6&7 of the paper. We state that all the training settings and hyperparameter choices for the NLM model strictly follow previous work [3] instead of being cherry-picked by ourselves. 
We have to state that it is inappropriate for the reviewer to state that: “hyperparameters are (as best I can tell) identical between the training methods, which may be an unfair comparison.” as it is an extremely common practice to set hyperparameters (such as learning rate) the same for the evaluated baseline models. Please feel free to refer to the cited literature for details [4][5]. Nevertheless, we conduct the hyperparameter analysis of the learning rate, the results are shown in Table 2: We conduct hyperparameter analysis for the IsGrandparent task from the family tree benchmark and 2-outdegree from the graph reasoning benchmark. Specifically, we experiment with the following learning rates: 0.005 (default of NLM), 0.01, 0.05, 0.1 for both NLM and NLM w/ FRGR. 
For IsGrandparent, the results are shown in the table below. Concretely, for the original NLM model, it performs the best under the default setting (0.005 learning rate) while for NLM w/ FRGR, it performs even better under the 0.1 learning rate. Specifically, the learning iterations can be decreased by +49.82% (from 55.80 to 28.00). 

| IsGrandparent|lr |grad-rate|n=20 |n=100|Iterations  |
|:--|:--|:--|:-|:-|:--|
|NLM|0.005|100.00% |100.00%|100.00%|96.20±21.87|
|NLM w/ FRGR|0.1|100.00% |100.00%|100.00%|28.00±6.73 |

[**Table 2**: learning rate analysis.]

For 2-Outdegree, the best results for both the NLM and NLM w/ FRGR are achieved under the default learning rate (0.005), which are the same results shown in Table 1 of our paper. Therefore, with the same budget of hyperparameter tuning, NLM w/ FRGR steadily outperforms the original NLM.

Reviewer comment (Experiments):
Regarding evaluation—Regularization: as mentioned above, another effect of the additional term in the loss function is weight decay. The paper does not provide evidence that the proposed approach outperforms a baseline with weight decay (or other regularization).

Author response:
The weight decay mechanism is already implemented in the original NLM model, which further demonstrates the effectiveness of FRGR’s effectiveness over this regularization approach. Besides, we further conduct an ablation study by comparing our approach with a random regularization baseline (i.e., randomly penalizing the same percentage of the model’s weights as that of FRGR), denoted as NLM w/ random. The results are shown in Table 3:

|2-Outdegree|grad-rate|n=10|n=20|Iterations|
|:-|:--|:-|:--|:--|
|NLM|90.00%|96.52%±0.10|90.80%±0.28|77.9±150.5|
|NLM w/ random|80.00%|82.10%±0.36|81.18%±0.38|114±203.70|
|NLM w/ FRGR|100.00%|100.00%±0.00|100.00%±0.00|13.40±11.13|

[**Table 3**: regularization comparisons.]

The results illustrate that by adding random regularization, the model's performance decreases compared to that of the original NLM. Specifically, compared to the original NLM model, the grad-ratio decreases by +10.00%, and the performance decreases by -14.42%.
While with FRGR, it is effective in improving grad-ratio, performance, generalization, and training efficiency.
The results demonstrate that the vanilla training loss, random regularization as well as weight decay regularization are not sufficient enough to capture the repetitious error pattern during training thus incapable of improving the NLM model.

Reviewer comment (Reproducibility):
How are hyperparameters for the evaluation tuned? What are $\gamma$ and $\tau$ set to?

Author response:
We haven’t conducted specific hyperparameter tuning for our method. And we directly adopt the hyperparameter settings from the original NLM paper and used them for all evaluated methods. $\gamma$ is set to be 0.99 and $\tau$ is set to be 100 for all tasks.

Reviewer comment (Experiments):
Does NLM w/ FRGR still outperform standard NLM when both approaches are given the same budget for hyperparameter tuning? When NLM is run with weight decay?

Author response:
The details of the training settings and hyperparameter choices are delivered in Appendix B&C with clear reference on page 6&7 of the paper. We state that all the training settings and hyperparameter choices for the NLM model strictly follow previous work [3] instead of being cherry-picked by ourselves. 
We have to state that it is inappropriate for the reviewer to state that: “hyperparameters are (as best I can tell) identical between the training methods, which may be an unfair comparison.” as it is an extremely common practice to set hyperparameters (such as learning rate) the same for the evaluated baseline models. Please feel free to refer to the cited literature for details [4][5]. Nevertheless, we conduct the hyperparameter analysis of the learning rate, the results are shown in Table 2: We conduct hyperparameter analysis for the IsGrandparent task from the family tree benchmark and 2-outdegree from the graph reasoning benchmark. Specifically, we experiment with the following learning rates: 0.005 (default of NLM), 0.01, 0.05, 0.1 for both NLM and NLM w/ FRGR. 
For IsGrandparent, the results are shown in the table below. Concretely, for the original NLM model, it performs the best under the default setting (0.005 learning rate) while for NLM w/ FRGR, it performs even better under the 0.1 learning rate. Specifically, the learning iterations can be decreased by +49.82% (from 55.80 to 28.00). 

| IsGrandparent|lr |grad-rate|n=20 |n=100|Iterations  |
|:--|:--|:--|:-|:-|:--|
|NLM|0.005|100.00% |100.00%|100.00%|96.20±21.87|
|NLM w/ FRGR|0.1|100.00% |100.00%|100.00%|28.00±6.73 |

[**Table 2**: learning rate analysis.]

For 2-Outdegree, the best results for both the NLM and NLM w/ FRGR are achieved under the default learning rate (0.005), which are the same results shown in Table 1 of our paper. Therefore, with the same budget of hyperparameter tuning, NLM w/ FRGR steadily outperforms the original NLM.

Reviewer comment (Writing):
The paper is written for a specialist audience, with little context or background given about NLP, about what the challenges in it are, about how ML methods can help solve it, or even just about what the tasks look like. In general, a formulation of the problem as a task that can be used to train a neural network on a certain experience and with a certain performance metric is omitted completely and taken for granted. As a result, most readers (as well as this referee) will not be able to get much from this paper.

Author response:
Our work is unrelated to NLP and the term “NLP” or “natural language processing” never exists throughout our paper. The focus of our work is on relational reasoning and decision-making tasks which is clearly stated many times in our paper as well as supplementary materials. Furthermore, it is a factual error to state that the description of the tasks and evaluation metrics are omitted. The corresponding details are all clearly delivered in section 4.1 and Appendix C with clear reference in the footnote of page 6. We sincerely recommend the reviewer carefully reread the paper and provide a more objective assessment.

Reviewer comment (Writing):
My main recommendation would be for the authors to provide context on NLP, NLMs, and on how their task can be solved via ML methods to begin with, and on how one would train a neural net to do so. Without such context, it's for me impossible to do justice to the content of this paper, but it is also going to be impossible for a mainstream ML reader (such as most of those attending ICLR) to profit from the paper to begin with.

Author response:
Our work is unrelated to NLP. The concrete details of the NLM model are delivered in section 2.2 and Appendix D with clear reference in the footnote of page 4. The training details can be found in section 4.1 and Appendix B with clear reference on page 7. Again, We sincerely recommend the reviewer to carefully reread the paper and provide a more objective judgment.

Reviewer comment (Novelty):
On the contribution side, the authors should prove that their regularisation scheme does have any general applicability beyond NLP tasks carried out with NLMs.

Author response:
Our work is unrelated to NLP. The benchmarks evaluated in our work range from relational reasoning tasks and decision making tasks, which are all broadly used by many domains such as LLMs [1], program induction [2], SAT-based program synthesis [3], etc.

Reviewer comment (Writing):
The papers still suffers from unclear writing, which results in the context of the author's work not being properly outlined for anyone not already familiar with it. The authors have not seemingly addressed these concerns in their revision, which is also missing markers to highlight changes with respect to the original version.

Author response:
Thanks for the response. We have not submitted the revised version of the paper before as we think the reviewer has not carefully read the paper due to the misleading use of "NLP" as well as the previous unfair claims. We have now submitted the revised version  of the main paper as well as the supplementary materials which explain NLM model design more plainly in the preliminary and we use a concrete case to explain the rationale of FRGR in the methodology section. Besides, we have revised all the mentioned details by all the reviewers as well. The revised parts are highlighted in blue. We hope the reviewer to inspect our revised version and take it into consideration for the rating.

Reviewer comment (Experiments):
There is not much attempt to understand why the technique works. Given how novel the approach is to me, I have a hard time forming a mental model connecting the technique's description with the resulting performance. No ablations or comparisons of design choices, or even motivation given for the design choices, are presented. The paper would be stronger with experiments (which could be toy examples) or examples that show how the approach is working, or with a discussion comparing the usefulness of provenance information for SAT-based synthesis with how the provenance information is helping the neural models. There should be more description in section 3 of why the technique works: I guess the model is a bunch of logical combinations simultaneously during training, and due to the logical nature of the NN activations, it's possible to get information about "incorrect search" happening just how the SAT solver generates erroneous formula. See questions section for more questions attempting to understand and contextualize the technique.

Author response:
We thank the reviewer for the detailed review and comments. Please see the following for our response!
### 1. No ablations or comparisons of design choices, or even motivation given for the design choices.

We conduct an ablation study by comparing our approach with a random regularization baseline (i.e., randomly penalizing same percentage of the model’s weights as that of FRGR), denoted as NLM w/ random. The results are shown in Table 1:

|2-Outdegre | grad-rate |n=10| n=20| Iterations |
| :--| :--| :-- | :--- | :--- |
| NLM| 90.00% | 96.52%±0.10  |  90\.80%±0.28  | 77\.9±150.5  |
| NLM w/ random | 80.00%   | 82\.10%±0.36  |  81\.18%±0.38  | 114±203.70  |
| NLM w/ FRGR | 100.00%| 100.00%±0.00 |100\.00%±0.00 | 13\.40±11.13 |

[**Table 1**: regularization comparisons.]

The results illustrate that by adding random regularization, the model's performance decreases compared to that of the original NLM. Specifically, compared to the original NLM model, the grad-ratio decreases by +10.00%, and the performance decreases by -14.42%.
While with FRGR, it is effective in improving grad-ratio, performance, generalization, and training efficiency.
The results demonstrate that the vanilla training loss, random regularization as well as weight decay regularization are not sufficient enough to capture the repetitious error pattern during training thus incapable of improving the NLM model. 

### 2. The paper would be stronger with experiments (could be toy) or examples that show how the approach is working, or a discussion comparing the usefulness of provenance information for SAT-based synthesis with how the provenance information is helping the neural models

We present an error pattern that FRGR manages to mitigate. Concretely, for the IsMGUncle and IsUncle tasks, their ground-truth programs only involve using binary predicates.  The ground-truth programs are shown as follows:

IsUncle:

$$
 \operatorname{IsUncle}(X, Y) \leftarrow \exists Z,  ((\operatorname{IsMother}(X, Z) \wedge \operatorname{IsBrother}(Z, Y))) \\
 \vee(\operatorname{IsFather}(X, Z) \wedge \operatorname{IsBrother}(Z, Y)) \\
$$
$$
\operatorname{IsBrother}(X, Y) \leftarrow \exists Z,  ((\operatorname{IsSon}(Z, Y) \wedge \operatorname{IsSon}(Z, X)) \\
 \vee(\operatorname{IsSon}(Z, Y) \wedge \operatorname{IsDaughter}(Z, X)))
$$

IsMGUncle:
$$
\operatorname{IsMGUncle}(X, Y) \leftarrow \exists Z,(\operatorname{IsMother}(X, Z) \wedge \operatorname{Is} \operatorname{Uncle}(Z, Y))
$$
Therefore, it is considered an erroneous behavior if the model strongly attributes its induction based on unary predicates. We calculate the ratio of all output binary predicates whose most attributed input predicate is unary during the training process. We evenly sample 5 epochs from the training process till NLM w/ FRGR converges to an optimal solution. The results are shown in Table 2. It is obvious that with the use of FRGR, the ratio of unary predicates is rapidly decreased compared with the vanilla NLM. The results indicate the FRGR is effective in recognizing repetitious error patterns during the training process and actively regularizing the model based on it, which results in higher training efficiency and better generalization.

|| **IsMGUncle** || |  |  |
| :-- | :-- | :-- | :-- | :-- | :- |
| NLM  |0\.417 | 0\.444 | 0\.426 | 0\.486 | 0\.394 |
| NLM w/ FRGR | 0\.417| 0\.426 | 0\.454 | 0\.394 | 0\.356 |
| | **IsUncle** || | | |
|NLM | 0\.433 | 0\.458 | 0\.450 | 0\.433 | 0\.483 |
|NLM w/ FRGR  |0\.433| 0\.442 | 0\.467 | 0\.358 | 0\.375 |

[**Table 2**: interpretation analysis.]

This process is similar to that of the provenance information used in the SAT-based synthesis, in which the solver's error is actively summarized based on previous derivation trees and is used to reduce future search space.

### 3. I think there should be more description in section 3 of why the technique works: I guess the model is a bunch of logical combinations simultaneously during training, and due to the logical nature of the NN activations, it's possible to get information about "incorrect search" happening just how the SAT solver generates erroneous formula.

We agree with the explanation of the reviewer that due to the similarity of the neuro-symbolic models' design and the SAT solving process, we can therefore effectively leverage the error pattern of the model's previous learning process (similar to the provenance information summarized from derivation tree which is used in SAT-based synthesis) to improve models' generalization and training efficiency. We will add the corresponding explanation of our method's effectiveness in the paper.

Reviewer comment (Writing):
The description of the approach in section 3 is poor. It leaves out numerous details, or mentions them unsystematically. For example, things like "what is an error?", equation 1 is challenging to understand, and the error list updating is never described clearly outside of the pseudocode—based on the pseudocode, you store a list of the T most recent errors, and once T are found, you run the pattern mining algorithm, and then you start over with a new list. This could be quickly described with a sentence in the text.

Author response:
### 4. The description of the approach in section 3 is poor. it leaves out numerous details, or mentions them unsystematically. things like "what is an error?", eq 1 is challenging to understand, the error list updating is never described clearly outside of the pseudocode— based on the pseudocode, you store a list of the T most recent errors, and once T is found, you run the pattern mining algorithm. and then you start over with a new list, right? this could be quickly described with a sentence in the text.

An error simply denotes a wrong final conclusion (i.e., classification results for relational reasoning tasks, actions for decision-making tasks). The understanding of the reviewer in terms of the error list update process is mostly correct: we store a list of the T most recent erroneous behavior in the erroneous behavior list, and once T is found, we run the pattern mining algorithm. And we continuously update the list from the beginning of the erroneous behavior list. We will add more explanation of the update process in our paper.

Reviewer comment (Writing):
It is unclear what the Apriori algorithm returns. A single thing? Based off elsewhere, it returns a set of weights, but this should be described when Apriori is brought up.

Author response:
### 5. What does the Apriori algorithm return? a single thing? based off elsewhere, it returns a set of weights, but this should be described when Apriori is brought up.

Apriori returns the position index of the frequently occurred weights of the error behavior representations stored in the erroneous behavior list, as described with a concrete example after Apriori brought up (last paragraph of section 3.1): “After the mining process, FRGR manages to identify the error pattern representation that stores the position index of the frequently intersected weights of all historical erroneous behavior representations, …”

Reviewer comment (Experiments):
There is a lack of justification of the design choices, ablations, other experiments, discussion, or comparison to related techniques, to help understand why this approach works.

Author response:
We thank the reviewer for the detailed review and comments. Please see the following for our response!
### 1. No ablations or comparisons of design choices, or even motivation given for the design choices.

We conduct an ablation study by comparing our approach with a random regularization baseline (i.e., randomly penalizing same percentage of the model’s weights as that of FRGR), denoted as NLM w/ random. The results are shown in Table 1:

|2-Outdegre | grad-rate |n=10| n=20| Iterations |
| :--| :--| :-- | :--- | :--- |
| NLM| 90.00% | 96.52%±0.10  |  90\.80%±0.28  | 77\.9±150.5  |
| NLM w/ random | 80.00%   | 82\.10%±0.36  |  81\.18%±0.38  | 114±203.70  |
| NLM w/ FRGR | 100.00%| 100.00%±0.00 |100\.00%±0.00 | 13\.40±11.13 |

[**Table 1**: regularization comparisons.]

The results illustrate that by adding random regularization, the model's performance decreases compared to that of the original NLM. Specifically, compared to the original NLM model, the grad-ratio decreases by +10.00%, and the performance decreases by -14.42%.
While with FRGR, it is effective in improving grad-ratio, performance, generalization, and training efficiency.
The results demonstrate that the vanilla training loss, random regularization as well as weight decay regularization are not sufficient enough to capture the repetitious error pattern during training thus incapable of improving the NLM model. 

### 2. The paper would be stronger with experiments (could be toy) or examples that show how the approach is working, or a discussion comparing the usefulness of provenance information for SAT-based synthesis with how the provenance information is helping the neural models

We present an error pattern that FRGR manages to mitigate. Concretely, for the IsMGUncle and IsUncle tasks, their ground-truth programs only involve using binary predicates.  The ground-truth programs are shown as follows:

IsUncle:

$$
 \operatorname{IsUncle}(X, Y) \leftarrow \exists Z,  ((\operatorname{IsMother}(X, Z) \wedge \operatorname{IsBrother}(Z, Y))) \\
 \vee(\operatorname{IsFather}(X, Z) \wedge \operatorname{IsBrother}(Z, Y)) \\
$$
$$
\operatorname{IsBrother}(X, Y) \leftarrow \exists Z,  ((\operatorname{IsSon}(Z, Y) \wedge \operatorname{IsSon}(Z, X)) \\
 \vee(\operatorname{IsSon}(Z, Y) \wedge \operatorname{IsDaughter}(Z, X)))
$$

IsMGUncle:
$$
\operatorname{IsMGUncle}(X, Y) \leftarrow \exists Z,(\operatorname{IsMother}(X, Z) \wedge \operatorname{Is} \operatorname{Uncle}(Z, Y))
$$
Therefore, it is considered an erroneous behavior if the model strongly attributes its induction based on unary predicates. We calculate the ratio of all output binary predicates whose most attributed input predicate is unary during the training process. We evenly sample 5 epochs from the training process till NLM w/ FRGR converges to an optimal solution. The results are shown in Table 2. It is obvious that with the use of FRGR, the ratio of unary predicates is rapidly decreased compared with the vanilla NLM. The results indicate the FRGR is effective in recognizing repetitious error patterns during the training process and actively regularizing the model based on it, which results in higher training efficiency and better generalization.

|| **IsMGUncle** || |  |  |
| :-- | :-- | :-- | :-- | :-- | :- |
| NLM  |0\.417 | 0\.444 | 0\.426 | 0\.486 | 0\.394 |
| NLM w/ FRGR | 0\.417| 0\.426 | 0\.454 | 0\.394 | 0\.356 |
| | **IsUncle** || | | |
|NLM | 0\.433 | 0\.458 | 0\.450 | 0\.433 | 0\.483 |
|NLM w/ FRGR  |0\.433| 0\.442 | 0\.467 | 0\.358 | 0\.375 |

[**Table 2**: interpretation analysis.]

This process is similar to that of the provenance information used in the SAT-based synthesis, in which the solver's error is actively summarized based on previous derivation trees and is used to reduce future search space.

### 3. I think there should be more description in section 3 of why the technique works: I guess the model is a bunch of logical combinations simultaneously during training, and due to the logical nature of the NN activations, it's possible to get information about "incorrect search" happening just how the SAT solver generates erroneous formula.

We agree with the explanation of the reviewer that due to the similarity of the neuro-symbolic models' design and the SAT solving process, we can therefore effectively leverage the error pattern of the model's previous learning process (similar to the provenance information summarized from derivation tree which is used in SAT-based synthesis) to improve models' generalization and training efficiency. We will add the corresponding explanation of our method's effectiveness in the paper.