Paper: Faithful Rule Extraction for Differentiable Rule Learning Models (ICLR-2024)

Reviewer comment (Theory):
I have some concerns about the motivation of the proposed method. It is not clear to me why faithfulness is an important metric for rule learning. What kind of benefits can we obtain for the rule learning problem if the rules can respect faithfulness, e.g., say better generalizability? The authors may need more effort to justify this. One motivating example might be helpful for readers to understand this.

Author response:
As requested, we have incorporated an example in the introduction, highlighting the benefits of ensuring faithfulness. We now also refer to this running example in other parts of the paper. Given the tight space restrictions and the need to rigorously present our technical results, this description is necessarily concise. Nevertheless, we anticipate that it will enhance the overall readability of the paper.

Reviewer comment (Novelty):
I also feel that the proposed solution is too specific. It is only applicable to the DRUM model rather than the general rule learning model, which may limit the applicability of the proposed method in general settings.

Author response:
While it is necessary to independently analyse the faithfulness of various rule-learning models, we expect that the strategies employed in our approach will contribute to a broader understanding of the faithfulness of rule-learning models. Indeed, we anticipate that these strategies can serve as a foundation for the future analysis of other models.

Reviewer comment (Presentation):
The overall presentation may need to be improved. I feel that the authors have a lot of technical content to present. However, putting all of them together without appropriate illustrations makes it hard for readers to digest. One suggestion would be using some running examples to intuitively explain the meaning of the notations and equations rather than just leaving them in the paper. Even the prior work such as DRUM paper or Neural ILP paper, there are a lot of visualizations that help users understand the intuitions of the proposed method.

Author response:
As requested, we have incorporated an example in the introduction, highlighting the benefits of ensuring faithfulness. We now also refer to this running example in other parts of the paper. Given the tight space restrictions and the need to rigorously present our technical results, this description is necessarily concise. Nevertheless, we anticipate that it will enhance the overall readability of the paper.

Reviewer comment (Evaluation):
I feel that the experimental evaluations are also concerning. The authors only focus on evaluating the performance of the proposed methods. However, no appropriate evaluations (except some simple case studies) are conducted for the faithfulness part, e.g., how many rules from the vanilla DRUM model violate the faithfulness criterion and what is the bad effect of including those rules?

Author response:
Please note that in Section 6 we perform an extensive analysis of the faithfulness of the rules extracted from the original DRUM model (results are summarised in Table 2). The soundness of the extracted rules is guaranteed by Theorem 1. However, Table 2 shows that 
the rules are not complete, and in fact they only produce a very small fraction (less than 7\%) of the facts predicted by the DRUM model. Therefore, *none* of the rule sets extracted from the original DRUM model is faithful.