From: ICLR.cc/2023/Conference/Paper1839/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper1839/Authors
title: Participation in discussion
comment: Dear reviewers,

We hope to have answered your concerns in our individual responses to the provided reviews. We would be happy to answer any further queries you might have before the end of the discussion period. Do let us know if you found our response satisfactory or/and wish to take forward the discussion.

Regards,

The Authors

From: ICLR.cc/2023/Conference/Paper1839/Authors
title: Response to reviewer AC8Z (part2/2)
comment: > "DLMI is claimed to be a differentiable model, but the model is never formally presented: what is the differentiable model that DLMI uses? How are logic programs represented in the differentiable model? What is the differentiable objective and how is the model optimized?"

The key idea to realize DLMI is using differentiable forward-chaining reasoning in first-order logic. Following the idea of [1-3], the differentiable forward-chaining reasoning in first-order logic works as:
* Step 1: A tensor that holds the relationships between clauses and facts is computed.
* Step 2: Each clause is compiled into a differentiable function that performs forward reasoning using the tensor.
* Step 3: A differentiable logic program is composed of the clause functions and their weights. $T$-time step inference is computed by amalgamating the inference results recursively. 

In DLMI, each clause is encoded to a differentiable function. DLMI performs differentiable forward reasoning by calling the function using weights of clauses. The gradients of the weights of clauses can be obtained since the reasoning function consists of only differentiable functions.

> "It is unclear what solve and clause do after reading Def 1"

*solve* and *clause* are preserved predicates to represent clauses in the form of atoms. Logic programs with meta predicates can describe how to perform reasoning, i.e., we can write meta-interpreter for logic programs.

For example, the atom with predicate *clause* represents a clause in the form of an atom. Given the clause *same_shape_pair(X,Y):-shape(X,A),shape(Y,A).* on the object-level reasoning. We use the atom *clause(same_shape_pair(X,Y),f(shape(X,A),shape(Y,A)))* to represent the same clause on the meta-level reasoning. The atoms, which have the form *shape(X,A)* on the object-level reasoning, will be *solve(shape(X,A))* on the meta-level reasoning. 

> "I find it difficult to understand the role of naïve and the modified interpreters; the codes used to define them on page 4 are not properly introduced"

By introducing the naive and modified meta interpreter, we demonstrate that being on the meta-level grants DLMI with initial benefits of realizing many important functionalities (such as proof tree)
on the meta-level without touching original/object-level systems, which are for other systems hard to realize.

Naive interpreter: The first rule defines the conjunction of logic. The second rule defines the goal. 

For example, given that the rule *same_shape_pair(X,Y):-shape(X,A),shape(Y,A).* exists in the object-level reasoning part. If the meta interpreter wants to check whether the goal *same_shape_pair(X,Y)* has a high weight. It will call the second rule and use the clause meta predicate to find a rule whose head is the goal and then call *solve* on the body of the rule *f(shape(X,A),shape(Y,A))*. Since the predicate *solve* has only one entry. One step is still needed, and that is to construct the atom *f(shape(X,A),shape(Y,A))* using the first rule *solve(f(shape(X,A),shape(Y,A))):-solve(shape(X,A)),solve(shape(Y,A)).*

Modified interpreter: The first rule defines the conjunction. The second rule defines the goal. The solve predicate is extended to have two entries, the first entry is the atom and the second entry is the corresponding proof of the atom. In solving a goal A using a rule A :- B, we construct the proof of B and return the structure (A :- ProofB). In solving a conjunction of two goals, A and B, we simply conjoin the proof trees for each goal: (ProofA, ProofB).

> "The objective and the environment of "repairing Kandinsky patterns" is not formally defined. It is unclear what the program in Figure 4 and 5 does"

On page 7 the fist paragraph of section 4.3, we introduced the ”repairing Kandinsky patterns” experiment. It is proposed to test the intelligence of a system. It requires a system to have the ability of classifying the Kandinsky patterns, also be able to edit the objects in an image so that it agrees with a given logical concept. Merely neural network is hard or even not possible to solve even the vanilla Kandinsky patterns classification problem. 

As we have introduced in the paper on page 5 the last paragraph and also in the figure captions. Figure 4 describes the reasoning clauses used by DLMI and NSFR to classify the Kandinsky patterns. To identify whether a plot is Kandinsky two pairs, it should meet two criteria: 1. The Kandinsky plot should have two pairs of objects in the same shape, 2. one pair of objects should be of the same color and the other pair should have a different color. The four clauses listed in figure 4 is NSFR used to determine whether a plot is Kandinsky pattern or not on the object-level. On the meta level, DLMI use the two rules to realize NSFR within DLMI. 

Figure 5 describes part of the output atoms of these two systems.

From: ICLR.cc/2023/Conference/Paper1839/Authors
title: Response to reviewer AC8Z (part1/2)
comment: Thank you very much for your review. We tried to answer all of your questions. Please don't hesitate to ask if anything is unclear or new questions come up.

> "This work claims DLMI is capable of meta-reasoning because it can "trace back" and provide the proof tree of a logic program. This property, however, is also true for most of the existing generic theorem provers or logic program learners. It is unclear why the proposed method excels on this front."

DLMI achieves *meta-level programming* and *differentiable forward reasoning*. In contrast to backward reasoning, forward reasoning does not require queries to compute, thus, the reasoner can be directly combined with Neural Networks without any discrete search.
However, previous differentiable forward reasoners [1-3] and theorem provers [4] lack the aspect of meta-level programming, which is an important advantage of logic programming. If a user needs to check the history of the proof but hold the scalable differentiable reasoning computing, meta-level programming to realize the feature on top of the differentiable forward reasoner is a solution.  DLMI is the first framework that achieves both of the two aspects and realizes scalable differentiable computing on meta-level programs.

> "The concept of meta-reasoning is not formally defined in this work. Since this is one of the main contributions, I recommend the authors revise this part carefully. In particular, in the proposed formalism, the authors should show why the existing logic reasoning methods lack the such capability and why the proposed method could address the problem."

On page 1 the second paragraph, we introduce the concept of meta-reasoning, which refers to the process of “reason about reasoning”. This means that a system using meta-level programs is able to reason about its own capabilities. Meta-level programs are defined by the meta-predicates, which enable the reasoner to represent clauses as atoms. Previous work lacked either the ability of differentiable reasoning or of meta-level programming.

> "The proposed method is poorly supported: many design choices lack justifications (see clarity). The proposed "repairing Kandinsky patterns" experiment is a sensible setup, but it doesn't seem to bring in any new challenges compared to the generic symbolic benchmarks. The authors should demonstrate how this benchmark is different from the existing ones."

We set up a completely novel task "repairing Kandinsky patterns" which requires meta-level reasoning.
The difference from previous task is that it requires a system not only to have the ability of classifying the Kandinsky patterns, but also to be able to edit the objects in an image so that it agrees with a given logical concept. 

To solve the task within the system itself, the new task requires an intelligent system is capable of performing meta-level reasoning, so that it is able to introspect its own capabilities. If a system is not able to perform meta-level reasoning, the operation must be made outside the system. Thus, the new task cannot be solved by existing deep learning methods as well as probabilistic programming languages which are capable of solving the vanilla Kandinsky classification problem within the system itself. 

> "More importantly, this work lacks a systematic empirical analysis that quantitatively characterizes the proposed method and compares it against prior methods. Without it, it is difficult to assess the quality of the work."

We have compared DLMI and the baseline method NSFR [3]. There are two reasons we only compare DLMI with the baseline NSFR [3]: 1. To the best of our knowledge, DLMI is the first to realize a differentiable meta-interpreter, 2. Moreover, [1] reported that the method we are comparing has outperformed neural-based baselines, which are CNN-based and object detection-based baselines. In our work, we demonstrated that NSFR can be realised within DLMI, thus, DLMI also outperforms these neural-based baselines. We have added these baselines comparisons in the appendix B of our draft.

[1] Evans R, Grefenstette E. Learning explanatory rules from noisy data[J]. Journal of Artificial Intelligence Research, 2018, 61: 1-64.

[2] Shindo H, Nishino M, Yamamoto A. Differentiable inductive logic programming for structured examples[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 35(6): 5034-5041.

[3] Shindo H, Dhami D S, Kersting K. Neuro-Symbolic Forward Reasoning[J]. arXiv preprint arXiv:2110.09383, 2021.

[4] Rocktäschel and Rieldel, End-to-End Differentiable Proving, NeurIPS 2017

From: ICLR.cc/2023/Conference/Paper1839/Authors
title: Response to reviewer nWag (part 2/2)
comment: > "It seems that whether some ground atom is true or false depends on the weights as described in Sec 4.2. However, I find the description "This atom has a weight of 0.98, which indicates this atom is true" pretty vague in the sense that it seems to implicitly assume that there is a threshold for the weights and if a certain weight is higher than a threshold then the corresponding ground atom is true. It is unclear what the threshold should be and how the threshold is chosen."

To better explain the results in the paper, we simply said that an atom with a weight of 0.98 can be seen as almost true, but DLMI doesn't perform discretization using a threshold, the outputs of DLMI are atoms with corresponding weights. As a matter of fact, the rules get weights either above 0.95 or below 0.1 thus providing a natural threshold and a nice separation.

From: ICLR.cc/2023/Conference/Paper1839/Authors
title: Response to reviewer nWag (part 1/2)
comment: Thank you for your encouraging and constructive feedback. We tried to answer all of your questions. Please don't hesitate to ask if anything is unclear or new questions come up.

> "I find comparisons between the proposed DLMI and other baselines are somewhat lacking. Only one baseline is presented and the comparison is only done on one experiment, the vanilla Kandinsky task. I wonder if there are more benchmarks/baselines such that a more extensive comparison can be carried out to show if DLMI is able to deliver similarly good results as others."

There are two reasons we only compare DLMI with the baseline NSFR [1]: 1. To the best of our knowledge, DLMI is the first to realize a differentiable meta-interpreter, 2. Moreover, [1] reported that the method we are comparing has outperformed neural-based baselines, which are CNN-based and object detection-based baselines. In our work, we demonstrated that NSFR can be realised within DLMI, thus, DLMI also outperforms these neural-based baselines. We have added these baselines comparisons in the appendix B of our draft.

> "Even though in this paper, it is mentioned a lot that the proposed DLMI is able to learn its own operations, it is still unclear to me what it means exactly. From the DLMI framework as shown in Fig 3, I wonder which parts/operations are from learning."

Being able to learn its own operations means it is able to introspect itself, and give the reason why it should make such a decision. For example, in the "repairing Kandinsky patterns" experiment, to determine in which horizontal direction object 2 should move. DLMI will output the atom *solve(move_h(obj2,right))*, instead of *solve(move_h(obj2,left)* or other atoms. Since after the meta reasoning, the atom *solve(move_h(obj2,right))* has a much larger weight than the other atoms. 

The entire architecture of DLMI is differentiable, thus DLMI can perform gradient-based learning using meta-level programs. For example, in Fig. 3, DLMI can learn weights of clauses (top left in Fig. 3) to perform classification on Kandinsky patterns by differentiable reasoning and also producing proof trees. In our experiments, we didn’t perform learning of logic programs. However, learning by gradient descent can be easily done since the entire pipeline of DLMI is differentiable
with respect to the weights of clauses.

> "The first rule in the meta program for the naive interpreter is confusing to me and the exampling of applying this rule three times recursively on A is even more confusing. I wonder what is the semantic explanation of this rule and why we need this rule."

The first rule defines the conjunction of logic. We use definite clauses to reason with, and a definite clause consists of a head and a body. A body is represented as a conjunction of atoms. The first rule is used to compute the conjunction of body atoms to perform reasoning over definite clauses. 

For example, given that the rule *same_shape_pair(X,Y):-shape(X,A),shape(Y,A)*.  exist in the object-level reasoning part. If the meta interpreter wants to check whether the goal *same_shape_pair(X,Y)*  has a high weight in the object-level reasoning. It will call the second rule, then it will use the clause meta predicate to find a rule whose head is the goal and then call solve on the body of the rule *f(shape(X,A),shape(Y,A))*.  Since the predicate solve has only one entry, one step is still needed, and that is using the first rule 
 *solve(f(shape(X,A),shape(Y,A))):-solve(shape(X,A)),solve(shape(Y,A))*  to construct the atom  *f(shape(X,A),shape(Y,A))*.

> "The comparison of the amount of code in Fig 4 seems unfair since DLMI should also include those meta probabilistic atoms. When the meta probabilistic atoms are also included, it seems to me DLMI does not use less code to achieve the same thing. Or maybe I misunderstand in some way."

We agree with the reviewer. what we want to express is the efficiency on the meta level compared with the object level. By adding few meta codes, the reasoner is able to introspect itself and even with more functionalities by small changes on the meta level, which on the object level is hard or even not possible without touching the object level system.

[1] Shindo H, Dhami D S, Kersting K. Neuro-Symbolic Forward Reasoning[J]. arXiv preprint arXiv:2110.09383, 2021.

From: ICLR.cc/2023/Conference/Paper1839/Authors
title: Response to reviewer suta
comment: Thank you very much for your review. We tried to answer all of your questions. Please don't hesitate to ask if anything is unclear or new questions come up.

> "In other words: o you use the chain rule D(M|P)D(P) ? You must describe how this works"

We don't use the chain rule, instead, we use the forward chaining inference in first-order logic. The process works as: 
* Step 1: A tensor that holds the relationships between clauses and facts is computed.
* Step 2: Each clause is compiled into a differentiable function that performs forward reasoning using the tensor.
* Step 3: A differentiable logic program is composed of the clause functions and their weights. $T$-time step inference is computed by amalgamating the inference results recursively.

> "Why bottom up execution/ Can you handle append & lstms?"

The reason why we use the the bottom up execution is we are using forward reasoning, and forward chaining reasoning determines that the algorithm should perform in a bottom up style. As for the second question, we think in the neural network (perception) part of DLMI, it is possible to append LSTMs.

> "4.1seems logical inf (with abduction) Do you use \ gradient searc?"

In our experiments, we didn't perform learning of logic programs (structure learning). However, learning by gradient descent can be easily done since the entire pipeline of DLMI is differentiable with respect to the weights of clauses.

> "4.2 this is the accumulator, used in polog intro courses. Your approach has advaaaatages like?"

The programs presented in section 4.2 are not trivial because they represent meta-level programs. DLMI can handle meta-level programs in a differentiable way. Compared with Prolog, one of the greatest advantages is that our framework is fully differentiable, so we can combine our method with neural networks like we have shown in the "repairing the Kandinsky" experiment. DLMI uses a neural network as the perception module and combines the individual strengths of lower level perception and higher level reasoning (object-level reasoning). Moreover, DLMI is able to reason on the reasoning on the meta-level reasoning.

From: ICLR.cc/2023/Conference/Paper1839/Authors
title: Response to reviewer BCjx
comment: Thank you very much for your review. We tried to answer all of your questions. Please don't hesitate to ask if anything is unclear or new questions come up.

> "This paper fails to include a few major neural symbolic reasoning works. For example, both DeepprobLog[1] and Scallop[2] adopt a similar pipeline, while providing more systematic weighted model counting-based differentiation and proof tracing."

We respectfully disagree with the reviewer. Note that, in this work, we focused on the problem of meta-level reasoning, which are not addressed in either of these works and thus using any of them as the baselines is not completely fair. 

> "This paper has a relatively weak experimental section, where no baselines and no standard tasks are chosen. It is very hard to evaluate this work."

We have compared DLMI and the baseline method NSFR [1]. There are two reasons we only compare DLMI with the baseline NSFR [1]: 1. To the best of our knowledge, DLMI is the first to realize a differentiable meta-interpreter, 2. Moreover, [1] reported that the method we are comparing has outperformed neural-based baselines, which are CNN-based and object detection-based baselines. In our work, we demonstrated that NSFR can be realised within DLMI, thus, DLMI also outperforms these neural-based baselines. We have added these baselines comparisons in the appendix B of our draft.

Also we set up a completely novel task of **repairing Kandinsky patterns** which requires meta-level reasoning and thus cannot be solved by existing deep learning methods as well as probabilistic programming languages. Thus we argue that calling our experimental section "weak" is unjustifiable.

> "This paper's theory section is quite unclear. What is the formal definition of proof? how is the proof grounded? What's the efficiency overhead of adopting this method compared to the baselines? These theoretical analyses are missing."

The definition of proof is DLMI is the proof tree, which is widely used in logic programming. We kindly refer the reviewer to [2] for more details. To better explain the experiments, we select several output atoms, such as the atom:

*solve(same_shape_pair(obj0,obj2),
(same_shape_pair(obj0,obj2),
((shape(obj0,triangle),0.98), (shape(obj2,triangle),0.98)))).* 

The second entry of the predicate *solve* is the proof of the first entry. Due to limited space in the figures, to better explain the atom in the figure, we write "proof" to stands for the proof of the atom. The proof of the atoms are grounded since they are constructed, we kindly refer to the definition of the modified meta interpreter (page 4, bottom paragraph): In the modified meta interpreter, the solve predicate is extended to have two entries, the first entry is the atom and the second entry is the corresponding proof of the atom. In solving a goal A using a rule *A :- B*, we construct the proof of B and return the structure *(A :- ProofB)*. In solving a conjunction of two goals, A and B, we simply conjoin the proof trees for each goal: *(ProofA, ProofB)*. In these two cases, the corresponding meta-atoms are *solve(A,(A :- ProofB))* and *solve(f(A,B),((ProofA, ProofB)))*

One of the most important advantages of our approach is that DLMI can realize different meta-level functionalities just by having different meta-programs without touching the object-level system. For example, a differentiable reasoner cannot produce proof-trees, i.e., the history of the reasoning. On the contrary, DLMI can have a meta program to perform reasoning saving all of the histories of reasoning. Moreover, different functionalities such as counting the inner inference steps, using different search strategies in the reasoning (depth-first or width-first) can be added by simply modifying the meta-program.

[1] Shindo H, Dhami D S, Kersting K. Neuro-Symbolic Forward Reasoning[J]. arXiv preprint arXiv:2110.09383, 2021.

[2] Ferrand G, Lesaint W, Tessier A. Explanations and proof trees[C]//International Symposium on Explanation-Aware Computing, ExaCt 2005. AAAI Press, 2005: 76-85.

From: ICLR.cc/2023/Conference/Paper1839/Reviewer_nWag
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This work proposes a differentiable logical meta interpreter (DLMI) that leverages differentiable forward-chaining reasoning in first-order logic. It aims to achieve similar high-quality results as object-level reasoners and meanwhile provides proof trees for decisions. Further, it supports augmenting the given inputs with certain logical concepts where it would provide evidence for making the desired decisions. Empirical results are shown on both vanilla Kandinsky tasks and a proposed repairing Kandinsky pattern experiment.
strength_and_weaknesses: I find the motivation of this work to build a reasoner that is able to explain its decision with a proof tree is tempting since it would increase the explainability of the reasoner. Also, the task of "repairing Kandinsky patterns" could be potentially a good test for the capability of a reasoner to integrate certain logical concept. Still, here are a few concerns/confusions when I go through this work:

- I find comparisons between the proposed DLMI and other baselines are somewhat lacking. Only one baseline is presented and the comparison is only done on one experiment, the vanilla Kandinsky task. I wonder if there are more benchmarks/baselines such that a more extensive comparison can be carried out to show if DLMI is able to deliver similarly good results as others.
- Even though in this paper, it is mentioned a lot that the proposed DLMI is able to learn its own operations, it is still unclear to me what it means exactly. From the DLMI framework as shown in Fig 3, I wonder which parts/operations are from learning.
- The first rule in the meta program for the naive interpreter is confusing to me and the exampling of applying this rule three times recursively on A is even more confusing. I wonder what is the semantic explanation of this rule and why we need this rule.
- The comparison of the amount of code in Fig 4 seems unfair since DLMI should also include those meta probabilistic atoms. When the meta probabilistic atoms are also included, it seems to me DLMI does not use less code to achieve the same thing. Or maybe I misunderstand in some way.
- It seems that whether some ground atom is true or false depends on the weights as described in Sec 4.2. However, I find the description "This atom has a weight of 0.98, which indicates this atom is true" pretty vague in the sense that it seems to implicitly assume that there is a threshold for the weights and if a certain weight is higher than a threshold then the corresponding ground atom is true. It is unclear what the threshold should be and how the threshold is chosen.
clarity,_quality,_novelty_and_reproducibility: Some of the technical details are unclear to me and the readability of this work needs to be further improved.
summary_of_the_review: My main concern is the readability of this work.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1839/Reviewer_AC8Z
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: In this work, the authors investigate and characterize the ability of meta-reasoning of machine learning models. The author proposes DLMI, a differentiable model that learns meta programs from a set of logic clauses. A new task, i.e., "repairing Kandinsky patterns", is introduced, which challenges the model to reason its own operations. In the experiments, DLMI can learn accurate programs using less code.
strength_and_weaknesses: Strength 
- The proposed meta-reasoning is interesting

Weaknesses
- Writing needs to be largely improved
- Lacks quantitative comparison with prior methods

clarity,_quality,_novelty_and_reproducibility: ### Novelty

This work might have some novelties but it is difficult to tell from the current draft. The authors claim DLMI to be the first differentiable interpreter capable of meta-reasoning. However, neither the model nor the concept of "meta-reasoning" is properly defined in the paper. The paper also lacks discussions and empirical comparisons with existing methods, making it difficult to assess the contributions.



### Quality

This work claims DLMI is capable of meta-reasoning because it can "trace back" and provide the proof tree of a logic program.  This property, however, is also true for most of the existing generic theorem provers or logic program learners. It is unclear why the proposed method excels on this front.

The concept of meta-reasoning is not formally defined in this work. Since this is one of the main contributions, I recommend the authors revise this part carefully. In particular, in the proposed formalism, the authors should show why the existing logic reasoning methods lack the such capability and why the proposed method could address the problem.

The proposed method is poorly supported: many design choices lack justifications (see clarity). The proposed "repairing Kandinsky patterns" experiment is a sensible setup, but it doesn't seem to bring in any new challenges compared to the generic symbolic benchmarks. The authors should demonstrate how this benchmark is different from the existing ones. 

More importantly, this work lacks a systematic empirical analysis that quantitatively characterizes the proposed method and compares it against prior methods. Without it, it is difficult to assess the quality of the work.




### Clarity

The paper needs an overhaul. The current draft is difficult to follow: many technical details are missing and the presentation of the proposed method is rather informal. I also recommend reorganizing the contents, especially section 4. For example, definitions such as Def 2 and technical details are buried in experimental results.

Many technical details are missing; concepts are not properly defined
- DLMI is claimed to be a differentiable model, but the model is never formally presented: what is the differentiable model that DLMI uses? How are logic programs represented in the differentiable model? What is the differentiable objective and how is the model optimized?
- It is unclear what solve and clause do after reading Def 1
- I find it difficult to understand the role of naïve and the modified interpreters; the codes used to define them on page 4 are not properly introduced.

The objective and the environment of "repairing Kandinsky patterns" is not formally defined. It is unclear what the program in Figure 4 and 5 does. 

summary_of_the_review: This work seeks to formalize the ability of meta-reasoning and propose a model and a new task to demonstrate this aspect. However, the current draft is largely incomplete and misses important technical details and experiments to support the claims. That said, I recommend rejection.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1839/Reviewer_suta
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: See below
strength_and_weaknesses: See below
clarity,_quality,_novelty_and_reproducibility: See below
summary_of_the_review: ok, I was really keen to review this paper. Like the authors, I believe that meta-programmming may be the key to address true reasoning problems. The problem with this paper is that it does not explain what it actually does? 

In other words: o you use the chain rule D(M|P)D(P) ? You must describe how this works

Why bottom up execution/  Can you handle append & lstms?

4.1seems logical inf (with abduction) Do you use \ gradient searc?

4.2 this is the accumulator, used in polog intro courses. Your approach has advaaaatages like?,

4.3 if you just focused here, beginning to end\, I would be happy to accept te paper

In conclusion, something good may be happening, but I can´t say for sure
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1839/Reviewer_BCjx
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors claim that 
1. They propose the first differentiable logic-meta interpreter DLMI.
2. DLMI can provide underlying proofs for a query result.
3. They implement a differentiable planner using DLMI
4. They propose a new task, "repair Kandinsky patterns", and DLMI can solve it. 
strength_and_weaknesses: Weakness:
1. This paper fails to include a few major neural symbolic reasoning works. For example, both DeepprobLog[1] and Scallop[2] adopt a similar pipeline, while providing more systematic weighted model counting-based differentiation and proof tracing. 
2. This paper has a relatively weak experimental section, where no baselines and no standard tasks are chosen. It is very hard to evaluate this work. 
3. This paper's theory section is quite unclear. What is the formal definition of proof? how is the proof grounded? What's the efficiency overhead of adopting this method compared to the baselines? These theoretical analyses are missing. 

[1] Manhaeve, Robin, et al. "Deepproblog: Neural probabilistic logic programming." Advances in Neural Information Processing Systems 31 (2018).
[2] Huang, Jiani, et al. "Scallop: From probabilistic deductive databases to scalable differentiable reasoning." Advances in Neural Information Processing Systems 34 (2021): 25134-25145.
clarity,_quality,_novelty_and_reproducibility: Clarity: 1/5 
The lack of formalism in explaining the major novel part, proof, causes significant clarity issues. 

Quality: 1.5/5
The experiment does support the authors' claim on DLMI can solve the "repair Kandinsky patterns" task. However, even with this task, I cannot agree with the authors' claim that this differentiation feature of the reasoning engine alone is "meta-learning". 

Originality: 1.5/5
I think both the neural symbolic architecture and the differentiation method are well discussed in related works. 
summary_of_the_review: Although the authors claims they propose the first differentiable logic-meta interpreter DLMI, I believe this statement is not well justified. To improve this work, more baselines and experiments are required. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough