Paper: Harnessing the Power of Large Language Models for Natural Language to First-Order Logic Translation (ICLR-2024)

Reviewer comment (Novelty):
The techniques in the paper are not novel and fine-tuning LLAMA for a downstream task can be considered engineering at this point instead of active research.

Author response:
We thank the reviewer for acknowledging the novelty of our proposed RLHF method. We would like to further highlight two important aspects of our work to clarify the overall novelty:

**LogicLLaMA and MALLS are significant contributions to the logic and NLP literature.**
NL-FOL translation has been a long-standing challenge in both NLP and the formal logic literature, and it plays a central role in many logic-based AI systems [1-4].
Solving this task could open up a wide range of applications, yet there lacks such a translation model that scales to real-world data, preventing these systems from applying to real-world NLP problems.

We believe LogicLLaMA is a significant contribution to the literature (As also acknowledged by reviewer wVqn) as it, for the first time, provides a GPT4-level performance translation model that is cheap, open-source, and can be extended for downstream tasks.



**The creation of MALLS dataset is non-trivial; prior work with similar scope is widely appreciated.**
This work shares a similar scope as those self-instruct LLM work such as Alpaca [5], Vicuna [6], and LLaVa [7]: most of them use the existing models and training algorithms, and the main contributions are “engineering” work such as dataset creation, prompting techniques, and so on, yet they are highly influential and significant to the community.

We note the creation of the MALLS dataset is also one of the main contributions of this work and should not be neglected. Given the importance of NL-FOL translation in many logic-based NLP systems, a high-quality real-world NL-FOL pair dataset is highly valuable to the literature and can open up possibilities for many downstream applications. In this regard, MALLS stands out as it is significantly larger (14x times compared to FOLIO) and more diverse in terms of logical expression and context.

Reviewer comment (Novelty):
The novelty of the proposed approach is not totally clear.

Author response:
We thank the reviewer for acknowledging the novelty of our proposed RLHF method. We would like to further highlight two important aspects of our work to clarify the overall novelty:

**LogicLLaMA and MALLS are significant contributions to the logic and NLP literature.**
NL-FOL translation has been a long-standing challenge in both NLP and the formal logic literature, and it plays a central role in many logic-based AI systems [1-4].
Solving this task could open up a wide range of applications, yet there lacks such a translation model that scales to real-world data, preventing these systems from applying to real-world NLP problems.

We believe LogicLLaMA is a significant contribution to the literature (As also acknowledged by reviewer wVqn) as it, for the first time, provides a GPT4-level performance translation model that is cheap, open-source, and can be extended for downstream tasks.



**The creation of MALLS dataset is non-trivial; prior work with similar scope is widely appreciated.**
This work shares a similar scope as those self-instruct LLM work such as Alpaca [5], Vicuna [6], and LLaVa [7]: most of them use the existing models and training algorithms, and the main contributions are “engineering” work such as dataset creation, prompting techniques, and so on, yet they are highly influential and significant to the community.

We note the creation of the MALLS dataset is also one of the main contributions of this work and should not be neglected. Given the importance of NL-FOL translation in many logic-based NLP systems, a high-quality real-world NL-FOL pair dataset is highly valuable to the literature and can open up possibilities for many downstream applications. In this regard, MALLS stands out as it is significantly larger (14x times compared to FOLIO) and more diverse in terms of logical expression and context.

Reviewer comment (Reproducibility):
Even though the paper says that the LogicLLAMA is finetuned using RLHF there is no human feedback involved. It should probably be called something else?

Author response:
Sorry for the confusion. Yes, there is indeed no human feedback involved. Technically, it should be RL with FOL verifier feedback, but since the overall algorithm remains the same we keep the same naming convention. We have included clarification for this in the draft.

Reviewer comment (Novelty):
The paper lacks novelty. Most contributions are either engineering work or published work including: prompting LLM to generate data, various prompting and dataset filtering methods, supervised fine-tuning, and RLHF. Moreover, the conclusions are predictable: fine-tuning LLM on a specific domain can improve its performance, potentially surpassing the best LLM for open domains.

Author response:
***”The paper lacks novelty”***

We clarify this point on three aspects:


**The iterative correction with RLHF is technically novel.**
We emphasize that the iterative correction with RLHF is a novel extension (as also acknowledged by reviewer 8JQp) to the original RLHF setup. Prior work either realized iterative correction via pure prompting [1] with closed-source models such as ChatGPT, or utilized RLHF for other tasks [2] such as instruction-following fine-tuning. Ours is the first method that uses RLHF to fine-tune language models for logical understanding with non-trivial settings, such as FOL rule perturbation and logical equivalence evaluation.


**LogicLLaMA and MALLS are significant contributions to the logic and NLP literature.**
NL-FOL translation has been a long-standing challenge in both NLP and the formal logic literature, and it plays a central role in many logic-based AI systems [3-6].
Solving this task could open up a wide range of applications, yet there lacks such a translation model that scales to real-world data, preventing these systems from applying to real-world NLP problems.

We believe LogicLLaMA is a significant contribution to the literature (As also acknowledged by reviewer wVqn) as it, for the first time, provides a GPT4-level performance translation model that is cheap, open-source, and can be extended for downstream tasks.



**The creation of MALLS dataset is non-trivial; prior work with similar scope is widely appreciated.**
This work shares a similar scope as those self-instruct LLM work such as Alpaca [7], Vicuna [8], and LLaVa [9]: most of them use the existing models and training algorithms, and the main contributions are those so-called “engineering work”: dataset creation, prompting techniques, and so on, yet they are highly influential and significant to the community.

We emphasize the creation of the MALLS dataset is also one of the main contributions of this work and should not be neglected. Given the importance of NL-FOL translation in many logic-based NLP systems, a high-quality real-world NL-FOL pair dataset is highly valuable to the literature (for benchmarking or fine-tuning) and can open up possibilities for many downstream applications. In this regard, MALLS stands out as it is significantly larger (14x times compared to FOLIO) and more diverse in terms of logical expression and context.

Reviewer comment (Evaluation):
The proposed methods do not show significant improvement. As per Table 2, the majority of the gain comes from vanilla supervised fine-tuning on MALLS, and the newly proposed methods only bring marginal gains, especially when compared to the gain from the baseline to vanilla fine-tuning.

Author response:
***” The proposed methods do not show significant improvement”***

As stated above, we disagree that one views RLHF as the sole proposed method and the rest two modes as the baselines—they are different modes of a LLaMA model proposed in this work for NL-FOL translation, and they are trained and evaluated on a dataset is non-trivial to collect and process.

Nevertheless, we are glad to share that we have resolved a few implementational issues with RLHF and rerun the experiments. We find the updated iterative correction performs significantly better compared to the two other modes, leading to ~10% increase in both FOL BLEU and FOL LE on FOLIO and MALLS (whereas scores for  LogicNLI are already saturated). We have updated the RLHF scores in Table 2.

Reviewer comment (Experiments):
Can your method generate more examples, since your generation and filtering are automatically executed?

Author response:
***” Can your method generate more examples?”***

Yes, and we plan to collect more FOL rules in future work.

Reviewer comment (Experiments):
How do you assess the coverage or diversity of your dataset? LLM may only generate high-frequency knowledge.

Author response:
***” How do you assess the coverage or diversity of your dataset? LLM may only generate high-frequency knowledge.”***

We believe the **Pair diversity** paragraph in section 3.3 has sufficiently addressed this concern. During the data collection phase, we implemented several prompting techniques to encourage rule and term diversity. As a result, *“MALLS has a total term vocabulary size of 49394 and the most frequent terms occur less than 2K times (Figure 8 in Appendix B), suggesting a diverse vocabulary distribution.”*. We also show in Table 1, Figure 1, and 7 that MALLS covers a wide range of concepts as well.

Reviewer comment (Experiments):
While the natural language input generated from LLM is often fluent and grammatically correct, real human inputs can be noisy. How do you address this issue?

Author response:
***”While the natural language input generated from LLM is often fluent and grammatically correct, real human inputs can be noisy. How do you address this issue?”***

As stated in Table 1 and Section 2, FOLIO consists of NL-FOL pairs written by humans, and we believe evaluating LogicLLaMA on this benchmark shows how well the model performs with *real human inputs*. As a result, Table 2 suggests that LogicLLaMA performs well on it.

Reviewer comment (Theory):
Page 7, regarding logical equivalence, which exact semantic models of FOL do you refer to? It is better to explicitly name the exact semantic models, such as Herbrand structure/universe if there is any; otherwise there might be ambiguity rendering the FOL being just a subset of FOL --- even if only a subset of FOL is supported it is already meaningful but please be specific.

Author response:
***”Is it a fully expressive First-order logic language or just the logic program subset?”***

Yes, both MALLS dataset and LogicLLaMA are for general FOL rules. When collecting and validating the FOL rules, we do not enforce specific semantic models such as the Herbrand structure. On the logic expression level, the dataset does have some biases, where it contains more entailment-type FOL rules and fewer rules with rare logical operators such as xor. For measuring logical equivalence, we follow a similar protocol as in FOLIO[1] that converts them to propositional rules and compares the truth table.