Paper: Certified Deductive Reasoning with Language Models (ICLR-2024)

Reviewer comment (Reproducibility):
The proposed method necessitates a reliance on a complex formalization process during training and inference.

Author response:
While this is true for problems that can be adequately formalized, our results (Section 4.4) show that bootstrapping GPT-3.5 Turbo on such problems, where formalization helps during training, can also improve reasoning when not formalizing at inference time.

Reviewer comment (Experiments):
The scenarios considered in the paper seem a bit limited. Despite experimenting on diverse datasets, the nature of problems within them appear quite similar. In more generalized contexts, it might be challenging to formalize and identify corresponding actions, such as `objects`, `relations`, etc.

Author response:
We agree that this can be hard in more naturalistic problems. Our experiments in Section 4.4 show that bootstrapping on simpler problems can still help even when not formalizing at inference time.

Reviewer comment (Novelty):
The paper's primary contribution, namely, how to harness logic to ensure output consistency, seems to overlap with prior work on the Peano theorem and the constrained Semantic Decoding algorithm, which weakens the novelty of the current research.

Author response:
Our framework of guides is a novel interface between these components. Peano itself is simply a formal theorem proving environment, with no natural language component. Moreover, while Constrained Semantic Decoding was originally used in cases where the entire output is constrained, our framework of guides generalizes this to allow the model to selectively turn the guide tool (and therefore the constraints) on and off during generation. Thus, we build on these prior works but ultimately implement new capabilities.

Reviewer comment (Evaluation):
How likely that encountering a formalization failure may happen, and are there strategies in place to minimize formalization errors?

Author response:
It varies from model to model - the OpenAI models were fairly good at formalization in our experiments, whereas LLaMA lagged behind. For the formalization blocks, LogicGuide still imposes constraints on well-formedness during generation to minimize errors. Namely, the constraints guarantee that the model will output expressions that are syntactically valid and coherent with its own proposition and object annotations, thus avoiding errors due to such inconsistencies. These constraints were triggered in around 5-10% of the cases depending on the dataset for text-davinci-003 and gpt-3.5-turbo, where otherwise there would have been formalization errors. We are running a more in-depth analysis on this and will add it to the paper soon.  

As we mention in the paper (Section 3.2), when activated by the model during formalization, LogicGuide enforces syntactic constraints to ensure that the model outputs well formed expressions. Thus, we are always able to extract some formalization from the model's output *by construction*. We ran an additional experiment where we use the LogicGuide prompt but don't apply such constraints, instead letting the model output the formalization on its own. Note that this approach is similar to LogicLM, in that the model is prompted to translate the problem into a formal form without any interference in its formalization.

Without our constraints during generation, out of 400 problems (in a mixture of ProofWriter and PrOntoQA), GPT-3.5 Turbo produced malformed formalizations in 46 cases (*11.5%* of the time). In these cases, the formalization is unusable, due to syntax errors or inconsistent arities in predicates. One example from ProofWriter is shown below:

```
(...) If something is [[prop:rough]] and [[prop:furry]] then it is [[prop:young]]. [[axiom:((rough 'x) -> (furry 'x)) -> (young 'x))]]
```

There are two subtle syntax errors here: (a) the first two premises are enclosed in parentheses where in Peano they shouldn't, and (b) there's an extra closing parentheses at the end. Since we leverage a simple grammar of first-order expressions to apply constraints with LogicGuide, we completely avoid these errors by construction, always sampling a formula that we're able to parse. Thus, with these constraints, we obtain 100% of valid formalizations. Semantic errors can still persist, since we cannot formally guarantee that the model has correctly interpreted the input sentences, even if it produces a valid formal interpretation. However, this step significantly improves our approach. We'd like to also notice that applying constraints to aid in formalization is unique to our method, since prior work (like LogicLM or Faithful-CoT) work in a two-phase manner, where first the LLM fully formalizes the input, and then the solver is called on the result. In this case, however, if the LLM outputs an invalid formalization, the solver cannot do anything to its input.

Reviewer comment (Evaluation):
To what extent does using constrained generation reduce the reasoning space, so as to mitigate the issue of "logical inferences made next can have a potentially large set of answers"? Is it possible that still there may be a considerably large set of answers, if so, how does your method decide on the most appropriate content to generate next?

Author response:
The number of possible valid inferences for the datasets we used were in the order of dozens in the worst case. In these cases, the language model is constrained to generate from that set when it opens an [[infer]] block, and uses its own log-probabilities to decide what is most appropriate in the current context / solution. Thus, we allow the model to choose its logical inferences while constraining it to logically sound ones. Allowing the language model to decide amongst logically valid continuations is unique to our guide approach.

Reviewer comment (Experiments):
Discussions on generalization involve models bootstrapped from other formalizable tasks. In scenarios challenging to formalize, what amount of preparatory work, such as the number of samples of formalizable tasks, is essential to ensure the model with strengthened generalization inference capabilities? If in the absence of abundant corresponding simpler tasks, how to generalize "guide" in broader scenarios?

Author response:
This is a good question. In general, we cannot strictly guarantee that positive transfer will happen between the training and inference tasks. However, using low-rank fine-tuning methods [1], we observe that even a few hundred high-quality training examples can improve reasoning performance. In turn, the guided models are able to produce higher quality training examples for this purpose, as our results show.

[1] HU, Edward J. et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022