Revised: L

Revised: ANGUAGE

Revised: M ODELS

Revised: A

Revised: RE

Original: L ANGUAGE M ODELS C AN ( KIND OF ) R EASON : A S YSTEMATIC F ORMAL A NALYSIS OF C HAIN - OF -T HOUGHT
Revised: G REEDY R EASONERS : A S YSTEMATIC F ORMAL A NALYSIS OF C HAIN - OFT HOUGHT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Large language models (LLMs) have shown remarkable reasoning capabilities given chain-of-thought prompts (examples with intermediate reasoning steps).
Revised: Large language models (LLMs) have shown remarkable reasoning capabilities given chain-of-thought prompts (examples with intermediate reasoning steps).

Original: Existing benchmarks measure reasoning ability indirectly, by evaluating accuracy on downstream tasks such as mathematical reasoning.
Revised: Existing benchmarks measure reasoning ability indirectly, by evaluating accuracy on downstream tasks such as mathematical reasoning.

Original: However, it is unclear how these models obtain the answers and whether they rely on simple heuristics rather than the generated chain-of-thought.
Revised: However, it is unclear how these models obtain the answers and whether they rely on simple heuristics rather than the generated chain-of-thought.

Original: To enable systematic exploration of the reasoning ability of LLMs, we present a new synthetic question-answering dataset called P R O NTO QA, where each example is generated from a synthetic world model represented in first-order logic.
Revised: To enable systematic exploration of the reasoning ability of LLMs, we present a new synthetic question-answering dataset called P R O NTO QA, where each example is generated from a synthetic world model represented in first-order logic.

Original: This allows us to parse the generated chain-ofthought into symbolic proofs for formal analysis.
Revised: This allows us to parse the generated chain-ofthought into symbolic proofs for formal analysis.

Original: Our analysis on I NSTRUCT GPT and GPT-3 shows that LLMs are quite capable of making correct individual deduction steps, and so are generally capable of reasoning, even in fictional contexts.
Revised: Our analysis on I NSTRUCT GPT and GPT-3 shows that LLMs are quite capable of making correct individual deduction steps, and so are generally capable of reasoning, even in fictional contexts.

Original: However, they have difficulty with proof planning : When multiple valid deduction steps are available, they are not able to systematically explore the different options.
Revised: However, they have difficulty with proof planning : When multiple valid deduction steps are available, they are not able to systematically explore the different options.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The ability to reason—drawing new conclusions from provided facts—is a hallmark of human intelligence.
Revised: The ability to reason—drawing new conclusions from provided facts—is a hallmark of human intelligence.

Original: Recently, chain-of-thought (CoT) prompting has enabled large language models (LLMs) to perform logical reasoning tasks with impressive accuracy (Wei et al., 2022b; Chowdhery et al., 2022;
Revised: Recently, chain-of-thought (CoT) prompting has enabled large language models (LLMs) to perform logical reasoning tasks with impressive accuracy (Wei et al., 2022; Chowdhery et al., 2022;

Original: Lewkowycz et al., 2022).
Revised: Lewkowycz et al., 2022).

Original: In CoT prompting, each example consists of a question (e.g., “ 63 − 1 ?”), a short description of the reasoning required to answer the question called the “chainof-thought” (e.g., “ 63 is 2 .
Revised: In CoT prompting, each example consists of a question (e.g., “ 63 − 1 ?”), a short description of the reasoning required to answer the question called the “chain-of-thought” (e.g., “ 63 is 2 .

Original: 2 − 1 is 1 .”), and a label (e.g., “ 1 ”).
Revised: 2 − 1 is 1 .”), and a label (e.g., “ 1 ”).

Original: When prompted with a few CoT examples, the elicited reasoning allows LLMs to predict the label with much higher accuracy than standard question-answer prompting.
Revised: When prompted with a few CoT examples, the elicited reasoning allows LLMs to predict the label with much higher accuracy than standard question-answer prompting.

Original: However, it is unclear to what extent these models can reason due to several confounding factors.
Revised: However, it is unclear to what extent these models can reason due to several confounding factors.

Original: First, existing studies primarily rely on question-answering  tasks from real-world settings such as math word problems (Cobbe et al., 2021; Han et al., 2022; Weston et al., 2016).
Revised: First, existing studies primarily rely on question-answering (QA) tasks from real-world settings such as math word problems (Cobbe et al., 2021; Han et al., 2022; Weston et al., 2016).

Original: It is likely that LLMs have already acquired the knowledge through pretraining and simply retrieve the answer rather than reason over it.
Revised: It is likely that LLMs have already acquired the knowledge through pretraining and simply retrieve the answer rather than reason over it.

Original: Second, the reasoning task may contain spurious correlations that allow the model to obtain the correct answer through shortcuts (Zhang et al., 2022).
Revised: Second, the reasoning task may contain spurious correlations that allow the model to obtain the correct answer through shortcuts (Zhang et al., 2022b).

Original: In this work, we systematically investigate the reasoning capability of LLMs by directly evaluating their predicted chains-of-thought (the interpretable proof steps), rather than the predicted label.
Revised: In this work, we systematically investigate the reasoning capability of LLMs by directly evaluating their predicted chains-of-thought (the interpretable proof steps), rather than the predicted label.

Original: To enable easy analysis of the CoT, we construct a new synthetic question-answering dataset called P R O NTO QA, for Pr oof and Onto logy-Generated Q uestion-
Revised: To enable easy analysis of the CoT, we construct a new synthetic QA dataset called P R O NTO QA, for Pr oof and Onto logy-Generated Q uestion-

Original: A nswering, where each example  is generated from an ontology and has a unique proof (see figure 1 for an example).
Revised: A nswering. Inspired by the P ROOF W RITER dataset (Tafjord et al., 2021), each example in P R O NTO QA is generated from an ontology and has a unique proof (see figure 1 for an example).

Original: We convert the proofs into syntactically simple sentences using asimple grammar so that the inverse process is relatively easy:
Revised: We convert the proofs into syntactically simple sentences using a grammar such that the inverse process is relatively easy:

Original: From the predicted CoT, weare able to semantically parse each sentence into a formal language and reconstruct the underlying proof steps.
Revised: From the predicted CoT, we semantically parse each sentence into a formal language and reconstruct the underlying proof steps.

Original: We then directly analyze the model’s reasoning by inspecting each step in the reconstructed proof and comparing them against the groundtruth proof.
Revised: We then directly analyze the model’s reasoning by inspecting each step in the reconstructed proof and comparing them against the gold proof.

Original:  We systematically evaluate I NSTRUCT GPT  (Ouyang et al., 2022) and the original GPT-3{+GPT-
+} (Brown et al., 2020) on P R O NTO QA by controlling a number of variables that characterize the complexity of the reasoning task, such as the ontology type and the number of proof steps required.
Revised: 1 We emphasize here that while the dataset is an important contribution of this paper, the main contribution is the analysis that is facilitated by the dataset. We systematically evaluate I NSTRUCT GPT 2 (Ouyang et al., 2022) and the original {+GPT-
+} (Brown et al., 2020) on P R O NTO QA by controlling a number of variables that characterize the complexity of [Table]

Original: Our analysis shows that these models are quite good at producing valid individual proof steps, even on fictional and counterfactual ontologies.
Revised: [Table] [Table]

Original: However, LLMs have difficulty with proof planning : when the models encounter a point in the proof where multiple valid proof steps are available, they sometimes select
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: the  wrong step, and this often leads to an incomplete proof and subsequently an incorrect answer.
Revised: the reasoning task, such as the ontology type and the number of proof steps required. Our analysis shows that these models are quite good at producing valid individual proof steps, even on fictional and counterfactual ontologies. However, LLMs have difficulty with proof planning : when the models encounter a point in the proof where multiple valid proof steps are available, they sometimes select the wrong step, and this often leads to an incomplete proof and subsequently an incorrect answer.

Original: Interestingly, the models are much less likely to be misled with a true ontology, suggesting that the world knowledge acquired during pretraining plays an important role in LLM reasoning.
Revised: Interestingly, the models are much less likely to be misled with a true ontology, suggesting that the world knowledge acquired during pretraining plays an important role in LLM reasoning.

Revised: We also find that our results generalize when using self-consistency prompting (Wang et al., 2022), a more sophisticated prompting strategy.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: There are a number of research directions related to  our work.
Revised: Our proposed dataset is most closely related to P ROOF W RITER (Tafjord et al., 2021) and FOLIO (Han et al., 2022) which are QA datasets designed to test reasoning ability. P ROOF W RITER provides multi-hop proofs for each example. However, there are a number of key properties that led us to develop our own dataset (see table 1 for a summary). FOLIO does not provide easily-parseable proofs/CoTs in their examples, and evaluation is done by inspecting the predicted labels, which may not necessarily be a good measure of reasoning ability.

Original: There are efforts to tweak or extend
Revised: In our analysis, we focus on more specific variables that may affect the reasoning of the model, such as: (1) Is the model’s reasoning dependent on whether the example is consistent with pretraining (“true”), inconsistent (“false”), or neither (“fictional”)? (2) Is the model’s reasoning sensitive to whether the predicates in the examples are unary or binary? (3) Is the model’s reasoning dependent on the rules of deduction in the examples? These variables are not controllable in existing datasets.

Revised: Further, in some datasets, the code to generate examples is not available.

Original:  CoT prompting to elicit more sophisticated reasoning behavior (Creswell et al., 2022;  Anil et al., 2022; Dohan et al., 2022), and they have shown that these prompting extensions to CoT can improve the elicited reasoning behavior of LLMs, even with smaller models.
Revised: There are efforts to tweak or extend CoT prompting to elicit more sophisticated reasoning behavior (Creswell et al., 2022; Wang et al., 2022; Creswell & Shanahan, 2022; Anil et al., 2022; Dohan et al., 2022), and they have shown that these prompting extensions to CoT can improve the elicited reasoning behavior of LLMs, even with smaller models.

Original: Rather than presenting a new prompting approach, the goal of this work is to measure the reasoning ability elicited by CoT.
Revised: Rather than presenting a new prompting that in our experiments, we also evaluate text-ada-001 , text-babbage-001 , text-curie-001 , davinci , and text-davinci-001 . Dataset Provides easily semanticallyparseable proofs Controls for true vs false vs fictional contexts

Revised: Controls for unary vs binary predicates

Revised: Controls for specific rules of deduction

Original: There are other datasets that have been designed or used to measure the reasoning capabilities of transformer-based models and LLMs (Han et al., 2022;
Revised: Tests reasoning beyond the domain of math word problems Generation code available GSM8K Cobbe et al. (2021) ✗ ✗ ✗ ✗ ✗

Revised: [Table]

Revised: ProofWriter Tafjord et al.

Revised: (2021) ✓ ✗ ✗ ∼ ✓ ✗

Original: Tafjord et al., 2021; Weston et al., 2016).
Revised: FOLIO Han et al. (2022) ✗ ✗ ✗ ✗ ✓ human- [Table] SimpleLogic Zhang et al.

Revised: (2022b) ✗ ✗ ✓ ✓ ✓ ✓

Revised: P

Revised: [Table]

Revised: NTO QA (proposed dataset) ✓ ✓ ✓ ✓ ✓ ✓ approach, the goal of this work is to measure the reasoning ability elicited by CoT. There are other datasets that have been designed or used to measure the reasoning capabilities of transformer-based models and LLMs (Han et al., 2022; Weston et al., 2016).

Original: They show that LLMs are able to answer questions that require reasoning in the few-shot setting with reasonable accuracy.
Revised: They show that LLMs are able to answer questions that require reasoning in the few-shot setting with reasonable accuracy.

Revised: Similar to our approach, Betz (2020) converts logical forms into fairly simple natural language using templates.

Original: These datasets arelargely consistent with the real-world, and so they may confound measuring reasoning ability with retrieval ability.
Revised: However, the examples in these datasets are consistent with the real-world, and so they may confound measuring reasoning ability with retrieval ability.

Original: In addition, they evaluate the predicted labels, which may not necessarily be a good measure of  reasoning ability.
Revised: Valmeekam et al. (2022) found that LLMs had difficulty with a fairly simple planning task, but it is not clear whether this was due to an inability to reason or other abilities instrumental in planning, such as world modeling, keeping track of state changes, and reasoning about events that occur sequentially in time. Our work aims to address this gap.

Original: As in our approach, Dasgupta et al.
Revised: As in our approach, Dasgupta et al.

Original: (2022) specifically looked at whether LLMs can reason in fictional or counterfactual settings and found that reasoning ability is indeed negatively affected in these settings.
Revised: (2022) specifically looked at whether LLMs can reason in fictional or counterfactual settings and found that reasoning ability is indeed negatively affected in these settings.

Original: However they did not analyze individual steps of reasoning to better understand the cause of the errors.
Revised: However they did not analyze individual steps of reasoning to better understand the cause of the errors.

Original: Since we are able to formally evaluate the LLM’s predicted CoT step-by-step, we are able to perform a more fine-grained analysis of their reasoning ability.
Revised: Since we are able to formally evaluate the LLM’s predicted CoT step-by-step, we are able to perform a more fine-grained analysis of their reasoning ability.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2022) showed that BERT is not able to learn to reason robustly, but they did not use CoT prompting and it is not obvious if their results generalize to LLMs, which we evaluate.
Revised: (2022b) showed that BERT is not able to learn to reason robustly, but they did not use CoT prompting and it is not obvious if their results generalize to LLMs, which we evaluate.

Original: There are two broad research approaches for reasoning in NLP: (1) reasoning over a formal symbolic language, possibly with neuro-symbolic methods and/or semantic parsing (Saparov & Mitchell, 
Revised: There are two broad research approaches for reasoning in NLP: (1) reasoning over a formal symbolic language, possibly with neuro-symbolic methods and/or semantic parsing (Saparov & Mitchell, 2022;

Original: 2022; Kapanipathi et al., 2021; Dong et al., 2019; Rocktäschel & Riedel, 2017), or (2) reasoning directly over natural language (Chen et al., 2021; Bostrom et al., 2022; 2021; Welleck et al., 2021; Bhagavatula et al., 2020).
Revised: Zhang et al., 2022a; Kapanipathi et al., 2021; Dong et al., 2019; Rocktäschel & Riedel, 2017), or (2) reasoning directly over natural language (Chen et al., 2021; Bostrom et al., 2022; 2021; Welleck et al., 2021; Bhagavatula et al., 2020; Angeli & Manning, 2014; MacCartney & Manning, 2009).

Original: While P R O NTO QA is generated from symbolic ontologies, the examples themselves are in natural language, and so provides value to both research approaches.
Revised: While P R O NTO QA is generated from symbolic ontologies, the examples themselves are in natural language, and so provides value to both research directions.

Revised: Recent work has examined in-context learning and found that performance on certain tasks is sensitive to the prompt (Razeghi et al., 2022; Lu et al., 2022).

Revised: However, they focused on sentiment classification and simple arithmetic tasks, and it is not clear if their results generalize to reasoning.

Revised: The LLM could feasibly use retrieval, rather than reasoning, to perform those tasks.

Revised: Our experiments on the fictional ontology show that the model is able to reason even when there is nothing to retrieve from.

Original:  P R O NTO QA: A SYNTHETIC DATASET FOR LOGICAL REASONING
Revised:  P R O NTO QA: A SYNTHETIC DATASET FOR LOGICAL REASONING

Original: We create a new dataset, called P R O NTO QA for Pr oof and Onto logy-Generated Q uestion-
Revised: We create a new dataset, called P R O NTO QA for Pr oof and Onto logy-Generated Q uestion-

Original: A nswering, where each question is generated from a symbolic ontology and proof to facilitate formal analysis of the predicted CoT. To focus the scope of our exploration, and to limit the complexity of the generated questions to those within reach of current LLMs, we only consider questions that are answerable using repeated applications of the modus ponens deduction rule.
Revised: A nswering, where each question is generated from a symbolic ontology and proof to facilitate formal analysis of the predicted CoT. To focus the scope of our exploration, and to limit the complexity of the generated questions to those within reach of current LLMs, we only consider questions that are answerable using repeated applications of the modus ponens deduction rule.

Original: More formally, modus ponens is a simple deduction rule where given the premises
Revised: More formally, modus ponens is a simple deduction rule where given the premises

Original: ∀ x ( f ( x ) → g ( x )) and f ( a ) , we conclude g ( a ) (e.g., given “All cats are carnivores” and “Fae is a cat,” we conclude “Fae is a carnivore;” see figurein the Appendix).
Revised: ∀ x ( f ( x ) → g ( x )) and f ( a ) , we conclude g ( a ) (e.g., given “All cats are carnivores” and “Fae is a cat,” we conclude “Fae is a carnivore;” see figurein the appendix).

Original: This rule caneasily be  chained together to construct proofs witha controllable number of steps.
Revised: This rule can be easily chained together to construct proofs with controllable size.

Original: We generate CoT examples consisting of: the context, query, CoT, and label, where the context is a short paragraph containing information relevant to answer the query (see figure 1 for an example).
Revised: We generate CoT examples consisting of: the context, query, CoT, and label, where the context is a short paragraph containing information relevant to answer the query (see figure 1 for an example).

Original: Each example is translated from a proof and ontology such that the inverse process is simple: the sentences in an example can be easily and uniquely parsed into symbolic logical forms amenable to formal analysis.
Revised: Each example is translated from a proof and ontology such that the inverse process is simple: the sentences in an example can be easily and uniquely parsed into symbolic logical forms amenable to formal analysis.

Original: More specifically, as shown in figure 2, we: (1) first generate an ontology from a set of concepts, (2) generate a proof by traversing the ontology, (3) translate the ontology into the natural language context, and (4) translate the proof into the query, CoT, and label by mapping logical forms to natural language sentences.
Revised: More specifically, as shown in figure 2, we: (1) first generate an ontology from a set of concepts, (2) generate a proof by traversing the ontology, (3) translate the ontology into the natural language context, and (4) translate the proof into the query, CoT, and label by mapping logical forms to natural language sentences.

Original: We describe each step in further detail below.
Revised: We describe each step in further detail below.

Original: Ontology generation.
Revised: Ontology generation.

Original: The first step is to generate a small hierarchical ontology .
Revised: The first step is to generate a small hierarchical ontology .

Original: The ontology is a collection of concepts (e.g., mammal , cat , carnivore , etc) and subtype relations between them (e.g., ∀ x ( cat ( x ) → carnivore ( x )) ).
Revised: The ontology is a set of concepts (e.g., mammal , cat , carnivore , etc) and subtype relations between them (e.g., ∀ x ( cat ( x ) → carnivore ( x )) ).

Original: The ontology also describes properties of concepts (e.g., ∀ x ( mammal ( x ) → ¬ cold_blooded ( x )) ).
Revised: The ontology also describes properties of concepts (e.g., ∀ x ( mammal ( x ) → ¬ cold_blooded ( x )) ).

Original: To generate questions that are not overly complex for LLMs, we restrict the ontologies to be linear (i.e. in the tree, every node has exactly 0 or 1 child nodes). 
Revised: To generate questions that are not overly complex, we restrict the ontologies to be linear (i.e., in the tree, every node has exactly 0 or 1 child nodes). Since ontologies are randomly generated, they vary in size from as few as 3 concepts to as many as 10 .

Original: Proof generation.
Revised: Proof generation.

Original: We generate proofs from the ontology by choosing a starting node uniformly at random, and generating the initial axiom indicating that an entity has a specific type (e.g., cat ( fae ) ).
Revised: We generate proofs from the ontology by choosing a starting node uniformly at random, and generating the initial axiom indicating that an entity has a specific type (e.g., cat ( fae ) ).

Original: Then, we walk up the tree, with each step corresponding to an application of a deduction rule (i.e., a proof step ).
Revised: Then, we walk up the tree, with each step corresponding to an application of a deduction rule (i.e., a proof step ).

Original: Each proof step consists of zero or more premises and one conclusion .
Revised: Each proof step consists of zero or more premises and one conclusion .

Original: We stop when we reach a node (e.g., carnivore ( fae ) ), or a node property (e.g., ¬ herbivorous ( fae ) ), such that the number of generated proof steps matches the target number of steps.
Revised: We stop when we reach a node (e.g., carnivore ( fae ) ), or a node property (e.g., ¬ herbivorous ( fae ) ), such that the number of generated proof steps matches the target number of steps.

Original: Translation to natural language example.
Revised: Translation to natural language example.

Original: Given a generated ontology and proof, we now translate it into a natural language CoT example consisting of the question (context and query), CoT, and label.
Revised: Given a generated ontology and proof, we now translate it into a natural language CoT example consisting of the question (context and query), CoT, and label.

Original: We describe how each component is generated below:
Revised: We describe how each component is generated below:

Original: We use a simple grammar to convert the formal statements of the ontology into the natural language utterances that make up the context.
Revised: We use a simple grammar to convert the formal statements of the ontology into the natural language utterances that make up the context.

Original: Every edge in the ontology is converted into sentences such as “All cats are carnivores” or “Every cat is a carnivore.”
Revised: Every edge in the ontology is converted into sentences such as “All cats are carnivores” or “Every cat is a carnivore.”

Original: Properties of nodes are also converted into sentences of the form “All mammals are not cold-blooded,” etc.
Revised: Properties of nodes are also converted into sentences of the form “All mammals are not cold-blooded,” etc.

Original: The query is generated by using the same grammar to convert the initial axiom in the proof into a natural language sentence (e.g., “Fae is a cat”).
Revised: The query is generated by using the same grammar to convert the initial axiom in the proof into a natural language sentence (e.g., “Fae is a cat”).

Original: We then determine with probability 0.5 whether to ask if the conclusion of the proof is true or if its negation is false, and convert it into a natural language “true or false” query (e.g., “True or false: Fae is not herbivorous.”) and label (e.g., “True”).
Revised: We then determine with probability 0.5 whether to ask if the conclusion of the proof is true or if its negation is false, and convert it into a natural language “true or false” query (e.g., “True or false: Fae is not herbivorous.”) and label (e.g., “True”).

Original: We convert the ordered sequence of proof steps into the CoT by translating the conclusion of each proof step into a CoT sentence.
Revised: We convert the ordered sequence of proof steps into the CoT by translating the conclusion of each proof step into a CoT sentence.

Original: Avoiding shortcuts.
Revised: Avoiding shortcuts.

Original: We have to be careful to removeany shortcutsin the question that would allow the model to “guess” the answer without reasoning.In fact, we find that without any distractors, I NSTRUCT GPT is able to predict the “true”/“false” label almost perfectly. I NSTRUCT GPT can utilize
Revised: In section B in the appendix, we describe how we add distractor sentences in order to remove shortcuts that would allow the model to “guess” the answer without reasoning.

Original: [Table]

Original: the heuristic that whether the queried property is mentioned in the context implies whether or not it is true.

Original: For instance, if the example is asking “Sally is a cat.

Original: True or false: Sally is a vertebrate,” the model can simply look for a string “Every is (not) a vertebrate,” regardless of the content in the blank.

Original: Due to the generative process of these examples, this kind of sentence is guaranteed to appear exactly once in the context.

Original: Thus, to ensure that such a heuristic is not informative, we add a distractor sentence by generating a novel concept that is disconnected from the ontology tree, and we assign to this new concept the negation property that is queried by the question.

Original: So in the above example, if the ontology has the rule “Every mammal is a vertebrate,” a possible distractor sentence is “Every insect is not a vertebrate.”

Original: We insert this distractor sentence into a random position in the context.

Original: A unique feature of P R O NTO QA is that it is easily programmable, with a handful of tunable knobs which we use to generate examples with varying degrees of complexity and study different aspects of reasoning in LLMs.
Revised: A unique feature of P R O NTO QA is that it is easily programmable, with a handful of tunable knobs which we use to generate examples with varying degrees of complexity and study different aspects of reasoning in LLMs.

Original: These variables are described in greater detail in section 5.1.
Revised: These variables are described in greater detail in section 5.1.

Original:  F ORMAL ANALYSIS OF PREDICTED PROOFS
Revised:  F ORMAL ANALYSIS OF PREDICTED PROOFS

Original: Instead of measuring the accuracy of the predicted answers (i.e. “true” or “false”), we would like to directly evaluate the predicted CoT to check if the model derives the right answer for the right reason.
Revised: Instead of measuring the accuracy of the predicted answers (i.e., “true” or “false”), we would like to directly evaluate the predicted CoT to check if the model derives the right answer for the right reason.

Original: We endeavor to analyze whether the model is able to apply deduction rules correctly at each proof step (i.e., local correctness), but also whether the model can plan ahead and work toward proving the answer for the query (i.e., global correctness).
Revised: We endeavor to analyze whether the model is able to apply deduction rules correctly at each proof step (i.e., local correctness), but also whether the model can plan ahead and work toward proving the answer for the query (i.e., global correctness).

Original: To measure the local correctness of a given proof step, we compute whether the step follows from one or more applications of deduction rules, and whether it requires additional rules beyond those of the gold proofs.
Revised: To measure the local correctness of a given proof step, we compute whether the step follows from one or more applications of deduction rules, and whether it requires additional rules beyond those of the gold proofs.

Original: To measure the global correctness, we wish to identify proof steps that deviate from the gold proof.
Revised: To measure the global correctness, we wish to identify proof steps that deviate from the gold proof.

Original: To achieve this, we parse each sentence of the predicted CoT into logical form.
Revised: To achieve this, we parse each sentence of the predicted CoT into logical form via recursive-descent parsing using the simple grammar from the generative process.

Original: We then compute whether that logical form is provable from previous logical forms via one or more applications of deduction rules.
Revised: We then compute whether that logical form is provable from previous logical forms via one or more applications of deduction rules.

Original: This logical form corresponds to the conclusion of a proof step.
Revised: This logical form corresponds to the conclusion of a proof step.

Original: We then evaluate the correctness of each proof step by categorizing it according to three dimensions:
Revised: We then evaluate the correctness of each proof step by categorizing it according to three dimensions:

Original: Validity: Is the current proof step provable from previous steps?
Revised: Validity: Is the current proof step provable from previous steps?

Original: If it is provable using only the deduction rules that appear in the gold proofs, we say the step is strictly-valid .
Revised: If it is provable using only the deduction rules that appear in the gold proofs, we say the step is strictly-valid .

Original: If it is provable with a more powerful proof calculus, like natural deduction, we say the step is broadly-valid .
Revised: If it is provable with a more powerful proof calculus, like natural deduction, we say the step is broadly-valid .

Original: Otherwise, we say the step is invalid .
Revised: Otherwise, we say the step is invalid .

Original: For example, given the premises, “Cats are carnivores” and “Carnivores are mammals,” the step with conclusion “Cats are mammals” is broadly-valid since an additional deduction rule is required  to prove it: given ∀ x ( f ( x ) → g ( x )) and ∀ x ( g ( x ) → h ( x )) , conclude ∀ x
Revised: For example, given the premises, “Cats are carnivores” and “Carnivores are mammals,” the step with conclusion “Cats are mammals” is broadly-valid since an additional deduction rule is required [Table] to prove it: given ∀ x ( f ( x ) → g ( x )) and ∀ x ( g ( x ) → h ( x )) [Table]

Revised: conclude ∀ x

Original: ( f ( x ) → h ( x )) .
Revised: ( f ( x ) → h ( x )) .

Revised: Notice that this is distinct from a strictly-valid non-atomic step since this conclusion is not provable via repeated applications of modus ponens.

Revised: We note that this the only additional rule that we check, as we did not encounter any instances of other broadly-valid rules.

Original: Atomicity:
Revised: Atomicity:

Original: Is the current proof step provable from previous steps with exactly one application of a deduction rule?
Revised: Is the current proof step provable from previous steps with exactly one application of a deduction rule?

Original: If so, we say the proof step is atomic .
Revised: If so, we say the proof step is atomic .

Original: Otherwise, it is non-atomic .
Revised: Otherwise, it is non-atomic .

Original: Note that since all broadly-valid steps are non-atomic, this distinction is only useful for strictly-valid steps.
Revised: Note that since all broadly-valid steps are non-atomic, this distinction is only useful for strictly-valid steps.

Original: For example, given the premises, “Fae is a cat,” “Cats are carnivores,” and “Carnivores are mammals,” the step with conclusion “Fae is a mammal” is non-atomic since the step “Fae is a carnivore” was skipped.
Revised: For example, given the premises, “Fae is a cat,” “Cats are carnivores,” and “Carnivores are mammals,” the step with conclusion “Fae is a mammal” is non-atomic since the step “Fae is a carnivore” was skipped.

Original: Misleading: If the current proof step’s premises are part of the gold proof, but its conclusion is not, then we say the proof step is misleading .
Revised: Utility: If the current proof step’s premises are part of the gold proof, but its conclusion is not, then we say the proof step is misleading .

Original: Otherwise, it is correct .
Revised: Otherwise, it is correct .

Original: For example, given the premises “Fae is a carnivore,” “All carnivores are not herbivorous,” and “Carnivores are mammals,” and the goal is to prove “Fae is not herbivorous,” the step “Fae is a mammal” is misleading since even though the step is strictly-valid, it does not help in proving the goal.
Revised: For example, given the premises “Fae is a carnivore,” “All carnivores are not herbivorous,” and “Carnivores are mammals,” and the goal is to prove “Fae is not herbivorous,” the step “Fae is a mammal” is misleading since although the step is strictly-valid, it does not help to prove the goal.

Original: The types of proof steps are listed in table 1 along with examples.
Revised: The types of proof steps are listed in table 2 along with examples.

Revised: Unparseable proof steps are marked as incorrect.

Original: For brevity, we refer to strictly-valid atomic correct steps as canonical steps .
Revised: For brevity, we refer to strictly-valid atomic correct steps as canonical steps .

Original: Psuedocode of the procedure to evaluate proofs is given in algorithm 1 in the Appendix.
Revised: Psuedocode of the procedure to evaluate proofs is given in algorithm 1 in the Appendix.

Original: Metrics.
Revised: Metrics.

Original: Given the above categorization of proof steps, we could define the correctness of a proof strictly by requiring the gold proof to be a subset of the predicted proof (so all corresponding proof steps must be canonical).
Revised: Given the above categorization of proof steps, we could define the correctness of a proof strictly by requiring the gold proof to be a subset of the predicted proof (so all corresponding proof steps must be canonical).

Original: But it is not obvious that this metric, which we call strict proof accuracy , would accurately measure the reasoning ability of the model.
Revised: But it is not obvious that this metric, which we call strict proof accuracy , would accurately measure the reasoning ability of the model.

Original: As such, we also consider more relaxed metrics: (a) we allow predicted proof steps to be strictly-valid non-atomic correct, which we call “skip” proof accuracy , (b) we allow predicted proof steps to be broadly-valid, which we call broad proof accuracy , or (c) we allow predicted proof steps to be strictly- or broadly-valid, which we call valid proof accuracy .
Revised: As such, we also consider more relaxed metrics: (a) we allow predicted proof steps to be strictly-valid non-atomic correct, which we call “skip” proof accuracy , (b) we allow predicted proof steps to be broadly-valid, which we call broad proof accuracy , or (c) we allow predicted proof steps to be strictly- or broadly-valid, which we call valid proof accuracy .

Original: The ordering of the sentences in the CoT is important: the conclusion of each proof step must come after the premises.
Revised: The ordering of the sentences in the CoT is important: the premises of each proof step must precede the conclusion.

Revised: If not, then we consider that step to be incorrect.

Original:  R ESULTS 5. E XPERIMENTAL SETUP
Revised:  R ESULTS 5. E XPERIMENTAL SETUP

Original: In each experiment, we generate question-answering examples, perform CoT prompting on the LLMs, and analyze the predicted CoTs.
Revised: In each experiment, we generate QA examples, perform CoT prompting on the LLMs, and analyze the predicted CoTs.

Original: We run the experiments on I NSTRUCT GPT(model sizes 350M, 1.3B, 6.7B, 175B versions 001 and 002 ) and the original GPT-3 (model size 175B), with greedy decoding (Ouyang et al., 2022; Brown et al., 2020).
Revised: We run the experiments on I NSTRUCT GPT and the original GPT-3 (OpenAI models text-ada-001 , text-babbage-001 , text-curie-001 , davinci , text-davinci-001 , text-davinci-002 ), with greedy decoding (Ouyang et al., 2022; Brown et al., 2020).

Original: We use 8-shot in-context learning, so each input to the LLM consists of 8 fully-labeled questions followed by a single test question with missing CoT and label.
Revised: We use 8-shot in-context learning, so each input to the LLM consists of 8 fully-labeled questions followed by a single test question with missing CoT and label.

Original: The model’s task is to predict the CoT and label for the test question.
Revised: The model’s task is to predict the CoT and label for the test question.

Original: Note that all examples across all inputs are independently and identically generated from P R O NTO QA. 
Revised: Note that all examples across all inputs are independently and identically generated from P R O NTO QA. [Figure]

Original:  There are a number of variables that we control when generating examples in P R O NTO QA: (1) the number of hops, (2) the ordering of the sentences in the context, and (3) the type of the ontology.
Revised: r i c t p r oo f a cc u r a c y [Figure] There are a number of variables that we control when generating examples in P R O NTO QA: (1) the number of hops, (2) the ordering in which the sentences are generated from the ontology, and (3) the type of the ontology.

Original: The number of hops directly controls the difficulty of the generated example, and we experiment with 1, 3, and 5 hops.
Revised: The number of hops directly controls the difficulty of the generated example, and we experiment with 1, 3, and 5 hops.

Original: We control the ordering of the context sentences produced by the ontology: We either traverse the tree top-down (i.e. preorder) or bottom-up (i.e. postorder), generating a sentence for each traversed edge/node.
Revised: We control the ontology traversal direction: We either traverse the tree top-down (i.e., preorder) or bottom-up (i.e., postorder), generating a sentence for each traversed edge/node.

Original: The ordering also affects the difficulty of the generated example: if thecontext sentences are generated bottom-up, they will follow the same order as the steps in the gold proof.
Revised: The ordering also affects the difficulty of the generated example: if the sentences are generated bottom-up, they will follow the same order as the steps in the gold proof.

Original: On the other hand, if they are generated top-down, the order is reversed, and the task may be more difficult.
Revised: On the other hand, if they are generated top-down, the order is reversed, and the task may be more difficult.

Original: To avoid any confounding effects from knowledge acquired during pretraining, P R O NTO QA generates examples with fictional concept names (e.g. “wumpus” instead of “cat,” etc).
Revised: To avoid any confounding effects from knowledge acquired during pretraining, P R O NTO QA generates examples with fictional concept names (e.g., “wumpus” instead of “cat,” etc).

Original: But we are also interested in measuring this confounding effect, and so in addition to fictional ontologies, we also generate “true” and “false” ontologies.
Revised: But we are also interested in measuring this confounding effect, and so in addition to fictional ontologies, we also generate “true” and “false” ontologies.

Original: True ontologies use real concept names and are consistent with the real-world (we randomly sample from a list of three hand-coded real ontologies).
Revised: True ontologies use real concept names and are consistent with the real-world (we randomly sample from a list of three hand-coded real ontologies).

Original: False ontologies use real concept names but the trees are generated using the random process described above, and so it is very likely to generate a false statement, such as “All mammals are cats.”
Revised: False ontologies use real concept names but the trees are generated using the random process described in section 3, and so it is very likely to generate a false statement, such as “All mammals are cats.”

Original: For each combination of variables, we run the model on 400 examples, for a total of 48 experiments.
Revised: For each combination of variables, we run the model on 400 examples generated from the testbed, for a total of 48 experiments.

Original: We compute 95% confidence intervals for each experiment since the number of correct proofs is distributed according to a binomial with a latent parameter that represents the model’s accuracy (Wilson, 1927).[Figure] r oo f a cc u r a c y [Figure]
Revised: We compute 95% confidence intervals for each experiment, as the number of correct proofs is distributed as Binomial (400 , p ) with p being the model’s accuracy (Wilson, 1927).

Original: 5. D O CORRECT ANSWERS IMPLY CORRECT REASONING ?
Revised: 5. D O CORRECT ANSWERS IMPLY CORRECT REASONING ?

Original: Label accuracy may not necessarily measure whether the model is performing reasoning correctly, since the model may find ways to guess the label via heuristics or spurious correlations.
Revised: Label accuracy may not necessarily measure whether the model is performing reasoning correctly, since the model may find ways to guess the label via heuristics.

Original: To gauge whether label accuracy is a good metric and which proof accuracy metric is best to measure reasoning ability, we investigate how label accuracy is related to the various proof accuracy metrics.
Revised: To gauge whether label accuracy is a good metric and which proof accuracy metric is best to measure reasoning ability, we investigate how label accuracy is related to the various proof accuracy metrics.

Original: We plotthe proof accuracy vs label accuracy (i [Figure] e., simply checking whether the predicted label “True” or “False” is correct) of every experiment that we conducted in figure 3.
Revised: We plot proof accuracy vs label accuracy (i.e., simply checking whether the predicted label “True” or “False” is correct) of every experiment that we conducted in figure 3.

Original: Each point in the scatter plot corresponds to one of our 48 experiments described above.
Revised: Each point in the scatter plot corresponds to one of ourexperiments described above.

Original: We observe that the label accuracy is poorly correlated with strict proof accuracy [Figure] and that strict proof accuracy may underestimate the model’s reasoning performance.
Revised: Observe that the label accuracy is poorly correlated with strict proof accuracy, and that strict proof accuracy may underestimate the model’s reasoning ability.

Original: Rather, the most permissive accuracy metric has the highest correlation with label accuracy, suggesting that GPT-3 is indeed performing the reasoning to answer the question, rather than relying on heuristics.
Revised: Rather, the most permissive accuracy metric has the highest correlation with label accuracy, suggesting that label accuracy is appropriate to measure reasoning accuracy.

Original: It also suggests that the most permissive proof accuracy metric is most appropriate for measuring the reasoning ability of the model.
Revised: It also suggests that the most permissive proof accuracy metric is most appropriate for measuring the reasoning ability of the model.

Original: 5. P ROOF ANALYSIS RESULTS
Revised: 5. P ROOF ANALYSIS RESULTS

Original: Reasoning is an emergent ability.
Revised: Only the largest model is able to reason.

Original: We investigated how reasoning ability is affected by model size.
Revised: We investigated how reasoning ability is affected by model size.

Original: In figure 7 in the Appendix, proof accuracy increases considerably when increasing the model size from 350M to 1.3B and 6.7B.
Revised: In figure 8 in the Appendix, proof accuracy increases considerably when increasing the model size from 350M to 1.3B and 6.7B.

Original: However, only version 002 of the 175B I NSTRUCT GPT model is able to perform better than chance, suggesting that the reasoning ability tested by our experiments could be an emergent ability (Wei et al., 2022a).
Revised: However, only text-davinci-002 is able to perform better than chance.

Original: We were not able to conclusively discern the cause of the significant difference in performance between version 001 and 002 .
Revised: We were not able to conclusively discern the cause of the significant difference in performance between version 001 and 002 .

Original: One possible factor is the maximum token limit of version 002 is roughly twice that of version 001 .
Revised: One possible factor is the maximum token limit of version 002 is roughly twice that of version 001 .

Original: In fact, the original GPTseems to perform as well as, if not slightly better than, version 001 of I NSTRUCT GPT.
Revised: In fact, the model davinci seems to perform as well as, if not slightly better than, text-davinci-001 .

Original: In addition, we notice that the frequency of invalid steps decreases as the model size increases, and so larger models seem to be better at making valid steps, whether or not those steps are actually useful. 
Revised: In addition, we notice that the frequency of invalid steps decreases as the model size increases, and so larger models seem to be better at making valid steps, whether or not those steps are actually useful. the of correct that contain an The are detailed

Original: For the remainder of the paper, our results focus on the 175B I NSTRUCT GPT model (version 002 ).
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: v a li d p r oo f a cc u r a c y

Revised: [Table]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: False ontology

Original:  Our main results are in figure 4 where we show the proof accuracy and distribution of proof step types for all experiments.
Revised: [Figure] True ontology For the remainder of the paper, our results focus on text-davinci-002 . Our main results are in figure 4 where we show the proof accuracy and distribution of proof step types for all experiments.

Original: Real-world knowledge helps reasoning.
Revised: Real-world knowledge helps reasoning.

Original: We investigate the extent to which reasoning ability is affected by whether the ontology is fictional, “true,” or “false.”
Revised: We investigate the extent to which reasoning ability is affected by whether the ontology is fictional, “true,” or “false.”

Original: Evidently from figure 4, the LLM seems to perform comparably in the fictional and “false” ontology settings (accuracy is slightly worse with a “false” ontology).
Revised: Evidently from figure 4, the LLM seems to perform comparably in the fictional and “false” ontology settings (accuracy is slightly worse with a “false” ontology).

Original: However, when using the “true” ontology, the model performs considerably better, and its performance does not deteriorate when increasing the number of hops from 3 to 5.
Revised: But when using the “true” ontology, the model performs much better, and its performance does not drop when increasing the number of hops from 3 to 5.

Original: Thelanguage model is able to utilize its background knowledge from pretraining to “skip” hops, and is therefore not as negatively affected by the increasednumber of hops.
Revised: The model is able to utilize its background knowledge from pretraining to “skip” hops, and is thus not as negatively affected by the increased hops.

Original: This is consistent with the findings of Dasgupta et al.
Revised: This is consistent with the findings of Dasgupta et al.

Original: Evidently, the model’s reasoning is heavily reliant on real-world knowledge, and this may be a problem for generalizability: such as when applying LLMs to novel scenarios or to scenarios that are not well-represented in the training data.[Table] [Table] [Figure]
Revised: Evidently, the model’s reasoning is heavily reliant on real-world knowledge, and this may be a problem for generalizability, such as when applying LLMs to novel scenarios or to settings that are not well-represented in the training data.

Original: [Table]

Original: [Figure]

Original: True ontology

Original: Longer proofs are still challenging.
Revised: Longer proofs are still challenging.

Original: We investigate the extent to which reasoning ability is affected by the number of hops in the proof.
Revised: We investigate the extent to which reasoning ability is affected by the number of hops in the proof.

Original: Wealso see from figure 4 that the model handles 1- and 3-hop examples quite accurately but struggles with 5-hop top-down examples, with accuracy falling to chance.
Revised: We see from figure 4 that the model handles 1- and 3-hop examples quite well but struggles with 5-hop top-down examples, with accuracy falling to chance.

Original: So while it is able to perform reasoning to an extent, its ability becomes more limited as the number of hops increases.
Revised: So while it is able to perform reasoning to an extent, it is more limited as the number of hops increases.

Original: Sentence ordering affects reasoning.
Revised: Traversal direction affects reasoning.

Original: We also tested how reasoning ability is affected by the ordering of the context sentences.
Revised: We also tested how reasoning ability is affected by the traversal direction of the ontology.

Original: We notice in figure 4 that as the number of hops increases, the model becomes sensitive to the ordering of the sentences (top-down vs bottom-up traversal of the ontology).
Revised: We notice in figure 4 that as the number of hops increases, the [Figure] o o r t i

Revised: o

Original: This may be due to the fact that the order of the gold proof steps mirrors the bottom-up traversal, and is the reverse of the top-down traversal.
Revised: n o f i n c o rr e c t p oo f [Table] [Figure] False ontology, 3 hops, top-down traversal direction

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: True ontology, 3 hops, top-down traversal direction

Revised: model becomes sensitive to the traversal direction of the ontology (top-down vs bottom-up).

Revised: This may be due to the fact that the order of the gold proof steps mirrors the bottom-up traversal, and is the reverse of the top-down traversal.

Original: Thus, the task may be made more difficult for language models if the context sentences are ordered top-down.
Revised: Thus, the task may be made more difficult for language models if the context sentences are ordered top-down.

Original: How do LLMs reason step-by-step?
Revised: How do LLMs reason step-by-step?

Original: We explore how often certain types ofproof steps appear in correct vs incorrect proofs, and whether  proofaccuracy is correlated with the presence of specific types of proof steps.
Revised: We investigate the fraction of correct and incorrect proofs that contain various types of proof steps, and whether the correctness of the proof is correlated with the presence of specific types of proof steps.

Original: Figure 4 breaks down the bars further (in darker red and blue) to indicate the fraction of examples that contain proof steps other than canonical steps. We make the following observations: [Figure] [Table]
Revised: Figure 4 breaks down the bars further (in darker red and blue) to indicate the fraction of proofs that contain proof steps other than canonical steps, since most predicted proof steps were canonical (in the 5-hop experiments with fictional ontology, they constitute 93 .

Original: [Figure] [Table] [Figure] [Table]
Revised: 2% of proof steps).

Original: [Figure] [Table] [Figure] [Figure]
Revised: We make the following observations:

Original: [Table] [Figure] [Table] [Figure]
Revised: Most predicted proof steps are strictly-valid (in the 5-hop experiments with fictional ontology, 93 .

Original: [Table] [Figure] [Table] [Figure]
Revised: 2% of proof steps are strictly-valid, 2 .

Original: [Table] [Figure] [Table]
Revised: 4% are broadly-valid, and 5 .

Original: [Figure] [Table] [Figure] [Table]
Revised: 9% are invalid).

Original: [Figure] Most predicted proof steps are strictly-valid, in both correct and incorrect proofs. LLMs tend to skip steps by producing non-atomic steps, just as humans do when they verbalize their reasoning.
Revised: LLMs tend to skip steps by producing non-atomic steps, just as humans do when they verbalize their reasoning (in the 5-hop experiments with fictional ontology, 2 . 4% of proof steps are nonatomic, even though all steps in the few-shot examples are atomic).

Original: Most incorrect proofs contain misleading steps and invalid steps.
Revised: Most incorrect proofs contain misleading steps and invalid steps.

Original: This suggests that the source of the incorrect reasoning is either a due to a misleading step or an invalid step that causes the model to produce steps that do not belong to the gold proof.
Revised: This suggests that the source of the incorrect reasoning is either a due to a misleading step or an invalid step that causes the model to produce steps that do not belong to the gold proof

Original: Intriguingly, some correct proofs also contain misleading steps and invalid steps, which implies that the model is sometimes able to recover from these “mistakes” and return to the gold proof.
Revised: Intriguingly, some correct proofs also contain misleading steps and invalid steps, which implies that the model is sometimes able to recover from these “mistakes” and return to the gold proof.

Original: We analyze this behavior in greater detail in section 5.4.
Revised: We analyze this behavior in greater detail in section 5.4.

Original: 5. W HAT LEADS TO A MISTAKE ?
Revised: 5. W HAT LEADS TO A MISTAKE ?

Original: We investigate whether specific types of proof steps are causingthe the largest I NSTRUCT GPTmodel to produce reasoning errors.
Revised: We investigate whether specific types of proof steps are causing I NSTRUCT GPT to produce reasoning errors.

Original: To do so, we identify the first step in each incorrect proof that is not a canonical step.
Revised: To do so, we identify the first step in each incorrect proof that is not a canonical step.

Original: We observe in figure 5, among incorrect proofs, strictly-valid atomic misleading steps appear in the proof first far more often than other non-canonical step types, including invalid steps.
Revised: We observe in figure 5, among incorrect proofs, strictly-valid atomic misleading steps appear in the proof first far more often than other non-canonical step types, including invalid steps.

Revised: See figure 7 in the appendix for an example prediction where a misleading step causes the model to fail to prove the goal and produce an invalid step.

Original: This indicates that for the best-performing models, the main source of reasoning error is from misleading steps, since most predicted steps are strictly-valid and atomic.
Revised: This indicates that for the best-performing models, the main source of reasoning error is from misleading steps, since most predicted steps are strictly-valid and atomic.

Original: That is, imagining the space of proof steps as a graph where each edge represents a single valid step, I NSTRUCT GPT almost always performs a walk in this graph.
Revised: That is, imagining the space of proof steps as a graph where each edge represents a single valid step, I NSTRUCT GPT almost always performs a walk in this graph.

Original: Once I NSTRUCT GPT encounters a branch where one path at the fork follows the correct proof and the other paths do not, I NSTRUCT GPT will select the incorrect direction with some frequency and is then not able to return to the correct path.
Revised: Once I NSTRUCT GPT encounters a branch where one path at the fork follows the correct proof and the other paths do not, I NSTRUCT GPT will select the incorrect direction with some frequency and is then not able to return to the correct path.

Original: Therefore, it seems that while LLMs are able to produce valid proof steps with high probability, they have difficulty with proof planning/strategizing.
Revised: Therefore, it seems that while LLMs are able to produce valid proof steps with high probability, they have difficulty with proof planning/strategizing.

Original: We werealso curiousto see if this relationship held in smaller models.
Revised: We were curious if this relationship held in smaller models.

Original: We see inthe bottom three rows of figure 5 that smaller models are more prone to making invalidsteps or non-atomic steps as their first non-canonical step.
Revised: We see in figure 5 that smaller models are more prone to make invalid or non-atomic steps as their first non-canonical step.

Original: But as model size increases,the frequency of these types of steps decreases, and is instead superseded by misleading steps.
Revised: But as model size increases, these types of steps become rarer, and is instead superseded by misleading steps.

Original: Looking again at figure 4, we note that many correct proofs also contain misleading steps, and so it must be the case that I NSTRUCT GPT sometimes returns to the correct proof path at some point after making a misleading step.
Revised: Looking again at figure 4, we note that many correct proofs also contain misleading steps, and so it must be the case that I NSTRUCT GPT sometimes returns to the correct proof path at some point after making a misleading step.

Original: To investigate this behavior more closely, we count the number of steps that the model takes after making a misleading step until it produces a step in the gold proof and plot the histogram in figure 9 in the Appendix.
Revised: To investigate this behavior more closely, we count the number of steps that the model takes after making a misleading step until it produces a step in the gold proof and plot the histogram in figure 10 in the appendix.

Original: We observe that, in general, the more time the model spends outside the correct proof path, the less likely it becomes to return to the correct proof.
Revised: We observe that, in general, the more time the model spends outside the correct proof path, the less likely it becomes to return to the correct proof.

Revised: We demonstrate in section G in the appendix that our findings generalize to more sophisticated prompting strategies via experiments using self-consistency prompting (Wang et al., 2022).

Original:  C ONCLUSION AND FUTURE WORK
Revised:  C ONCLUSION AND FUTURE WORK

Original: In this work, we introduced a synthetic fictional question-answering dataset called P R O NTO QA designed to evaluate the reasoning ability of LLMs.
Revised: In this work, we introduced a synthetic fictional QA dataset called P R O NTO QA designed to evaluate the reasoning ability of LLMs.

Original: We evaluated I NSTRUCT GPT and GPT-3 on P R O NTO QA and found that while the largest model was generally able to perform reasoning,even with fictional and counterfactual inputs, it had difficulty with proof planning and selecting the correct proof step when there are multiple available.
Revised: We evaluated I NSTRUCT GPT and GPT-3 on P R O NTO QA and found that while the largest model was generally able to perform reasoning, it had difficulty with proof planning and selecting the correct proof step when there are multiple available.

Original: P R O NTO QA, and our high-level approach more broadly, could be used to compare LLM reasoning with that of humans, and to explore which aspects of human reasoning were acquired by LLMs from their pretraining.
Revised: P R O NTO QA, and our high-level approach more broadly, could be used to compare LLM reasoning with that of humans, and to explore which aspects of human reasoning were acquired by LLMs from their pretraining.

Original: As our work has shown that LLMs are able to reason to a degree, it is yet unclear where the model acquired this ability.
Revised: As our work has shown that LLMs are able to reason to a degree, it is yet unclear where the model acquired this ability.

Original: Are there portions of the pretraining data that teach the model to reason?
Revised: Are there portions of the pretraining data that teach the model to reason?

Original: Our work shows thatrelying on CoT prompting is not sufficient for more complex reasoning, such as in mathematical domains.
Revised: Our work shows that CoT prompting is not sufficient for more complex reasoning, such as in mathematical domains, since the reasoning tested in this work is a strict subset of that of general mathematical reasoning.

Original:  Rather, our results suggest that reasoning systems stand to benefit from more sophisticated proof planning/search strategies, such as neurosymbolic approaches where part of the reasoning can be done over interpretable symbolic structures.
Revised: Mathematical proofs contain steps with much higher branching factor, where robust proof planning is instrumental. Rather, our results suggest that reasoning systems may benefit from more sophisticated proof planning/search strategies, such as neurosymbolic approaches where part of the reasoning is done over interpretable symbolic structures.

Original: P R O NTO QA can be used to train new reasoning systems, or to pretrain/fine-tune LLMs tohopefully improve their reasoning capability.
Revised: P R O NTO QA can be used to train new reasoning systems, or to pretrain/fine-tune LLMs to improve their reasoning capability.

Original: The inability of LLMs to plan ahead in their reasoning might be related to recent work illuminating the theoretical computational limitations of such models (Merrill et al., 2022).
Revised: The inability of LLMs to plan ahead in their reasoning might be related to recent work illuminating the theoretical computational limitations of such models (Merrill et al., 2022).

Original: Since our analysis was limited to modus ponens, proof lengths of at most 5, and semantically simple sentences, it remains to be seen whether LLMs are able to produce longer proofs, or reason with other deduction rules, or over more semantically complex sentences/logical forms.
Revised: Since our analysis was limited to modus ponens, proof lengths of at most 5, and semantically simple sentences, it remains to be seen whether LLMs are able to produce longer proofs, or reason with other deduction rules, or over more semantically complex sentences/logical forms.

Original: Our analysis also revealed a possible way forward to rectify this shortcoming: Even after making a misleading step, I NSTRUCT GPT sometimes “returned” to the correct proof.
Revised: Our analysis also revealed a possible way forward to rectify this shortcoming: Even after making a misleading step, I NSTRUCT GPT sometimes “returned” to the correct proof.

Original: We could relax the constraint that the chains-of-thought always reflect the shortest correct proof of the answer.
Revised: We could relax the constraint that the chains-of-thought always reflect the shortest correct proof of the answer.

Original: It is reasonable for humans to explore a space of possible solutions before arriving at the correct answer, and we could mimic this in LLMs by allowing the CoT to explore alternative paths in the space of proof steps, even if those paths are not ultimately part of the underlying shortest proof.
Revised: It is reasonable for humans to explore a space of possible solutions before arriving at the correct answer, and we could mimic this in LLMs by allowing the CoT to explore alternative paths in the space of proof steps, even if those paths are not ultimately part of the underlying shortest proof.

Original: This kind of search strategy is analogous to depth-first search (DFS) in graphs.
Revised: This kind of search strategy is analogous to depth-first search (DFS) in graphs.

Original: We could provide in-context examples of chains-of-thought that follow a DFS, with the hope that the model is able to learn to perform DFS when given a new test example, thereby improving the likelihood that it finds the correct answer.
Revised: We could provide in-context examples of chains-of-thought that follow a DFS, with the hope that the model is able to learn to perform DFS when given a new test example, thereby improving the likelihood that it finds the correct answer.

Original: Our results suggest that the model’s greedily-predicted CoT may not necessarily maximize the likelihood globally, according to the LLM.
Revised: R EPRODUCIBILITY STATEMENT

Original: As such, we can extend the CoT paradigm where after every predicted step in the CoT, we perform beam search decoding to find the top- k most likely values for the next step. We can then perform beam search over the proof steps (i.e., sentences) rather than over the tokens, in order to hopefully find more globally optimal predicted proofs that lead to the correct answer.
Revised: All our experiments were run using the OpenAI API on September 9 th , 10 th and 11 th , 2022. For the sake of reproducibility of the analysis, all model outputs, the code for data generation and the analysis code are freely available with a permissive open-source license at [anonymized] . The command python analyze_results.py produces all figures used in this paper

Original: A D EDUCTION RULES
Revised: A D EDUCTION RULES

Original: Figure 6 outlines the two deduction rules that we utilize in P R O NTO QA.
Revised: Figure 6 outlines the two deduction rules that we utilize in P R O NTO QA.

Original: Deduction rules in general form
Revised: Deduction rules in general form

Original: Examples
Revised: Examples

Original: f ( a ) ∀ x ( f ( x ) → g ( x ))
Revised: [Equation] cat [Equation]

Original: Hop g ( a ) cat ( fae ) ∀ x ( cat ( x ) → carnivore ( x ))
Revised: cat [Equation] x ) → carnivore ( x )

Original: Hop
Revised: [Equation]

Original: carnivore ( fae ) i.e. Given that “Fae is a cat” and “All cats are carnivores,” we conclude that “Fae is a carnivore.” 
Revised: carnivore ( fae ) i.e., Given that “Fae is a cat” and “All cats are carnivores,” we conclude that “Fae is a carnivore.” [Equation]

Original: AxA
Revised: Ax

Revised: [Equation]

Original: Ax cat ( fae ) i.e. Assume that “Fae is acat” is an axiom.
Revised: [Table] ) i.e., Assume that [Table] is a

Revised: [Table]

Revised: is an axiom.

Original: B H OW WE EVALUATE THE CHAIN - OF - THOUGHT
Revised: B A VOIDING SHORTCUTS

Revised: When generating examples in P R O NTO QA, we have to be careful to remove any shortcuts in the question that would allow the model to “guess” the answer without reasoning. In fact, we find that without any distractors, I NSTRUCT GPT is able to predict the “true”/“false” label almost perfectly. I NSTRUCT GPT can utilize the heuristic that whether the queried property is mentioned in the context implies whether or not it is true.

Revised: For instance, if the example is asking “Sally is a cat.

Revised: True or false: Sally is a vertebrate,” the model can simply look for a string “Every is (not) a vertebrate,” regardless of the content in the blank.

Revised: Due to the generative process of these examples, this kind of sentence is guaranteed to appear exactly once in the context.

Revised: Thus, to ensure that such a heuristic is not informative, we add a distractor sentence by generating a novel concept that is disconnected from the ontology tree, and we assign to this new concept the negation property that is queried by the question.

Revised: So in the above example, if the ontology has the rule “Every mammal is a vertebrate,” a possible distractor sentence is “Every insect is not a vertebrate.”

Revised: We insert this distractor sentence into a random position in the context.

Revised: C E XAMPLE I NSTRUCT GPT MISPREDICTION

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: 1 function evaluate_cot( context sentences Q 1 , . . . , Q m , predicted chain-of-thought sentences C 1 , . . , C n , gold chain-of-thought sentences T 1 , . .

Original: [Table]
Revised: , T r ) for i ∈ 1 , . . . , m do /* parse the context */ L

Original: [Table] [Table] [Table]
Revised: = semantic_parse( i ) for i ∈ 1 , . . .

Original:  do /
Revised: , r do /

Original: [Table]
Revised: * parse the L Ti = semantic_parse( i ) initialize S as an empty set for i . . , n do /* parse and evaluate the predicted

Original: [Table]
Revised: -of

Original: [Table] [Table]
Revised: L Ci = semantic_parse( C i ) P, k is_provable

Original: [Table]
Revised: L Q

Original:  ) if k ≥/* if we wish to use a stricter metric for proof accuracy, we can add conditions here (e.g., requiring atomicity by checking k = 1 ) */ add L Ci to S if P ⊆ { L T 1 , . . .
Revised: , L Qm } , S ) if k ≥/* if we wish to use a stricter metric for proof accuracy, we can add conditions here (e.g., requiring atomicity by checking k = 1 ) */ add L Ci to S if P ⊆ { L T 1 , . . .

Original: , L Tr } and L Ci / ∈ { L T 1 , . . .
Revised: , L Tr } and L Ci / ∈ { L T 1 , . . .

Original: , L Tr } /* the premises are in the gold proof but the conclusion is not */ mark L return
Revised: , L Tr } /* the premises are in the gold proof but the conclusion is not */ mark L Ci step

Original: 15 functionis_provable set
Revised: 15 function

Original: else if φ∈ S return
Revised: else if φ S return

Original: else if φ has form g c or ¬ g ( c  constants g and cfor a ∈ A∪ do if 
Revised: else if φ has form g c or ¬ g ( c for any constants g and c A do if a has form ∀ x

Original: ∀ ψ γwhere γ [ c φ else graph */
Revised: → γ γ x φ /* note we precompute this graph if

Original: return 
Revised: return to path

Original: C P ROOF ACCURACY VS MODEL SIZE
Revised: E P ROOF ACCURACY VS MODEL SIZE

Original: [Figure]
Revised: [Figure]

Original: v a li d p r oo f a cc u r a c y
Revised: v a li [Figure] p r oo f a cc u r a c y

Original: Fictional ontology, 3 hops 
Revised: Fictional ontology, 3 hops c

Original: [Table]
Revised: [Table]

Original: False ontology, 3 hops I NSTRUCT GPT 350M I NSTRUCT GPT
Revised: rr ec t

Original: [Table]
Revised: [Table]

Original: .3B

Original: I NSTRUCT GPT

Original: [Table]
Revised: [Figure]

Original: .7B Original GPT-
Revised: False ontology, 3 hops

Original: [Table]
Revised: [Figure]

Original: 175B I NSTRUCT GPT 175B (version 001 ) I NSTRUCT GPT 175B (version 002 )
Revised: [Table] [Table]

Original: [Table]
Revised: [Figure]

Original: True ontology,3 hops
Revised: True ontology,

Original: I NSTRUCT GPT 350M
Revised: [Table]

Original: [-I NSTRUCT GPT 1.3B I NSTRUCT GPT 6.7B Original GPT-
 175B I NSTRUCT GPT 175B (version 001 ) I NSTRUCT GPT 175B (-]
Revised: [-I NSTRUCT GPT 1.3B I NSTRUCT GPT 6.7B Original GPT-
 175B I NSTRUCT GPT 175B (version 001 ) I NSTRUCT GPT 175B (-][Figure]

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Figure]

Original: Fictional ontology, 1 hopI
Revised: Fictional ontology, 1 hop

Original: [Table]
Revised: [Table]

Original: GPT

Original: [Table]
Revised: [Figure]

Original: I

Original: [Table]
Revised: [Table]

Original: GPT

Original: [Table]
Revised: [Figure]

Original: GPT

Original: [Table]
Revised: [Table]

Original: Original

Original: [Table]
Revised: [Figure]

Original: I

Original: [Table]
Revised: [Table]

Original: I

Original: [Table]
Revised: [Figure]

Original: GPT 175B

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Figure]

Original: False ontology, 1 hop
Revised: False ontology, 1 hop

Original: I NSTRUCT GPT 350M

Original: I NSTRUCT GPT

Original: [Table]
Revised: [Table]

Original: .3B
Revised: [Figure] [Table] [Figure]

Original: I NSTRUCT GPT

Original: [Table]
Revised: [Table]

Original: .7B Original GPT-3 175B I NSTRUCT GPT 175B (version 001 ) I NSTRUCT GPT 175B (version 002 )
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Figure]

Original: True ontology, 1 hop
Revised: True ontology, [Table]

Original: D A DDITIONAL ERROR ANALYSIS
Revised: F A DDITIONAL ERROR ANALYSIS

Original: [Figure]
Revised: [Figure]

Original:  o r t i o n o f i n 
Revised: r o p o r t i o n o f i n c o rr e c

Original: [Figure]
Revised: [Figure]

Original: rr
Revised: p r oo

Original: [Figure]
Revised: [Figure]

Original: c [Figure] [Table][Figure]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original:  the 2049 token limit for all models (except I NSTRUCT GPT 175B version 002 ).
Revised: The proof step types are detailed in figure 2. We note that in the 3-hop exper- iments with fictional ontology, four of the 400 exam- ples surpassed the 2049 token limit for all models (ex- cept text-davinci-002 ).

Original: These examples were ignored (so the effective number of examples is 396).
Revised: These examples were ig- nored (so the effective number of examples is 396).

Original: We omit the results for the 1-hop experiments here since there were too few incorrect proofs. 
Revised: We omit the results for the 1hop experiments here since there were too few incorrect proofs. [Figure] [Figure] c

Original: [Figure]
Revised: [Figure]

Revised: p

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: G D O OTHER PROMPTING STRATEGIES HELP ?

Revised: To what extent do our findings generalize to prompting strategies other than CoT with greedy decoding?

Revised: To test this, we experimented with self-consistency prompting (Wang et al., 2022), where for each example, we queried the LLM for 40 sample predictions of the CoT, using a temperature setting of 0 .

Revised: For each sample CoT s i , we compute the following quantity:

Revised: [Equation]

Revised: We parse each predicted CoT sample into a sequence of logical forms, and we find the logical form sequence with the highest sum of the above quantity over all the CoT samples that share the same semantic parse.

Revised: This logical form sequence is the final prediction.

Revised: We run this experiment in our setting with 5 hops, fictional ontology, and top-down traversal direction, with 100 examples.

Revised: The resulting valid proof accuracy is 0 .

Revised: 56 compared to 0 .

Revised: which is not significantly different.

Revised: Furthermore, inspecting specific examples of CoT samples (see figure 11), we see that for examples that the model gets wrong, the model is actually assigning higher overall probability to the incorrect proof than to the correct proof.

Revised: This suggests that our results do in fact generalize to more sophisticated prompting/decoding strategies, and that strategies that endeavor to find proofs with higher probability globally (e.g. beam search) will not help the model in proof planning.

Revised: [Table]

Revised: Sample predicted answer: Polly is an impus.

Revised: Impuses are zumpuses.

Revised: Polly is a zum pus.

Revised: Zumpuses are numpuses.

Revised: Polly is a numpus.

Revised: Numpuses are tumpuses.

Revised: Polly is a tumpus.

Revised: Each tumpus is a yumpus.

Revised: Polly is a yumpus.

Revised: Yumpuses are vumpuses.

Revised: Polly is a vumpus.

Revised: Vumpuses are jompuses.

Revised: Polly is a jompus.

Revised: Every jompus is a rompus.

Revised: Polly is a rompus.

Revised: Rompuses are not

Original: [Figure]
Revised: [Table] [Table] is not [Table]