Paper: MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning (ICLR-2024)

Reviewer comment (Experiments):
Have humans checked whether the stories are coherent, and reference answers are correct?

Author response:
MuSR was annotated with a 96% human accuracy across all datasets. We consider this close to perfect. Reiterating from section 5.1 in the human evaluation subsection, our analysis showed that a majority of misannotations are due to inattention due to the task requiring careful reading.

It is possible that a failure in the reasoning tree construction or narrative translation process can introduce inconsistencies.  However, this is very rare.  One instance we found of this was in the Object Placements domain:

“[...] the headphones found their resting place on the equipment rack thanks to Mark. […]. Austin […] shifts his attention from the console, rising from his chair […]”
 
This led the annotators to believe that Austin did not see Mark move the headphones, but in the reasoning tree Austin is said to have seen it because he is near Mark rearranging some equipment. (Note that our high consensus evaluation numbers show that a panel of humans generally *do* agree on who observed what in this domain.)

Reviewer comment (Experiments):
Concern that the human results are far from perfect, with uncertainty about whether this is due to poor crowdsourcing incentive structures or crowdworker filtering processes, good annotators being careless, or the references actually being incorrect (or other problems in the story).

Author response:
MuSR was annotated with a 96% human accuracy across all datasets. We consider this close to perfect. Reiterating from section 5.1 in the human evaluation subsection, our analysis showed that a majority of misannotations are due to inattention due to the task requiring careful reading.

It is possible that a failure in the reasoning tree construction or narrative translation process can introduce inconsistencies.  However, this is very rare.  One instance we found of this was in the Object Placements domain:

“[...] the headphones found their resting place on the equipment rack thanks to Mark. […]. Austin […] shifts his attention from the console, rising from his chair […]”
 
This led the annotators to believe that Austin did not see Mark move the headphones, but in the reasoning tree Austin is said to have seen it because he is near Mark rearranging some equipment. (Note that our high consensus evaluation numbers show that a panel of humans generally *do* agree on who observed what in this domain.)

Reviewer comment (Novelty):
There have been many recent benchmarks (Open Assistant, advanced reasoning benchmark, https://arxiv.org/abs/2307.02477, etc.), so there is a question of how the authors expect this particular benchmark to be used.

Author response:
Reiterating from our conclusion, we expect our dataset to test the limits of current reasoning systems that require vast amounts of commonsense, deductive reasoning, and “system 2” deliberation (decision making).  To our knowledge, no dataset effectively combines all three of these components, yet we consider them vital for real world reasoning (for example, reasoning about real murder mysteries).  Furthermore, the reasoning trees in MuSR allow for more in-depth analysis of intermediate reasoning in both prompt-based and neurosymbolic methods that have previously not been possible.

Reviewer comment (Experiments):
Given that the data is synthetic and arbitrarily complex synthetic problems can be generated, the importance of LLMs being able to solve these particular problems (over other benchmarks) is unclear. The reviewer would be more convinced if the technique could be used to generate arbitrary stories and ask arbitrary questions.

Author response:
Although the dataset is synthetic, we believe the problems themselves are realistic by virtue of being grounded in real-world reasoning scenarios. The text is not at the level of a great human writer, but it presents the necessary information for the reasoning problem in a coherent narrative. Most importantly, we believe that this kind of long-form narrative reasoning provides an important counterpoint to benchmarks like test questions (e.g., LSAT questions, math questions) which are artificial by design and optimized to test humans’ reasoning capabilities.

I would be more convinced if this technique can be used to generate arbitrary story and ask arbitrary questions.

We believe that MuSR is as close as you can get here with existing methods. Sections 3 and 4 detail how once a basic template is engineered, any number of stories can be generated and with minor tweaks you can ask various questions.  To the best of our knowledge, it’s as close to any work that has ever come to allow for an unguided creation of a narrative QA dataset. In our experience, less scaffolded versions of the dataset creation (e.g., generate a story and then ask questions) lead to either easy questions, simple narratives, or inconsistencies between the story and the generated questions’ answers.

Reviewer comment (Evaluation):
A missing analysis appears to be counting the number of responses which cannot be clearly parsed for each method, since non-parseable responses do not necessarily indicate that the model cannot solve the problem. Is the number of non-parseable responses zero?

Author response:
GPT and Llama 2 70B very rarely produce unparsable answers.  Vicuna 13B produced unparsable answers 7.8% of the time.  Smaller models (7B-scale models) can produce high rates of unparsable answers (either no murderer or the instructions were not followed).  When this happens, we treat the model as having guessed randomly.

Reviewer comment (Presentation):
Minor: The Table 1 framing needs improvement. For example, if Sophia knows about the shooting and has a grudge toward Emily because Emily stole Sophia’s fortune, then the reference says Sophia is the murderer. In reality, killer identification is not that simple; we can only guess based on existing evidence, and we do not want the LLM to be certain that Sophia is the killer. (The prompts in the appendix seem okay.)

Author response:
We assume this is referring to Figure 1. First, we note that we don’t include the example due to the length of the mysteries in MuSR, but complete examples can be seen in the appendix. The context is proving a motive, not the murderer.

Reviewer comment (Experiments):
Did the authors try the “let’s think step by step” zero-shot prompt?

Author response:
Yes, see table 7 “CoT” row, which is a fairly minimal chain-of-thought prompt similar in spirit to “let’s think step-by-step”.

Reviewer comment (Novelty):
What are the advantages over existing long-document question answering benchmarks? For example, generated data will be new so there won’t be data leakage issues from pretraining, and questions can be made arbitrarily long.

Author response:
See Table 1 and our conclusion.   To reiterate: our text is not templated, we require multiple steps of reasoning coupled with commonsense, we have intermediate data structures proving the correct answer for each question, and there is no rule-based solution for the examples we create in MuSR. 

Furthermore, we can create an arbitrary number of new examples, new domains are easily created (hours with a single engineer), an example is cheap to create (cost of the API call), and there is no dataset leakage.

Reviewer comment (Evaluation):
The dataset size is small according to Table 2. Does the dataset size affect the evaluation of model performances?

Author response:
Increasing the size of the dataset would lower the variance of model accuracy numbers, but we do not believe it would change the core conclusion of this paper. Furthermore, given the expense of running LLMs on long narratives, particularly for researchers who want to explore neurosymbolic reasoning methods like decomposed prompting that require several prompts per example, we felt that the size of this benchmark balanced affordability of running on our dataset with high enough statistical power. Because human eval is averaging 96% and the models are underperforming by a decent margin, we believe that this gap will remain and is still useful for researchers to build on.

Reviewer comment (Reproducibility):
In Section 3.2, when recursively calling the GPT-4 to produce the reasoning trees, how to deal with the possible logical or factual inconsistency between the statements/facts?

Author response:
Section 3.2 and appendix C.1 describe our validation algorithms in detail.  These validators aim to ensure that the facts being generated in the reasoning tree are consistent both with themselves and the generated narratives. Ultimately, the human performance reflects that humans agree on the answers, which is an extrinsic measure of this kind of factual consistency.

Reviewer comment (Writing):
What does “CMCR” in the title of Section 5.2 refer to?

Author response:
This is an erroneous reference to an earlier name of the dataset. It is fixed now.

Reviewer comment (Experiments):
It's unclear to me how much human engineering is needed in the dataset construction process. The major confusion comes from the "scenario-specific facts" generation. Looking at the prompt in Appendix C.1, at the end of page 30, it seems that human engineering is needed to provide the scenario setup, e.g., the following. If my understanding is correct, then the dataset is not "scalable" (which the authors claimed is a main advantage over human-annotated ones).
```
Scenario: Victim: Tessa
Crime Scene: kitchen
Murder Weapon: poisonous gas
Suspect: Penelope
Role in story: Tarot Reader
The suspect’s motive: To protect a secret
```

Author response:
Thank you for your review.  We are encouraged that you have found our dataset scalable while still being grounded in natural text. To that point, your question:
 
### “how much human engineering is needed in the dataset construction process”

The facts you list from the prompt in “Scenario: Victim: Tessa…” are described in Appendix D.1 in more depth. This template is constructed through engineering, but the values are sampled from LLMs to provide diverse scenario facts. That is, once a domain (like a murder mystery) is set, we prompted GPT-4 for “100 murder weapons” or “100 crime scenes” and sample from those lists.

Therefore, the extent of engineering for each domain is designing the templates themselves and the prompts. The actual engineering for each specific example is zero, as these templates are filled automatically from our generated lists.

We have clarified this in the new version of Section 3.

Reviewer comment (Evaluation):
The current experiment exploration seems a bit shallow, as the discussions are mostly about comparing numbers achieved by different LLMs or prompting methods. It's suggested that the authors could provide some qualitative analyses and elaborate on the bottlenecks for LLMs to perform well in such tasks.

Author response:
### “The current experiment exploration seems a bit shallow, as the discussions are mostly about comparing numbers achieved by different LLMs or prompting methods.”

We would like to point out that our goal is to benchmark existing methods on this dataset, most of which do consist of variants of chain-of-thought prompting. Few other neurosymbolic methods can work broadly here (see 5.2). Finally, we can expand our qualitative analysis in a future version, but largely left this in the appendix due to space constraints.

Reviewer comment (Experiments):
Could the authors clarify the human engineering in the dataset construction?

Author response:
Thank you for your review.  We are encouraged that you have found our dataset scalable while still being grounded in natural text. To that point, your question:
 
### “how much human engineering is needed in the dataset construction process”

The facts you list from the prompt in “Scenario: Victim: Tessa…” are described in Appendix D.1 in more depth. This template is constructed through engineering, but the values are sampled from LLMs to provide diverse scenario facts. That is, once a domain (like a murder mystery) is set, we prompted GPT-4 for “100 murder weapons” or “100 crime scenes” and sample from those lists.

Therefore, the extent of engineering for each domain is designing the templates themselves and the prompts. The actual engineering for each specific example is zero, as these templates are filled automatically from our generated lists.

We have clarified this in the new version of Section 3.

Reviewer comment (Experiments):
Could the authors provide more justification for the creation ablation analysis?

Author response:
### “The authors suggested low GPT-4 performance (Acc) from the ablated baselines as a good indicator of the effectiveness of each creation component”

Actually, the point in Section 5.1 is a bit more nuanced. “Although our goal is to make a challenging dataset, in the context of this table, low GPT-4 performance usually means that the examples are ambiguous or unsolvable.”  We argue that low model accuracy, high human accuracy, *and* other criteria like length and fact recall are required to have a good dataset. Thus changes in performance on that ablated dataset could improve due to consistent reasoning be applied in the narrative (more solvable questions) or the performance could decrease due to more complex reasoning be adhered to (narrative is more faithful to the reasoning tree). 

Looking at only one of those columns does not lead to a complete picture.  

### “no comparison was done for Team Allocation. It's suggested that the authors could provide some qualitative analyses and elaborate on the bottlenecks for LLMs to perform well in such tasks.”

We mention in Appendix C.3 that Team Allocation does not require validators: “Team Allocation required no validators in the construction of the reasoning tree nor the story to create valid contexts.”  This is why 2 out of 5 ablations are left blank in Table 4 for this domain. We further clarified this in the new version of the paper.

Reviewer comment (Experiments):
By far the biggest concern I have, which is derived from working on attempts at similar dataset creation, is that there is a point where despite designing the reasoning chains to be deductively entailed from premises, somewhere in the translation to ambiguous natural language and the accumulation of steps, reasonable humans start to disagree on whether a particular conclusion follows or not, at least with certainty. This can lead to there possibly being a sizable number of cases in the dataset where in fact a single gold answer cannot (even in principle) be attributed to the problem. I am wondering if the authors have considered this issue and how they may say their dataset construction procedure guards against it. Would they claim the best performing or majority human annotator label be considered a lower bound on the percentage of problems for which a definite true answer exists?

Author response:
We have, and this is a difficult issue to fully address. We describe validation algorithms in section 3.2 and go into specific details about entailment validation in Appendix C.1:

“[...] we also implemented a story validator during the narrative generation. The story validator prompts GPT-4 to check the entailment of each scenario-specific fact S(T) given the story. If any fact is not entailed by the story, we prompt GPT-4 to rewrite the story and ensure that fact is entailed.”

The human accuracy also serves as an end-to-end measure of correctness of each example, giving us a combination of automated stepwise validation and human end-to-end validation. These are precision-focused measures that show that the ground truth reasoning is sound. However, we acknowledge that human evaluation of each step would be beneficial. Furthermore, we agree it would be useful to have another eval focused on whether alternative reasoning paths are possible.

Reviewer comment (Experiments):
I am also slightly concerned about the self-referential use of GPT-4 in constructing the dataset. I think the authors have put in a lot of work to minimize the problem, since they use it only for specific steps after the bulk of the reasoning chain has been fleshed out by the reasoning tree algorithm and the chaptering approach introduces tight constraints so that it seems that the main effect of GPT-4 is 'simply' to turn the abstract narrative into natural language. However, recently there have been concerns raised about the practice of the field as a whole relying on LMs for every stage of generation, evaluation, and so on (see [1]) so it would be good to discuss this. Indeed the jump in performance from LLama 2 to GPT-4 is large enough to make me a bit suspicious that the use of GPT-4 in the dataset construction is biasing these results in a subtle manner. Would it have been helpful to have used a mixture of LMs to generate the text for the narratives?

Author response:
Reiterating a bit from section 2 “Why a synthetic benchmark”, because the human eval is at 96% on average, we believe that language models should be able to solve MuSR regardless of how it was constructed. The structured nature of the creation process means that GPT-4 is not quite as privileged as if we simply generated instances from GPT-4 directly and asked GPT-4 to solve them, but we agree it still might find the text more “in-distribution” than other language models do.

Maybe! This would be an exciting experiment to try out. We used GPT-4 because it generated the highest-quality narratives.

Reviewer comment (Experiments):
A minor point: It seems the second part of the benchmarking is not as complete as may be hoped for. I do appreciate the attempt to explore neuro-symbolic approaches, and the authors used a fair number (5), but why was each only applied to a limited number of domains (were there limitations to each that prevented this?). Also, LAMBADA [2] is a recent general purpose approach to neurosymbolic reasoning that seems like it would be very helpful in all 3 domains (since it performs backward reasoning which is generally more effective on complex reasoning problems) that the authors could have tried.

Author response:
Yes. Each neuro-symbolic method is best fit for the domain we tested on.  Expanding them to work on other domains would require engineering a new algorithm which wasn’t the point of the evaluation.  

For example, SymbolicToM wouldn’t work on Murder Mysteries since the facts for motive/means/opportunity could span multiple paragraphs that are unrelated. Thus, no graph could be constructed.  Similarly, deconstructed prompting would require modification for object placement and team allocation as they require some form of decision-making after recovering higher-level facts.

Unfortunately, LAMBADA does not allow for natural narratives as input; it needs structured input as rules & facts. An extended version of this algorithm would be exciting to try in future work.