Paper: ARB: Advanced Reasoning Benchmark for Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
As the limitation states, this benchmark could have potentially be seen through the training dataset and lead to the contamination problem. For example, an answer only have the correct final step but not the correct intermediate step.

Author response:
Thank you for pointing this out. We do not have access to the training set of any of the tested models. In spite of this, we can test for memorization in this manner: we ask models to complete problem statements randomly sampled from the dataset, and compare their outputs against the true continuation of the problem statement. (Please see Appendix G for details.)

As the similarity scores are quite low, we claim that models we test do not easily complete the exact problem statements, which rules out the naive memorization hypothesis.

However, memorization in LLMs is a complex topic, and the fact that the text of a problem is not memorized does not necessarily mean the model has not memorized a similar problem or fact; we also discuss this in Appendix G. We are not aware of other works that successfully resolve this issue.

Reviewer comment (Experiments):
The solutions to these problems sets from textbooks may already in the training data. How to deal with this situation?

Author response:
Unfortunately, we do not have access to the training set for any of the tested models. In spite of this, we can test for memorization in this manner: we ask models to complete problem statements randomly sampled from the dataset, and compare their outputs against the true continuation of the problem statement. (See Appendix G for details.)  

As the similarity scores are quite low, we claim that models we test do not easily complete the exact problem statements, which rules out the naive memorization hypothesis.  

However, memorization in LLMs is a complex topic, and the fact that the text of a problem is not memorized does not necessarily mean the model has not memorized a similar problem or fact; we also discuss this in Appendix G. We are not aware of other works that successfully resolve this issue.

Reviewer comment (Evaluation):
How the current low performance on the ARB benchmark is not due to under-claiming?

Author response:
Thank you for bringing this up. We definitely agree that there is no way to prove that models cannot do this, as our prompts are surely not optimal. It is not clear to us, at this stage, what the “best” way to prompt and use models for our benchmark; and we think finding out the answer is outside the scope of this benchmark paper. We now additionally do one-shot testing on the Numerical part of the benchmark to see if anything improves, and it doesn’t.

Reviewer comment (Evaluation):
How do you grade the types of mistakes GPT-4 make in Table 3? Is it human evaluation or are there some hard-coded rules.

Author response:
It is human evaluation. We say in Section 4.2:  
"It is possible that our graders underestimate the rate of arithmetic mistakes in some cases, especially when the approach is clearly wrong, or it is not clear whether a given error is due to faulty reasoning or due to a missed term in the calculations."  

The graders were given the descriptions of the five general types of errors as guidance; in addition, the graders were familiar with the relevant section of [1]. We are not sure automated evaluation of detailed error types is possible using current technology; the only way to reduce noise would be to employ more graders.

Reviewer comment (Novelty):
I somewhat disagree with the motivation behind this work, which suggests that other benchmarks are not challenging enough. For instance, [1] and [2] have shown that even the best models either cannot generate plans or solve JEE problems, respectively, with an accuracy below 50%. While I concur that there are easier benchmarks, there are also more challenging ones available. I view this work as an additional benchmark to the existing ones that involves a broader set of domains.

Author response:
Thank you for emphasizing those lines of work! We do cite [1], which is a similar work as ours, developed concurrently.  The main differences are: (1) their quantitative problems are slightly easier than in our benchmark, and require much less deep math knowledge; (2) our math/physics questions are not multiple-choice; (3) their quantitative problems are drawn from a single source which is somewhat more likely to be thoroughly discussed online.
The updated (Oct 2023) version of [1] also discusses a version of self-evaluation in the “Can GPT-4 find and correct its mistakes” section, and find a negative result using some simple prompts. We introduce rubric-based self-evaluation as a way to make progress on this.

We also cite [3] (the followup to the mentioned [2]), which is indeed quite challenging on a second glance, but is in a different area. We moderated our language (in the paper and in the abstract) to avoid claiming that our benchmark is uniquely challenging for current models.

Reviewer comment (Evaluation):
The paper appears to lack a dataset quality verification process. Moreover, it’s not entirely clear how solving the benchmark (especially MCAT or Law questions) would correlate with reasoning ability.

Author response:
We agree that the term “reasoning ability” does not have a clear definition in the literature.
Our estimate is that some of the law problems are quite nontrivial for a non-trained human, and the sources agree. It is possible that the main abilities required to solve it are reading comprehension and pattern matching from memorizing a lot of legal cases, and not “reasoning” as in the math or physics problems.

Reviewer comment (Evaluation):
Is a non-parsable answer deemed as an incorrect answer? Or is that question not even considered in the final evaluation?

Author response:
Thank you for bringing this point up. We do discuss this in the Evaluation section.

For numerical problems, we prompt the model to respond with the final answer, and if the final expression is not parseable using Python’s SymPy library, it is marked as incorrect. (For physics problems, there is the additional step of specifying what units to use for the final answer.) 

The process is harder for symbolic answers since they have less structure and are generally harder to parse. We use SymPy to check equivalence between model output and the ground truth up to a permutation of variables. If the two are not equivalent or the model output is unparseable, then the answer is marked incorrect. Manual inspection revealed little discrepancy in the scoring compared to an "ideal" equivalence checker.

Reviewer comment (Experiments):
Given the rapid iterations of LLMs with new and more extensive training data, a diagonalization procedure could be worth investigating, as most of the benchmarks can become fodder for the LLM in the next iteration.

Author response:
We do agree that novel methods that maintain utility of all recent benchmarks after publication would be great; however, this was somewhat outside the scope of this project. We try to keep our data behind an API, accessible by requests that are not explicit links, so simple crawlers will not just ingest it; and “smart” LLM-based data collection mechanisms hopefully realize it’s a benchmark. In the current paradigm, it is impossible to prevent overfitting in evaluation if model creators are not careful.