Paper: NLPBench: Evaluating Large Language Models on Solving NLP Problems (ICLR-2024)

Reviewer comment (Reproducibility):
How did you get access to the final exams included in the dataset? How do you ensure that these exams were not already online and that some recent models like GPT-4 have already included them in their training dataset? It was not clear from the paper how you checked that. You mentioned that you curate questions that are not readily accessible online and couldn’t be easily extracted or transformed into text, but how do you measure that specific content cannot be easily extracted or transformed into text? What reliable techniques did you use to do that?

Author response:
We collected 1,000 NLP exam questions over a decade, from 2013 to 2023. For the integrity and fairness of the final exams, these questions have not been available online, so they cannot be part of other datasets, and there is no risk of training data contamination. Professors and teaching assistants (TAs) contributed new questions to this repository each semester. This extensive set comprises three types of questions: 1) Online-sourced questions, refined by TAs to differentiate them from their original versions, ensuring their uniqueness for final exams, along with a thorough verification of answers. 2) Original questions formulated by professors, drawing from their teaching experience. 3) Original questions developed by TAs. Over 30 TAs have been involved in the completion of this dataset. We have made this clear in the updated version.

Reviewer comment (Experiments):
In what years were these final exams given? This information might help in the development of future related datasets.

Author response:
We collected 1,000 NLP exam questions over a decade, from 2013 to 2023. For the integrity and fairness of the final exams, these questions have not been available online, so they cannot be part of other datasets, and there is no risk of training data contamination. Professors and teaching assistants (TAs) contributed new questions to this repository each semester. This extensive set comprises three types of questions: 1) Online-sourced questions, refined by TAs to differentiate them from their original versions, ensuring their uniqueness for final exams, along with a thorough verification of answers. 2) Original questions formulated by professors, drawing from their teaching experience. 3) Original questions developed by TAs. Over 30 TAs have been involved in the completion of this dataset. We have made this clear in the updated version.

Reviewer comment (Experiments):
In the data selection process, you mentioned that you selected 400 questions out of 1000 total questions, what were the criteria? Was it a random selection?

Author response:
In the raw data, many questions include figures and tables in the context. We discard those questions since we cannot easily translate them into text formats like images and tables. We have made this clear in the updated version. Considering that multimodal LLM (like GPT-4V) has been announced after this work, we will add more experiments of multimodal models in our future work.

Reviewer comment (Experiments):
The dataset size looks small; why in Table 6 was there not any comparison for the NLPBench size to other benchmarks?

Author response:
A good benchmark dataset should be able to reveal the differences among LLMs through their performance. According to Table 3, Figures 2 and 3, NLPBench can achieve this with the current amount of questions. We also provided in-depth error analyses in Section 4. Therefore, we did not focus on tweaking the number of questions in each dataset but more on the features (e.g., new types of questions or new evaluation processes).

Reviewer comment (Novelty):
The very closely related benchmark (SciBench) also includes math problems; did you check if NLPBench's math-related problems are not also in SciBench? I did not see this evaluation in the paper.

Author response:
SciBench extracts questions from ten textbooks in the fields of physics, chemistry, and pure math, while the questions in NLPBench are created by professors and TAs. Therefore, there is no overlap between SciBench and NLPBench.

Reviewer comment (Experiments):
How many expert human annotators were involved in the dataset processing?

Author response:
The main contributors to the original dataset are at least 30 TAs, each of them completing part of the questions’ or answers’ completion. Furthermore, three human experts are involved in the data processing stage: one for transiting the original format into the text format with a front-end interface and two for checking the results. It took about one week to complete the development of the interface and dataset transformation. Two of them were further involved in the free-response answer evaluation and error analysis stage. Evaluating each answer needs ~5 minutes per annotator, and it took about 13 hours per annotator to finish the evaluation.

Reviewer comment (Writing):
Minor typos — Page 3 in the paragraph that describes the complex structure: multi-tern -> multi-term; Page 4: zero-shot and few-shot(FS+ToT, FS+ToT) -> (ZS+ToT, FS+ToT)

Author response:
Thank you for your valuable and meaningful feedback and for pointing out typos in our work; we updated our manuscript, reflected reviews and fixed typos, and carefully addressed your questions and concerns below.

Reviewer comment (Novelty):
The gap filled by the proposed by introducing NLPBench (Table 6) is rather narrow and needs stronger justification. It covers a broader context than the claimed main focus of NLP-related topics such as Math.

Author response:
Unlike pure math, math problems in the NLP domain are closer to applications. For example, the language generation models are constructed based on the conditional probability. The context of the math problem in NLPBench is strongly correlated with the NLP context (see examples in Figure 1). Therefore, in general, NLPBench covers different aspects of  NLP-related topics, including related math questions.

Reviewer comment (Novelty):
The analysis conclusion are mostly known, such as small LLMs have inconsistent results with advanced prompting. LLM limitation on problem-solving tasks.

Author response:
To the best of our knowledge, this is the first LLM benchmark in the domain of NLP, which not only verifies several long-discussed conjectures but also shows several novel discoveries that have not been explored by other works by the time we submit. For example, (1) The failures of chain-of-thought and tree-of-thought on 13B-sized, 70B-sized, or even larger models, which set up a boundary for the application of advanced prompting strategies; (2) We also discovered a significant drawback in logical thinking skills, which can be further used to, for example, explain why training on some specific resources (like code) will improve the performance on others because many logical symbols in those resources may be involved in the construction of logical thinking skills and can also be used to instruct the pre-training and post-training stage of LLMs.

Reviewer comment (Evaluation):
How open questions have been evaluated? The paper mentions only human annotation of errors.

Author response:
We first prompt a GPT-4 to evaluate the free-response questions’ answers, and two human domain experts further read through and carefully provide feedback. On average, for each question, they need ~5 minutes; it took about 13 hours per annotator to finish the work. They evaluated answers in the following procedure: (1) figure out the scoring point from the ground truth answer, and (2) compare the generated answer with those scoring points at the semantic level.

Reviewer comment (Experiments):
Under Inaccessibility, details are missing. How did you make sure that the questions are not part of other datasets? What are risks of contamination by LLM training data?

Author response:
For the integrity and fairness of the final exams, these questions are made originally or refined by professors and teaching assistants and unavailable online, so they cannot be part of other datasets, and there is no risk of contamination of training data.

Reviewer comment (Experiments):
I expected that the top-3 models is including PalM but in 4.2, Llama-2 was taken instead. Is there a justification for this choice?

Author response:
We included LLAMA-2 (70b) in the error analysis because LLAMA-2 (70b) is an open-source LLM, which will make our evaluation more trackable and will not influenced by the online API update. On the other hand, we chose GPT-3.5/4 because (1) it is one of the most common online API LLMs; and (2) unlike PaLM, its API is accessible to all users.

Reviewer comment (Evaluation):
Could this approach be adapted to other areas or domains? Would be nice to discuss about it and also the transfer cost.

Author response:
Our automated evaluation can be directly adapted to other domains. We provided a complete codebase, which only requires minor changes for the new domain (e.g., rewriting the prompt and a JSON formatted dataset) to complete automated evaluations and obtain results like Table 3, 4, and 5. The human evaluation (like in Figure 5 and the evaluation of free-response answers) can be costly, considering that the domain-specific evaluations may require experienced domain experts and cannot be approached automatically. We tried our best to reduce the evaluation cost by involving GPT-4 as an assistant and caching the result with the same parameters by leveraging AutoGen.