Paper: Efficiently Measuring the Cognitive Ability of LLMs: An Adaptive Testing Perspective (ICLR-2024)

Reviewer comment (Reproducibility):
How does the "human response" calibration in IRT work? Does it mean that the question difficulty parameter is pre-estimated using students' responses, and is then fixed during the subsequent CAT procedures?

Author response:
Yes. Just like the general CAT method, we can use human response to estimate the question parameters. For example, we can use the BCE loss on the binary responses data to estimate the parameters, and fix them for direct use in the subsequent CAT process.

Reviewer comment (Reproducibility):
Following up the previous question, what IRT parameters are fixed and are estimated at each estimation step \(t\) during CAT? The reviewer's impression from the paper is that \(\theta\) is being estimated, \(\beta\) is fixed (or pre-estimated or calibrated), but the status of the remaining parameters is unclear.

Author response:
All parameters (except ability $\theta$) need to be pre-estimated. All the parameters of question are fixed (e.g., difficulty $\beta$, discrimination $\alpha$, guess factor $c$). The only one that can update continuously during the CAT process is the ability estimate $\theta$.

Reviewer comment (Experiments):
Is the calibration process done independently for each dataset, or are the responses from all three datasets combined to estimate \(\beta\)?

Author response:
Yes, the difficulty of questions in different datasets/domains cannot be aligned, so parameter estimation are independent for each dataset.

Reviewer comment (Theory):
The calibration process is done using human data, and the calibrated IRT model is used to evaluate LLMs. The reviewer thinks the underlying assumption is that LLMs behave like humans and thus can be evaluated using a human-data-calibrated IRT model. The reviewer questions whether such an assumption is valid or can be verified. If not, it might not make sense to use a human-calibrated IRT model to evaluate LLMs because they may not be comparable. The reviewer asks whether it would make more sense to have LLMs answer the questions, use these answers (in addition to human ones) to calibrate the IRT model, and then evaluate.

Author response:
This is a very good question. We use students' data to estimate parameters and apply them to the LLM, because we want to achieve human and LLM comparability. "Cognitive Ability" is actually viewed from a human perspective, even if ChatGPT can answer global challenges correctly, but fails to answer simple addition and subtraction, we still think it is problematic in cognitive ability. If we want to achieve a simple ranking between models, of course we can also use model response data to estimate parameters, or even use a pre-trained difficulty predictor to predict the characteristics of any question in the world. This depends on the purpose of evaluating the LLM.

Reviewer comment (Experiments):
From Appendix A.3, it seems that some of the questions in the MATH dataset contain figures. How are models evaluated when the questions contain both text and figures? The reviewer’s impression is that all models being evaluated are text-only models.

Author response:
Sorry for causing you misunderstanding. The figure in Appendix A.3 do not exist in the dataset. For readers to better understand, we drew this picture manually to illustrate fickle-minded issue of ChatGPT.

Reviewer comment (Presentation):
In Figure 4c, what does the student curve mean? Is it one student, or the average of many students (the caption says "students" whereas the text says "student")? The student curve appears to be more uncertain; does it imply that the student(s) is/are (on average?) more careless than the model?

Author response:
This figure represents the estimated uncertainty averaged over many students. A smaller value indicates a more precise or reliable ability estimate [1] and this is not directly related to the degree of carelessness. We find that after adding various types of perturbations (guess and slip) the curve will be lower and the estimate will be more reliable. The reason may be that the parameters in IRT (e.g., guess and discrimination factor) can directly model these perturbations. Actually, by comparing the positions of four curves, the ChatGPT is more careless than the normal students.

Reviewer comment (Evaluation):
The emphasis on evaluation efficiency was not immediately evident to me. While I recognize that inference can carry significant costs, it's relatively more economical compared to the (pre-)training of LLMs. Furthermore, evaluations are typically conducted once, rendering them comparatively affordable. I can also imagine LLM providers would want to evaluate their LLMs very accurately, to demonstrate that their LLM is better than others with high confidence. In contrast to the fact that humans become tired after answering too many questions, LLM performance will not degrade as we ask more questions. Given these considerations, it would be helpful if the paper can provide more motivating situations where efficiency can be important.

Author response:
Thank you for your valuable feedback on our paper. We appreciate the time and effort you have invested in reviewing our work. We understand that there might be room for improvement in terms of clarity and depth.

[The necessity of evaluation efficiency.]

- **Save the cost of model inference**. Simply put, it saves money. It is difficult to estimate the time and money costs incurred by model inference on such a large scale as GPT3.5. We can calculate the cost of using the API. Take WMT benchmark in machine translation as an example. It has a total of 11 million tokens. Considering the API price of ChatGPT, a complete evaluation of one model costs 20$. This seems acceptable, but if we want to evaluate and rank 10 models on 10 different domain benchmarks, it may not be. Recently, more and more comprehensive large benchmarks, various LLMs (general/specific to a certain domain) have emerged, and the iteration speed of versions of the same model is also accelerating. In order to achieve fast, timely and accurate assessment, CAT is a good choice (obviously not the only).

- **Save the cost of human experts**: In order to automatically evaluate the correctness of model responses, most benchmarks can only include multiple-choice questions. For free response questions (e.g., subjective, creative questions) that truly test the model's generation capabilities, human experts' judgments are more reliable than GPT-4 and ordinary people (on crowdsourcing platforms) as stated in the paper. However, the energy of human experts is limited (humans may become tired as you said), we need to find truly valuable questions for LLM to answer and let the experts judge their responses.

- Last but not least, as mentioned in the paper, evaluation methods based on cognitive science such as CAT can be used for building a scientific evaluation paradigm, which can achieve comparisons between models and humans, and between models.

Reviewer comment (Reproducibility):
What is the ChatGPT model? Is it gpt-3.5-turbo?

Author response:
> **Q**: What is the ChatGPT model? Is it gpt-3.5-turbo?

**A**: Yes. We will include more LLMs for adaptive evaluations in future work.

Reviewer comment (Evaluation):
One main/initial advantage of using CAT for evaluation is to save human effort during evaluation in human studies. For example, to evaluate the student's academic performance (e.g., GRE, which is the example used in the paper), CAT can shorten the length/time duration of the evaluation. It is unclear why this advantage applies to LLM evaluation, since the candidates are machines rather than humans, and thus "human effort" or "labor cost" may not be a significant concern. In LLM evaluation, evaluation time could be easily shortened by running models in parallel.

Author response:
> **Concern 1**: The necessity of evaluation efficiency.

- **Save the cost of model inference**. Simply put, it saves money. It is difficult to estimate the time and money costs incurred by model inference on such a large scale as GPT3.5. We can calculate the cost of using the API. Take WMT benchmark in machine translation as an example. It has a total of 11 million tokens. Considering the API price of ChatGPT, a complete evaluation of one model costs 20$. This seems acceptable, but if we want to evaluate and rank 10 models on 10 different domain benchmarks, it may not be. Recently, more and more comprehensive large benchmarks, various LLMs (general/specific to a certain domain) have emerged, and the iteration speed of versions of the same model is also accelerating. In order to achieve fast, timely and accurate assessment, CAT is a good choice (obviously not the only).

- **Save the cost of human experts**: In order to automatically evaluate the correctness of model responses, most benchmarks can only include multiple-choice questions. For free response questions (e.g., subjective, creative questions) that truly test the model's generation capabilities, human experts' judgments are more reliable than GPT-4 and ordinary people (on crowdsourcing platforms) as stated in the paper. However, the energy of human experts is limited, we need to find truly valuable questions for LLM to answer and let the experts judge their responses.

- Last but not least, as mentioned in the paper, evaluation methods based on cognitive science such as CAT can be used for **building a scientific evaluation paradigm**, which can achieve comparisons between models and humans, and between models.

Reviewer comment (Evaluation):
If every model has 20%-30% difference in their evaluation questions (as mentioned in the paper), it is unclear how to ensure the evaluation is fair to each model. Questions arise regarding whether the authors calibrate the final score by difficulty level or other factors. Even if the method can produce more accurate results, it may not make it easier to compare LLMs with each other (e.g., in a leaderboard), because they are evaluated on different test sets. This may introduce high communication costs, and while calibration might mitigate this, it could also introduce more noise into the evaluation pipeline.

Author response:
> **Concern 2**: Fairness about evaluation.

We deeply appreciate your thoughtful consideration of the fairness issues associated with CAT itself. Like recommendation system (different users will be recommended different items), CAT is also a personalized question in Education and has fairness issues of course e.g., [1], [2], which is another interesting topic. The typical application of CAT is GRE exam: when a test-taker begins the GRE, the computer provides a question of medium difficulty. If the response is correct, the computer presents a slightly more challenging question. Conversely, if the response is incorrect, the following question will be slightly easier. That's why some candidates may perceive the test becoming progressively more easier and their final score will be relatively low. The final score or ranking is related to the accuracy rate and the difficulty of the questions (answered by him). The CAT tests of each candidates are independent, and there are no communication costs. As mentioned in the paper, in Stage 1, we can collect some responses to estimate and fix the question parameters. Then, any model (that may do not appear in Stage 1) can successfully proceed to Stage 2 testing and evaluation. In an improved version, we will give a brief overview of fairness issues in CAT itself.

Reviewer comment (Evaluation):
Are there any other baseline methods? In Table 1, the paper shows some rankings in terms of CAT evaluation results. If the models were evaluated with the original static test sets of these tasks, would the same rankings be obtained? If not, what could be the reason?

Author response:
> **Q1**: Are there any other baseline methods? 

**A1**: In this work, No. Because this paper is the first attempt to apply CAT to the assessment of LLM. We abandon the complex methods and technical implementations in CAT and choose the simplest Fisher selection method, which is the most interpretable among them. We would like to introduce the CAT idea to more LLM researchers. We are trying to introduce the CAT idea to more LLM researchers and ICLR readers in a friendly way.
 
 
> **Q2**: In Table 1 you show some rankings in terms of CAT evaluation results. What if you evaluate your models with the original static test sets of these tasks? Will you arrive at the same rankings? If not, what could be the reason?

**A2**: Sure. In fact, we compared two leaderboards generated using the full dataset (containing at least thousands of samples) and 20 questions selected by CAT. We found that the consistency between these two leaderboards can reach 100%. We did not add this results to the paper because the number of candidate LLMs is small, making this ranking unreliable. On the other hand, as stated in the paper, all questions in the question bank/dataset are not equally important. Some low-quality questions (or noise data) will interfere with the evaluation and ranking. Filtering out these data is what CAT is trying to do. Therefore, we resort to the metric in traditional CAT field, such as simulation experiments (Figure 4a) and estimation uncertainty analysis (Figure 4c).

Reviewer comment (Experiments):
If we are assessing Subject Knowledge, can't that be done by MCQA, which doesn't require expert annotation once the benchmark is created? If we are assessing programming, can't we check that with pre-specified unit tests? Indeed, the appendix shows that the unidentifiable datasets used in this paper are multiple choice, as opposed to free response. In these cases, how many forward passes through the model are we actually saved by CAT, and is that substantial? I can't tell from the paper. But I don't think it works to motivate a paper by saying "free response is expensive so we propose this efficient method and evaluate it on multiple choice."

Author response:
First of all, I need to clarify that CAT does not require whether the model/human responses are multiple choice or free response questions. CAT can run successfully as long as the correctness of the response can be judged by some methods. In this paper we only consider two types: multiple choice and fill-in-the-blank questions. Multiple-choice questions can be automatically judged, while the fill-in-the-blank are judged for correctness by human experts.

[The necessity of evaluation efficiency]: 
- **Save the cost of model inference**. Simply put, it saves money. It is difficult to estimate the time and money costs incurred by model inference on such a large scale as GPT3.5. We can calculate the cost of using the API. Take WMT benchmark in machine translation as an example. It has a total of 11 million tokens. Considering the API price of ChatGPT, a complete evaluation of one model costs 20$. This seems acceptable, but if we want to evaluate and rank 10 models on 10 different domain benchmarks, it may not be. Recently, more and more comprehensive large benchmarks, various LLMs (general/specific to a certain domain) have emerged, and the iteration speed of versions of the same model is also accelerating. In order to achieve fast, timely and accurate assessment, CAT is a good choice (obviously not the only).

- **Save the cost of human experts**: In order to automatically evaluate the correctness of model responses, most benchmarks can only include multiple-choice questions. For free response questions (e.g., subjective, creative questions) that truly test the model's generation capabilities, human experts' judgments are more reliable than GPT-4 and ordinary people (on crowdsourcing platforms) as stated in the paper. However, the energy of human experts is limited, we need to find truly valuable questions for LLM to answer and let the experts judge their responses.

- Last but not least, as mentioned in the paper, evaluation methods based on cognitive science such as CAT can be used for **building a scientific evaluation paradigm**, which can achieve comparisons between models and humans, and between models.

Reviewer comment (Theory):
False or at the very least fringe claims asserted without evidence or even argument: "Large language models (LLMs), like ChatGPT, have shown human-level cognitive ability" and "We believe this is just the tip of the iceberg of its latent knowledge."

Author response:
> **Q2**: false or at the very least fringe claims asserted without evidence or even argument: "Large language models (LLMs), like ChatGPT, have shown human-level cognitive ability","We believe this is just the tip of the iceberg of its latent knowledge."

**A2**: LLMs like ChatGPT, have shown human-level cognitive. How to evaluate a person's "human-level" cognitive ability? We usually conduct a series of tests on him, and the same for the LLM. OpenAI's official report has already reported that ChatGPT has achieved amazing results in various human subject examinations, and we quote it in the first paragraph; "We believe this is just the tip of the iceberg of its latent knowledge.": This is what we believe after reading so many works related to LLM's various abilities. The LLM is constantly developing and improving, and this is our opinion in this paper.

Reviewer comment (Evaluation):
Where in the paper do you explain how much efficiency we get from using this method, if this is indeed, the motivation? Doesn't the efficiency argument only hold in free response questions, as opposed to MCQA, as you've evaluated on here?

Author response:
> **Q3**: Where in the paper do you explain how much efficiency we get from using this method, if this is indeed, the motivation?

**A3**: Page 8 of the paper (Evaluation Efficiency) explains the accuracy comparison between random selection (traditional benchmark) and adaptive testing. We can conclude that for a given test length or sample size limit, the CAT method is much better than random selection in terms of ability estimation accuracy.