Paper: Measuring Vision-Language STEM Skills of Neural Models (ICLR-2024)

Reviewer comment (Presentation):
[Missing Limitation and Potential Impact] It would be nice to show the limitations and potential impacts of the proposed dataset.

Author response:
**Limitation and impact**: We understand the importance of limitations and potential impact. We have included the discussion in the ethics statement. We plan to expand the discussions to focus on the potential use of models after deep analysis of our dataset. For example, it will follow common limitations of the foundation models such as misuse, bias, and hallucinations [4, 5, 6, 7, 8].

Reviewer comment (Evaluation):
[Confusions in Figure 9] In Figure 9, for RN50, RN101 and ViT-B/16, the performance decreases when the model size becomes larger. Could the authors explain this a bit more?

Author response:
**Figure 9 clarification**: Thank you for pointing this out. Although the general trend is that the model performance increases when models become larger, as the scaling law indicated [9],  the model’s performance does not only depend on the size, other factors could also contribute (e.g., the length of the training). It is possible for a model with fewer parameters to outperform another model, which might explain why ViT-B/16 is outperformed by RN50 and RN101.

Reviewer comment (Experiments):
[Unconvincing Human Study] In the human study, the inclusion of only seven university students may not be fully representative, potentially limiting the generalizability of the findings to the broader population and raising considerations about the robustness of the observed human performance.

Author response:
**Human study**: As described in Sec. 2.4 and Sec. 3.3, our exam scores are calculated based on millions of IXL elementary users [10]. This is one of the main advantages of using our dataset, as the human performances are simulated in a really large environment. The IXL SmartScore [10] serves as the main resource for our human performance comparison. The performance of the graduate students mainly serves as an expert performance, setting up an upper bound for human performance, while the IXL exam scores are from elementary students. We agree that the pool of test takes can be expanded to more people, but it seems a common approach in recent literature such as MATH [11]. We will consider adding more expert test takers in future work.

Reviewer comment (Experiments):
Could the authors show many examples for better understanding the dataset?

Author response:
**More dataset examples**: Due to space limitations, we have included dataset examples for all skills of our dataset in Appendix C and a case study in Section 3.4.

Reviewer comment (Experiments):
Despite the analyses of foundation models, e.g., CLIP and ChatGPT, there are still some new and better models missing, such as EVA-CLIP, Kosmos-2, BLIP-2 and etc. What are the results of these models?

Author response:
**Other models**: We agree that the evaluation of more models will be valuable. The main contribution of our paper is the creation of our dataset to evaluate the multimodal STEM understanding. We choose the foundation models like CLIP for a pilot study on this dataset. We anticipate performance improvements from the recent models and will add the discussion on the models you suggested in the new version.

Reviewer comment (Experiments):
Does the quality of the caption model influence the performance of zero-shot performance on vision-language models. How about changing the caption model to a more accurate one, e.g., BLIP-2, GPT4 (I understand that the API may be not available now)?

Author response:
**Caption model**: Thank you for the helpful suggestion and we believe that caption models will influence the performance to some extent. As discussed in Appendix B.5, it seems the textual input (e.g., prompt) does not impact the performance significantly. But we will investigate this in future work.

Reviewer comment (Experiments):
What is the performance on other datasets after finetuning on STEM, e.g., VQA (2015)? Which is better when comparing with the model before finetuning? It would be important to verify that STEM is important and finetuning on STEM can maintain or improve the generalization of the model.

Author response:
**Performance of the finetuned model**: We agree that the generalization after finetuning on STEM is important. Our main contribution is the creation of the STEM dataset. The dataset is designed for evaluating the vision-language STEM Skills of neural models. Therefore, the generalization might be out of the scope of the current paper. However, we plan to add the results on VQA (as you suggested) in the new version.  

Per your suggestion, we added VQA results of the zero-shot CLIP model and models finetuned on each subject in Appendix B.7. The average increase of the finetuned models over the zero-shot setting is 1.2%. This shows finetuning on our dataset can potentially help improve the generalization of models.

Reviewer comment (Evaluation):
The paper does not present strong enough evidence for its claims. The human evaluation part is not clear enough. Therefore, I am not convinced that current neural models fall far behind the human performance on the STEM dataset.

Author response:
**Human evaluation clarification**: We have included descriptions of the human evaluation e.g., setup, procedure, and metrics in Sec. 2.4 and Sec. 3.3. Overall, not only the average exam score of the tested models is below 60, but also their average accuracy is below 60 (Figure 1b). The results support our findings that model performance falls behind human performance on our dataset. Regarding the sources of the human scores, first, the exam scores of elementary students are sourced based on the IXL SmartScore [1] from grades 1 to 6 and we chose 90 as the reference. So if a skill belongs to grade 1, then 90 is for students at grade 1. According to the official guideline of SmartScore [1], a score higher than 90 is considered excellence for a mastered skill, and this standard also takes into account the grade level of the test takers, which are elementary students if we focus on grades 1 to 6. It means that a score of 90 is achievable and a reasonable goal for elementary students. Note that this score is still at the elementary level, and is considered excellent only for elementary students. Second, for expert-level human performance, we do not consider the grade level information so we test graduate students to get the upper bound performance of humans, which we refer to the expert-level performance. Since the SmartScore is only designed for the corresponding grades, we use graduate students’ test scores for the expert-level evaluation. We will add these clarifications in the new version.

Reviewer comment (Experiments):
When benchmarking on SOTA models, the paper does not use the newest language model, i.e., GPT-4. If GPT-4 is tested, we might see quite significantly different results.

Author response:
**GPT-4**: The main factor for excluding the GPT-4 results was cost. The approximate cost of evaluating GPT-4 on our benchmark is $6500. Although we don’t include GPT-4, we believe the main contribution of our paper is the multimodal dataset. Given the current results including GPT-3.5 Turbo, we expect the latest models like GPT-4 could improve the performance on our benchmark. This is indeed aligned with the value of contributing our dataset to the research community.

Reviewer comment (Evaluation):
The paper touched the bad case analysis, but not deep enough. As part of its biggest contribution, it will be better to make a more thorough analysis on the examples and patterns where the models fail.

Author response:
**Error analysis**: We did include the overall error analysis in Sec. 3.4 and more detailed analysis in the appendix (Appendix B.3) due to space limit. We also investigate the performance variance along many other aspects including questions with different skills (Table 8, Figure 23-28), different numbers of answers (Figure 17), questions with different lengths (Figure 18), different question types (Figure 19), different grades (Figure 21). In addition, we will add more error analysis including the top-5 error rate skills for science and math subjects in the Appendix and give an example for each skill.

Reviewer comment (Evaluation):
The claims that the paper concluded seem inconsistent with the results it presented. In the abstract, the paper claims that "these models are still well below (averaging 54.7%) the performance of elementary students, not to mention near expert-level performance." My questions and concerns are: How do we define "the performance of elementary students"? I see that the paper compares the model performance with a score of 90, which "according to IXL, is considered excellent for a mastered skill", but it does not make sense to take a score of excellence as "the performance of elementary students", because it is an "expert-level" human performance.

Author response:
**Human evaluation clarification**: We have included descriptions of the human evaluation e.g., setup, procedure, and metrics in Sec. 2.4 and Sec. 3.3. Overall, not only the average exam score of the tested models is below 60, but also their average accuracy is below 60 (Figure 1b). The results support our findings that model performance falls behind human performance on our dataset. Regarding the sources of the human scores, first, the exam scores of elementary students are sourced based on the IXL SmartScore [1] from grades 1 to 6 and we chose 90 as the reference. So if a skill belongs to grade 1, then 90 is for students at grade 1. According to the official guideline of SmartScore [1], a score higher than 90 is considered excellence for a mastered skill, and this standard also takes into account the grade level of the test takers, which are elementary students if we focus on grades 1 to 6. It means that a score of 90 is achievable and a reasonable goal for elementary students. Note that this score is still at the elementary level, and is considered excellent only for elementary students. Second, for expert-level human performance, we do not consider the grade level information so we test graduate students to get the upper bound performance of humans, which we refer to the expert-level performance. Since the SmartScore is only designed for the corresponding grades, we use graduate students’ test scores for the expert-level evaluation. We will add these clarifications in the new version.

Reviewer comment (Evaluation):
The claims that the paper concluded seem inconsistent with the results it presented. In the abstract, the paper claims that "these models are still well below (averaging 54.7%) the performance of elementary students, not to mention near expert-level performance." My questions and concerns are: Where is the number 54.7% from?

Author response:
**The number of 54.7%**: Thank you for pointing this out. We calculate the average exam scores of zero-shot CLIP from grades 1 to 6 which equals 40.8. This is 54.7% lower than the exam score of 90.

Reviewer comment (Evaluation):
The claims that the paper concluded seem inconsistent with the results it presented. In the abstract, the paper claims that "these models are still well below (averaging 54.7%) the performance of elementary students, not to mention near expert-level performance." My questions and concerns are: Besides, it doesn't seem fair to use Ph.D. students' test scores for the human evaluation of these elementary school questions.

Author response:
**Human evaluation clarification**: We have included descriptions of the human evaluation e.g., setup, procedure, and metrics in Sec. 2.4 and Sec. 3.3. Overall, not only the average exam score of the tested models is below 60, but also their average accuracy is below 60 (Figure 1b). The results support our findings that model performance falls behind human performance on our dataset. Regarding the sources of the human scores, first, the exam scores of elementary students are sourced based on the IXL SmartScore [1] from grades 1 to 6 and we chose 90 as the reference. So if a skill belongs to grade 1, then 90 is for students at grade 1. According to the official guideline of SmartScore [1], a score higher than 90 is considered excellence for a mastered skill, and this standard also takes into account the grade level of the test takers, which are elementary students if we focus on grades 1 to 6. It means that a score of 90 is achievable and a reasonable goal for elementary students. Note that this score is still at the elementary level, and is considered excellent only for elementary students. Second, for expert-level human performance, we do not consider the grade level information so we test graduate students to get the upper bound performance of humans, which we refer to the expert-level performance. Since the SmartScore is only designed for the corresponding grades, we use graduate students’ test scores for the expert-level evaluation. We will add these clarifications in the new version.

Reviewer comment (Experiments):
What is the reason of not experimenting on GPT-4? It would be beneficial to test it since its performance has largely increased compared to GPT-3.5.

Author response:
**GPT-4**: The main factor for excluding the GPT-4 results was cost. The approximate cost of evaluating GPT-4 on our benchmark is $6500. Although we don’t include GPT-4, we believe the main contribution of our paper is the multimodal dataset. Given the current results including GPT-3.5 Turbo, we expect the latest models like GPT-4 could improve the performance on our benchmark. This is indeed aligned with the value of contributing our dataset to the research community.