From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: The submission presents a new dataset of machine learning exam questions. The paper initially had a wide divergence in reviewer opinion, reflecting disagreements over the usefulness of an additional dataset. Machine learning exams comprise an interesting and challenging domain for evaluating models. However, the proposed dataset is quite small, which reviewers felt would limit interest. Given the vast number of existing datasets, new datasets need to be compelling to make an impact. It would significantly improve the submission to expand the dataset, perhaps to a wider range of subjects than machine learning. However, the idea has much potential, and I would encourage the authors to continue working in this direction and resubmit.
justification_for_why_not_higher_score: After discussion, reviewers felt that the paper needs further work to be accepted. Particularly, expanding the dataset to be larger or cover exams in other subjects would substantially strengthen the paper.
justification_for_why_not_lower_score: n/a
summary_of_AC-reviewer_meeting: During discussion, the consensus was that overall the idea here is great, but the execution needs further work. There were legitimate differences of opinion over the importance of dataset papers, but overall the consensus was that the dataset may be unlikely to be widely used in its current form, but that the authors should be encouraged to expand the dataset and resubmit.

From: ICLR.cc/2023/Conference/Paper6152/Authors
title: Response to reviewers' comments
comment: We thank the reviewers for their time and have addressed all the comments.

Reviewer ffgb
1. Reviewer's comment: "Seems like one more dataset from BIG-bench, https://github.com/google/BIG-bench . Not too much novelty"
Response: BIG-bench is based on the paper: https://arxiv.org/pdf/2206.04615.pdf by Srivastava, Aarohi, et al. "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models." arXiv preprint arXiv:2206.04615 (2022).
BIG-bench consists of a broad range of task topics, including linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, and software development; however, it does not consist of university-level final exams. Our work is vastly different from the BIG-bench dataset. We curate a challenging dataset of questions from university machine learning final exams in top universities' machine learning classes, specifically at MIT, Harvard, and Cornell - see the updated version.
2. Reviewer's comment: "Not sure why Figure 1 is shown in the paper. It only shows students taking the final exam which is not related to the dataset."
Response: We removed figure 1 and improved the paper's readability.

Reviewer FkVg
1. Reviewers comment: The references should be in parentheses
Response: We fixed the references to be in parenthesis (\citep instead of \cite)
2. Reviewer's comment: "In Table 5, there is the exact same result between GPT3-ZS and GPT-3 FS in Fall 2021, is that a typo?"
Response: This typo has been fixed with the correct result.
3. Reviewers comment: "We extended this work to include final exams from Harvard's and Cornell's machine learning classes as well. "Are these questions included or to be included in the future?
Response: We included the Harvard and Cornell final exams. Please see the revised version including a dozen exams spanning 17 topics.

Reviewer bhJL
1. Reviewer's comment: “A lot of details are missing with regards to how the dataset was curated. An illustration/figure to show the method used in this work would be advantageous.”
Response: We have added details explaining the curation of the dataset. Please see the supplementary material for data and code.
2. Reviewers comment: “The recommended models do not consider questions that rely on images for their automated solution.”
Response: Images make up only 27.5% of the questions. Gleaning text from input images is beyond the scope of this work.

From: ICLR.cc/2023/Conference/Paper6152/Reviewer_ffgb
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors provide a new dataset and benchmark of questions from machine learning final exams and code for automatically answering these questions and generating new questions. They build baselines by applying zero-shot and few-shot learning to GPT-3 and Codex, adding chain-of-thought prompting for GPT-3. 
strength_and_weaknesses: Strength:
1. A quite interesting dataset
2. Solid baselines

Weakness:
1. Seems like one more dataset from BIG-bench, https://github.com/google/BIG-bench . Not too much novelty
2. The paper is not well-written. Not sure why Figure 1 is shown in the paper. It only shows students taking the final exam which is not related to the dataset.
clarity,_quality,_novelty_and_reproducibility: What's the difference between the proposed dataset and the BIG-bench datasets?
summary_of_the_review: I think this is an interesting dataset, but it is like one dataset from BIG-bench. I think we don't need every dataset from BIG-bench to be published.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper6152/Reviewer_FkVg
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors introduce a new university exam dataset spanning questions on 12 machine learning topics covered at MIT's Introduction to Machine Learning course. This joins a thin body of work of exam datasets that have mostly focused on high school and lower levels. The dataset and code are open-sourced and uploaded as part of the Supplemental Material. The exams span multiple quarters of this class. The experiments include benchmarking LMs on this dataset, as well as generating new exam questions. The results show that the generated questions are of equal quality as the human-written ones, have the same average difficulty rating, and get recognized as human-written with the same probability.

UPDATE: After discussion with my fellow reviewers and area chair, I agree that there are valid concerns about the size of the dataset. While we all praise the authors for the originality of the work and consider it valuable to the community, the dataset as presented may be too small for the community to use. We recommend trying to expand the dataset and resubmitting the paper.
strength_and_weaknesses: Strengths:
- Much needed dataset for the community and interesting topic
- Strong GPT- and Codex-based baselines, as well as baselines with Open LMs
- Extensive experiments provide benchmarking results to start off of
- Great experiments and promising results with generated questions

Weaknesses:
- No major weakness

Notes:
- The references should be in parentheses
- In Table 5, there is the exact same result between GPT3-ZS and GPT-3 FS in Fall 2021, is that a typo?
- "We extended this work to include final exams from Harvard’s and Cornell’s machine learning classes as well. " Are these questions included or to be included in the future?
clarity,_quality,_novelty_and_reproducibility: The work is clear, well evaluated, novel and reproducible with the supplemental material.
summary_of_the_review: This work introduces a novel dataset of university level machine learning exam questions and answers. This is a much needed resource for the community, and the results of the benchmarking is valuable, as well as the ones for exam question generation.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper6152/Reviewer_bhJL
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This work presented a new dataset that contains final exam questions, and compares the performance of popular transformer models for answering them. The dataset includes questions that arise from a wide range of topics in machine learning including loss functions, RNNs, reinforcement learning, and etc. 
strength_and_weaknesses: Strength: This is the first study to present a dataset of machine learning final exams
Weakness 1: A lot of details are missing with regards to how the dataset was curated. An illustration/figure to show the method used in this work would be advantageous.
 Weakness 2: The recommended models do not consider questions that rely on images for their automated solution. 
clarity,_quality,_novelty_and_reproducibility: 
This work is of high quality since it presents a novel approach by being the first study to curate a machine learning dataset of final exams. This work is also reproducible since the authors made their data and code publicly available for the machine learning community. 
summary_of_the_review: This paper presents an appealing and novel approach by generating a new dataset of machine learning final exams. This work also provided a benchmark of models for answering the final exam questions. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
details_of_ethics_concerns: None. 
recommendation: 8: accept, good paper