Original: A UTOMATICALLY A NSWERING AND G ENERATING M ACHINE L EARNING F INAL E XAMS
Revised: A UTOMATICALLY A NSWERING AND G ENERATING M ACHINE L EARNING F INAL E XAMS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Can a machine learn machine learning?
Revised: Can a machine learn machine learning?

Original: We propose to answer this question using the same criteria we use to answer a similar question: can a human learn machine learning?
Revised: We propose to answer this question using the same criteria we use to answer a similar question: can a human learn machine learning?

Original: We automatically answer final exams in MIT’s recent large machine learning course and generate new questions at a human level.
Revised: We automatically answer final exams in MIT’s, Harvard’s and Cornell’s large machine learning courses and generate new questions at a human level.

Original: Recently, program synthesis and few-shot learning solved university-level problem set questions in mathematics and STEM courses at a human level.
Revised: Recently, program synthesis and few-shot learning solved university-level problem set questions in mathematics and STEM courses at a human level.

Original: In this work, we solve questions from final exams that differ from problem sets in several ways: the questions are longer, have multiple parts, are more complicated, and span a broader set of topics.
Revised: In this work, we solve questions from final exams that differ from problem sets in several ways: the questions are longer, have multiple parts, are more complicated, and span a broader set of topics.

Original: We provide a new dataset and benchmark of questions from machine learning final exams and code for automatically answering these questions and generating new questions.
Revised: We provide a new dataset and benchmark of questions from machine learning final exams and code for automatically answering these questions and generating new questions.

Original: To make our dataset a reproducible benchmark, we use automatic checkers for multiple choice questions, questions with numeric answers, and questions with expression answers, and evaluate a large free language model, Meta’s OPT, and compare the results with Open AI’s GPT-3 and Codex.
Revised: To make our dataset a reproducible benchmark, we use automatic checkers for multiple choice questions, questions with numeric answers, and questions with expression answers, and evaluate a large free language model, Meta’s OPT, and compare the results with Open AI’s GPTand Codex.

Original: A student survey comparing the quality, appropriateness, and difficulty of machinegenerated questions with human-written questions shows that across multiple aspects, machine-generated questions are indistinguishable from human-generated questions and are suitable for final exams.
Revised: A student survey comparing the quality, appropriateness, and difficulty of machine-generated questions with human-written questions shows that across multiple aspects, machine-generated questions are indistinguishable from humangenerated questions and are suitable for final exams.

Original: We perform ablation studies comparing zero-shot learning with few-shot learning, chain-of-thought prompting,GPT-3 and
Revised: We perform ablation studies comparing zero-shot learning with few-shot learning, chain-of-thought prompting,

Original:  OPT pre-trained on text and Codex fine-tuned on code on a range of machine learning topics and find that few-shot learning methods perform best.
Revised: GPT-3 and OPT pre-trained on text and Codex fine-tuned on code on a range of machine learning topics and find that few-shot learning methods perform best.

Original: We make our data and code publicly available for the machine learning community.
Revised: We make our data and code publicly available for the machine learning community.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Can a machine learn machine learning?
Revised: Can a machine learn machine learning?

Original: This work presents a new dataset of machine learning final exams with around 500 question parts and a benchmark of baselines using transformers and their respective grade performance, demonstrating that the best baseline performs at a human level.
Revised: This work presents a new dataset of machine learning final exams with 646 question parts and a benchmark of baselines using transformers and their respective grade performance, demonstrating that the best baseline performs at a human level.

Original: In university-level STEM courses, students complete assignments (including problem sets and labs) and exams throughout the course.
Revised: In university-level STEM courses, students complete assignments (including problem sets and labs) and exams throughout the course.

Original: Recent work has opened the door for a machine to solve course problem sets Drori et al. using language models and few-shot learning.
Revised: Recent work has opened the door for a machine to solve course problem sets (Drori et al., 2022) using language models and few-shot learning.

Original: However, final exams remain challenging, and this work is the first to present a structured dataset of machine learning finals and a benchmark of baseline methods for answering them.
Revised: However, final exams remain challenging, and this work is the first to present a structured dataset of machine learning finals and a benchmark of baseline methods for answering them.

Original: Final exams differ from problem sets because they serve as a benchmark of cumulative understanding of material learned over a semester and evaluate the students’ depth and breadth of expertise.
Revised: Final exams differ from problem sets because they serve as a benchmark of cumulative understanding of material learned over a semester and evaluate the students’ depth and breadth of expertise.

Original: Further, questions on final exams are longer, have multiple parts, span a broader set of topics, and are more complicated and nuanced.
Revised: Further, questions on final exams are longer, have multiple parts, span a broader set of topics, and are more complicated and nuanced.

Original: All the above holds for MIT’s  Introduction to Machine Learning class, an undergraduate courses with 500 students each semester,as shown in Figure 1, making this one of largest undergraduate courses offered.
Revised: All the above holds for MIT’s and Cornell’s Introduction to Machine Learning classes and Harvard’s Machine Learning course. These are undergraduate courses with hundreds of students each semester, making them the largest undergraduate courses offered.

Original: Introduction to Machine Learning is a core class in the computer science program.
Revised: Introduction to Machine Learning is a core class in the computer science program.

Original: The prerequisites for the course are Python Programming and Multivariate Calculus, with Introduction to Algorithms and Linear Algebra recommended.
Revised: The prerequisites for the course are Python Programming and Multivariate Calculus, with Introduction to Algorithms and Linear Algebra recommended.

Original: The class typically consists of weekly exercises, labs, quizzes, homework, a midterm, and a final exam.
Revised: The class typically consists of weekly exercises, labs, quizzes, homework, a midterm, and a final exam.

Original: There were no final exams in  Fall 2020 and Springdue to COVID-19.
Revised: There were no final exams in at MIT for Fall 2020 and Spring 2020 due to COVID-19.

Original: (no student faces or identifying information appears in the image to maintain anonymity).

Original: The exam is three hours long and is taken in an indoor track-and-field stadium accommodating many students.

Original: Introdduction to Machine Learning final exams differ from problem sets in several ways, and the experience of solving each varies.
Revised: Introduction to Machine Learning final exams differ from problem sets in several ways, and the experience of solving each varies.

Original: First, finals are long, containing around nine questions with around seven parts each.
Revised: First, finals are long, containing around nine questions with around seven parts each.

Original: Final exam questions are also multifaceted and multi-stepped: different parts of a single question require applying different concepts and problem-solving skills, and parts may build upon each other.
Revised: Final exam questions are also multifaceted and multi-stepped: different parts of a single question require applying different concepts and problem-solving skills, and parts may build upon each other.

Original: While weekly problem sets focus on a single topic, finals span topics from the entire semester.
Revised: While weekly problem sets focus on a single topic, finals span topics from the entire semester.

Original: Further, final questions are often story-based problems that may require mathematical modeling.
Revised: Further, final questions are often story-based problems that may require mathematical modeling.

Original: Due to the time constraint of these exams, finals are also designed to test core understanding and application of course material over rote calculations.
Revised: Due to the time constraint of these exams, finals are also designed to test core understanding and application of course material over rote calculations.

Original: Thus, asking a machine to answer questions from finals allows for testing whether the model is able to learn a breadth and depth of topics beyond problem sets.
Revised: Thus, asking a machine to answer questions from finals allows for testing whether the model is able to learn a breadth and depth of topics beyond problem sets.

Original: In this work, we present a new dataset curated from final exams of MIT’s  Introduction to Machine Learning course, totaling around 500 question parts spanning the broad range of topics in machine learning: regression, classifiers, logistic regression, features, loss functions, neural networks, convolutional neural networks (CNNs), Markov decision processes (MDPs), recurrent neural networks (RNNs), reinforcement learning, clustering,and decision trees.
Revised: In this work, we present a new dataset curated from final exams of MIT’s and Cornell’s Introduction to Machine Learning course and Harvard’s Machine Learning class, totaling 646 question parts spanning the broad range of topics in machine learning: regression, classifiers, logistic regression, features, loss functions, neural networks, convolutional neural networks (CNNs), Markov decision processes (MDPs), recurrent neural networks (RNNs), reinforcement learning, clustering, decision trees, model selection, ensemble methods, Bayesian networks, hidden Markov models (HMMs), and optimization.

Original: Our dataset covers therecent finals given  forthe semesters  Fall 2017, Spring 2018, Fall 2018, Spring 2019, Fall 2019, Spring 2021, Fall 2021, and Spring 2022.
Revised: Our dataset covers the finals given at MIT for semesters of Fall 2017, Spring 2018, Fall 2018, Spring 2019, Fall 2019, Spring 2021, Fall 2021, and Spring 2022, Harvard Springand Spring 2021, and Cornell Spring 2017 and Fall 2018.

Original: Due to the COVID-19 pandemic, no finals were in the  course during 2020.
Revised: Due to the COVID-19 pandemic, no finals were in the MIT course during 2020.

Original: [Table]
Revised: [Table]

Original: Given a set of data D train = { ( x ( i ) , y ( i ) ) } , a weighted nearest neighbor regressor has the form h ( x
Revised: Given a set of data D train = { ( x ( i ) , y ( i ) ) } , a weighted nearest neighbor regressor has the form [Equation] [Equation] y

Original: [Equation]
Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: ) ∈D

Revised: [Equation]

Revised: f

Revised: [Equation]

Revised: [Equation]

Original: A typical choice for f is
Revised: A typical choice for f is

Original: [Equation]
Revised: [Equation]

Original: where θ is a scalar and ∥ x − x ′ ∥ 

Revised: where θ is a scalar and ∥ x − x ′ ∥

Original: = (cid:80) d j =1 ( x j − x ′ j ) 2 .
Revised: = (cid:80) dj =1 ( x j − x ′ j ) 2 .

Original: Assume our training data D train =
Revised: Assume our training data D train =

Original: What is h (10; 0) ?
Revised: What is h (10; 0) ?

Original: That is, letting θ = 0 , what is our prediction for x = 10 ?
Revised: That is, letting θ = 0 , what is our prediction for x = 10 ?

Original: Solution
Revised: Solution

Original: Codex Zero-Shot Answer import numpy as np def f(x, x_i, theta): return np.exp(-theta * np.linalg.norm(x - x_i)**2) def h(x, theta): D = np.array([(1, 1), (2, 2), (3, 6)])
Revised: Codex Zero-Shot Answer import numpy as np def f(x, x_i, theta): return np.exp(-theta * np.linalg.norm(x - x_i)**2) def h(x, theta): D = np.array([(1, 1), (2, 2), (3, 6)])

Original: numerator = 0 denominator =for x_i, y_i in D: numerator += f(x, x_i, theta) *
Revised: numerator = 0 denominator =for x_i, y_i in D: numerator += f(x, x_i, theta) *

Original: y_i denominator += f(x, x_i, theta) return numerator / denominator print(h(10, 0))
Revised: y_i denominator += f(x, x_i, theta) return numerator / denominator print(h(10, 0))

Original: Program Output 
Revised: Program Output 3 .

Original: [Table]
Revised: [Table]

Original: 3 Zero-

Original: [Table]

Original: h (10; 0) =

Original: f (10 , 1 , 0) y (1) + f (10 , 2 , 0)

Original: y (2) + f (10 , 3 , 0) y (3) f

Original: (10 , 1 , 0)+ f (10 , 2 , 0)+ f (10 , 3 , 0) = 1 · 1+1 · 2+1 · 6 1+1+ = 93 =

Original: [Table]
Revised: [Table]

Original: with

Original: [Table]

Original: Answer

Original: [Equation] [Table] [Equation] We verify that our results are not due to overfitting by including  finals from 2021 and 2022 that are unavailable online.
Revised: We verify that our results are not due to overfitting by including MIT finals from 2021 and 2022 that are unavailable online.

Original: Also, the  Spring 2022 final exam was given after GPT-3 and Codex were last updated, which means that the models were not trained on this data.
Revised: Also, the MIT Spring 2022 final exam was given after GPT-3 and Codex were last updated, which means that the models were not trained on this data.

Original: The final exam questions have many parts, each posing a new problem, and each question in the dataset corresponds to one part.
Revised: The final exam questions have many parts, each posing a new problem [Table] and each question in the dataset corresponds to one part.

Original: The questions in the finals are varied in topics and solution types.
Revised: The questions in the finals are varied in topics and solution types.

Original: Solutions are primarily open-ended questions with some true/false and multiple-choice questions on theory, math, and code implementations.
Revised: Solutions are primarily open-ended questions with some true/false and multiple-choice questions on theory, math, and code implementations.

Original: We make the dataset publicly available and welcome others to use it to aid in developing and assessing new language models and methods.
Revised: We make the dataset publicly available and welcome others to use it to aid in developing and assessing new language models and methods.

Original: Due to the diversity of Intro to ML final questions, our dataset uniquely assesses advanced problem-solving and reasoning skills in machine learning, math, and natural language processing.
Revised: Due to the diversity of Intro to ML final questions, our dataset uniquely assesses advanced problem-solving and reasoning skills in machine learning, math, and natural language processing.

Original: This dataset opens the door to achieving breakthroughs in machine learning performance in machine learning final exams.
Revised: This dataset opens the door to achieving breakthroughs in machine learning performance in machine learning final exams.

Original: In addition to the dataset, we present a benchmark using several baseline methods.
Revised: In addition to the dataset, we present a benchmark using several baseline methods.

Original: We apply zero-shot and few-shot learning to GPT-3 and Codex, adding chain-of-thought prompting for GPT-3.
Revised: We apply zero-shot and few-shot learning to GPTand Codex, adding chain-of-thought prompting for GPT-3.

Original: We find that few-shot learning methods perform best.
Revised: We find that few-shot learning methods perform best.

Original: As shown in Table 2 the best performing methods pass the final exams, and their grade is comparable with human grades of MIT students on the same machine learning finals evaluated by the same human graders.
Revised: As shown in Table 2 the best performing methods pass the final exams, and their grade is comparable with human grades of MIT students on the same machine learning finals evaluated by the same human graders.

Original: We generate new final exam questions that are indistinguishable from human-written questions.
Revised: We generate new final exam questions that are indistinguishable from human-written questions.

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: There is often thought that humans are generalists, whereas machines are specialists.
Revised: There is often thought that humans are generalists, whereas machines are specialists.

Original: However, large language models based on transformers such as GPT-3 Brown et al. (2020), Gopher Rae et al. (2021), and Palm Chowdhery et al. (2022), also called foundation models, are generalist learners.
Revised: However, large language models based on transformers such as GPT-3 (Brown et al., 2020), Gopher (Rae et al., 2021), and PaLM (Chowdhery et al., 2022), also called foundation models, are generalist learners.

Original: Specifically, in our setting, while humans care about the number of topics in an exam and therefore find finals more difficult than problem sets, foundation models effortlessly scale to many topics without re-training.
Revised: Specifically, in our setting, while humans care about the number of topics in an exam and therefore find finals more difficult than problem sets, foundation models effortlessly scale to many topics without re-training.

Original: Language models may be pre-trained on text and fine-tuned on specific datasets such as code, for example OpenAI’s Codex Chen et al. (2021), which allows generating programs from text.[Table]
Revised: Language models may be pre-trained on text and fine-tuned on specific datasets such as code, for example OpenAI’s Codex (Chen et al., 2021), which allows generating programs from text.

Original: [Table]

Original: There are several ways to improve the mathematical reasoning ability of language models: (1) using chain-of-thought (CoT) prompting Kojima et al. ; Wei et al. (2022), (2) using the top-k ranking solutions Li et al. and merging them by voting Wang et al. (2022) or least-to-most prompting 
Revised: There are several ways to improve the mathematical reasoning ability of language models: (1) using chain-of-thought (CoT) prompting (Kojima et al., 2022; Wei et al., 2022), (2) using the top-k ranking solutions (Li et al., 2022) and merging them by voting (Wang et al., 2022) or least-to-most prompting (Zhou et al., 2022), and (3) using program synthesis and few-shot learning to generate code that answers questions (Drori et al., 2022).

Original: Zhou et al.
Revised: [Table]

Original: and (3) using program synthesis and few-shot learning to generate code that answers questions Drori et al.
Revised: [Table]

Original: Much of the prior work focuses on high school or middle school level material Qu et al.
Revised: Much of the prior work focuses on high school or middle school level material (Qu et al., 2021).

Original: The first work to tackle university-level machine learning course problem set questions Tran et al. (2021) used a transformer and GNN architecture and heavily relied on data augmentation.
Revised: The first work to tackle university-level machine learning course problem set questions (Tran et al., 2021) used a transformer and GNN architecture and heavily relied on data augmentation.

Original: This resulted in overfitting and did not scale up to other types of questions or courses.
Revised: This resulted in overfitting and did not scale up to other types of questions or courses.

Original: Probability and statistics course problem-set questions have been answered Tang et al. (2022) by probabilistic program synthesis with human performance.
Revised: Probability and statistics course problem-set questions have been answered (Tang et al., 2022) by probabilistic program synthesis with human performance.

Original: Problem-set questions from the core university math courses Drori et al. (2022) have been automatically solved using few-shot learning and program synthesis at a human level.
Revised: Problem-set questions from the core university math courses (Drori et al., 2022) have been automatically solved using few-shot learning and program synthesis at a human level.

Original: Other work considers university-level course questions across a variety of domains Hendrycks et al. (2021) and identifying theorems Srivastava et al.
Revised: Other work considers university-level course questions across a variety of domains (Hendrycks et al., 2021) and identifying theorems (Srivastava et al., 2022).

Original: Prior work on question generation includes question–answer pair generation based on a text passage Qu et al. and question text generation based on other questions Drori et al.
Revised: Prior work on question generation includes question–answer pair generation based on a text passage (Qu et al., 2021) and question text generation based on other questions (Drori et al., 2022).

Original:  D ATASET
Revised:  D ATASET

Original: We present a new dataset of 496 questions from the eight most recent final exams of MIT’s  Introduction to Machine Learning course.
Revised: We present a new dataset of 646 question parts from a dozen recent final exams of MIT’s and Cornell’s Introduction to Machine Learning courses and Harvard’s Machine Learning class.

Original: The dataset spans questions on the 12 machine learning topics covered in the course: (1) regression, (2) classifiers, (3) logistic regression, (4) features, (5) loss functions, (6) neural networks, (7) convolutional neural networks (CNNs), (8) Markov decision processes (MDPs), (9) recurrent neural networks (RNNs), (10) reinforcement learning, (11) clustering,and (12) decision trees.
Revised: The dataset spans questions on the 17 machine learning topics covered in the courses: (1) regression, (2) classifiers, (3) logistic regression, (4) features, (5) loss functions, (6) neural networks, (7) convolutional neural networks (CNNs), (8) Markov decision processes (MDPs), (9) recurrent neural networks (RNNs), (10) reinforcement learning, (11) clustering, (12) decision trees, (13) model selection, (14) ensemble methods, (15) Bayesian networks, (16) hidden Markov models (HMMs), and (17) optimization.

Original: We make our data and code publicly available.
Revised: We make our data and code publicly available.

Original: The breakdown of questions, parts, points, and non-image points by each semester and topic are shown in Table 2.
Revised: The breakdown of questions, parts, points, and non-image points by each semester and topic are shown in Table 2.

Original: Each question in a final exam consists of multiple parts.
Revised: Each question in a final exam consists of multiple parts.

Original: Questions are written by providing set-up and context information first, followed by the question parts (which may come with additional information).
Revised: Questions are written by providing set-up and context information first, followed by the question parts (which may come with additional information).

Original: Set-up and context information may contain (1) story elements (ex., character names, and motivations), (2) relevant definitions and equations, and (3) data points.
Revised: Set-up and context information may contain (1) story elements (ex., character names, and motivations), (2) relevant definitions and equations, and (3) data points.

Original: We format questions in the dataset by concatenating the question context, any context or solutions from prior parts of the question required for answering the part, and the part’s context and question.
Revised: We format questions in the dataset by concatenating the question context, any context or solutions from prior parts of the question required for answering the part, and the part’s context and question.

Original: We split the questions into their corresponding parts.
Revised: We split the questions into their corresponding parts.

Original: Questions consist of English text, mathematical notation, and images.
Revised: Questions consist of English text, mathematical notation, and images.

Original: Mathematical notation is represented in the dataset by LaTeX and images by screenshots from pdfs files.
Revised: Mathematical notation is represented in the dataset by LaTeX and images by screenshots from pdfs files.

Original: The types of question answers are diverse.
Revised: The types of question answers are diverse.

Original: A few are multiple-choice or true/false questions.
Revised: A few are multiple-choice or true/false questions.

Original: Most are open-ended, for which the evaluation requires modeling the problem, mathematical manipulation, or code writing.
Revised: Most are open-ended, for which the evaluation requires modeling the problem, mathematical manipulation, or code writing.

Original: Many questions require providing an explanation.
Revised: Many questions require providing an explanation.

Original: We used eight final exams from different semesters for data curation.
Revised: We used twelve final exams from different semesters for data curation.

Original: We had access to the Latex version for the three most recent semesters of  Spring 2021, Fall 2021, and Spring 2022, and therefore did not require transcription.
Revised: We had access to the Latex version for the three most recent semesters of MIT Spring 2021, Fall 2021, and Spring 2022, and therefore did not require transcription.

Original: For the five remaining exams,  Fall 2017, Spring 2018, Fall 2018, Spring 2019, Fall 2019,  we had access to the pdf versions.
Revised: For the nine remaining exams, MIT Fall 2017, Spring 2018, Fall 2018, Spring 2019, Fall 2019, Harvard Spring 2015 and Spring 2021, and Cornell Springand Fall 2018, we had access to the pdf versions.

Original: In these cases, we used mathpix mathpix.com for an initial transcription, and curators then evaluated and manually corrected the input questions and verified the correctness of each input question.
Revised: In these cases, we used mathpix mathpix.com for an initial transcription, and curators then evaluated and manually corrected the input questions and verified the correctness of each input question.

Original: We extract questions and solutions for all parts of all types of questions, including those that rely on images.
Revised: We extract questions and solutions for all parts of all types of questions, including those that rely on images.

Original: We curated the five examsbetween 2017 and 2019 from publicly available pdf files.
Revised: We curated nine exams from publicly available pdf files.

Original:  Spring 2020 and Fall 2020 do not have final exams due to COVID-19.
Revised: MIT Spring 2020 and Fall 2020 do not have final exams due to COVID-19.

Original: The three  exams between 2021 and 2022 were unavailable online; therefore, the model does not overfit their solutions.
Revised: The three MIT exams between 2021 andwere unavailable online; therefore, the model does not overfit their solutions.

Original: The aggregate average grades were available to the students and did not contain any personally identifiable information.
Revised: The aggregate average grades were available to the students and did not contain any personally identifiable information.

Original: Three duplicate questions were originally on the final exam of  Fall 2017 (questions 1, 3, 6) and appeared again in the final exam of  Spring 2022.
Revised: Three duplicate questions were originally on the final exam of MIT Fall 2017 (questions 1, 3, 6) and appeared again in the final exam of MIT Spring 2022.

Original:  B ENCHMARK 3. B ASELINES
Revised:  B ENCHMARK 3. B ASELINES

Original: We provide a benchmark by comparing six baselines for answering the final exam questions: (1)
Revised: We provide a benchmark by comparing six baselines for answering the final exam questions: (1)

Original: GPT-3 with zero-shot learning, (2) GPT-3 with few-shot learning, (3) GPT-3 with zero-shot learning and chain-of-thought (CoT) prompting, (4) GPT-3 with few-shot learning and chain-of-thought (CoT) prompting, (5) Codex with zero-shot learning, and (6) Codex with few-shot learning.
Revised: GPT-3 with zero-shot learning, (2) GPT-3 with few-shot learning, (3) GPT-3 with zero-shot learning and chain-of-thought (CoT) prompting, (4) GPT-3 with few-shot learning and chain-of-thought (CoT) prompting, (5) Codex with zero-shot learning, and (6) Codex with few-shot learning.

Original: Table 4 shows the prompt used for each approach.
Revised: Table 4 shows the prompt used for each approach.

Original: GPT-3 zero-shot uses the question as-is, whereas
Revised: GPT-3 zero-shot uses the question as-is, whereas

Original: GPT-3 zero-shot with CoT uses the suffix “Let’s think step by step.”
Revised: GPT-3 zero-shot with CoT uses the suffix “Let’s think step by step.”

Original: after the question to encourage multi-step output.
Revised: after the question to encourage multi-step output.

Original: Codex zero-shot uses the prefix “Write a program that answers” before the question within Python comments denoted by triple quotes """ to encourage Codex to write code.
Revised: Codex zero-shot uses the prefix “Write a program that answers” before the question within Python comments denoted by triple quotes """ to encourage Codex to write code.

Original: GPTfew-shot finds the closest questions in the embedding space, measured by cosine similarity, and uses them and their corresponding answers before the new question as examples in the prompt.
Revised: GPTfew-shot finds the closest questions in the embedding space, measured by cosine similarity, and uses them and their corresponding answers before the new question as examples in the prompt.

Original: Codex few-shot finds the closest questions in the embedding space also as measured by cosine similarity and uses these questions and their corresponding code as examples.
Revised: Codex few-shot finds the closest questions in the embedding space also as measured by cosine similarity and uses these questions and their corresponding code as examples.

Original: For students, a good study technique is to use previous final exams to review and practice for their upcoming final.
Revised: For students, a good study technique is to use previous final exams to review and practice for their upcoming final.

Original: We model this method by few-shot learning using the question–answer pairs (for
Revised: We model this method by few-shot learning using the question–answer pairs (for

Original: GPT-3) or question–code (for Codex) with the closest question embeddings from previous finals.
Revised: GPT-3) or question–code (for Codex) with the closest question embeddings from previous finals.

Original: We implement this by considering all the exam questions, marking each question by its semester and year, and using only previous semesters’ questions for few-shot learning.
Revised: We implement this by considering all the exam questions, marking each question by its semester and year, and using only previous semesters’ questions for few-shot learning.

Original: The  Fall 2017 and Springexams contain three duplicate questions, and we handle these same questions the same way humans do by allowing few-shot learning in  Spring 2022 based on successful Fall 2017 zero-shot answers.It [Table]
Revised: The MIT Fall 2017 and Spring 2022 exams contain three duplicate questions, and we handle these same questions the same way humans do by allowing few-shot learning in MIT Spring 2022 based on successful Fallzero-shot answers.

Original:  is reasonable that if a student studies all previous exams, there may be 8.5% of repeated question points.
Revised: It is reasonable that if a student studies all previous exams, there may be 8.5% of repeated question points.

Original: Since  Fall  2017 is the first final exam, we do not perform few-shot learning this
Revised: Since MIT Fall 2017, Harvard Spring 2015, and Cornell Spring 2017 are the first final exams in the corresponding universities, we do not perform few-shot learning on these.

Original: 3.1. C OMPARISON WITH O PEN L ANGUAGE M ODELS
Revised: 3.1. C OMPARISON WITH O PEN L ANGUAGE M ODELS

Original: We also evaluated our dataset on an open-source language model, Meta’s OPT-175B. OPT-175B is a model consisting of 175 billion parameters.
Revised: We also evaluated our dataset on an open-source language model, Meta’s OPT-175B. OPT-175B is a model consisting of 175 billion parameters.

Original: Our dataset consists of final exam questions from machine learning courses and fit to be used by OPT-175B. Tables 5 and 6 compare the results of
Revised: Our dataset consists of final exam questions from machine learning courses and fit to be used by OPT-175B. Tables 5 and 6 compare the results of

Original: OpenAI GPT-3, OpenAI Codex [Table] and Meta OPT.
Revised: OpenAI GPT-3, OpenAI Codex, and Meta OPT.

Original: We evaluated OPT on only 163 question parts, since
Revised: We evaluated OPT on only 163 question parts, since

Original: OPT was limited to handling questions under 256 characters in length.
Revised: OPT was limited to handling questions under 256 characters in length.

Original: We implement the inference for the OPT-175B model using Alpa.
Revised: We implement the inference for the OPT-175B model using Alpa.

Original: Alpa is a particular framework designed for training and inference of large models.
Revised: Alpa is a particular framework designed for training and inference of large models.

Original: For the hardware, we use an 8x A100 PCIE cluster.
Revised: For the hardware, we use an 8x A100 PCIE cluster.

Original: The model requires about 560 GB of VRAM in our run case and each example takes nine minutes for inference.
Revised: The model requires about 560 GB of [Table]

Original: 3. G RADING 3.2. H UMAN G RADING
Revised: 3. G RADING 3.2. H UMAN G RADING

Original: The questions are of different types: multiple-choice, numerical, expressions, and open-text.
Revised: The questions are of different types: multiple-choice, numerical, expressions [Table] and open-text.

Original: We grade answers and aim to keep all factors equal in grading human and machine answers.
Revised: We grade answers and aim to keep all factors equal in grading human and machine answers.

Original: Human and machine answers are graded based on the number of points allocated to each question part, giving full, partial, or no credit for each answer.
Revised: Human and machine answers are graded based on the number of points allocated to each question part, giving full, partial, or no credit for each answer.

Original: We approximate partial credit by assigning half-credit.
Revised: We approximate partial credit by assigning half-credit.

Original: The course staff graded student final exams, which included graduate TAs and instructors.
Revised: The course staff graded student final exams, which included graduate TAs and instructors.

Original: Two of the same graduate TAs and the instructor that graded the student answers also graded the machine answers.
Revised: Two of the same graduate TAs and the instructor that graded the student answers also graded the machine answers.

Original: Grading instructions are the same for grading student answers as grading machine answers.
Revised: Grading instructions are the same for grading student answers as grading machine answers.

Original: 3.2. A UTOMATIC G RADING
Revised: 3.2. A UTOMATIC G RADING

Original: We label each question’s answer type into one or two categories out of four options - multiple choice (MC), numerical, expression, or open. We consider answers multiple choice if the test-taker is
Revised: We label each question’s answer type into one or two categories out of four options - multiple choice (MC), numerical, expression

Original: [Table]
Revised: [Table]

Original:  presented with an enumerated list of choices, numerical if the answer is a number, expression if the answer includes variables or other notation, and open if the answer calls for free-response text.
Revised: or open. We consider answers multiple choice if the test-taker is presented with an enumerated list of choices, numerical if the answer is a number, expression if the answer includes variables or other notation, and open if the answer calls for free-response text.

Original: We categorize questions that have additional questions nested within them by the multiple relevant categories.
Revised: We categorize questions that have additional questions nested within them by the multiple relevant categories.

Original: Most often, this is the case when a question with one of MC, numerical, or expression, is followed by a follow-up question asking the student to explain their previous answer.
Revised: Most often, this is the case when a question with one of MC, numerical, or expression, is followed by a follow-up question asking the student to explain their previous answer.

Original: The breakdown of the questions is: 76 are multiple-choice, 100 numerical, 89 expressions, and 214 are open.
Revised: The breakdown of the questions is: 98 are multiple-choice, 84 numerical, 81 expressions, and 204 are open.

Original: The ’Non-Open Points’ column of Tables 7 and 8 show the answer type breakdown by number of points.
Revised: The ’Non-Open Points’ column of Tables 7 and 8 show the answer type breakdown by number of points.

Original: Table 7 shows the number of question parts that do not rely on images, the number of points that do not rely on images, and the number of non-open question points in Introduction to Machine Learning finals for each semester.
Revised: Table 7 shows the number of question parts that do not rely on images, the number of points that do not rely on images, and the number of non-open question points in Introduction to Machine Learning finals for each semester.

Original:  Spring 2020 and Fall 2020 did not have final exams due to COVID-19.
Revised: MIT Spring 2020 and Fall 2020 did not have final exams due to COVID-19.

Original: Table 8 shows the breakdown by topic.
Revised: Table 8 shows the breakdown by topic.

Original: Our automatic grading uses string matching and regular expressions.
Revised: Our automatic grading uses string matching and regular expressions.

Original: In the case of multiple-choice results, we check that the output of the code is equal to the solution.
Revised: In the case of multiple-choice results, we check that the output of the code is equal to the solution.

Original: In the case of numerical answers, we look for a matching integer or real number.
Revised: In the case of numerical answers, we look for a matching integer or real number.

Original: 3. P ERFORMANCE
Revised: 3. P ERFORMANCE

Original: Table 5 shows the machine grades by semester and Table 6 shows the machine grades by topic, excluding question parts that rely on images.
Revised: Table 5 shows the machine grades by semester and Table 6 shows the machine grades by topic, excluding question parts that rely on images.

Original: We compare the average grade of GPT-3 with zero-shot (ZS), GPT-3 with ZS and chain-of-thought (CoT) prompting, GPT-3 with few-shot (FS) learning,
Revised: We compare the average grade of GPT-3 with zero-shot [Table] MIT Fall 2017, Cornell Spring 2017, and Harvard

Revised: Spring 2015 were the first semester for each university, so few-shot learning results based on previous semesters are unavailable (NA).

Revised: MIT Spring 2020 and MIT Fall 2020 did not have final exams due to

Revised: COVID-19.

Original:  GPT-3with FS andCoT prompting, Codex with ZS, Codex  with FS, and OPT with ZS.
Revised: MIT Spring 2021, MIT Fall 2021, and MIT Spring 2022 final exams were unavailable online when GPT-3 and Codex were trained, ensuring that the model is not overfitting. The result of the best-performing method for each semester is marked in bold. [Table] (ZS), GPT-3 with ZS and chain-of-thought (CoT)

Revised: , GPT

Revised: 3 with shot learning, and CoT prompting, Codex with

Revised: , and OPT with

Original: Fallis the first semester, so few-shot learning results based on previous semesters are unavailable (NA).
Revised: Fallis the first semester, so few-shot learning results based on previous semesters are unavailable (NA).

Original: Spring 2020 and Fall 2020 did not have final exams due to COVID-19.
Revised: Spring 2020 and Fall 2020 did not have final exams due to COVID-19.

Original: Spring 2021, Fall 2021, and
Revised: Spring 2021, Fall 2021, and

Original: Spring 2022 final exams were unavailable online when GPT-3 and Codex were trained, ensuring that the model is not overfitting content it has seen previously.
Revised: Spring 2022 final exams were unavailable online when GPT-3 and Codex were trained, ensuring that the model is not overfitting content it has seen previously.

Original: The results consistently demonstratethat few-shot learning methods perform best across semesters and topics, as marked in bold.
Revised: results consistently demonstrate few-shot learning methods perform best across semesters and topics, as marked in bold.

Original: 3. L IMITATIONS
Revised: 3. L IMITATIONS

Original: Our dataset consists of all question parts and their solutions, including images.
Revised: Our dataset consists of all question parts and their solutions, including images.

Original: However, our baseline methods do not handle questions that rely on an image containing the information required to solve the question since GPT-3 and Codex do not handle images.
Revised: However, our baseline methods do not handle questions that rely on an image containing the information required to solve the question since GPT-3 and Codex do not handle images.

Original: Tables 7 and 8 show the breakdown of the number of question parts and points of questions that do not rely on image information for answering the question.
Revised: Tables 7 and 8 show the breakdown of the number of question parts and points of questions that do not rely on image information for answering the question.

Original: On average, 28.42% of the question parts, which make up 35.63% of the points in final  exams, arequestions that rely on image information.
Revised: On average, 27.55% of the question parts, which make up 30.32% of the points in final [Table] [Table] exams, are that rely on tallied,

Original: The points attributed to the non-image parts are tallied, recorded, and used to calculate non-image percentage grades.

Original: 3. G ENERATING N EW Q UESTIONS
Revised: 3. G ENERATING N EW Q UESTIONS

Original: The creation of new, high-quality questions by course instructors and TA is often a time-consuming, high-effort process.
Revised: The creation of new, high-quality questions by course instructors and TA is often a time-consuming, high-effort process.

Original: These new questions must be varied from past questions while still testing the [Table] [Table]
Revised: These new questions must be varied from past questions while still testing the same core concepts.

Original: [Table] same core concepts. We explore the potential of using GPT-3 to write exam content efficiently by using the dataset of exam questions to generate new questions automatically.
Revised: We explore the potential of using GPT-3 to write exam content efficiently by using the dataset of exam questions to generate new questions automatically.

Original: We use questions from our dataset as prompts to create new high-quality questions not present in our dataset.
Revised: We use questions from our dataset as prompts to create new high-quality questions not present in our dataset.

Original: We create a list of various questions in our curated dataset and use the resulting list to prompt GPT-3 to create a new question.
Revised: We create a list of various questions in our curated dataset and use the resulting list to prompt GPT-3 to create a new question.

Original: The supplementary material demonstrates the results of this process for each topic in the course.
Revised: The supplementary material demonstrates the results of this process for each topic in the course.

Original: The Appendix consists of new generated questions and the closest question from our dataset as measured by the cosine similarity of the embedding of each question.
Revised: The Appendix consists of new generated questions and the closest question from our dataset as measured by the cosine similarity of the embedding of each question.

Original: These new questions are diverse and qualitatively similar to questions on previous MIT final exams.
Revised: These new questions are diverse and qualitatively similar to questions on previous MIT final exams.

Original: This provides an efficient way for course TAs and instructors to generate new final questions.
Revised: This provides an efficient way for course TAs and instructors to generate new final questions.

Original: 3.5. S TUDENT S URVEY
Revised: 3.5. S TUDENT S URVEY

Original: To evaluate the machine-generated questions, we conducted an anonymous online student survey comparing them with the human-written questions in terms of quality, appropriateness relative to the course, and question difficulty.
Revised: To evaluate the machine-generated questions, we conducted an anonymous online student survey comparing them with the human-written questions in terms of quality, appropriateness relative to the course, and question difficulty.

Original: We surveyed 15 students who have taken the Introduction to
Revised: We surveyed 15 students who have taken the Introduction to

Original: Machine Learning course or its equivalent.
Revised: Machine Learning course or its equivalent.

Original: The survey was optional and included informed consent, with the following description: “We are conducting a survey to assess the quality and difficulty of automatically generated questions for an introductory machine learning course final.
Revised: The survey was optional and included informed consent, with the following description: “We are conducting a survey to assess the quality and difficulty of automatically generated questions for an introductory machine learning course final.

Original: You will be presented with a series of questions, either human-written (taken from an actual course final exam) or machine generated, but you will not be told the source of a given question.
Revised: You will be presented with a series of questions, either human-written (taken from an actual course final exam) or machine generated, but you will not be told the source of a given question.

Original: For each question, you will be asked (a) whether you think the question is human-written or machine-generated, (b) whether the question is appropriate for the given course final, and finally (c) how you would rate the difficulty of the question.
Revised: For each question, you will be asked (a) whether you think the question is human-written or machine-generated, (b) whether the question is appropriate for the given course final, and finally (c) how you would rate the difficulty of the question.

Original: Please carefully read each question and answer to the best of your ability”.
Revised: Please carefully read each question and answer to the best of your ability”.

Original: We randomly sampled one generated question and its closest (measured by cosine similarity) original, human-written question for each of the twelve machine learning topics.
Revised: We randomly sampled one generated question and its closest (measured by cosine similarity) original, human-written question for each of the twelve machine learning topics.

Original: Students were asked to read these 24 questions in the survey, mixed and presented randomly, and then answer three questions for each: (1) “Is the question human-written or machine-generated?”, (2) “Is the question appropriate or not appropriate for the specific course final?”, and (3) “What is the question’s difficulty level on a scale between 1 (easiest) and 5 (hardest)?”.
Revised: Students were asked to read these 24 questions in the survey, mixed and presented randomly, and then answer three questions for each: (1) “Is the question human-written or machine-generated?”, (2) “Is the question appropriate or not appropriate for the specific course final?”, and (3) “What is the question’s difficulty level on a scale between 1 (easiest) and 5 (hardest)?”.

Original: We ask the students to provide ratings and not to solve the questions.
Revised: We ask the students to provide ratings and not to solve the questions.

Original: The results of our survey are as follows:
Revised: The results of our survey are as follows:

Original: Out of the human-written questions, students identified 56.11% of them correctly as human-written and 43.89% incorrectly as machine-generated.
Revised: Out of the human-written questions, students identified 56.11% of them correctly as human-written and 43.89% incorrectly as machine-generated.

Original: Of the machine-generated questions, students identified 45% of them correctly as machine-generated and 55% of them incorrectly as human-written.
Revised: Of the machine-generated questions, students identified 45% of them correctly as machine-generated and 55% of them incorrectly as human-written.

Original: The difficulty ratings were between 1 (the easiest) and 5 (the hardest).
Revised: The difficulty ratings were between 1 (the easiest) and 5 (the hardest).

Original: Students rated machine-generated questions with a difficulty level of 2.55 with a 1.11 standard deviation and rated human-written questions with a difficulty level of 2.85 with a 1.
Revised: Students rated machine-generated questions with a difficulty level of 2.55 with a 1.11 standard deviation and rated human-written questions with a difficulty level of 2.85 with a 1.

Original: standard deviation.
Revised: standard deviation.

Original: Students rated machine-generated questions as appropriate 82.6% of the time and human-written questions as appropriate 85.0% of the time.
Revised: Students rated machine-generated questions as appropriate 82.6% of the time and human-written questions as appropriate 85.0% of the time.

Original: The conclusions we draw from the survey are that (1) survey participants considered human-written questions to be as likely to be human-written or machine-generated, and similarly, machine-generated questions were considered equally likely to be machine-generated as human-written, (2) survey participants considered the machine-generated questions slightly easier than human-written questions, and (3) survey participants considered machine-generated questions as appropriate as human-written questions.
Revised: The conclusions we draw from the survey are that (1) survey participants considered human-written questions to be as likely to be human-written or machine-generated, and similarly, machine-generated questions were considered equally likely to be machine-generated as human-written, (2) survey participants considered the machine-generated questions slightly easier than human-written questions, and (3) survey participants considered machine-generated questions as appropriate as human-written questions.

Original: Based on these results, we conclude that across multiple aspects, the machine-generated questions are highly similar to human-generated questions and can be adapted to generate questions for machine learning courses.
Revised: Based on these results, we conclude that across multiple aspects, the machine-generated questions are highly similar to human-generated questions and can be adapted to generate questions for machine learning courses.

Original: 3. I MPLEMENTATION D ETAILS
Revised: 3. I MPLEMENTATION D ETAILS

Original: We use the latest OpenAI GPT-3 and Codex models and do not re-train these very large language models.
Revised: We use the latest OpenAI GPT-3 and Codex models and do not re-train these very large language models.

Original: We fix all the hyperparameters of the models so that the answers are deterministic and reproducible.
Revised: We fix all the hyperparameters of the models so that the answers are deterministic and reproducible.

Original: Specifically, we set both top P, which controls diversity, and sampling temperature, which controls randomness, to 0 .
Revised: Specifically, we set both top P, which controls diversity, and sampling temperature, which controls randomness, to 0 .

Original: The frequency and presence penalties are also set to 0 , and we do not halt on any stop sequences.
Revised: The frequency and presence penalties are also set to 0 , and we do not halt on any stop sequences.

Original: We allow diversity for generating new questions by setting the top
Revised: We allow diversity for generating new questions by setting the top

Original: P and temperature to 0 .
Revised: P and temperature to 0 .

Original: We run Codex with an upper bound of generating programs withtokens.
Revised: We run Codex with an upper bound of generating programs withtokens.

Original: We use the OpenAI text-davinci-002 and code-davinci-002 engines for generating text and programs.
Revised: We use the OpenAI text-davinci-002 and code-davinci-002 engines for generating text and programs.

Original: For few-shot-learning and question generation, we use the text-similarity-babbage-  engine to embed the questions and find the closest questions in the dataset by cosine similarity.
Revised: For few-shot-learning and question generation, we use the text-similarity-babbageengine to embed the questions and find the closest questions in the dataset by cosine similarity.

Original: The running time for answering or generating each question part is a few seconds.
Revised: The running time for answering or generating each question part is a few seconds.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We present a dataset and benchmark for answering and generating university-level final exams in machine learning.
Revised: We present a dataset and benchmark for answering and generating university-level final exams in machine learning.

Original: Machine performance and human performance are evaluated by the same graders and grading instructions, as well as by automatic checkers.
Revised: Machine performance and human performance are evaluated by the same graders and grading instructions, as well as by automatic checkers.

Original: A comparison of baselines shows that few-shot learning methods perform best across semesters and topics.
Revised: A comparison of baselines shows that few-shot learning methods perform best across semesters and topics.

Original: A limitation of our work is that our benchmark does not consider questions that rely on images for their solution.
Revised: A limitation of our work is that our benchmark does not consider questions that rely on images for their solution.

Original: This work may result in improving students learning for final exams, help course staff generate questions for finals, and compare levels of difficulty of exams across semesters and schools.
Revised: This work may result in improving students learning for final exams, help course staff generate questions for finals, and compare levels of difficulty of exams across semesters and schools.

Original: We extended this work to include final exams from Harvard’s and Cornell’s machine learning classes as well.
Revised: This work includes final exams from MIT’s and Cornell’s Introduction to Machine Learning classes and Harvard’s Machine Learning course.

Original: We hope this dataset and benchmark serve the machine learning community and advance the state-of-the-art in the field.
Revised: We hope this dataset and benchmark serve the machine learning community and advance the state-of-the-art in the field.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: [Table]
Revised: [Table]

Original: Generated Question: “We’re given a data set D
Revised: Generated Question: “We’re given a data set D

Original: =
Revised: =

Original: (cid:8)(cid:0) x
Revised: (cid:8)(cid:0) x

Original: ( i ) , y ( i ) (cid:1)(cid:9) n i
Revised: ( i ) , y ( i ) (cid:1)(cid:9) n i

Original: =1 , where x ( i ) ∈ R d and y ( i ) ∈ R .
Revised: =1 , where x ( i ) ∈ R d and y ( i ) ∈ R .

Original: Let X be a d × n matrix in which the x ( i ) are the columns and let Y be a 1 × n vector containing the values of y ( i ) .
Revised: Let X be a d × n matrix in which the x ( i ) are the columns and let Y be a 1 × n vector containing the values of y ( i ) .

Original: Using the ordinary least-squares formula, we can compute
Revised: Using the ordinary least-squares formula, we can compute

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: = (cid:0) XX T (cid:1) − 1 XY T Using ridge regression, we can compute
Revised: = (cid:0) XX T (cid:1) − 1 XY T Using ridge regression, we can compute

Original: W ridge =
Revised: W ridge =

Original: (cid:0) XX T + λI (cid:1) − 1 XY T
Revised: (cid:0) XX T + λI (cid:1) − 1 XY T

Original: We decide to try to use these methods to initialize a single-unit neural network with a linear activation function.
Revised: We decide to try to use these methods to initialize a single-unit neural network with a linear activation function.

Original: Assume that XX T is neither singular nor equal to the identity matrix, and that neither W ols nor W ridge is equal to (0 , 0 , . .
Revised: Assume that XX T is neither singular nor equal to the identity matrix, and that neither W ols nor W ridge is equal to (0 , 0 , . .

Original: Consider a neuron initialized with W ridge .
Revised: Consider a neuron initialized with W ridge .

Original: Provide an objective function J ( W ) that depends on the data, such that batch gradient descent to minimize J will have no effect on the weights, or argue that one does not exist."
Revised: Provide an objective function J ( W ) that depends on the data, such that batch gradient descent to minimize J will have no effect on the weights, or argue that one does not exist."

Original: Closest Question: “We’re given a data set D
Revised: Closest Question: “We’re given a data set D

Original: =
Revised: =

Original: (cid:8)(cid:0) x
Revised: (cid:8)(cid:0) x

Original: ( i ) , y ( i ) (cid:1)(cid:9) n i
Revised: ( i ) , y ( i ) (cid:1)(cid:9) n i

Original: =1 , where x ( i ) ∈ R d and y ( i ) ∈ R .
Revised: =1 , where x ( i ) ∈ R d and y ( i ) ∈ R .

Original: Let X be a d × n matrix in which the x ( i ) are the columns and let Y be a 1 × n vector containing the values of y ( i ) .
Revised: Let X be a d × n matrix in which the x ( i ) are the columns and let Y be a 1 × n vector containing the values of y ( i ) .

Original: Using the analytical regression (ordinary least-squares) formula, we can compute
Revised: Using the analytical regression (ordinary least-squares) formula, we can compute

Original: [Equation]
Revised: [Equation]

Original: Using ridge regression, we can compute
Revised: Using ridge regression, we can compute

Original: [Equation]
Revised: [Equation]

Original: (cid:0) XX T + λI (cid:1) − 1 XY T
Revised: (cid:0) XX T + λI (cid:1) − 1 XY T

Original: We decide to try to use these methods to initialize a single-unit "neural network" with a linear activation function and no offset: h ( x ; W ) =
Revised: We decide to try to use these methods to initialize a single-unit "neural network" with a linear activation function and no offset: h ( x ; W ) =

Original: W T x. Assume that XX T is invertible and not equal to the identity matrix, and that neither W ols nor W ridge is equal to (0 , 0 , . . . , 0) .
Revised: W T x. Assume that XX T is invertible and not equal to the identity matrix, and that neither W ols nor W ridge is equal to (0 , 0 , . . . , 0) .

Original: Note also that we are not using an explicit offset/bias term.
Revised: Note also that we are not using an explicit offset/bias term.

Original: Rory has solved many problems from this particular domain before and the solution has typically been close to W ∗
Revised: Rory has solved many problems from this particular domain before and the solution has typically been close to W ∗

Original: =
Revised: =

Original: . , 1) T .
Revised: . , 1) T .

Original: Define an objective function J ( W ) that we could minimize in order to obtain good estimates for Rory’s next problem, even with very little data."
Revised: Define an objective function J ( W ) that we could minimize in order to obtain good estimates for Rory’s next problem, even with very little data."

Original: Classifiers
Revised: Classifiers

Original: Generated Question: “Consider a binary classification problem with two classes, +1 and -1.
Revised: Generated Question: “Consider a binary classification problem with two classes, +1 and -1.

Original: Assume that the training data is linearly separable.
Revised: Assume that the training data is linearly separable.

Original: Consider the following two models:
Revised: Consider the following two models:

Original: Model 1: g ( x ) = sgn ( w T x ) Model 2: g ( x ) = sgn ( w T x + b )
Revised: Model 1: g ( x ) = sgn ( w T x ) Model 2: g ( x ) = sgn ( w T x + b )

Original: Assume that the training data is linearly separable.
Revised: Assume that the training data is linearly separable.

Original: Which model is more likely to overfit?
Revised: Which model is more likely to overfit?

Original: Closest Question: “In some cases, we will have a validation set in addition to training and test sets.
Revised: Closest Question: “In some cases, we will have a validation set in addition to training and test sets.

Original: Assume the validation set is approximately the same size as the test set.
Revised: Assume the validation set is approximately the same size as the test set.

Original: This validation set is often used to tune hyperparameters such as λ .
Revised: This validation set is often used to tune hyperparameters such as λ .

Original: Imagine we have trained a classifier using regularization, with λ chosen based on performance on the training set.
Revised: Imagine we have trained a classifier using regularization, with λ chosen based on performance on the training set.

Original: Which will have the highest accuracy the training set, the validation set or the test set?"
Revised: Which will have the highest accuracy the training set, the validation set or the test set?"

Original: [Table]
Revised: [Table]

Original: Logistic Regression
Revised: Logistic Regression

Original: Generated Question: “In logistic regression, we use the sigmoid function to "squish" the output of our linear function to be between 0 and 1.
Revised: Generated Question: “In logistic regression, we use the sigmoid function to "squish" the output of our linear function to be between 0 and 1.

Original: What is the output of the sigmoid function when the input is 0?"
Revised: What is the output of the sigmoid function when the input is 0?"

Original: Closest Question: “Consider the simplest of all neural networks, consisting of a single unit with a sigmoid activation function: h ( x ; w = σ ( w 0 + w 1 x ) where σ ( z ) =
Revised: Closest Question: “Consider the simplest of all neural networks, consisting of a single unit with a sigmoid activation function: h ( x ; w = σ ( w 0 + w 1 x ) where σ ( z ) =

Original: (1 + exp ( − z ))
Revised: (1 + exp ( − z ))

Original: − 1 Let’s start with a classifier defined by w 0 = − 1 and w 1 = 1 .
Revised: − 1 Let’s start with a classifier defined by w 0 = − 1 and w 1 = 1 .

Original: Which range of input values x are classified as positive?
Revised: Which range of input values x are classified as positive?

Original: Which as negative?"
Revised: Which as negative?"

Original: [Table]
Revised: [Table]

Original: Features
Revised: Features

Original: Generated Question: “You are working on a project to predict how successful a given Kickstarter campaign will be, based on information about the campaign (such as the amount of money pledged, the number of backers, the category of the project, the country where the project is based, and so forth).
Revised: Generated Question: “You are working on a project to predict how successful a given Kickstarter campaign will be, based on information about the campaign (such as the amount of money pledged, the number of backers, the category of the project, the country where the project is based, and so forth).

Original: One of the features you want to use is the project’s title.
Revised: One of the features you want to use is the project’s title.

Original: What is a good way to encode the project’s title for use in a machine-learning predictor?"
Revised: What is a good way to encode the project’s title for use in a machine-learning predictor?"

Original: Closest Question: “Mac O’Larnin is considering selling an app on Frugal Play.
Revised: Closest Question: “Mac O’Larnin is considering selling an app on Frugal Play.

Original: You have a friend with inside info at Frugal, and they’re able to share data on how previous apps have performed on the store.
Revised: You have a friend with inside info at Frugal, and they’re able to share data on how previous apps have performed on the store.

Original: Mac decides that he will learn a neural network with no hidden layer (i.e., consisting only of the output layer).
Revised: Mac decides that he will learn a neural network with no hidden layer (i.e., consisting only of the output layer).

Original: He needs help in figuring out the precise formulation for machine learning.
Revised: He needs help in figuring out the precise formulation for machine learning.

Original: What is the best way to encode the app characteristic ’Suitable for people ages (2–4, 5–10, 11–15, 16 and over)’ as a feature for an input to the neural network?
Revised: What is the best way to encode the app characteristic ’Suitable for people ages (2–4, 5–10, 11–15, 16 and over)’ as a feature for an input to the neural network?

Original: Choose from among the following: multiple unary features (one-hot encoding), multiple binary features (thermometer encoding), an integer or real-valued feature.
Revised: Choose from among the following: multiple unary features (one-hot encoding), multiple binary features (thermometer encoding), an integer or real-valued feature.

Original: Also give the exact function that maps each input to its corresponding feature(s)."
Revised: Also give the exact function that maps each input to its corresponding feature(s)."

Original: [Table]
Revised: [Table]

Original: Loss Functions
Revised: Loss Functions

Original: Generated Question: “We have a data set of images, each of which is either a picture of a dog or a picture of a cat.
Revised: Generated Question: “We have a data set of images, each of which is either a picture of a dog or a picture of a cat.

Original: We want to train a binary classifier to distinguish between the two classes.
Revised: We want to train a binary classifier to distinguish between the two classes.

Original: Which of the following is a better choice of loss function for this task:
Revised: Which of the following is a better choice of loss function for this task:

Original: (a) The 0-1 loss, which gives a loss of 1 if the classifier predicts the wrong class and a loss of 0 otherwise.
Revised: (a) The 0-1 loss, which gives a loss of 1 if the classifier predicts the wrong class and a loss of 0 otherwise.

Original: (b) The cross-entropy loss, which is given by -log(p) if the correct class is predicted and -log(1-p) otherwise, where p is the predicted probability of the correct class.
Revised: (b) The cross-entropy loss, which is given by -log(p) if the correct class is predicted and -log(1-p) otherwise, where p is the predicted probability of the correct class.

Original: (c) The hinge loss, which is given by max(0, 1-s) if the correct class is predicted and max(0, 1+s)
Revised: (c) The hinge loss, which is given by max(0, 1-s) if the correct class is predicted and max(0, 1+s)

Original: otherwise, where s is the predicted score for the correct class."
Revised: otherwise, where s is the predicted score for the correct class."

Original: Closest Question: “Consider the following 2D dataset in (x,y) format: ((1,-1), +1), ((1,1), +1), ((1,2.5),+1), ((2,-2),-1), ((2,1),+1),((2,3),+1),((5,- 1),-1),((5,-2),-1).
Revised: Closest Question: “Consider the following 2D dataset in (x,y) format: ((1,-1), +1), ((1,1), +1), ((1,2.5),+1), ((2,-2),-1), ((2,1),+1),((2,3),+1),((5,- 1),-1),((5,-2),-1).

Original: Break ties in distance by choosing the point with smaller x 1 coordinate, and if still tied, by smaller x 2 coordinate.
Revised: Break ties in distance by choosing the point with smaller x 1 coordinate, and if still tied, by smaller x 2 coordinate.

Original: Compute the leave-one-out cross validation accuracy of the 3-nearest-neighbor learning algorithm on this dataset."
Revised: Compute the leave-one-out cross validation accuracy of the 3-nearest-neighbor learning algorithm on this dataset."

Original: [Table]
Revised: [Table]

Original: Neural Networks
Revised: Neural Networks

Original: Generated Question: “Consider a neural network with two hidden layers, each with two units.
Revised: Generated Question: “Consider a neural network with two hidden layers, each with two units.

Original: The input layer has two units, and the output layer has one unit.
Revised: The input layer has two units, and the output layer has one unit.

Original: The hidden layers use the sigmoid activation function, and the output layer uses the linear activation function.
Revised: The hidden layers use the sigmoid activation function, and the output layer uses the linear activation function.

Original: The weights from the input layer to the first hidden layer are w11 = 1, w12 = 1, w21 = 1, and w22 = 1. The weights from the first hidden layer to the second hidden layer are w11 = 1, w12 = 1, w21 = 1, and w22 = 1.
Revised: The weights from the input layer to the first hidden layer are w11 = 1, w12 = 1, w21 = 1, and w22 = 1. The weights from the first hidden layer to the second hidden layer are w11 = 1, w12 = 1, w21 = 1, and w22 = 1.

Original: The weights from the second hidden layer to the output layer are w11 = 1, w21 = 1.
Revised: The weights from the second hidden layer to the output layer are w11 = 1, w21 = 1.

Original: The bias terms are all zero.
Revised: The bias terms are all zero.

Original: What is the output of the neural network for the input x1 = 1, x2 = 1?"
Revised: What is the output of the neural network for the input x1 = 1, x2 = 1?"

Original: Closest Question: “A neural network is given as Z 1 = X ∗ W 1 ,
Revised: Closest Question: “A neural network is given as Z 1 = X ∗ W 1 ,

Original: A 1 = f 1( Z 1 ) , Z 2 =
Revised: A 1 = f 1( Z 1 ) , Z 2 =

Original: W 2 ∗
Revised: W 2 ∗

Original: A 1 , ˆ y =
Revised: A 1 , ˆ y =

Original: f 2 ( Z 2 ) .
Revised: f 2 ( Z 2 ) .

Original: Specifically, the input X is a 4 × 1 column vector, ˆ y is a 1 × 1 scalar.
Revised: Specifically, the input X is a 4 × 1 column vector, ˆ y is a 1 × 1 scalar.

Original: W 2 is a 3 × 1 vector.
Revised: W 2 is a 3 × 1 vector.

Original: We also know that, Z 1 = ( W 1 ) T X and Z 2 =
Revised: We also know that, Z 1 = ( W 1 ) T X and Z 2 =

Original: ( W 2 ) T A 1 .
Revised: ( W 2 ) T A 1 .

Original: What are the dimensions of Z 2 ?"
Revised: What are the dimensions of Z 2 ?"

Original: CNNs
Revised: CNNs

Original: Generated Question: “Suppose we have a 3x3 image and we use a 2x filter with stride 1.
Revised: Generated Question: “Suppose we have a 3x3 image and we use a 2x filter with stride 1.

Original: What are the dimensions of the output image?"
Revised: What are the dimensions of the output image?"

Original: Closest Question: “A neural network is given as Z 1 = X ∗ W 1 ,
Revised: Closest Question: “A neural network is given as Z 1 = X ∗ W 1 ,

Original: A 1 = f 1 ( Z 1 ) , Z 2 =
Revised: A 1 = f 1 ( Z 1 ) , Z 2 =

Original: W 2 ∗
Revised: W 2 ∗

Original: A 1 , ˆ y =
Revised: A 1 , ˆ y =

Original: f 2 ( Z 2 ) .
Revised: f 2 ( Z 2 ) .

Original: There is only one data point which is: X = [1 , 1 , 1 , 1] T and y =
Revised: There is only one data point which is: X = [1 , 1 , 1 , 1] T and y =

Original: If W 1 and W 2 are both matrices/vectors of all ones, what is the resulting Loss where the Loss =
Revised: If W 1 and W 2 are both matrices/vectors of all ones, what is the resulting Loss where the Loss =

Original: ( y − ˆ y ) 2 ?" 0.
Revised: ( y − ˆ y ) 2 ?" 0.

Original: [Table]
Revised: [Table]

Original: Generated Question: “Consider an MDP with four states, called A, B, C , and D , and with two actions called Move and Stay.
Revised: Generated Question: “Consider an MDP with four states, called A, B, C , and D , and with two actions called Move and Stay.

Original: The discount factor γ = 0 .
Revised: The discount factor γ = 0 .

Original: Here is a reminder of the Q-learning update formula, based on experience tuple ( s, a, r, s ′ ) :
Revised: Here is a reminder of the Q-learning update formula, based on experience tuple ( s, a, r, s ′ ) :

Original: [Equation]
Revised: [Equation]

Original: Let α = 1 .
Revised: Let α = 1 .

Original: Assume we see the following state-action-reward sequence:
Revised: Assume we see the following state-action-reward sequence:

Original: A, Move, 0 B, Move, 0 C, Move, 1 A, Move, 0 B, Move, 0 C, Move,
Revised: A, Move, 0 B, Move, 0 C, Move, 1 A, Move, 0 B, Move, 0 C, Move,

Original: With Q-values all starting at 0, we run the Q-learning algorithm on that state-action sequence.
Revised: With Q-values all starting at 0, we run the Q-learning algorithm on that state-action sequence.

Original: Provide the q-learning value for Q(C, Move)."
Revised: Provide the q-learning value for Q(C, Move)."

Original: Closest Question: “Consider an MDP with four states, called
Revised: Closest Question: “Consider an MDP with four states, called

Original: A, B, C , and D , and with two actions called Move and Stay.
Revised: A, B, C , and D , and with two actions called Move and Stay.

Original: The discount factor γ = 0 .
Revised: The discount factor γ = 0 .

Original: Here is a reminder of the Q-learning update formula, based on experience tuple ( s, a, r, s ′ ) :
Revised: Here is a reminder of the Q-learning update formula, based on experience tuple ( s, a, r, s ′ ) :

Original: [Equation]
Revised: [Equation]

Original: Let α = 1 .
Revised: Let α = 1 .

Original: Assume we see the following state-action-reward sequence:
Revised: Assume we see the following state-action-reward sequence:

Original: A, Move, 0 B, Move, 0 C, Move, 1 A, Move, 0 With Q-values all starting at 0, we run the Q-learning algorithm on that state-action sequence.
Revised: A, Move, 0 B, Move, 0 C, Move, 1 A, Move, 0 With Q-values all starting at 0, we run the Q-learning algorithm on that state-action sequence.

Original: Provide the q-learning value for Q(A, move).”
Revised: Provide the q-learning value for Q(A, move).”

Original: [Table]
Revised: [Table]

Original: Generated Question: “Consider the following RNN: s t = tanh ( w 1 x t + w 2 s t − 1 + b ) , y t =
Revised: Generated Question: “Consider the following RNN: s t = tanh ( w 1 x t + w 2 s t − 1 + b ) , y t =

Original: w 3 s t + b 2 .
Revised: w 3 s t + b 2 .

Original: Assume s 0 = 0 and b 2 = 0 .
Revised: Assume s 0 = 0 and b 2 = 0 .

Original: What values of w 1 , w 2 , w 3 and b would generate output sequence [0 , 0 , 0 , 1 , 1 , 1 , 1] given input sequence [0 , 0 , 0 , 1 , 0 , 1 , 0] ”
Revised: What values of w 1 , w 2 , w 3 and b would generate output sequence [0 , 0 , 0 , 1 , 1 , 1 , 1] given input sequence [0 , 0 , 0 , 1 , 0 , 1 , 0] ”

Original: Closest Question: “Ronnie makes a simple RNN with state dimensionand a step function for f 1 , so that s t
Revised: Closest Question: “Ronnie makes a simple RNN with state dimensionand a step function for f 1 , so that s t

Original: = step ( w 1 x t + w 2 s t − 1 + b )
Revised: = step ( w 1 x t + w 2 s t − 1 + b )

Original: where step ( z )
Revised: where step ( z )

Original: = 

Revised: =

Original: if z > 0 . 0 and equals 0 otherwise, and where the output y t =
Revised: if z > 0 . 0 and equals 0 otherwise, and where the output y t =

Original: s t .
Revised: s t .

Original: Assuming s 

Revised: Assuming s

Original: = 1 , we want to generate output sequence [0 , 0 , 0 , 1 , 1 , 1 , 1] given input sequence [0 , 0 , 0 , 1 , 0 , 1 , 0] .
Revised: = 1 , we want to generate output sequence [0 , 0 , 0 , 1 , 1 , 1 , 1] given input sequence [0 , 0 , 0 , 1 , 0 , 1 , 0] .

Original: Rennie thinks this is not possible using Ronnie’s architecture.
Revised: Rennie thinks this is not possible using Ronnie’s architecture.

Original: Rennie makes an argument based on the relationships in the table above.
Revised: Rennie makes an argument based on the relationships in the table above.

Original: Is Rennie right?"
Revised: Is Rennie right?"

Original: Reinforcement Learning
Revised: Reinforcement Learning

Original: Generated Question: “What is the tabular Q-learning update equation, based on experience tuple ( s, a, r, s ′ ) ?"
Revised: Generated Question: “What is the tabular Q-learning update equation, based on experience tuple ( s, a, r, s ′ ) ?"

Original: Closest Question: “b is the bias for classifier network.
Revised: Closest Question: “b is the bias for classifier network.

Original: What are dimensions of b for multi k-class classification?"
Revised: What are dimensions of b for multi k-class classification?"

Original: [Table]
Revised: [Table]

Original: Generated Question: “Suppose that we have a dataset with n data points, k clusters, and d features.
Revised: Generated Question: “Suppose that we have a dataset with n data points, k clusters, and d features.

Original: After running the k -means algorithm, the within-cluster sum of squared errors (WCSS) is given by:
Revised: After running the k -means algorithm, the within-cluster sum of squared errors (WCSS) is given by:

Original: [Equation]
Revised: [Equation]

Original: where y i is the cluster label of the i th data point, and µ y i is the cluster center associated with the i th data point.
Revised: where y i is the cluster label of the i th data point, and µ y i is the cluster center associated with the i th data point.

Original: The within-cluster sum of squared errors (WCSS) is a measure of how well the clusters fit the data.
Revised: The within-cluster sum of squared errors (WCSS) is a measure of how well the clusters fit the data.

Original: Suppose that we have two datasets, X 1 and X 2 , where X 1 has n 1 data points and X 2 has n 2 data points.
Revised: Suppose that we have two datasets, X 1 and X 2 , where X 1 has n 1 data points and X 2 has n 2 data points.

Original: We run the k -means algorithm on both datasets.
Revised: We run the k -means algorithm on both datasets.

Original: We find that the WCSS for X 1 is smaller than the WCSS for X 2 .
Revised: We find that the WCSS for X 1 is smaller than the WCSS for X 2 .

Original: Does this imply that the clusters for X 1 are better than the clusters for X 2 ?
Revised: Does this imply that the clusters for X 1 are better than the clusters for X 2 ?

Original: Why or why not?"
Revised: Why or why not?"

Original: Closest Question: “Consider the following 2D dataset in (x,y) format: ((1,-1), +1), ((1,1), +1), ((1,2.5),+1), ((2,-2),-1), ((2,1),+1),((2,3),+1),((5,- 1),-1),((5,-2),-1).
Revised: Closest Question: “Consider the following 2D dataset in (x,y) format: ((1,-1), +1), ((1,1), +1), ((1,2.5),+1), ((2,-2),-1), ((2,1),+1),((2,3),+1),((5,- 1),-1),((5,-2),-1).

Original: We will construct a tree using a greedy algorithm that recursively minimizes weighted average entropy.
Revised: We will construct a tree using a greedy algorithm that recursively minimizes weighted average entropy.

Original: Recall that the weighted average entropy of a split into subsets A and B is: (fraction of points in A ) · H (cid:0) R Aj,s (cid:1) + ( fraction of points in B ) · H (cid:0) R Bj,s (cid:1) where the entropy H ( R m ) of data in a region R m is given by
Revised: Recall that the weighted average entropy of a split into subsets A and B is: (fraction of points in A ) · H (cid:0) R Aj,s (cid:1) + ( fraction of points in B ) · H (cid:0) R Bj,s (cid:1) where the entropy H ( R m ) of data in a region R m is given by

Original: H ( R m )
Revised: H ( R m )

Original: =
Revised: =

Original: − (cid:80) k
Revised: − (cid:80) k

Original: ˆ P mk log 2 ˆ P mk .
Revised: ˆ P mk log 2 ˆ P mk .

Original: The ˆ P mk is the empirical probability, which is in this case the fraction of items in region m that are of class k .
Revised: The ˆ P mk is the empirical probability, which is in this case the fraction of items in region m that are of class k .

Original: Some facts that might be useful to you: H(0) = 0, H(3/5) = 0.97,
Revised: Some facts that might be useful to you: H(0) = 0, H(3/5) = 0.97,

Original: H(3/8)
Revised: H(3/8)

Original: = 0.95, H(3/4)
Revised: = 0.95, H(3/4)

Original: = 0.81, H(5/6)
Revised: = 0.81, H(5/6)

Original: = 0.65, H(1)
Revised: = 0.65, H(1)

Original: = 0.
Revised: = 0.

Original: Draw the decision tree that would be constructed by our tree algorithm for this dataset.
Revised: Draw the decision tree that would be constructed by our tree algorithm for this dataset.

Original: Clearly label the test in each node, which case (yes or no) each branch corresponds to, and the prediction that will be made at each leaf.
Revised: Clearly label the test in each node, which case (yes or no) each branch corresponds to, and the prediction that will be made at each leaf.

Original: Assume there is no pruning and that the algorithm runs until each leaf has only members of a single class."
Revised: Assume there is no pruning and that the algorithm runs until each leaf has only members of a single class."

Original: Decision Trees
Revised: Decision Trees

Original: Generated Question: “The Gini score is a measure of how often a randomly chosen element would be incorrectly labeled if it was randomly labeled according to the distribution of labels in the subset.
Revised: Generated Question: “The Gini score is a measure of how often a randomly chosen element would be incorrectly labeled if it was randomly labeled according to the distribution of labels in the subset.

Original: It ranges from 0 to 1, with 0 meaning that there are no mislabeled elements and 1 meaning that the subset is perfectly mixed.
Revised: It ranges from 0 to 1, with 0 meaning that there are no mislabeled elements and 1 meaning that the subset is perfectly mixed.

Original: Explain whether or not it would be a good idea to use the Gini score as a scoring function for pruning decision trees."
Revised: Explain whether or not it would be a good idea to use the Gini score as a scoring function for pruning decision trees."

Original: Closest Question: “There are different strategies for pruning decision trees.
Revised: Closest Question: “There are different strategies for pruning decision trees.

Original: We assume that we grow a decision tree until there is one or a small number of elements in each leaf.
Revised: We assume that we grow a decision tree until there is one or a small number of elements in each leaf.

Original: Then, we prune by deleting individual leaves of the tree until the score of the tree starts to get worse.
Revised: Then, we prune by deleting individual leaves of the tree until the score of the tree starts to get worse.

Original: The question is how to score each possible pruning of the tree.
Revised: The question is how to score each possible pruning of the tree.

Original: Here is a definition of the score: The score is the percentage correct of the tree on a separate validation set.
Revised: Here is a definition of the score: The score is the percentage correct of the tree on a separate validation set.

Original: Explain whether or not it would be a good idea and give a reason why or why not."
Revised: Explain whether or not it would be a good idea and give a reason why or why not."