Giving Feedback on Interactive Student Programs with Meta-Exploration
Anonymous Author(s)
AffiliationAddress email
Abstract
Creating interactive software, such as websites or games, is a particularly engagingway to learn computer science.
However, teaching and giving feedback on suchsoftware is hard — standard approaches require instructors to hand grade studentimplemented interactive programs.
As a result, online platforms that serve millions,like Code.org, are unable to provide any feedback on assignments for implementinginteractive programs, which critically hinders students’ ability to learn.
Recentwork proposes to train reinforcement learning agents to interact with a student’sprogram, aiming to explore states indicative of errors.
However, this approach onlyprovides binary feedback of whether a program is correct or not, while studentsrequire finer-grained feedback on the specific errors in their programs to understandtheir mistakes.
In this work, we show that exploring to discover errors can be cast asa meta-exploration problem.
This enables us to construct a principled objective fordiscovering errors and an algorithm for optimizing this objective, which providesfine-grained feedback.
We evaluate our approach on a set of 700K real anonymizedstudent programs from a Code.org interactive assignment.
Our approach providesfeedback with 94.3% accuracy, improving over existing approaches by over 17.7%and coming within 1.5% of human-level accuracy.
Introduction
[Figure] [Figure]
provide [
We focus on onearea where providing feedback isparticularly burdensome: moderncomputer science education, wherestudents are often tasked with developing interactive programs, such aswebsites or games (e.g., see Figure 1).
While developing such programsis highly engaging [29] and hasbecome ubiquitous in contemporaryclassrooms [12], these programscan include stochastic or creativeelements, so they cannot be gradedwith traditional unit tests, and mustinstead be manually graded.
However,such manual grading is increasinglyinfeasible with the growing demand for computer science education and rise of massive online learning platforms.
For example, one popular platform, Code.org has enrolled over 72M students [7].
As manually grading a submission can take up to 6 minutes, a single assignment creates decadesof grading labor.
Consequently, Code.org cannot yet provide feedback about whether an interactiveassignment submission is correct or not, let alone more fine-grained feedback.
To alleviate this enormous grading burden, Nie et al.
[25] introduce the Play-to-Grade paradigm forautomatically providing feedback by training a reinforcement learning agent to grade a program thesame way humans do: by interacting or playing with the program.
The idea is for the agent to visitstates that reveal errors in the program, and then aggregate this information as feedback.
Such an agentis trained on a set of training programs labeled with feedback (e.g., provided by an instructor), and thegoal is to generalize to new student programs.
Figure 1 shows an example learned agent that tests whathappens when the ball is hit into the goal, exposing an error where the ball bounces off the goal insteadof entering and scoring a point.
The state-of-the-art approach in this paradigm provides highly accuratecoarse feedback of whether the program is completely correct or not [25].
However, to understandtheir mistakes, students usually require feedback about what specific errors are in their programs.
Learning an agent to explore and discover the errors in a program to provide such fine-grainedfeedback is challenging: Most errors cannot be discovered with simple random exploration, andinstead require targeted exploration, such as deliberately hitting the ball into the goal.
In addition, theagent must be able to adapt its exploration to different programs, which each behave differently andmay present unexpected obstacles, such as multiple balls.
Our key technical insight is that learningto discover errors connects with the meta-exploration problem in meta-reinforcement learning (metaRL).
This insight enables us to leverage techniques from the meta-exploration literature to constructand optimize a principled objective for producing fine-grained feedback.
Specifically, we follow the
Play-to-Grade paradigm and assume access to 3500 training programs, labeled with the errors in theprogram.
Then, we formulate the problem as maximizing the mutual information between states visited by our agent and the label.
Finally, we use techniques from the D REAM meta-RL algorithm [20]to decompose this objective into shaped rewards that enable learning sophisticated exploration.
Overall, the main contribution of this work is to connect the Play-to-Grade paradigm with themeta-RL literature, and consequently, to provide an effective method for providing fine-grainedfeedback for interactive student programs, which we call D REAM G RADER .
Additionally, we releaseour code to propose automatic feedback as a new meta-RL benchmark that fulfills an unmet needin the community for a meta-RL benchmark that is simultaneously readily accessible and directlyimpactful.
We evaluate our system on 711,274 anonymized student submissions of the Bounceassignment from Code.org [25].
Trained on 3500 programs, D REAM G RADER achieves an accuracyof 94.3%, which improves over existing approaches by 17.7% and comes within 1.5% of human-levelgrading accuracy.
In addition, our approach can significantly reduce instructor grading burden: whilemanually grading all student submissions would require 4 years of work, our system can grade thesame set of programs in only 3 hours on a single GPU.
Related Works
Educational feedback.
We consider the problem of automatically providing feedback, which playsan important role in student learning and motivation [27].
Though we specifically focus on feedback,other works also leverage machine learning for other aspects of education, including tracking whatstudents know [40, 8, 30], predicting student retention [9, 4, 35, 3], and building intelligent tutoringsystems [1, 26].
Work on automatically providing feedback for computer science assignments focuseson two main approaches: analyzing either the code or behavior of a program.
Methods that analyzecode provide feedback by passing the code through a neural network [31, 6, 21, 45], or by constructingsyntax trees [37, 43], e.g., to predict useful next implementation steps [33, 28].
Analyzing codeworks well for shorter programs (e.g., under 50 lines of code) and has even been deployed in onlinecourses [46].
However, this approach struggles to scale to lengthier or more complex programs.
Hence, we instead opt for the second approach of analyzing program behavior, which convenientlydoes not depend on program length, though it requires the program to be executable.
Arguably, the simplest method of analyzing program behavior is unit testing.
Unit testing canprovide automatic feedback to some extent when the desired output of each input is known, but thisis typically not the case with interactive programs, such as websites or games.
Instead, work onautomated testing can provide feedback by generating corner-case inputs that reveal errors via inputfuzzing [13], symbolic generation [19], or reinforcement learning exploration objectives [48, 14].
However, this line of work assumes that errors are easy to detect when revealed, while detectinga revealed error itself can be challenging [25].
Consequently, Nie et al.
[25] propose the Play-to-Grade paradigm to both learn an agent to discoverstates that reveal errors, and a model to detect and output the revealed errors.
Our work builds uponthe Play-to-Grade paradigm, but differs from Nie et al.
[25] in the provided feedback.
While Nieet al.
[25] only provide coarse binary feedback of whether a program is correct, we introduce anew principled objective to provide fine-grained feedback of what specific errors are present to helpstudents understand their mistakes.
Meta-reinforcement learning.
To provide fine-grained feedback, we connect the problem ofdiscovering errors with the meta-exploration problem in meta-RL.
There is a rich literature ofapproaches that learn to explore via meta-RL [15, 38, 32, 34, 49, 50, 17, 16, 18, 20].
We specificallyleverage ideas from the D REAM algorithm [20] to construct a shaped reward function for learningexploration.
Our work has two key differences from prior meta-RL research.
First, we introducea novel factorization of the D REAM objective that better generalizes to new programs.
Second,and more importantly, we focus on the problem of providing feedback on interactive programs.
This differs from a large body of meta-RL work that focuses on interesting, yet synthetic problems,such as 2D and 3D maze navigation [10, 22, 50, 20], simulated control problems [11, 32, 47], andalchemy [42].
While meta-RL has been applied to realistic settings in robotics [24, 2, 36], this workprovides a new benchmark meta-RL problem that is both realistic and readily accessible.
The Fine-Grained Feedback Problem
We consider the problem of automatically providing feedback on programs.
During training, weassume access to a set of programs labeled with the errors made in the program (i.e., ground-truthinstructor feedback).
During testing, the grading system is presented with a new student programand must output feedback of what errors are in the program.
To produce this feedback, the gradingsystem is allowed to interact with the program.
More formally, we consider a distribution over programs p ( µ ) , where each program µ defines a
Markov decision process (MDP) µ = ⟨S , A , T , R⟩ with states S , actions A , dynamics T , andrewards R .
We assume that the instructor creates a rubric : an ordered list of K potential errors thatcan occur in a program.
Each program µ is associated with a ground-truth label y ∈ { 0 , 1 } K ofwhich errors are made in the program.
The k th index y k denotes that the k th error of the rubric ispresent in the program µ .
During training, the grading system is given a set of N labeled training programs { ( µ n , y n ) } Nn =1 .
The goal is to learn a feedback function f that takes a program µ and predicts the label ˆ y =
f ( µ ) tomaximize the expected grading accuracy J grade ( f ) over test programs µ with unobserved labels y :
[Equation]
where I is an indicator variable.
Effectively, J grade measures the per-rubric item accuracy of predictingthe ground-truth label y .
To predict the label y , the feedback function may optionally interact withthe MDP µ defined by the program for any small number of episodes.
Bounce programming assignment.
Though the methods we propose in this work generally applyto any interactive programs with instructor-created rubrics, we specifically consider the Bounceprogramming assignment from Code.org, a real online assignment that has been completed nearlya million times.
As providing feedback for interactive assignments is challenging, the assignmentcurrently provides no feedback whatsoever on Code.org, and instead relies on the student to discovertheir own mistakes by playing their program.
This assignment is illustrated in Figure 1.
Each studentprogram defines an MDP, where we use the state representation from Nie et al.
[25] — each stateconsists of the ( x, y ) -coordinates of the paddle and balls, as well as the ( x, y ) -velocities of the balls.
There are three actions: moving the paddle left or right, or keeping the paddle in the current position.
In the dynamics of a correct program, the ball bounces off the paddle and wall.
When the ball hits thegoal or floor, it disappears and launches a new ball, which increments the player score and opponentscore respectively.
However, the student code may define other erroneous dynamics, such as the ballpassing through the paddle or bouncing off the goal.
The reward function is +1 when the playerscore increments and − 1 when the opponent score increments.
An episode terminates after 100 stepsor if either the player or opponent score exceeds 30 .
Our experiments use a dataset of 711,274 real anonymized student submissions to this assignment,released by Nie et al.
We use 0 .
5% of these programs for training, corresponding to N = 3556and uniformly sample from the remaining programs for testing.
[Table]
Possible errors in a student programtake the form of “when event occurs,there is an incorrect consequence ,”where the list of all events and consequences is listed in Table 1.
Forexample, Figure 1 illustrates the errorwhere the event is the ball hitting thegoal, and the consequence is that theball incorrectly bounces off the goal, rather than entering the goal.
For simplicity, we consider arepresentative rubric of K = 8 errors, spanning all event and consequence types, listed in Appendix A.
Prior approach for program feedback.
This problem is challenging because the agent must explorein a targeted way to discover all potential errors and must be able to adapt to variability in theprograms.
Prior work by Nie et al.
[25] sidesteps this challenge by determining whether a studentprogram is distinct from a reference solution program, which only results in coarse feedback ofwhether a program is correct or not.
In the next section, we present a new approach that instead targetsexploration toward uncovering specific misconceptions to effectively provide fine-grained feedback.
Automatically Providing Fine-Grained Feedback with D REAM G RADER
In this section, we detail our approach, D REAM G RADER , for automatically providing fine-grainedfeedback to help students understand their mistakes.
From a high level, D REAM G RADER learns twocomponents that together form the feedback function f :
(i)
An exploration policy π that acts on a program µ to produce a trajectory τ = ( s 0 , a 0 , r 0 , . . . ) .
(ii) A feedback classifier g ( y | τ ) that defines a distribution over labels y given a trajectory τ .
The idea is to explore states that either indicate or rule out errors with the exploration policy, and thensummarize the discovered errors with the feedback classifier.
To provide feedback on a new programµ , we first roll out the exploration policy π on the program to obtain a trajectory τ , and then obtainthe predicted label arg max y g
( y | τ ) by applying the feedback classifier.
Under this parametrizationof the feedback function f , we can rewrite the expected grading accuracy objective in Equation 1 as:
[Equation]
where π ( µ ) denotes the distribution over trajectories from rolling out the policy π on the program µ .
After this rewriting of the objective, our approach is conceptually straightforward: we learn boththe exploration policy and classifier to maximize our rewritten objective.
We can easily learn thefeedback classifier g by maximizing the probability of the correct label given a trajectory generatedby the exploration policy (i.e., cross-entropy loss), but learning the exploration policy π is morechallenging.
Note that we could directly optimize our objective in Equation 2 by treating the insideof the expectation as a reward received at the end of the episode and use this to learn the explorationpolicy π with reinforcement learning.
However, this reward signal makes credit assignment difficultfor learning the exploration policy, since it is given at the end of the episode, rather than at the stateswhere the exploration policy discovers errors.
Consequently, we empirically find that learning fromthis reward signal struggles to adequately explore (Section 5).
Hence, our goal is instead to construct a reward signal that helps assign credit for the explorationpolicy and provides rewards at the states that indicate or rule out errors in the program.
To do this,we propose an alternative objective that is sufficient to maximize the Play-to-Grade objective, butcan be decomposed into per-timestep rewards that correctly assign credit (Section 4.1).
Intuitively,these rewards leverage the feedback classifier to provide high reward when an action leads to a newstate that causes the classifier to become very certain about whether an error is present.
After, wedetail practical design choices for implementing this approach with neural networks (Section 4.2) andconclude by drawing a connection between learning to find errors and the meta-exploration problem,which motivates our choice of alternative objective and its subsequent decomposition (Section 4.3). [Figure] [Figure] [Figure]
[Figure]
launched
Error type: Does the into the score?
𝚫
Score 0 :
4.1 Assigning Credit to Learn Exploration
We now obtain a reward signal to help assign credit for learning exploration.
From a high level, wefirst propose to maximize a mutual information objective that is sufficient to maximize our objective
J D REAM G RADER ( π, g ) in Equation 2.
We then rewrite our mutual information objective in terms ofper-timestep exploration rewards related to the information gain of the feedback classifier on the truelabel y when it observes the transition ( s t , a t , r t ) .
This helps assign credit for learning the explorationpolicy, as the transitions that either indicate or rule out errors in the program are exactly those thathave high information gain.
Figure 2 illustrates an example of the derived exploration rewards.
Objective.
Intuitively, we want our exploration policy to visit states that either indicate or rule outerrors.
We can formalize this intuition by maximizing the mutual information I ( τ ; y ) between thetrajectories τ ∼ π visited by the policy and the feedback label of the program y , which causes thepolicy to visit states that make the feedback label highly predictable.
Importantly, maximizing thisobjective is sufficient to maximize the expected grading accuracy J D REAM G RADER ( π, g ) in Equation 2:
Let p ( y | τ ) be the true posterior over labels given trajectories τ ∼ π sampled from the policy.
Maximizing the mutual information I ( τ ; y ) produces trajectories that maximize the probability ofthe label under the true posterior p ( y | τ ) .
Then, if the feedback classifier is learned to matchthe true posterior g ( y | τ ) =
p ( y | τ ) while I ( τ ; y ) is maximized, the expected grading accuracy
J D REAM G RADER ( π, g ) is also maximized.
Optimization.
We can efficiently maximize our objective I ( τ ; y ) by maximizing a variational lowerbound [5] and decomposing the lower bound into a per-timestep reward that helps assign credit forlearning the exploration policy π .
We derive this for the case of learning only a single explorationpolicy π to uncover all errors, though we will later discuss how we can factorize this to learn Nexploration policies { π i } Ni =1 that each explore to uncover a single error type.
[Equation]
[Equation]
[Equation]
where r exp t
= log g
( y | τ : t +1 ) − log g ( y | τ : t ) and
T is the length of the trajectory τ =
( s 0 , a 0 , r 0 , . . .
, s T ) .
The inequality in (5) holds for replacing the true posterior p ( y | τ ) with any distribution, and (6)comes from expanding a telescoping series as done by D REAM [20], where τ : t = ( s 0 , a 0 , r 0 , . . .
, s t )denotes the the trajectory to the
This derivation provides a shaped reward function r exp t for learning the exploration policy.
Intuitively,this reward captures how much new information the transition ( s t , a t , r t , s t +1 ) provides to thefeedback classifier g on what errors are in the program: The reward is high if observing this transitioneither indicates an error (e.g., the ball enters the goal, but does not score a point) or rules out an error(e.g., the ball enters the goal and scores a point), and is low otherwise.
Additionally, we now have a recipe for maximizing the mutual information I ( τ ; y ) to learn both thepolicy π and feedback classifier g .
Only the second term in (6) depends on π and g .
Hence, we canmaximize this lower bound on I ( τ ; y ) by maximizing the log-likelihood of the label with respectto the feedback classifier J feedback ( g ) =
E µ ∼ p ( µ ) ,τ ∼ π [log g ( y | τ )] , and maximizing the rewardsr exp t = log g
( y | τ : t +1 ) − log g ( y | τ : t ) with respect to the policy π via reinforcement learning.
Factorizing our objective.
So far, our approach learns a single exploration policy that must uncoverall error types in the rubric.
However, learning such a policy can be challenging, especially ifuncovering different error types requires visiting very different states.
We instead propose to learn aseparate exploration policy π k for each error index of the rubric k = 1 , . . .
, K .
We can accomplishthis by observing that maximizing the mutual information
I (( τ 1 , . . .
, τ K ); y ) between K trajectoriesτ i ∼
π
i is also sufficient to maximize the expected grading accuracy.
Furthermore, maximizingthe mutual information with each dimension of the label I ( τ k ; y k ) , where τ k ∼ π k for each k ,is sufficient to maximize the mutual information with the entire label
I (( τ 1 , . . .
, τ K ); y ) .
Wetherefore can derive exploration rewards for each term
I ( τ k ; y k ) to learn each policy π k with rewardsr exp t = log g
( y k | τ : t +1 )
− log g ( y k | τ : t ) .
We find that this improves grading accuracy in ourexperiments (Section 5) and enables parallel training and testing for the K exploration policies.
4.2 A Practical Implementation
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Overall, D REAM G RADER consists of a feedbackclassifier g and K exploration policies { π k } Kk =1 ,where the k th policy π k tries to visit states indicative of whether the k th error type is present in theprogram.
We learn these components by repeatedlyrunning training episodes for each policy π k with
Algorithm 1.
We first sample a labeled training program and follow the policy on the program (lines 1–2).
Then, we maximize our mutual information objective by updating the policy with our explorationrewards (lines 3–4), and by updating the classifier to maximize the log-likelihood of the label (line 5).
In practice, we parametrize the exploration policies and feedback classifier as neural networks.
Sincethe exploration rewards r exp t depend on the past and are non-Markov, we make each explorationpolicy π k recurrent: At timestep t , the policy π k ( a t | ( s 0 , a 0 , r 0 , . . .
, s t ))
conditions on all past states,actions, and observed rewards for each k .
We parametrize each policy as a deep dueling double Qnetworks [23, 44, 39].
Consequently, our policy updates in line 4 consist of placing the trajectory in areplay buffer with rewards r exp t and sampling from the replay buffer to perform Q-learning updates.
Weparametrize each dimension of the feedback classifier g
( y k | τ ) for k = 1 , . . .
, K as a separate neuralnetwork.
We choose not to share parameters between the exploration policies and between the dimensions of the feedback classifier for simplicity.
See Appendix B for full architecture and model details.
4.3 Play-to-Grade as Meta-Exploration
Our choice to optimize the mutual information objective I ( τ ; y ) and decompose this objective intoper-timestep rewards using techniques from the D REAM meta-RL algorithm stems from the fact thatthe Play-to-Grade paradigm can be cast as a meta-exploration problem.
Specifically, meta-RL aimsto learn agents that can quickly learn new tasks by leveraging prior experience on related tasks.
Thestandard few-shot meta-RL setting formalizes this by allowing the agent to train on several MDPs(tasks).
Then, at time time, the agent is presented with a new MDP and is allowed to first explore thenew MDP for several episodes to gather information, before it must solve the MDP and maximizereturns.
Learning to efficiently spending these allowed few exploration episodes to best solve the test MDP is the meta-exploration problem.
In our setting, we can view each student program as a 1-step task of predicting the feedback label.
To predict this label, the Play-to-Grade paradigm first explores the program for several episodesto discover states indicative of errors, which is exactly the meta-exploration problem.
This bridgebetween identifying errors in programs and meta-exploration suggests that techniques from each body of literature could mutually benefit each other.
Indeed, D REAM G RADER leverages ideas from D REAM and future work could explore other techniques to transfer across the two areas.
Additionally,this connection also offers a new benchmark for meta-exploration and meta-RL research.
As discussedin Section 2, while existing meta-RL benchmarks tend to be either readily accessible or impactful andrealistic, automatically providing feedback simultaneously provides both, and we release code for ameta-RL wrapper of the Bounce programming assignment to spur further research in this direction.
Experiments
In our experiments, we aim to answer five main questions: (1) How does automated feedback gradingaccuracy compare to human grading accuracy?
(2) How does D REAM G RADER compare with Nieet al.
[25], the state-of-the art Play-to-Grade approach?
(3) What are the effects of our proposedfactorization and derived exploration rewards on D REAM G RADER ?
(4) How much human labor issaved by automating feedback?
(5) Interactive programs can be particularly challenging to gradebecause they can include unexpected creative elements not seen during training — how well doautomated feedback systems generalize to such creative elements?
To answer these questions weconsider the dataset of 700K real anonymized Bounce student programs, described in Section 3.
Below, we first establish the points of comparison to necessary answer these questions (Section 5.1).
Then, we evaluate these approaches to answer the first four questions (Section 5.2).
Finally, weanswer question (5) by evaluating D REAM G RADER on variants of Bounce student programs thatmodify the ball and paddle speeds, including speeds not seen during training (Section 5.3).
5.1 Points of Comparison
We compare with the following four approaches.
Unless otherwise noted, we train 3 seeds of eachautomated approach for 5M steps on N = 3556 training programs, consisting of 0 .
5% of the dataset.
Human grading.
To measure the grading accuracy of humans, we asked for volunteers to grade
Bounce programming assignments.
We obtained 9 volunteers consisting of computer science undergraduate and PhD students, 7 of whom had previously instructed or been a teaching assistant for acomputer science course.
Each volunteer received training on the Bounce programming assignmentand then was asked to grade 6 randomly sampled Bounce programs.
See Appendix C.1 for details.
Nie et al.
[25] extended to provide fine-grained feedback.
We extend the original Play-to-Gradeapproach, which provides binary feedback about whether a program is completely correct or not, toprovide fine-grained feedback.
Specifically, Nie et al.
[25] choose a small set of 10 training programs,curated so that each program exhibits a single error, and together, they span all errors.
Then, for eachtraining program, the approach learns (i) a distance function d ( s, a ) that takes a state-action tuple( s, a ) , trained to be large for tuples from the buggy training program and small for tuples from acorrect reference implementation; and (ii) an exploration policy that learns to visit state-actions whered ( s, a ) is large.
To provide feedback to a new student program, the approach runs each explorationpolicy on the program and outputs that the program has an error if any of the distance functions ishigh for any of the tuples visited by the exploration policies.
We extend this approach to provide fine-grained feedback by following a similar set up.
We followthe original procedure to train a separate policy and distance function on K = 8 curated trainingprograms, where the k th program exhibits only the k th error from the rubric we consider.
Then, toprovide fine-grained feedback on a new program, we run each policy on the new program and predictthat the k th error is present (i.e., ˆ y k
= 1 ) if the k th distance function is high on any state-action tuple.
We use code released by the authors without significant fine-tuning or modification.
We emphasizethat this approach only uses 8 curated training programs, as opposed to the N = 3556 randomlysampled programs used by other automated approaches, as this approach is not designed to use moretraining programs, and furthermore cannot feasibly scale to many more training programs, as it learnsa distance function and policy for every training program.
D REAM G RADER (direct max).
To study the effect of our derived exploration rewards r exp t , weconsider the approach of directly maximizing the D REAM G RADER objective in Equation 2, describedat the beginning of Section 4.1.
This approach treats the inside of the expectation as end-of-episodereturns, and does not provide explicit credit assignment.
This approach is equivalent to maximizingthe D REAM G RADER objective with the RL 2 meta-RL algorithm [10, 41]. [Figure]
D REAM G RADER (unfactorized).
Finally, to study the effect of our proposed factorization scheme,described at the end of Section 4.
[Figure]
we consider a variant of D REAM G RADER where we do notfactorize the objective and instead only learn a single exploration policy to uncover all errors.
5.2 Main Results
[Table]
We compare the approachesbased on grading accuracy,precision, recall, and F1scores averaged across the8 error types in the rubric.
Table 2 summarizes the results.
Overall, we find that
D REAM G RADER achievesthe highest grading accuracy of the automated grading approaches, providing feedback with 17.7%greater accuracy than Nie et al.
Furthermore, D REAM G RADER comes within 1.5% of humanlevel grading accuracy and actually achieves slightly superhuman F1.
D REAM G RADER achievesthis by learning exploration behaviors that probe each possible error event (see Appendix D forvisualizations of the learned behaviors).
Analysis.
To further understand these results, we plot the training curves of the grading accuracyon each error type in the rubric vs. the number of training steps of D REAM G RADER , as well asthe final grading accuracies of the other approaches in Figure 3.
The performance of variants of
D REAM G RADER underscores the importance of our design choices: D REAM G RADER (direct max)achieves significantly lower accuracy than D REAM G RADER across all error types, indicating theimportance of our shaped exploration rewards r exp t for learning effective exploration.
Additionally,while D REAM G RADER (unfactorized) achieves relatively high average accuracy, it still performsworse than D REAM G RADER .
This illustrates the difficulty of learning a single exploration policyto uncover all errors, which is alleviated by our factorization.
Additionally, we find that D REAM G RADER achieves human-level or even slightly superhumangrading accuracy on 5 of the 8 error types, but relatively struggles on the 3 errors on the right ofthe top row of Figure 3, which leads to overall lower average grading accuracy.
For the errorsinvolving incrementing the opponent score, we qualitatively find that D REAM G RADER most commonly struggles when there are multiple balls, which makes it difficult to ascertain which events arecausing the opponent score to increment.
D REAM G RADER similarly most frequently misclassifiesthe “no new ball after goal” error when multiple events launch balls.
Human grading was able to circumvent this issue, but humans required playing the program for up to 40 episodes, while welimited D REAM G RADER to a single episode per error type.
Reduction in human grading burden.
We found that our grading volunteers required between 1–6minutes to grade each program, averaging around 3 minutes.
At this grading speed, grading all ofthe 700K Code.org submissions would take close to 4 years of human labor.
In contrast, automaticgrading with D REAM G RADER requires only 1 second per program on a single NVIDIA RTX
GPU, which grades all 700K submissions in only 3 hours.
5.3 Feedback in the Face of Creativity
A key challenge in providing feedback for interactive programs is that student programs may includeunexpected creative elements, requiring generalization to unseen behaviors at test time.
The Bounceassignment provides one simple way for students to express creativity in their programs by selectingthe speeds of the ball and paddle.
There are five possible settings for the ball and paddle speeds: veryslow, slow, normal, fast, and very fast.
To test the ability of D REAM G RADER to generalize to unseenbehaviors at test time, we train D REAM G RADER on programs where we hold out the normal ball andpaddle speeds, and then test D REAM G RADER on programs with the held out speeds.
Specifically, weuse the same N = 3556 training programs as before, but we uniformly randomize the ball and paddlespeeds independently to be one of the four non-held out speeds.
Then we evaluate on test programswith (1) held out ball and paddle speeds; (2) held out ball speed with random training paddle speed;(3) held out paddle speed with random training ball speed; and (4) training ball and paddle speeds.
[Table]
D REAM G RADER success-
[Table]
fully generalizes to unseenball and paddle speeds attest time.
Compared to thestandard training in the previous section, where all balland paddles had the “normal” speed, performance drops.
However, accuracy remains relatively high, and D REAM G RADERperforms about the same on test programs regardless of whether the ball and paddle speeds wereseen during training or not.
This indicates some ability to generalize to unseen behaviors at test time.
Table 3 displays the full results.
Conclusion
In this work, we introduced D REAM G RADER , an automatic grading system for interactive programsthat provides fine-grained feedback at near human-level accuracy.
The key insight behind our systemis connecting the problem of automatically discovering errors with the meta-exploration problemin meta-RL, which yields important benefits for both sides.
On the one hand, this connection offersa powerful and previously unexplored toolkit to computer science education, and more generally,to discovering errors in websites or other interactive programs.
On the other hand, this connectionalso opens impactful and readily accessible applications for meta-RL research, which has formerlyprimarily focused on synthetic tasks due to the lack of more compelling accessible applications.
While D REAM G RADER nears human-level grading accuracy and even achieves superhuman gradingaccuracy on many error types, we caution against blindly replacing instructor feedback with automated grading systems, such as D REAM G RADER , which can lead to potentially negative educationaland societal impacts without conscientious application.
For example, practitioners must ensure thatautomated feedback is equitable and not biased against certain classes of solutions that may be correlated with socioeconomic status.
One option to mitigate the risk of potential negative consequenceswhile still reducing instructor labor is to use automated feedback systems to assist instructors, e.g.,by querying the instructor on examples where the system has low certainty, or presenting videosof exploration behavior from the system for the instructor to provide the final feedback.
Finally, this work takes an important step to reduce teaching burden and improve education, butwe also acknowledge D REAM G RADER still has important limitations to overcome.
Beyond theremaining small accuracy gap between D REAM G RADER and human grading, D REAM G RADER alsorequires a substantial amount of training data.
Though we only use 0.5% of the Bounce dataset fortraining, it still amounts to 3500 labeled training programs, and labeling this many programs can beprohibitive for smaller-scale classrooms, though feasible for larger online platforms.
We hope thatfuture work on meta-exploration via our released benchmark can help overcome these limitations.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary. • Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See experiments (Section 5)
(b) Did you describe the limitations of your work?
[Yes] See conclusion (Section 6)
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Seeconclusion (Section 6) (d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)?
[Yes] All code is
released in the supplemental materials and will be publicly released
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
[--tuple for each timestep t in thetrajectory τ . Then, we pass an LSTM with output dimension 128 over the embeddings of the tuples,and take the last hidden state of the LSTM as e ( τ ) . To embed each ( s t , a t , r t , s t +1 ) -tuple, we embedeach component separately, and then apply two final linear layers with output dimensions 
-]how they
were chosen)?
[Yes] See Section
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] Our work uses code
from Nie et al. [25], which we cite.
(b) Did you mention the license of the assets? [Yes] No license is provided.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We release our code in the supplemental material (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identi- fiable information or offensive content?
[N/A] Our data does not contain personally
identifiable information or offensive content
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[Yes] See Section C.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[Yes] See Section C.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[Yes] See Section C.