Original: D YNAMIC N EURAL P ROGRAM E MBEDDINGS FOR P ROGRAM R EPAIR
Revised: D YNAMIC N EURAL P ROGRAM E MBEDDINGS FOR P ROGRAM R EPAIR

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neural program embeddings have shown much promise recently for a variety of program analysis tasks, including program synthesis, program repair, fault localization, etc.
Revised: Neural program embeddings have shown much promise recently for a variety of program analysis tasks, including program synthesis, program repair, fault localization, etc.

Original: However, most existing program embeddings are based on syntactic features of programs, such as raw token sequences or abstract syntax trees.
Revised: However, most existing program embeddings are based on syntactic features of programs, such as raw token sequences or abstract syntax trees.

Original: Unlike images and text, a program has an unambiguous semantic meaning that can be difﬁcult to capture by only considering its syntax ( i.e. syntactically similar programs can exhibit vastly different run-time behavior), which makes syntax-based program embeddings fundamentally limited.
Revised: Unlike images and text, a program has an unambiguous semantic meaning that can be difﬁcult to capture by only considering its syntax ( i.e. syntactically similar programs can exhibit vastly different run-time behavior), which makes syntax-based program embeddings fundamentally limited.

Original: This paper proposes a novel semantic program embedding that is learned from program execution traces.
Revised: This paper proposes a novel semantic program embedding that is learned from program execution traces.

Original: Our key insight is that program states expressed as sequential tuples of live variable values not only captures program semantics more precisely, but also offer a more natural ﬁt for Recurrent Neural Networks to model.
Revised: Our key insight is that program states expressed as sequential tuples of live variable values not only captures program semantics more precisely, but also offer a more natural ﬁt for Recurrent Neural Networks to model.

Original: We evaluate different syntactic and semantic program embeddings on predicting the types of errors that students make in their submissions to an introductory programming class and two exercises on the CodeHunt education platform.
Revised: We evaluate different syntactic and semantic program embeddings on predicting the types of errors that students make in their submissions to an introductory programming class and two exercises on the CodeHunt education platform.

Original: Evaluation results show that our new semantic program embedding signiﬁcantly outperforms the syntactic program embeddings based on token sequences and abstract syntax trees.
Revised: Evaluation results show that our new semantic program embedding signiﬁcantly outperforms the syntactic program embeddings based on token sequences and abstract syntax trees.

Original: In addition, we augment a search-based program repair system with the predictions obtained from our semantic embedding, and show that search efﬁciency is also signiﬁcantly improved.
Revised: In addition, we augment a search-based program repair system with the predictions obtained from our semantic embedding, and show that search efﬁciency is also signiﬁcantly improved.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent breakthroughs of deep learning techniques in computer vision and natural language processing have triggered a growing interest in their applications in programming languages and software engineering.
Revised: Recent breakthroughs of deep learning techniques in computer vision and natural language processing have triggered a growing interest in their applications in programming languages and software engineering.

Original: Several well-explored areas include program classiﬁcation, similarity detection, program repair, and program synthesis.
Revised: Several well-explored areas include program classiﬁcation, similarity detection, program repair, and program synthesis.

Original: One of the key steps in using neural networks for such tasks is to design suitable program representations for the networks to exploit.
Revised: One of the key steps in using neural networks for such tasks is to design suitable program representations for the networks to exploit.

Original: Most existing approaches in the neural program analysis literature have used syntax-based program representations.
Revised: Most existing approaches in the neural program analysis literature have used syntax-based program representations.

Original: Mou et al.
Revised: Mou et al.

Original: (2016) proposed a convolutional neural network over Abstract Syntax Trees (ASTs) as the program representation to classify programs based on their functionalities and detecting different sorting routines.
Revised: (2016) proposed a convolutional neural network over Abstract Syntax Trees (ASTs) as the program representation to classify programs based on their functionalities and detecting different sorting routines.

Original: DeepFix (Gupta et al., 2017), SynFix (Bhatia & Singh, 2016), and sk p (Pu et al., 2016) are recent neural program repair techniques for correcting errors in student programs for MOOC assignments, and they all represent programs as sequences of tokens.
Revised: DeepFix (Gupta et al., 2017), SynFix (Bhatia & Singh, 2016), and sk p (Pu et al., 2016) are recent neural program repair techniques for correcting errors in student programs for MOOC assignments, and they all represent programs as sequences of tokens.

Original: Even program synthesis techniques that generate programs as output such as RobustFill (Devlin et al., 2017), also adopt a token-based program representation for the output decoder.
Revised: Even program synthesis techniques that generate programs as output such as RobustFill (Devlin et al., 2017), also adopt a token-based program representation for the output decoder.

Original: The only exception is Piech et al.
Revised: The only exception is Piech et al.

Original: (2015), which introduces a novel perspective of representing programs using input-output pairs.
Revised: (2015), which introduces a novel perspective of representing programs using input-output pairs.

Original: However, such representations are too coarse-grained to capture program properties — programs with the same input-output behaviors may possess very different characteristics.
Revised: However, such representations are too coarse-grained to capture program properties — programs with the same input-output behaviors may possess very different characteristics.

Original: Consequently, the embeddings learned from input-output pairs are not precise enough for many of program analysis tasks.
Revised: Consequently, the embeddings learned from input-output pairs are not precise enough for many of program analysis tasks.

Original: Although these pioneering efforts have made signiﬁcant contributions to bridge the gap between deep learning techniques and program analysis tasks, syntax-based program representations are fundamentally limited due to the enormous gap between program syntax ( i.e. static expression) and semantics ( i.e. dynamic execution).
Revised: Although these pioneering efforts have made signiﬁcant contributions to bridge the gap between deep learning techniques and program analysis tasks, syntax-based program representations are fundamentally limited due to the enormous gap between program syntax ( i.e. static expression) and semantics ( i.e. dynamic execution).

Original: This gap can be illustrated as follows.
Revised: This gap can be illustrated as follows.

Original: First, when a program is executed at runtime, its statements are almost never interpreted in the order in which the corresponding token sequence is presented to the deep learning models (the only exception being straightline programs, i.e. , ones without any control-ﬂow statements).
Revised: First, when a program is executed at runtime, its statements are almost never interpreted in the order in which the corresponding token sequence is presented to the deep learning models (the only exception being straightline programs, i.e. , ones without any control-ﬂow statements).

Original: For example, a conditional statement
Revised: For example, a conditional statement

Original: [Figure]
Revised: [Figure]

Original: Bubble
Revised: Bubble

Original: Insertion
Revised: Insertion

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Variable Trace
Revised: Variable Trace

Original: State Trace { max val : −∞} { max val : −∞ , item : ⊥} { item : 1 } { max val : −∞ , item : 1 } { max val : 1 } { max val : 1 , item : 1 } { item : 5 } { max val : 1 , item : 5 } { max val : 5 } { max val : 5 , item : 5 } { item : 3 } { max val : 5 , item : 3 } only executes one branch each time, but its token sequence is expressed sequentially as multiple branches.
Revised: State Trace { max val : −∞} { max val : −∞ , item : ⊥} { item : 1 } { max val : −∞ , item : 1 } { max val : 1 } { max val : 1 , item : 1 } { item : 5 } { max val : 1 , item : 5 } { max val : 5 } { max val : 5 , item : 5 } { item : 3 } { max val : 5 , item : 3 } only executes one branch each time, but its token sequence is expressed sequentially as multiple branches.

Original: Similarly, when iterating over a looping structure at runtime, it is unclear in which order any two tokens are executed when considering different loop iterations.
Revised: Similarly, when iterating over a looping structure at runtime, it is unclear in which order any two tokens are executed when considering different loop iterations.

Original: Second, program dependency ( i.e. data and control) is not exploited in token sequences and ASTs despite its essential role in deﬁning program semantics.
Revised: Second, program dependency ( i.e. data and control) is not exploited in token sequences and ASTs despite its essential role in deﬁning program semantics.

Original: Figure 2 shows an example using a simple max function.
Revised: Figure 2 shows an example using a simple max function.

Original: On line 8, the assignment statement means variable max val is data-dependent on item .
Revised: On line 8, the assignment statement means variable max val is data-dependent on item .

Original: In addition, the execution of this statement depends on the evaluation of the if condition on line 7, i.e. , max val is also control-dependent on item as well as itself.
Revised: In addition, the execution of this statement depends on the evaluation of the if condition on line 7, i.e. , max val is also control-dependent on item as well as itself.

Original: Third, from a pure program analysis standpoint, the gap between program syntax and semantics is manifested in that similar program syntax may lead to vastly different program semantics.
Revised: Third, from a pure program analysis standpoint, the gap between program syntax and semantics is manifested in that similar program syntax may lead to vastly different program semantics.

Original: For example, consider the two sorting functions shown in Figure 1.
Revised: For example, consider the two sorting functions shown in Figure 1.

Original: Both functions sort the array via two nested loops, compare the current element to its successor, and swap them if the order is incorrect.
Revised: Both functions sort the array via two nested loops, compare the current element to its successor, and swap them if the order is incorrect.

Original: However, the two functions implement different algorithms, namely Bubble Sort and Insertion Sort .
Revised: However, the two functions implement different algorithms, namely Bubble Sort and Insertion Sort .

Original: Therefore minor syntactic discrepancies can lead to signiﬁcant semantic differences.
Revised: Therefore minor syntactic discrepancies can lead to signiﬁcant semantic differences.

Original: This intrinsic weakness will be inherited by any deep learning technique that adopts a syntax-based program representation.
Revised: This intrinsic weakness will be inherited by any deep learning technique that adopts a syntax-based program representation.

Original: To tackle this aforementioned fundamental challenge, this paper proposes a novel semantic program embedding that is learned from the program’s runtime behavior, i.e. dynamic program execution traces.
Revised: To tackle this aforementioned fundamental challenge, this paper proposes a novel semantic program embedding that is learned from the program’s runtime behavior, i.e. dynamic program execution traces.

Original: We execute a program on a set of test cases and monitor/record the program states comprising of variable valuations.
Revised: We execute a program on a set of test cases and monitor/record the program states comprising of variable valuations.

Original: We introduce three approaches to embed these dynamic executions:
Revised: We introduce three approaches to embed these dynamic executions:

Original: (1) variable trace embedding — consider each variable independently, (2) state trace embedding — consider sequences of program states, each of which comprises of a set of variable values, and (3) hybrid embedding — incorporate dependencies into individual variable sequences to avoid redundant variable values in program states.
Revised: (1) variable trace embedding — consider each variable independently, (2) state trace embedding — consider sequences of program states, each of which comprises of a set of variable values, and (3) hybrid embedding — incorporate dependencies into individual variable sequences to avoid redundant variable values in program states.

Original: Our novel program embeddings address the aforementioned issues with the syntactic program representations.
Revised: Our novel program embeddings address the aforementioned issues with the syntactic program representations.

Original: The dynamic program execution traces precisely illustrate the program behaves at runtime, and the values for each variable at each program point precisely models the program semantics.
Revised: The dynamic program execution traces precisely illustrate the program behaves at runtime, and the values for each variable at each program point precisely models the program semantics.

Original: Regarding program dependencies, the dynamic execution traces, expressed as a sequential list of tuples (each of which represents the value of a variable at a certain program point), provides an opportunity for Recurrent Neural Network (RNN) to establish the data dependency and control dependency in the program.
Revised: Regarding program dependencies, the dynamic execution traces, expressed as a sequential list of tuples (each of which represents the value of a variable at a certain program point), provides an opportunity for Recurrent Neural Network (RNN) to establish the data dependency and control dependency in the program.

Original: By monitoring particular value patterns between interacting variables, the RNN is able to model their relationship, leading to more precise semantic representations.
Revised: By monitoring particular value patterns between interacting variables, the RNN is able to model their relationship, leading to more precise semantic representations.

Original: Reed & De Freitas (2015) recently proposed using program traces (as a sequence of actions/statements) for training a neural network to learn to execute an algorithm such as addition or sorting.
Revised: Reed & De Freitas (2015) recently proposed using program traces (as a sequence of actions/statements) for training a neural network to learn to execute an algorithm such as addition or sorting.

Original: Their notion of traces is different from our dynamic execution traces consisting of program states with variable valuations.
Revised: Their notion of traces is different from our dynamic execution traces consisting of program states with variable valuations.

Original: Our notion offers the following advantages: (1) a sequence of program states can be viewed as a sequence of input-output pairs of each executed statement, in other words, sequences of program states provide more robust information than that from sequences of executed statements, and (2) although a sequence of executed statements follows dynamic execution, it is still represented syntactically, and therefore may not adequately capture program semantics.
Revised: Our notion offers the following advantages: (1) a sequence of program states can be viewed as a sequence of input-output pairs of each executed statement, in other words, sequences of program states provide more robust information than that from sequences of executed statements, and (2) although a sequence of executed statements follows dynamic execution, it is still represented syntactically, and therefore may not adequately capture program semantics.

Original: We have evaluated our dynamic program embeddings in the context of automated program repair.
Revised: We have evaluated our dynamic program embeddings in the context of automated program repair.

Original: In particular, we use the program embeddings to predict common mistake patterns that students made to their Microsoft-DEV204.1X programming assignments and two additional Microsoft CodeHunt problems.
Revised: In particular, we use the program embeddings to predict common mistake patterns that students made to their Microsoft-DEV204.1X programming assignments and two additional Microsoft CodeHunt problems.

Original: The results show that our dynamic embeddings signiﬁcantly outperform syntax-based program embeddings, including those trained on token sequences and abstract syntax trees.
Revised: The results show that our dynamic embeddings signiﬁcantly outperform syntax-based program embeddings, including those trained on token sequences and abstract syntax trees.

Original: In addition, we show that our dynamic embeddings can be leveraged to signiﬁcantly improve the efﬁciency of a search-based program corrector S ARF G EN 1 (Please refer to the appendix for its algorithm).
Revised: In addition, we show that our dynamic embeddings can be leveraged to signiﬁcantly improve the efﬁciency of a search-based program corrector S ARF G EN 1 (Please refer to the appendix for its algorithm).

Original: More importantly, we believe that our dynamic program embeddings can be useful for many other program analysis tasks, such as program synthesis, fault localization, and similarity detection.
Revised: More importantly, we believe that our dynamic program embeddings can be useful for many other program analysis tasks, such as program synthesis, fault localization, and similarity detection.

Original: To summarize, the main contributions of this paper are: (1) we show the fundamental limitation of representing programs using syntax-level features; (2) we propose dynamic program embeddings learned from runtime execution traces to overcome key issues with syntactic program representations; (3) we evaluate our dynamic program embeddings for predicting common mistake patterns students make in program assignments, and results show that the dynamic program embeddings outperform state-of-the-art syntactic program embeddings; and (4) we show how the dynamic program embeddings can be utilized to improve an existing production program repair system.
Revised: To summarize, the main contributions of this paper are: (1) we show the fundamental limitation of representing programs using syntax-level features; (2) we propose dynamic program embeddings learned from runtime execution traces to overcome key issues with syntactic program representations; (3) we evaluate our dynamic program embeddings for predicting common mistake patterns students make in program assignments, and results show that the dynamic program embeddings outperform state-of-the-art syntactic program embeddings; and (4) we show how the dynamic program embeddings can be utilized to improve an existing production program repair system.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: This section brieﬂy reviews dynamic program analysis (Ball, 1999), an inﬂuential program analysis technique that lays the foundation for constructing our new program embeddings.
Revised: This section brieﬂy reviews dynamic program analysis (Ball, 1999), an inﬂuential program analysis technique that lays the foundation for constructing our new program embeddings.

Original: Unlike static analysis (Nielson et al., 1999), i.e. , the analysis of program source code, dynamic analysis focuses on program executions.
Revised: Unlike static analysis (Nielson et al., 1999), i.e. , the analysis of program source code, dynamic analysis focuses on program executions.

Original: An execution is modeled by a set of atomic actions, or events, organized as a trace (or event history).
Revised: An execution is modeled by a set of atomic actions, or events, organized as a trace (or event history).

Original: For simplicity, this paper considers sequential executions only (as opposed to parallel executions) which lead to a single sequence of events, speciﬁcally, the executions of statements in the program.
Revised: For simplicity, this paper considers sequential executions only (as opposed to parallel executions) which lead to a single sequence of events, speciﬁcally, the executions of statements in the program.

Original: Detailed information about executions is often not readily available, and separate mechanisms are needed to capture the tracing information.
Revised: Detailed information about executions is often not readily available, and separate mechanisms are needed to capture the tracing information.

Original: An often adopted approach is to instrument a program’s source code ( i.e. , by adding additional monitoring code) to record the execution of statements of interest.
Revised: An often adopted approach is to instrument a program’s source code ( i.e. , by adding additional monitoring code) to record the execution of statements of interest.

Original: In particular, those inserted instrumentation statements act as a monitoring window through which the values of variables are inspected.
Revised: In particular, those inserted instrumentation statements act as a monitoring window through which the values of variables are inspected.

Original: This instrumentation process may occur in a fully automated manner, e.g. , a common approach is to traverse a program’s abstract syntax tree and insert “write” statements right after each program statement that causes a side-effect
Revised: This instrumentation process may occur in a fully automated manner, e.g. , a common approach is to traverse a program’s abstract syntax tree and insert “write” statements right after each program statement that causes a side-effect

Original: ( i
Revised: ( i

Original: e. , changing the values of some variables)
Revised: e. , changing the values of some variables)

Original: Consider the two sorting algorithms depicted in Figure 1.
Revised: Consider the two sorting algorithms depicted in Figure 1.

Original: If we assume A to be the only variable of interest and subject to monitoring, we can instrument the two algorithms with
Revised: If we assume A to be the only variable of interest and subject to monitoring, we can instrument the two algorithms with

Original: Console.
Revised: Console.

Original: WriteLine(A) after each program location in the code whenever A is modiﬁed ( i.e. the lines marked by comments).
Revised: WriteLine(A) after each program location in the code whenever A is modiﬁed ( i.e. the lines marked by comments).

Original: Given the input vector A = [8 , 5 , 1 , 4 , 3] , the execution traces of the two sorting routines are shown on the right in Figure 1.
Revised: Given the input vector A = [8 , 5 , 1 , 4 , 3] , the execution traces of the two sorting routines are shown on the right in Figure 1.

Original: One of the key beneﬁts of dynamic analysis is its ability to easily and precisely identify relevant parts of the program that affect execution behavior.
Revised: One of the key beneﬁts of dynamic analysis is its ability to easily and precisely identify relevant parts of the program that affect execution behavior.

Original: As shown in the example above, despite the very similar program syntax of bubble sort and insertion sort, dynamic analysis is able to discover their distinct program semantics by exposing their execution traces.
Revised: As shown in the example above, despite the very similar program syntax of bubble sort and insertion sort, dynamic analysis is able to discover their distinct program semantics by exposing their execution traces.

Original: Since understanding program semantics is a central issue in program analysis, dynamic analysis has seen remarkable success over the past several decades and has resulted in many successful program analysis tools such as debuggers, proﬁlers, monitors, or explanation generators.
Revised: Since understanding program semantics is a central issue in program analysis, dynamic analysis has seen remarkable success over the past several decades and has resulted in many successful program analysis tools such as debuggers, proﬁlers, monitors, or explanation generators.

Original:  O VERVIEW OF THE A PPROACH
Revised:  O VERVIEW OF THE A PPROACH

Original: We now present an overview of our approach.
Revised: We now present an overview of our approach.

Original: Given a program and the execution traces extracted for all its variables, we introduce three neural network models to learn dynamic program embeddings.
Revised: Given a program and the execution traces extracted for all its variables, we introduce three neural network models to learn dynamic program embeddings.

Original: To demonstrate the utility of these embeddings, we apply them to predict common error patterns (detailed in Section 5) that students make in their submissions to an online introductory programming course.
Revised: To demonstrate the utility of these embeddings, we apply them to predict common error patterns (detailed in Section 5) that students make in their submissions to an online introductory programming course.

Original: Variable Trace
Revised: Variable Trace

Original: Embedding As shown in Table 1, each row denotes a new program point where a variable gets updated.
Revised: Embedding As shown in Table 1, each row denotes a new program point where a variable gets updated.

Original: The entire variable trace consists of those variable values at all program points.
Revised: The entire variable trace consists of those variable values at all program points.

Original: As a subsequent step, we split the complete trace into a list of sub-traces (one for each variable).
Revised: As a subsequent step, we split the complete trace into a list of sub-traces (one for each variable).

Original: We use one single RNN to encode each sub-trace independently and then perform max pooling on the ﬁnal states of the same RNN to obtain the program embedding.
Revised: We use one single RNN to encode each sub-trace independently and then perform max pooling on the ﬁnal states of the same RNN to obtain the program embedding.

Original: Finally, we add a one layer softmax regression to make the predictions.
Revised: Finally, we add a one layer softmax regression to make the predictions.

Original: The entire workﬂow is show in Figure 3.
Revised: The entire workﬂow is show in Figure 3.

Original: State Trace Embedding Because each variable trace is handled individually in the previous approach, variable dependencies/interactions are not precisely captured.
Revised: State Trace Embedding Because each variable trace is handled individually in the previous approach, variable dependencies/interactions are not precisely captured.

Original: To address this issue, we propose the state trace embedding.
Revised: To address this issue, we propose the state trace embedding.

Original: As depicted in Table 1, each program point l introduces a new program state expressed by the latest variable valuations at l .
Revised: As depicted in Table 1, each program point l introduces a new program state expressed by the latest variable valuations at l .

Original: The entire state trace is a sequence of program states.
Revised: The entire state trace is a sequence of program states.

Original: To learn the state trace embedding, we ﬁrst use one RNN to encode each program state ( i.e. , a tuple of values) and feed the resulting RNNs as a sequence to another RNN.
Revised: To learn the state trace embedding, we ﬁrst use one RNN to encode each program state ( i.e. , a tuple of values) and feed the resulting RNNs as a sequence to another RNN.

Original: Note that we do not assume that the order in which variables values are encoded by the RNN for each program state but rather maintain a consistent order throughout all program states for a given trace.
Revised: Note that we do not assume that the order in which variables values are encoded by the RNN for each program state but rather maintain a consistent order throughout all program states for a given trace.

Original: Finally, we feed a softmax regression layer with the ﬁnal state of the second RNN (shown in Figure 4).
Revised: Finally, we feed a softmax regression layer with the ﬁnal state of the second RNN (shown in Figure 4).

Original: The beneﬁt of state trace embedding is its ability to capture dependencies among variables in each program state as well as the relationship among program states.
Revised: The beneﬁt of state trace embedding is its ability to capture dependencies among variables in each program state as well as the relationship among program states.

Original: Dependency Enforcement for Variable Trace
Revised: Dependency Enforcement for Variable Trace

Original: Embedding Although state trace embedding can better capture program dependencies, it also comes with some challenges, the most signiﬁcant of which is redundancy.
Revised: Embedding Although state trace embedding can better capture program dependencies, it also comes with some challenges, the most signiﬁcant of which is redundancy.

Original: Consider a looping structure in a program.
Revised: Consider a looping structure in a program.

Original: During an iteration, whenever one variable gets modiﬁed, a new program state will be created containing the values of all variables, even of those unmodiﬁed by the loop.
Revised: During an iteration, whenever one variable gets modiﬁed, a new program state will be created containing the values of all variables, even of those unmodiﬁed by the loop.

Original: This issue becomes more severe for loops with larger numbers of iterations.
Revised: This issue becomes more severe for loops with larger numbers of iterations.

Original: To tackle this challenge, we propose the third and ﬁnal approach, dependency enforcement for variable trace embedding (hereinafter referred as dependency enforcement embedding ), that combines the advantages of variable trace embedding ( i.e. , compact representation of execution traces) and state trace embedding ( i.e. , precise capturing of program dependencies).
Revised: To tackle this challenge, we propose the third and ﬁnal approach, dependency enforcement for variable trace embedding (hereinafter referred as dependency enforcement embedding ), that combines the advantages of variable trace embedding ( i.e. , compact representation of execution traces) and state trace embedding ( i.e. , precise capturing of program dependencies).

Original: In dependency enforcement embedding, a program is represented by separate variable traces, with each variable being handled by a different RNN.
Revised: In dependency enforcement embedding, a program is represented by separate variable traces, with each variable being handled by a different RNN.

Original: In order to enforce program dependencies, the hidden states from different RNNs will be interleaved in a way that simulates the needed data and control dependencies.
Revised: In order to enforce program dependencies, the hidden states from different RNNs will be interleaved in a way that simulates the needed data and control dependencies.

Original: Unlike variable trace embedding, we perform an average pooling on the ﬁnal states of all RNNs to obtain the program embedding on which we build the ﬁnal layer of softmax regression.
Revised: Unlike variable trace embedding, we perform an average pooling on the ﬁnal states of all RNNs to obtain the program embedding on which we build the ﬁnal layer of softmax regression.

Original: Figure 5 describes the workﬂow.
Revised: Figure 5 describes the workﬂow.

Original:  D YNAMIC P ROGRAM E MBEDDINGS
Revised:  D YNAMIC P ROGRAM E MBEDDINGS

Original: We now formally deﬁne the three program embedding models.
Revised: We now formally deﬁne the three program embedding models.

Original: 4. V ARIABLE T RACE M ODEL
Revised: 4. V ARIABLE T RACE M ODEL

Original: Given a program P , and its variable set V ( v 0 , v 1 ,..., v n ∈ V ), a variable trace is a sequence of values a variable has been assigned during the execution of P .
Revised: Given a program P , and its variable set V ( v 0 , v 1 ,..., v n ∈ V ), a variable trace is a sequence of values a variable has been assigned during the execution of P .

Original: 3 Let x t v n denote the value from the variable trace of v n that is fed to the RNN encoder (GRU) at time t as the input, and h t v n as the resulting RNN’s hidden state.
Revised: 3 Let x t v n denote the value from the variable trace of v n that is fed to the RNN encoder (GRU) at time t as the input, and h t v n as the resulting RNN’s hidden state.

Original: We compute the variable trace embedding for P in Equation (3) as follows ( h T v n denotes the last hidden state of the encoder):
Revised: We compute the variable trace embedding for P in Equation (3) as follows ( h T v n denotes the last hidden state of the encoder):

Original: [Equation]
Revised: [Equation]

Original: Evidence =
Revised: Evidence =

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: Y
Revised: Y

Original: [Equation]
Revised: [Equation]

Original: softmax (
Revised: softmax (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: h t v 1 , ..., h t v n , h P ∈ R k where k denotes the size of hidden layers of the RNN encoder.
Revised: h t v 1 , ..., h t v n , h P ∈ R k where k denotes the size of hidden layers of the RNN encoder.

Original: 4. S TATE T RACE M ODEL
Revised: 4. S TATE T RACE M ODEL

Original: The key idea in state trace model is to embed each program state as a numerical vector ﬁrst and then feed all program state embeddings as a sequence to another RNN encoder to obtain the program embedding.
Revised: The key idea in state trace model is to embed each program state as a numerical vector ﬁrst and then feed all program state embeddings as a sequence to another RNN encoder to obtain the program embedding.

Original: Suppose x t v n is the value of variable v n at t -th program state, and h t v n is the resulting hidden state of the program state encoder.
Revised: Suppose x t v n is the value of variable v n at t -th program state, and h t v n is the resulting hidden state of the program state encoder.

Original: Equation (8) computes the t -th program state embedding.
Revised: Equation (8) computes the t -th program state embedding.

Original: Equations (9-11) encode the sequence of all program state embeddings ( i.e. , h t v n , h t +1 v n , ..., h t + m v n ) with another RNN to compute the program embedding.
Revised: Equations (9-11) encode the sequence of all program state embeddings ( i.e. , h t v n , h t +1 v n , ..., h t + m v n ) with another RNN to compute the program embedding.

Original: [Equation]
Revised: [Equation]

Original: 11) h t v 1 , ..., h t v n ∈ R k 1 ; h (cid:48) t v n , ..., h P ∈ R k 2 where k 1 and k 2 denote, respectively, the sizes of hidden layers of the ﬁrst and second RNN encoders.
Revised: 11) h t v 1 , ..., h t v n ∈ R k 1 ; h (cid:48) t v n , ..., h P ∈ R k 2 where k 1 and k 2 denote, respectively, the sizes of hidden layers of the ﬁrst and second RNN encoders.

Original: 4. D EPENDENCY E NFORCEMENT FOR V ARIABLE T RACE E MBEDDING
Revised: 4. D EPENDENCY E NFORCEMENT FOR V ARIABLE T RACE E MBEDDING

Original: The motivation behind this model is to combine the advantages of the previous two approaches, i.e. representing the execution trace compactly while enforcing the dependency relationship among variables as much as possible.
Revised: The motivation behind this model is to combine the advantages of the previous two approaches, i.e. representing the execution trace compactly while enforcing the dependency relationship among variables as much as possible.

Original: In this model, each variable trace is handled with a different RNN.
Revised: In this model, each variable trace is handled with a different RNN.

Original: A potential issue to be addressed is variable matching/renaming ( i.e. , α -renaming).
Revised: A potential issue to be addressed is variable matching/renaming ( i.e. , α -renaming).

Original: In other words same variables may be named differently in different programs.
Revised: In other words same variables may be named differently in different programs.

Original: Processing each variable id with a single RNN among all programs in the dataset will not only cause memory issues, but more importantly the loss of precision.
Revised: Processing each variable id with a single RNN among all programs in the dataset will not only cause memory issues, but more importantly the loss of precision.

Original: Our solution is to (1) execute all programs to collect traces for all variables, (2) perform dynamic time wrapping (Vintsyuk, 1968) on the variable traces across all programs to ﬁnd the top- n most used variables that account for the vast majority of variable usage, and (3) rename the top- n most used variables consistently across all programs, and rename all other variables to a same special variable.
Revised: Our solution is to (1) execute all programs to collect traces for all variables, (2) perform dynamic time wrapping (Vintsyuk, 1968) on the variable traces across all programs to ﬁnd the top- n most used variables that account for the vast majority of variable usage, and (3) rename the top- n most used variables consistently across all programs, and rename all other variables to a same special variable.

Original: Given the same set of variables among all programs, the mechanism of dependency enforcement on the top ones is to use the hidden states of multiple RNNs based on how a new value of a variable is produced.
Revised: Given the same set of variables among all programs, the mechanism of dependency enforcement on the top ones is to use the hidden states of multiple RNNs based on how a new value of a variable is produced.

Original: For example, in Figure 2 at line 8, the new value of max val is data-dependent on item , and control-dependent on both item and itself.
Revised: For example, in Figure 2 at line 8, the new value of max val is data-dependent on item , and control-dependent on both item and itself.

Original: So at the time step when the new value of max val is produced, the latest hidden states of the RNNs encode variable item as well as itself; they together determine the previous state of the RNN upon which the new value of max val is produced.
Revised: So at the time step when the new value of max val is produced, the latest hidden states of the RNNs encode variable item as well as itself; they together determine the previous state of the RNN upon which the new value of max val is produced.

Original: If a value is produced without any dependencies, this mechanism will not take effect.
Revised: If a value is produced without any dependencies, this mechanism will not take effect.

Original: In other words, the RNN will act normally to handle data sequences on its own.
Revised: In other words, the RNN will act normally to handle data sequences on its own.

Original: Equations (11 and 12) expose the inner workﬂow.
Revised: Equations (11 and 12) expose the inner workﬂow.

Original: h LT v m denotes the latest hidden state of the RNN encoding variable trace of v m up to the point of time t when x t v n is the input of the RNN encoding variable trace of v n .
Revised: h LT v m denotes the latest hidden state of the RNN encoding variable trace of v m up to the point of time t when x t v n is the input of the RNN encoding variable trace of v n .

Original: denotes element-wise matrix product.
Revised: denotes element-wise matrix product.

Original: [Equation]
Revised: [Equation]

Original: Given v n depends on v 1 and v m
Revised: Given v n depends on v 1 and v m

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  E VALUATION
Revised:  E VALUATION

Original: [Figure]
Revised: [Figure]

Original: We train our dynamic program embeddings on the programming submissions obtained from Assignment 2 from Microsoft-
Revised: We train our dynamic program embeddings on the programming submissions obtained from Assignment 2 from Microsoft-

Original: DEV204.1X: “Introduction to C#” offered on edx and two other problems on Microsoft
Revised: DEV204.1X: “Introduction to C#” offered on edx and two other problems on Microsoft

Original: CodeHunt platform.
Revised: CodeHunt platform.

Original: • Print Chessboard : print the pattern of chessboard using “X” and “O” characters to represent the squares as shown in
Revised: • Print Chessboard : print the pattern of chessboard using “X” and “O” characters to represent the squares as shown in

Original: Figure 6.
Revised: Figure 6.

Original: • Count Parentheses : count the depth of nesting parentheses in a given string.
Revised: • Count Parentheses : count the depth of nesting parentheses in a given string.

Original: • Generate Binary Digits : generate the string of binary digits for a given integer.
Revised: • Generate Binary Digits : generate the string of binary digits for a given integer.

Original: Regarding the three programming problems, the errors students made in their submissions can be roughly classiﬁed into low-level technical issues ( e.g. , list indexing, branching conditions or looping bounds) and high-level conceptual issues ( e.g. , mishandling corner case, misunderstanding problem requirement or misconceptions on the underlying data structure of test inputs).
Revised: Regarding the three programming problems, the errors students made in their submissions can be roughly classiﬁed into low-level technical issues ( e.g. , list indexing, branching conditions or looping bounds) and high-level conceptual issues ( e.g. , mishandling corner case, misunderstanding problem requirement or misconceptions on the underlying data structure of test inputs).

Original: In order to have sufﬁcient amount of data to train our models to predict the error patterns, we (1) convert each incorrect program into multiple programs such that each new program will have only one error, and (2) mutate all the correct programs to generate synthetic incorrect programs such that they exhibit similar errors that students made in real program submissions.
Revised: In order to have sufﬁcient amount of data to train our models to predict the error patterns, we (1) convert each incorrect program into multiple programs such that each new program will have only one error, and (2) mutate all the correct programs to generate synthetic incorrect programs such that they exhibit similar errors that students made in real program submissions.

Original: These two steps allow us to set up a dataset depicted in Table 2.
Revised: These two steps allow us to set up a dataset depicted in Table 2.

Original: Based on the same set of training data, we evaluate the dynamic embeddings trained with the three network models and compare them with the syntaxbased program embeddings (on the same error prediction task) on the same testing data.
Revised: Based on the same set of training data, we evaluate the dynamic embeddings trained with the three network models and compare them with the syntaxbased program embeddings (on the same error prediction task) on the same testing data.

Original: The syntaxbased models include (1) one trained with a RNN that encodes token sequences of programs, and (2) another trained with a RNN on abstract syntax trees of programs (Socher et al., 2013).
Revised: The syntaxbased models include (1) one trained with a RNN that encodes token sequences of programs, and (2) another trained with a RNN on abstract syntax trees of programs (Socher et al., 2013).

Original: Problem
Revised: Problem

Original: Program
Revised: Program

Original: [Table]
Revised: [Table]

Original: Correct
Revised: Correct

Original: Incorrect
Revised: Incorrect

Original: Training
Revised: Training

Original: Validation
Revised: Validation

Original: Testing
Revised: Testing

Original: Print Chessboard
Revised: Print Chessboard

Original: [Table]
Revised: [Table]

Original: All models are implemented in TensorFlow.
Revised: All models are implemented in TensorFlow.

Original: All encoders in each of the trace model have two stacked GRU layers with 200 hidden units in each layer except that the state encoder in the state trace model has one single layer of 100 hidden units.
Revised: All encoders in each of the trace model have two stacked GRU layers with 200 hidden units in each layer except that the state encoder in the state trace model has one single layer of 100 hidden units.

Original: We adopt random initialization for weight initialization.
Revised: We adopt random initialization for weight initialization.

Original: Our vocabulary has 5,568 unique tokens ( i.e. , the values of all variables at each time step), each of which is embedded into a 100-dimensional vector.
Revised: Our vocabulary has 5,568 unique tokens ( i.e. , the values of all variables at each time step), each of which is embedded into a 100-dimensional vector.

Original: All networks are trained using the Adam optimizer (Kingma & Ba, 2014) with the learning and the decay rates set to their default values (learning rate = 0.0001, beta1 = 0.9, beta2 = 0.999) and a mini-batch size of 500.
Revised: All networks are trained using the Adam optimizer (Kingma & Ba, 2014) with the learning and the decay rates set to their default values (learning rate = 0.0001, beta1 = 0.9, beta2 = 0.999) and a mini-batch size of 500.

Original: For the variable trace and dependency enforcement models, each trace is padded to have the same length across each batch; for the state trace model, both the number of variables in each program state as well as the length of the entire state trace are padded.
Revised: For the variable trace and dependency enforcement models, each trace is padded to have the same length across each batch; for the state trace model, both the number of variables in each program state as well as the length of the entire state trace are padded.

Original: During the training of the dependency enforcement model, we have observed that when dependencies become complex, the network suffers from optimization issues, such as diminishing and exploding gradients.
Revised: During the training of the dependency enforcement model, we have observed that when dependencies become complex, the network suffers from optimization issues, such as diminishing and exploding gradients.

Original: This is likely due to the complex nature of fusing hidden states among RNNs, echoing the errors back and forth through the network.
Revised: This is likely due to the complex nature of fusing hidden states among RNNs, echoing the errors back and forth through the network.

Original: We resolve this issue by truncating each trace into multiple sub-sequences and only back-propagate on the last sub-sequence while only feedforwarding on the rest.
Revised: We resolve this issue by truncating each trace into multiple sub-sequences and only back-propagate on the last sub-sequence while only feedforwarding on the rest.

Original: Regarding the baseline network trained on token sequences, we use the same encoder architecture ( i.e. , two layer GRU of 200 hidden units) processing the same 100-dimension embedding vector for each token.
Revised: Regarding the baseline network trained on token sequences, we use the same encoder architecture ( i.e. , two layer GRU of 200 hidden units) processing the same 100-dimension embedding vector for each token.

Original: As for the AST model, we learn an embedding (100-dimension) for each type of the syntax node by propagating the leaf (a simple look up) to the root through the learned production rules.
Revised: As for the AST model, we learn an embedding (100-dimension) for each type of the syntax node by propagating the leaf (a simple look up) to the root through the learned production rules.

Original: Finally, we use the root embeddings to represent programs.
Revised: Finally, we use the root embeddings to represent programs.

Original: [Table]
Revised: [Table]

Original: As shown in Table 3, our embeddings trained on execution traces signiﬁcantly outperform those trained on program syntax (greater than 92% accuracy compared to less than 21% for syntax-based embeddings).
Revised: As shown in Table 3, our embeddings trained on execution traces signiﬁcantly outperform those trained on program syntax (greater than 92% accuracy compared to less than 21% for syntax-based embeddings).

Original: We conjecture this is because of the fact that minor syntactic discrepancies can lead to major semantic differences as shown in Figure 1.
Revised: We conjecture this is because of the fact that minor syntactic discrepancies can lead to major semantic differences as shown in Figure 1.

Original: In our dataset, there are a large number of programs with distinct labels that differ by only a few number of tokens or AST nodes, which causes difﬁculty for the syntax models to generalize (some examples shown in Appendix).
Revised: In our dataset, there are a large number of programs with distinct labels that differ by only a few number of tokens or AST nodes, which causes difﬁculty for the syntax models to generalize (some examples shown in Appendix).

Original: Even for the simpler syntax-level errors, they are buried in large number of other syntactic variations and the size of the training dataset is relatively small for the syntax-based models to learn precise patterns.
Revised: Even for the simpler syntax-level errors, they are buried in large number of other syntactic variations and the size of the training dataset is relatively small for the syntax-based models to learn precise patterns.

Original: In contrast, dynamic embeddings are able to canonicalize the syntactical variations and pinpoint the underlying semantic differences, which results in the trace-based models learning the correct error patterns more effectively even with relatively smaller size of the training data.
Revised: In contrast, dynamic embeddings are able to canonicalize the syntactical variations and pinpoint the underlying semantic differences, which results in the trace-based models learning the correct error patterns more effectively even with relatively smaller size of the training data.

Original: In addition, we incorporated our dynamic program embeddings into S ARFGEN — a program repair system — to demonstrate the beneﬁt in producing ﬁxes to correct errors students made in their programming assignments.
Revised: In addition, we incorporated our dynamic program embeddings into S ARFGEN — a program repair system — to demonstrate the beneﬁt in producing ﬁxes to correct errors students made in their programming assignments.

Original: Given a set of potential repair candidates, S ARFGEN uses an enumerative search-based technique to ﬁnd minimal changes to an incorrect program.
Revised: Given a set of potential repair candidates, S ARFGEN uses an enumerative search-based technique to ﬁnd minimal changes to an incorrect program.

Original: We use the dynamic embeddings to learn a distribution over the corrections to prioritize the search for the repair algorithm.
Revised: We use the dynamic embeddings to learn a distribution over the corrections to prioritize the search for the repair algorithm.

Original: To establish the baseline, we obtain the set of all corrections from S ARFGEN for each of the real incorrect program to all three problems and enumerate each subset until we ﬁnd the minimum ﬁxes.
Revised: To establish the baseline, we obtain the set of all corrections from S ARFGEN for each of the real incorrect program to all three problems and enumerate each subset until we ﬁnd the minimum ﬁxes.

Original: On the contrary, we also run another experiment where we prioritize each correction according to the prediction of errors with the dynamic embeddings.
Revised: On the contrary, we also run another experiment where we prioritize each correction according to the prediction of errors with the dynamic embeddings.

Original: It is worth mentioning that one incorrect program may be caused by multiple errors.
Revised: It is worth mentioning that one incorrect program may be caused by multiple errors.

Original: Therefore, we only predict the top-1 error each time and repair the program with the corresponding corrections.
Revised: Therefore, we only predict the top-1 error each time and repair the program with the corresponding corrections.

Original: If the program is still incorrect, we repeat this procedure till the program is ﬁxed.
Revised: If the program is still incorrect, we repeat this procedure till the program is ﬁxed.

Original: The comparison between the two approaches is based on how long it takes them to repair the programs.
Revised: The comparison between the two approaches is based on how long it takes them to repair the programs.

Original: [Table]
Revised: [Table]

Original: As shown in Table 4, the more ﬁxes required, the more speedups dynamic program embeddings yield — more than an order of magnitude speedups when the number of ﬁxes is four or greater.
Revised: As shown in Table 4, the more ﬁxes required, the more speedups dynamic program embeddings yield — more than an order of magnitude speedups when the number of ﬁxes is four or greater.

Original: When the number of ﬁxes is greater than seven, the performance gain drops signiﬁcantly due to poor prediction accuracy for programs with too many errors.
Revised: When the number of ﬁxes is greater than seven, the performance gain drops signiﬁcantly due to poor prediction accuracy for programs with too many errors.

Original: In other words, our dynamic embeddings are not viewed by the network as capturing incorrect execution traces, but rather new execution traces.
Revised: In other words, our dynamic embeddings are not viewed by the network as capturing incorrect execution traces, but rather new execution traces.

Original: Therefore, the predictions become unreliable.
Revised: Therefore, the predictions become unreliable.

Original: Note that we ignored incorrect programs having greater than 10 errors when most experiments run out of memory for the baseline approach.
Revised: Note that we ignored incorrect programs having greater than 10 errors when most experiments run out of memory for the baseline approach.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: There has been recent interest in learning neural program representations for various applications, such as program induction and synthesis, program repair, and program completion.
Revised: There has been recent interest in learning neural program representations for various applications, such as program induction and synthesis, program repair, and program completion.

Original: Speciﬁcally for neural program repair techniques, none of the existing techniques, such as DeepFix (Gupta et al., 2017), SynFix (Bhatia & Singh, 2016) and sk p (Pu et al., 2016), have considered dynamic embeddings proposed in this paper.
Revised: Speciﬁcally for neural program repair techniques, none of the existing techniques, such as DeepFix (Gupta et al., 2017), SynFix (Bhatia & Singh, 2016) and sk p (Pu et al., 2016), have considered dynamic embeddings proposed in this paper.

Original: In fact, dynamic embeddings can be naturally extended to be a new feature dimension for these existing neural program repair techniques.
Revised: In fact, dynamic embeddings can be naturally extended to be a new feature dimension for these existing neural program repair techniques.

Original: Piech et al.
Revised: Piech et al.

Original: (2015) is a notable recent effort targeting program representation.
Revised: (2015) is a notable recent effort targeting program representation.

Original: Piech et al.
Revised: Piech et al.

Original: explore the possibility of using input-output pairs to represent a program.
Revised: explore the possibility of using input-output pairs to represent a program.

Original: Despite their new perspective, the direct mapping between input and output of programs usually are not precise enough, i.e. , the same input-output pair may correspond to two completely different programs, such as the two sorting algorithms in Figure 1.
Revised: Despite their new perspective, the direct mapping between input and output of programs usually are not precise enough, i.e. , the same input-output pair may correspond to two completely different programs, such as the two sorting algorithms in Figure 1.

Original: As we often observe in our own dataset, programs with the same error patterns can also result in different input-output pairs.
Revised: As we often observe in our own dataset, programs with the same error patterns can also result in different input-output pairs.

Original: Their approach is clearly ineffective for these scenarios.
Revised: Their approach is clearly ineffective for these scenarios.

Original: Reed & De Freitas (2015) introduced the novel approach of using execution traces to induce and execute algorithms, such as addition and sorting, from very few examples.
Revised: Reed & De Freitas (2015) introduced the novel approach of using execution traces to induce and execute algorithms, such as addition and sorting, from very few examples.

Original: The differences from our work are (1) they use a sequence of instructions to represent dynamic execution trace as opposed to using dynamic program states; (2) their goal is to synthesize a neural controller to execute a program as a sequence of actions rather than learning a semantic program representation; and (3) they deal with programs in a language with low-level primitives such as function stack push/pop actions rather than a high-level programming language.
Revised: The differences from our work are (1) they use a sequence of instructions to represent dynamic execution trace as opposed to using dynamic program states; (2) their goal is to synthesize a neural controller to execute a program as a sequence of actions rather than learning a semantic program representation; and (3) they deal with programs in a language with low-level primitives such as function stack push/pop actions rather than a high-level programming language.

Original: As for learning representations, there are several related efforts in modeling semantics in sentence or symbolic expressions (Socher et al., 2013; Zaremba et al., 2014; Bowman, 2013).
Revised: As for learning representations, there are several related efforts in modeling semantics in sentence or symbolic expressions (Socher et al., 2013; Zaremba et al., 2014; Bowman, 2013).

Original: These approaches are similar to our work in spirit, but target different domains than programs.
Revised: These approaches are similar to our work in spirit, but target different domains than programs.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented a new program embedding that learns program representations from their runtime execution traces.
Revised: We presented a new program embedding that learns program representations from their runtime execution traces.

Original: We have used the new embeddings to predict error patterns that student make in their online programming submissions.
Revised: We have used the new embeddings to predict error patterns that student make in their online programming submissions.

Original: Our evaluation shows that the dynamic program embeddings signiﬁcantly outperform those that are learned via program syntax.
Revised: Our evaluation shows that the dynamic program embeddings signiﬁcantly outperform those that are learned via program syntax.

Original: We also demonstrate, via an additional application, that our dynamic program embeddings yield more than 10x speedups compared to an enumerative baseline for search-based program repair.
Revised: We also demonstrate, via an additional application, that our dynamic program embeddings yield more than 10x speedups compared to an enumerative baseline for search-based program repair.

Original: Beyond neural program repair, we believe that our dynamic program embeddings can be fruitfully utilized for many other neural program analysis tasks such as program induction and synthesis.
Revised: Beyond neural program repair, we believe that our dynamic program embeddings can be fruitfully utilized for many other neural program analysis tasks such as program induction and synthesis.

Original: A PPENDIX E RROR P ATTERNS
Revised: A PPENDIX E RROR P ATTERNS

Original: Print Chessboard:
Revised: Print Chessboard:

Original: • Misprinting “O” to “0” or printing lower case instead of upper case characters.
Revised: • Misprinting “O” to “0” or printing lower case instead of upper case characters.

Original: • Switching across rows are supposed to be the other way around ( i
Revised: • Switching across rows are supposed to be the other way around ( i

Original: e. printing OXOXOXOX for odd number rows and XOXOXOXO for even number rows).
Revised: e. printing OXOXOXOX for odd number rows and XOXOXOXO for even number rows).

Original: • Printing the ﬁrst row correctly but failed to make a switch across rows.
Revised: • Printing the ﬁrst row correctly but failed to make a switch across rows.

Original: • Printing the entire chessboard as “X” or “O” only.
Revised: • Printing the entire chessboard as “X” or “O” only.

Original: • Printing the chessboard correctly but with extra unnecessary characters.
Revised: • Printing the chessboard correctly but with extra unnecessary characters.

Original: Printing the incorrect number of rows.
Revised: Printing the incorrect number of rows.

Original: • Printing the incorrect number of columns.
Revised: • Printing the incorrect number of columns.

Original: • Printing the characters correctly but in wrong format ( i.e. not correctly seperated with the spaces to form the rows).
Revised: • Printing the characters correctly but in wrong format ( i.e. not correctly seperated with the spaces to form the rows).

Original: • Others.
Revised: • Others.

Original: Count Parentheses:
Revised: Count Parentheses:

Original: • Miss the corner case of empty strings.
Revised: • Miss the corner case of empty strings.

Original: • Mistakenly consider the parenthesis to be symbols rather than “(” or “)”.
Revised: • Mistakenly consider the parenthesis to be symbols rather than “(” or “)”.

Original: • Mishandling the string of unmatched parentheses.
Revised: • Mishandling the string of unmatched parentheses.

Original: Counting the number of matching parentheses rather then depth.
Revised: Counting the number of matching parentheses rather then depth.

Original: • Incorrectly assume nested parentheses are always present.
Revised: • Incorrectly assume nested parentheses are always present.

Original: Miscounting the characters which should have been ignored.
Revised: Miscounting the characters which should have been ignored.

Original: • Others.
Revised: • Others.

Original: Generate Binary Digits:
Revised: Generate Binary Digits:

Original: • Miss the corner case of integer 0.
Revised: • Miss the corner case of integer 0.

Original: • Misunderstand the binary digits to be underlying bytes of a string.
Revised: • Misunderstand the binary digits to be underlying bytes of a string.

Original: • Mistakes in arithmetic calculation regrading shift operations.
Revised: • Mistakes in arithmetic calculation regrading shift operations.

Original: • Adding the binary digits rather than concatenating them to a string.
Revised: • Adding the binary digits rather than concatenating them to a string.

Original: • Miss the one on the most signiﬁcant bit.
Revised: • Miss the one on the most signiﬁcant bit.

Original: • Others.
Revised: • Others.

Original: S ARF G EN ’ S A LGORITHM
Revised: S ARF G EN ’ S A LGORITHM

Original: /* P : incorrect program;
Revised: /* P : incorrect program;

Original: : all correct solutions */ FixGeneration
Revised: : all correct solutions */ 1 function FixGeneration( s ) begin //

Revised: Among P s identify P cs to be reference programs to fix P

Original: P cs ← CandidatesIdentiﬁcation( P  s ) //
Revised: P cs ← CandidatesIdentiﬁcation( P , P s ) //

Original: Initialize  to be  k ← ∞ //
Revised: Initialize the minimum number of fixes k to be inifinity k ← ∞ //

Original: Initialize the minimum set of fixes F 
Revised: Initialize the minimum set of fixes F ( P )

Original: F  null  c  cs do 
Revised: F ( P ) ← null for P c ∈ P cs do // Generates the syntactic discrepencies w.r.t. each P c C ( P , P c ) ← DiscrepenciesGeneration( P , P s ) //

Original: Generates the discrepencies
Revised: Selecting subsets of C ( P , P c ) from size of one itll |C ( P , P c ) | for n ∈ [1 , 2 , ..., |C ( P , P c ) | ] do C x x P each subset of

Revised: P c ) for C sub c ) ∈ C subs c )

Revised: do

Revised: (cid:48)

Revised: PatchApplication necessary

Revised: if then

Revised: |

Revised: return F learned Model */ s

Revised: M ) identify cs to be reference programs to fix P cs ← CandidatesIdentiﬁcation( P , P s ) //

Revised: Initialize the minimum number of fixes k to be inifinity k

Revised: ∞ //

Revised: Initialize null

Revised: P c do //

Revised: Generates the syntactic discrepencies w

Original: P c
Revised: P c

Original: C
Revised: C

Original: P 
Revised: P , P c ← DiscrepenciesGeneration( P , P s ) // Executing P to extract dynamic execution trace T ( P ) ← DynamicTraceExtraction( P ) //

Revised: Prioritizing subsets

Revised: C

Revised: P pre model

Original: DiscrepenciesGeneration( P do do
Revised: C subs P P c Prioritization do

Original: E XAMPLE P ROGRAMS
Revised: E XAMPLE P ROGRAMS

Original: Here we given several examples the baseline models failed to generalize due to 1).
Revised: Here we given several examples the baseline models failed to generalize due to 1).

Original: small syntax differences cause distinct error patterns.
Revised: small syntax differences cause distinct error patterns.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure]
Revised: [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: (int

Original: [Equation]
Revised: [Table]

Original: for

Original: [Equation]
Revised: [Figure]

Original:  if [Equation] [Figure] [Equation]
Revised: ) { string s1 = "X"; string s2 = ""; for (int i = 1; i <= 8; i++) { for (int x = 1; x <= 8; x++) { s2 = s2 + s1; if (x < 8) { if (s1 == "X") { s1 = "0"; } else{ s1 = "X"; } } }

Original: [Equation]
Revised: Console.

Original: [Figure] [Figure] [Figure]
Revised: WriteLine(s2);s2=""; } } }

Original: large syntactic variation overwhelms real error (“O” instead of “0”)
Revised: large syntactic variation overwhelms real error (“O” instead of “0”)

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Original: do{
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: using System; namespace ConsoleApplication1 { class Program { public static void Main() { for (int i = 0; i < 8; i++) { for (int j = 0; j < 8; j++) { if (i % 2 == 0) { if (j % 2 == 0) {
Revised: using System; namespace ConsoleApplication1 { class Program { public static void Main() { for (int i = 0; i < 8; i++) { for (int j = 0; j < 8; j++) { if (i % 2 == 0) { if (j % 2 == 0) {

Original: Console.
Revised: Console.

Original: Write("X"); } else{
Revised: Write("X"); } else{

Original: Console.
Revised: Console.

Original: Write("0");} }else {if (j % 2 == 0) {
Revised: Write("0");} }else {if (j % 2 == 0) {

Original: Console.
Revised: Console.

Original: Write("0"); } else{
Revised: Write("0"); } else{

Original: Console.
Revised: Console.

Original: Write("X");} } }
Revised: Write("X");} } }

Original: Console.
Revised: Console.

Original: WriteLine(""); } } } }
Revised: WriteLine(""); } } } }

Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: int row 
Revised: int row =

Original: [Table]
Revised: [Table]

Original:  int squarecolor = 1; do{ do{ if (squarecolor == 1) {
Revised: = 0; int squarecolor = 1; do{ do{ if (squarecolor = = 1) {

Original: Console.
Revised: Console.

Original: Write("X"); if (row !
Revised: Write("X"); if (row !

Original: = 7) { squarecolor = 0; } } else{
Revised: = 7) { squarecolor = 0; } } else{

Original: [Table]
Revised: Console.

Original: .Write [Table] ");if(row!=7) {squarecolor
Revised: Write("0");if(row!=7) { squarecolor=1;} }

Original: [Table] [Table]
Revised: [Table] [Table]