Original: L EARNING TO R EPRESENT P ROGRAMS WITH G RAPHS
Revised: L EARNING TO R EPRESENT P ROGRAMS WITH G RAPHS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning tasks on source code ( i.e. , formal languages) have been considered recently, but most work has tried to transfer natural language methods and does not capitalize on the unique opportunities offered by code’s known syntax.
Revised: Learning tasks on source code ( i.e. , formal languages) have been considered recently, but most work has tried to transfer natural language methods and does not capitalize on the unique opportunities offered by code’s known syntax.

Original: For example, long-range dependencies induced by using the same variable or function in distant locations are often not considered.
Revised: For example, long-range dependencies induced by using the same variable or function in distant locations are often not considered.

Original: We propose to use graphs to represent both the syntactic and semantic structure of code and use graph-based deep learning methods to learn to reason over program structures.
Revised: We propose to use graphs to represent both the syntactic and semantic structure of code and use graph-based deep learning methods to learn to reason over program structures.

Original: In this work, we present how to construct graphs from source code and how to scale Gated Graph Neural Networks training to such large graphs.
Revised: In this work, we present how to construct graphs from source code and how to scale Gated Graph Neural Networks training to such large graphs.

Original: We evaluate our method on two tasks: V AR N AMING , in which a network attempts to predict the name of a variable given its usage, and V AR M ISUSE , in which the network learns to reason about selecting the correct variable that should be used at a given program location.
Revised: We evaluate our method on two tasks: V AR N AMING , in which a network attempts to predict the name of a variable given its usage, and V AR M ISUSE , in which the network learns to reason about selecting the correct variable that should be used at a given program location.

Original: Our comparison to methods that use less structured program representations shows the advantages of modeling known structure, and suggests that our models learn to infer meaningful names and to solve the V AR M ISUSE task in many cases.
Revised: Our comparison to methods that use less structured program representations shows the advantages of modeling known structure, and suggests that our models learn to infer meaningful names and to solve the V AR M ISUSE task in many cases.

Original: Additionally, our testing showed that V AR M ISUSE identiﬁes a number of bugs in mature open-source projects.
Revised: Additionally, our testing showed that V AR M ISUSE identiﬁes a number of bugs in mature open-source projects.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The advent of large repositories of source code as well as scalable machine learning methods naturally leads to the idea of “big code”, i.e. , largely unsupervised methods that support software engineers by generalizing from existing source code (Allamanis et al., 2017).
Revised: The advent of large repositories of source code as well as scalable machine learning methods naturally leads to the idea of “big code”, i.e. , largely unsupervised methods that support software engineers by generalizing from existing source code (Allamanis et al., 2017).

Original: Currently, existing deep learning models of source code capture its shallow, textual structure, e.g. as a sequence of tokens (Hindle et al., 2012; Raychev et al., 2014; Allamanis et al., 2016), as parse trees (Maddison & Tarlow, 2014;
Revised: Currently, existing deep learning models of source code capture its shallow, textual structure, e.g. as a sequence of tokens (Hindle et al., 2012; Raychev et al., 2014; Allamanis et al., 2016), as parse trees (Maddison & Tarlow, 2014;

Original: Bielik et al., 2016), or as a ﬂat dependency networks of variables (Raychev et al., 2015).
Revised: Bielik et al., 2016), or as a ﬂat dependency networks of variables (Raychev et al., 2015).

Original: Such models miss out on the opportunity to capitalize on the rich and well-deﬁned semantics of source code.
Revised: Such models miss out on the opportunity to capitalize on the rich and well-deﬁned semantics of source code.

Original: In this work, we take a step to alleviate this by including two additional signal sources in source code: data ﬂow and type hierarchies.
Revised: In this work, we take a step to alleviate this by including two additional signal sources in source code: data ﬂow and type hierarchies.

Original: We do this by encoding programs as graphs, in which edges represent syntactic relationships ( e.g. “token before/after”) as well as semantic relationships (“variable last used/written here”, “formal parameter for argument is called stream ”, etc. ).
Revised: We do this by encoding programs as graphs, in which edges represent syntactic relationships ( e.g. “token before/after”) as well as semantic relationships (“variable last used/written here”, “formal parameter for argument is called stream ”, etc. ).

Original: Our key insight is that exposing these semantics explicitly as structured input to a machine learning model lessens the requirements on amounts of training data, model capacity and training regime and allows us to solve tasks that are beyond the current state of the art.
Revised: Our key insight is that exposing these semantics explicitly as structured input to a machine learning model lessens the requirements on amounts of training data, model capacity and training regime and allows us to solve tasks that are beyond the current state of the art.

Original: We explore two tasks to illustrate the advantages of exposing more semantic structure of programs.
Revised: We explore two tasks to illustrate the advantages of exposing more semantic structure of programs.

Original: First, we consider the V AR N AMING task (Allamanis et al., 2014;
Revised: First, we consider the V AR N AMING task (Allamanis et al., 2014;

Original: Raychev et al., 2015), in which given some source code, the “correct” variable name is inferred as a sequence of subtokens.
Revised: Raychev et al., 2015), in which given some source code, the “correct” variable name is inferred as a sequence of subtokens.

Original: This requires some understanding of how a variable is used, i.e. , requires reasoning about lines of code far apart in the source ﬁle.
Revised: This requires some understanding of how a variable is used, i.e. , requires reasoning about lines of code far [Figure] [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Original: Secondly, we introduce the variable misuse prediction task (V AR M ISUSE ), in which the network aims to infer which variable should be used in a program location.
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: [Figure]

Revised: apart in the source ﬁle.

Revised: Secondly, we introduce the variable misuse prediction task (V AR M ISUSE ), in which the network aims to infer which variable should be used in a program location.

Original: To illustrate the task, Figure 1 shows a slightly simpliﬁed snippet of a bug our model detected in a popular open-source project (details anonymized).
Revised: To illustrate the task, Figure 1 shows a slightly simpliﬁed snippet of a bug our model detected in a popular open-source project.

Original: Speciﬁcally, instead of the variable clazz , variable first should have been used in the yellow highlighted slot.
Revised: Speciﬁcally, instead of the variable clazz , variable first should have been used in the yellow highlighted slot.

Original: Existing static analysis methods cannot detect such issues, even though a software engineer would easily identify this as an error from experience.
Revised: Existing static analysis methods cannot detect such issues, even though a software engineer would easily identify this as an error from experience.

Original: To achieve high accuracy on these tasks, we need to learn representations of program semantics.
Revised: To achieve high accuracy on these tasks, we need to learn representations of program semantics.

Original: For both tasks, we need to learn the semantic role of a variable ( e.g. , “is it a counter?”, “is it a ﬁlename?”).
Revised: For both tasks, we need to learn the semantic role of a variable ( e.g. , “is it a counter?”, “is it a ﬁlename?”).

Original: Additionally, for V AR M ISUSE , learning variable usage semantics ( e.g. , “a ﬁlename[Figure] is needed here”) is required.
Revised: Additionally, for V AR M ISUSE , learning variable usage semantics ( e.g. , “a ﬁlename is needed here”) is required.

Original: This “ﬁll the blank element” task is related to methods for learning distributed representations of natural language words, such as Word2Vec (Mikolov et al., 2013) and
Revised: This “ﬁll the blank element” task is related to methods for learning distributed representations of natural language words, such as Word2Vec (Mikolov et al., 2013) and

Original: GLoVe (Pennington et al., 2014).
Revised: GLoVe (Pennington et al., 2014).

Original: However, we can learn from a much richer structure such as data ﬂow information.
Revised: However, we can learn from a much richer structure such as data ﬂow information.

Original: This work is a step towards learning program representations, and we expect them to be valuable in a wide range of other tasks, such as code completion (“this is the variable you are looking for”) and more advanced bug ﬁnding (“you should lock before using this object”).
Revised: This work is a step towards learning program representations, and we expect them to be valuable in a wide range of other tasks, such as code completion (“this is the variable you are looking for”) and more advanced bug ﬁnding (“you should lock before using this object”).

Original: To summarize, our contributions are: (i)
Revised: To summarize, our contributions are: (i)

Original: We deﬁne the V AR M ISUSE task as a challenge for machine learning modeling of source code, that requires to learn (some) semantics of programs ( cf. section 3).
Revised: We deﬁne the V AR M ISUSE task as a challenge for machine learning modeling of source code, that requires to learn (some) semantics of programs ( cf. section 3).

Original: (ii) We present deep learning models for solving the V AR N AMING and V AR M ISUSE tasks by modeling the code’s graph structure and learning program representations over those graphs ( cf.
Revised: (ii) We present deep learning models for solving the V AR N AMING and V AR M ISUSE tasks by modeling the code’s graph structure and learning program representations over those graphs ( cf.

Original: section 4).
Revised: section 4).

Original: (iii)
Revised: (iii)

Original: We evaluate our models on a large dataset of 2.9 million lines of real-world source code, showing that our best model achieves 30.7% accuracy on the V AR N AMING task and 82.1% accuracy on the V AR M ISUSE task, beating simpler baselines ( cf. section 5).
Revised: We evaluate our models on a large dataset of 2.9 million lines of real-world source code, showing that our best model achieves 32.9% accuracy on the V AR N AMING task and 85.5% accuracy on the V AR M ISUSE task, beating simpler baselines ( cf. section 5).

Original: (iv) We document practical relevance of V AR M ISUSE by summarizing some bugs that we found in mature open-source software projects ( cf. subsection 5.3).
Revised: (iv) We document practical relevance of V AR M ISUSE by summarizing some bugs that we found in mature open-source software projects ( cf. subsection 5.3).

Original: We provide all data at [anonymized].
Revised: Our implementation of graph neural networks (on a simpler task) can be found at https://github.com/Microsoft/gated-graph-neural-network-samples .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work builds upon the recent ﬁeld of using machine learning for source code artifacts (Allamanis et al., 2017).
Revised: Our work builds upon the recent ﬁeld of using machine learning for source code artifacts (Allamanis et al., 2017).

Original: For example, Hindle et al.
Revised: For example, Hindle et al.

Original: ; Bhoopchand et al.
Revised: ; Bhoopchand et al.

Original: (2016) model the code as a sequence of tokens, while Maddison & Tarlow (2014); Raychev et al.
Revised: (2016) model the code as a sequence of tokens, while Maddison & Tarlow (2014); Raychev et al.

Original: (2016) model the syntax tree structure of code.
Revised: (2016) model the syntax tree structure of code.

Original: All works on language models of code ﬁnd that predicting variable and method identiﬁers is one of biggest challenges in the task.
Revised: All works on language models of code ﬁnd that predicting variable and method identiﬁers is one of biggest challenges in the task.

Original: Closest to our work is the work of Allamanis et al.
Revised: Closest to our work is the work of Allamanis et al.

Original: (2015) who learn distributed representations of variables using all their usages to predict their names.
Revised: (2015) who learn distributed representations of variables using all their usages to predict their names.

Original: However, they do not use data ﬂow information and we are not aware of any model that does so.
Revised: However, they do not use data ﬂow information and we are not aware of any model that does so.

Original: Raychev et al. (2015) and Bichsel et al.
Revised: Raychev et al. (2015) and Bichsel et al.

Original: (2016) use conditional random ﬁelds to model a variety of relationships between variables, AST elements and types to predict variable names and types (resp.
Revised: (2016) use conditional random ﬁelds to model a variety of relationships between variables, AST elements and types to predict variable names and types (resp.

Original: to deobfuscate Android apps), but without considering the ﬂow of data explicitly.
Revised: to deobfuscate Android apps), but without considering the ﬂow of data explicitly.

Original: In these works, all variable usages are deterministically known beforehand (as the code is complete and remains unmodiﬁed), as in Allamanis et al.
Revised: In these works, all variable usages are deterministically known beforehand (as the code is complete and remains unmodiﬁed), as in Allamanis et al.

Original: Our work is remotely related to work on program synthesis using sketches (Solar-Lezama, 2008) and automated code transplantation (Barr et al., 2015).
Revised: Our work is remotely related to work on program synthesis using sketches (Solar-Lezama, 2008) and automated code transplantation (Barr et al., 2015).

Original: However, these approaches require a set of speciﬁcations ( e.g. input-output examples, test suites) to complete the gaps, rather than statistics learned from big code.
Revised: However, these approaches require a set of speciﬁcations ( e.g. input-output examples, test suites) to complete the gaps, rather than statistics learned from big code.

Original: These approaches can be thought as complementary to ours, since we learn to statistically complete the gaps without any need for speciﬁcations, by learning common variable usages patterns from code.
Revised: These approaches can be thought as complementary to ours, since we learn to statistically complete the gaps without any need for speciﬁcations, by learning common variable usage patterns from code.

Original: Neural networks on graphs (Gori et al., 2005; Li et al., 2015; Defferrard et al., 2016; Kipf & Welling, 2016;
Revised: Neural networks on graphs (Gori et al., 2005; Li et al., 2015; Defferrard et al., 2016; Kipf & Welling, 2016;

Original: Gilmer et al., 2017) adapt a variety of deep learning methods to graph-structured input.
Revised: Gilmer et al., 2017) adapt a variety of deep learning methods to graph-structured input.

Original: They have been used in a series of applications, such as link prediction and classiﬁcation (Grover &
Revised: They have been used in a series of applications, such as link prediction and classiﬁcation (Grover &

Original: Leskovec, 2016) and semantic role labeling in NLP (Marcheggiani & Titov, 2017).
Revised: Leskovec, 2016) and semantic role labeling in NLP (Marcheggiani & Titov, 2017).

Original: Somewhat related to source code is the work of Wang et al.
Revised: Somewhat related to source code is the work of Wang et al.

Original: (2017) who learn graph-based representations of mathematical formulas for premise selection in theorem proving.
Revised: (2017) who learn graph-based representations of mathematical formulas for premise selection in theorem proving.

Original:  T HE V AR M ISUSE T ASK
Revised:  T HE V AR M ISUSE T ASK

Original: Detecting variable misuses in code is a task that requires understanding and reasoning about program semantics.
Revised: Detecting variable misuses in code is a task that requires understanding and reasoning about program semantics.

Original: To successfully tackle the task one needs to infer the role and function of the program elements and understand how they relate.
Revised: To successfully tackle the task one needs to infer the role and function of the program elements and understand how they relate.

Original: For example, given a program such as Fig. 1, the task is to automatically detect that the marked use of clazz is a mistake and that first should be used instead.
Revised: For example, given a program such as Fig. 1, the task is to automatically detect that the marked use of clazz is a mistake and that first should be used instead.

Original: While this task resembles standard code completion, it differs signiﬁcantly in its scope and purpose, by considering only variable identiﬁers and a mostly complete program.
Revised: While this task resembles standard code completion, it differs signiﬁcantly in its scope and purpose, by considering only variable identiﬁers and a mostly complete program.

Original: Task Description We view a source code ﬁle as a sequence of tokens t 0 . . .
Revised: Task Description We view a source code ﬁle as a sequence of tokens t 0 . . .

Original: t N = T , in which some tokens t λ 0 , t λ 1 . . .
Revised: t N = T , in which some tokens t λ 0 , t λ 1 . . .

Original: are variables.
Revised: are variables.

Original: Furthermore, let V t ⊂ V refer to the set of all type-correct variables in scope at the location of t , i.e. , those variables that can be used at t without raising a compiler error.
Revised: Furthermore, let V t ⊂ V refer to the set of all type-correct variables in scope at the location of t , i.e. , those variables that can be used at t without raising a compiler error.

Original: We call the location t where we want to predict the correct variable usage a slot .
Revised: We call the location t where we want to predict the correct variable usage a slot .

Original: We deﬁne a separate task for each slot t λ :
Revised: We deﬁne a separate task for each slot t λ :

Original: Given t 0 . . .
Revised: Given t 0 . . .

Original: t λ − 1 and t λ +1 , . . .
Revised: t λ − 1 and t λ +1 , . . .

Original: ,  λ N , correctly select t λ from V t λ .
Revised: , t λ N , correctly select t λ from V t λ .

Original: For training and evaluation purposes, a correct solution is one that simply matches the ground truth, but note that in practice, several possible assignments could be considered correct ( i.e. , when several variables refer to the same value in memory).
Revised: For training and evaluation purposes, a correct solution is one that simply matches the ground truth, but note that in practice, several possible assignments could be considered correct ( i.e. , when several variables refer to the same value in memory).

Original:  M ODEL : P ROGRAMS AS G RAPHS
Revised:  M ODEL : P ROGRAMS AS G RAPHS

Original: In this section, we discuss how to transform program source code into program graphs and learn representations over them.
Revised: In this section, we discuss how to transform program source code into program graphs and learn representations over them.

Original: These program graphs not only encode the program text but also the semantic information that can be obtained using standard compiler tools.
Revised: These program graphs not only encode the program text but also the semantic information that can be obtained using standard compiler tools.

Original: Gated Graph Neural Networks Our work builds on Gated Graph Neural Networks (Li et al., 2015) (GGNN) and we summarize them here.
Revised: Gated Graph Neural Networks Our work builds on Gated Graph Neural Networks (Li et al., 2015) (GGNN) and we summarize them here.

Original: A graph G =
Revised: A graph G =

Original: ( V , E , X ) is composed of a set of nodes
Revised: ( V , E , X ) is composed of a set of nodes

Original: V , node features X , and a list of directed edge sets E
Revised: V , node features X , and a list of directed edge sets E

Original: =
Revised: =

Original: ( E 1 , . . .
Revised: ( E 1 , . . .

Original: , E K ) where K is the number of edge types.
Revised: , E K ) where K is the number of edge types.

Original: We annotate each v  with a real-valued vector x ( v ) ∈ R D representing the features of the node ( e.g. , the embedding of a string label of that node).
Revised: We annotate each v ∈ V with a real-valued vector x ( v ) ∈ R D representing the features of the node ( e.g. , the embedding of a string label of that node).

Original: We associate every node v with a state vector h ( v ) , initialized from the node label x ( v ) .
Revised: We associate every node v with a state vector h ( v ) , initialized from the node label x ( v ) .

Original: The sizes of the state vector and feature vector are typically the same, but we can use larger state vectors through padding of node features.
Revised: The sizes of the state vector and feature vector are typically the same, but we can use larger state vectors through padding of node features.

Original: To propagate information throughout the graph, “messages” of type k are sent from each v to its neighbors, where each message is computed from its current state vector as m ( v ) k
Revised: To propagate information throughout the graph, “messages” of type k are sent from each v to its neighbors, where each message is computed from its current state vector as m ( v ) k

Original: = f k ( h ( v ) ) .
Revised: = f k ( h ( v ) ) .

Original: Here, f k can be an arbitrary function; we choose a linear layer in our case.
Revised: Here, f k can be an arbitrary function; we choose a linear layer in our case.

Original: By computing messages for all graph edges at the same time, all states can be updated at the same time.
Revised: By computing messages for all graph edges at the same time, all states can be updated at the same time.

Original: In particular, a new state for a node v is computed by aggregating all incoming messages as ˜ m ( v ) =
Revised: In particular, a new state for a node v is computed by aggregating all incoming messages as ˜ m ( v ) =

Original: g ( { m ( u ) k | there is an edge of type k from u to v } ) .
Revised: g ( { m ( u ) k | there is an edge of type k from u to v } ) .

Original: g is an aggregation function, which we implement as elementwise summation.
Revised: g is an aggregation function, which we implement as elementwise summation.

Original: Given the aggregated message ˜ m ( v ) and the current state vector h ( v ) of node v , the state of the next time step h (cid:48) ( v ) is computed as h (cid:48) ( v ) =
Revised: Given the aggregated message ˜ m ( v ) and the current state vector h ( v ) of node v , the state of the next time step h (cid:48) ( v ) is computed as h (cid:48) ( v ) =

Original: GRU ( ˜ m ( v ) , h ( v ) ) , where GRU is the recurrent cell function of gated recurrent unit (GRU) (Cho et al., 2014).
Revised: GRU ( ˜ m ( v ) , h ( v ) ) , where GRU is the recurrent cell function of gated recurrent unit (GRU) (Cho et al., 2014).

Original: The dynamics deﬁned by the above equations are repeated for a ﬁxed number of time steps.
Revised: The dynamics deﬁned by the above equations are repeated for a ﬁxed number of time steps.

Original: Then, we use the state vectors from the last time step as the node representations. 
Revised: Then, we use the state vectors from the last time step as the node representations. [Figure] Assert . [Figure]

Revised: [Figure]

Revised: Fig.

Original: Program Graphs We represent program source code as graphs and use different edges to model syntactic and semantic relationships between different tokens.
Revised: [Figure] are nodes [Figure] [Figure]

Revised: gular

Revised: [Figure]

Original:  The backbone of a program graph is the program’s abstract syntax tree (AST), consisting of syntax nodes (corresponding to non-terminals in the programming language’s grammar) and syntax tokens (corresponding to terminals).
Revised: tokens [Figure] Program Graphs We represent program source code as graphs and use different edge types to model syntactic and semantic relationships between different tokens. The backbone of a program graph is the program’s abstract syntax tree (AST), consisting of syntax nodes (corresponding to nonterminals in the programming language’s grammar) and syntax tokens (corresponding to terminals).

Original: We label syntax nodes with the name of the nonterminal from the program’s grammar, whereas syntax tokens are labeled with the string that they represent.
Revised: We label syntax nodes with the name of the nonterminal from the program’s grammar, whereas syntax tokens are labeled with the string that they represent.

Original: We use Child edges to connect nodes according to the AST.
Revised: We use Child edges to connect nodes according to the AST.

Original: As this does not induce an order on children of a syntax node, we additionally add NextToken edges connecting each syntax token to its successor.
Revised: As this does not induce an order on children of a syntax node, we additionally add NextToken edges connecting each syntax token to its successor.

Original: An example of this is shown in Fig.
Revised: An example of this is shown in Fig.

Original: 2a.
Revised: 2a.

Original: To capture the ﬂow of control and data through a program, we add additional edges connecting different uses and updates of syntax tokens corresponding to variables.
Revised: To capture the ﬂow of control and data through a program, we add additional edges connecting different uses and updates of syntax tokens corresponding to variables.

Original: For such a token v , let D R ( v ) be the set of syntax tokens at which the variable could have been used last.
Revised: For such a token v , let D R ( v ) be the set of syntax tokens at which the variable could have been used last.

Original: This set may contain several nodes (for example, when using a variable after a conditional in which it was used in both branches), and even syntax tokens that follow in the program code (in the case of loops).Similarly, [Figure] ArgumentList
Revised: This set may contain several nodes (for example, when using a variable after a conditional in which it was used in both branches), and even syntax tokens that follow in the program code (in the case of loops).

Original: Assert .

Original: NotNull (

Original: [Figure]

Original: Simpliﬁed

Original: [Figure]

Original: graph for line 2 of Fig.

Original: 1, where blue rounded boxes are syntax nodes, black rectangular boxes syntax tokens,

Original: [Figure]

Original: Child edges and

Original: [Figure]

Original: black edges NextToken edges

Original: [Figure] let D W ( v ) be the set of syntax tokens at which the variable was last written to.
Revised: Similarly, let D W ( v ) be the set of syntax tokens at which the variable was last written to.

Original: Using these, we add LastRead (resp.
Revised: Using these, we add LastRead (resp.

Original: LastWrite ) edges connecting v to all elements of D R ( v ) (resp.
Revised: LastWrite ) edges connecting v to all elements of D R ( v ) (resp.

Original: D W ( v ) ).
Revised: D W ( v ) ).

Original: Additionally, whenever we observe an assignment v = expr , we connect v to all variable tokens occurring in expr using ComputedFrom edges.
Revised: Additionally, whenever we observe an assignment v = expr , we connect v to all variable tokens occurring in expr using ComputedFrom edges.

Original: An example of such semantic edges is shown in
Revised: An example of such semantic edges is shown in

Original: Fig.
Revised: Fig.

Original: 2b.
Revised: 2b.

Original: We extend the graph to chain all uses of the same variable using LastLexicalUse edges (independent of data ﬂow, i.e. , in if (...) { ... v ...} else { ... v ...} , we link the two occurrences of v ).
Revised: We extend the graph to chain all uses of the same variable using LastLexicalUse edges (independent of data ﬂow, i.e. , in if (...) { ... v ...} else { ... v ...} , we link the two occurrences of v ).

Original: We also connect return tokens to the method declaration using ReturnsTo edges (this creates a “shortcut” to its name and type).
Revised: We also connect return tokens to the method declaration using ReturnsTo edges (this creates a “shortcut” to its name and type).

Original: Finally, inspired by Rice et al.
Revised: Inspired by Rice et al.

Original: , we connect arguments in method calls to the formal parameters that they are matched to with FormalArgName edges, i.e. , if we observe a call Foo(bar) and a method declaration Foo(InputStream stream) , we connect the bar token to the stream token.
Revised: , we connect arguments in method calls to the formal parameters that they are matched to with FormalArgName edges, i.e. , if we observe a call Foo(bar) and a method declaration Foo(InputStream stream) , we connect the bar token to the stream token.

Revised: Finally, we connect every token corresponding to a variable to enclosing guard expressions that use the variable with GuardedBy and GuardedByNegation edges.

Revised: For example, in if (x > y) { ... x ...} else { ... y ...} , we add a GuardedBy edge from x (resp.

Revised: a GuardedByNegation edge from y ) to the AST node corresponding to x > y .

Original: Finally, for all types of edges we introduce their respective backwards edges (transposing the adjacency matrix), doubling the number of edges and edge types.
Revised: Finally, for all types of edges we introduce their respective backwards edges (transposing the adjacency matrix), doubling the number of edges and edge types.

Original: Backwards edges help with propagating information faster across the GGNN and make the model more expressive.
Revised: Backwards edges help with propagating information faster across the GGNN and make the model more expressive.

Original: Leveraging Variable Type Information We assume a statically typed language and that the source code can be compiled, and thus each variable has a (known) type τ ( v ) .
Revised: Leveraging Variable Type Information We assume a statically typed language and that the source code can be compiled, and thus each variable has a (known) type τ ( v ) .

Original: To use it, we deﬁne a learnable embedding function r ( τ ) for known types and additionally deﬁne an “U NK T YPE ” for all unknown/unrepresented types.
Revised: To use it, we deﬁne a learnable embedding function r ( τ ) for known types and additionally deﬁne an “U NK T YPE ” for all unknown/unrepresented types.

Original: We also leverage the rich type hierarchy that is available in many object-oriented languages.
Revised: We also leverage the rich type hierarchy that is available in many object-oriented languages.

Original: For this, we map a variable’s type τ ( v ) to the set of its supertypes, i.e. τ ∗ ( v ) =
Revised: For this, we map a variable’s type τ ( v ) to the set of its supertypes, i.e. τ ∗ ( v ) =

Original: { τ : τ ( v ) implements type τ } ∪ { τ ( v ) } .
Revised: { τ : τ ( v ) implements type τ } ∪ { τ ( v ) } .

Original: We then compute the type representation r ∗ ( v ) of a variable v as the element-wise maximum of { r ( τ ) : τ ∈ τ ∗ ( v ) } .
Revised: We then compute the type representation r ∗ ( v ) of a variable v as the element-wise maximum of { r ( τ ) : τ ∈ τ ∗ ( v ) } .

Original: We chose the maximum here, as it is a natural pooling operation for representing partial ordering relations (such as type lattices).
Revised: We chose the maximum here, as it is a natural pooling operation for representing partial ordering relations (such as type lattices).

Original: Using all types in τ ∗ ( v ) allows us to generalize to unseen types that implement common supertypes or interfaces.
Revised: Using all types in τ ∗ ( v ) allows us to generalize to unseen types that implement common supertypes or interfaces.

Original: For example, List<K> has multiple concrete types ( e.g. List<int> ,
Revised: For example, List<K> has multiple concrete types ( e.g. List<int> ,

Original: List<string> ).
Revised: List<string> ).

Original: Nevertheless, these types implement a common interface ( IList ) and share common characteristics.
Revised: Nevertheless, these types implement a common interface ( IList ) and share common characteristics.

Original: During training, we randomly select a non-empty subset of τ ∗ ( v ) which ensures training of all known types in the lattice.
Revised: During training, we randomly select a non-empty subset of τ ∗ ( v ) which ensures training of all known types in the lattice.

Original: This acts both like a dropout mechanism and allows us to learn a good representation for all types in the type lattice.
Revised: This acts both like a dropout mechanism and allows us to learn a good representation for all types in the type lattice.

Original: Initial Node Representation To compute the initial node state, we combine information from the textual representation of the token and its type.
Revised: Initial Node Representation To compute the initial node state, we combine information from the textual representation of the token and its type.

Original: Concretely, we split the name of a node  into subtokens ( e.g. classTypes will be split into two subtokens class and types ) on camelCase and pascal_case .
Revised: Concretely, we split the name of a node representing a token into subtokens ( e.g. classTypes will be split into two subtokens class and types ) on camelCase and pascal_case .

Original: We then average the embeddings of all subtokens to retrieve an embedding for the node name.
Revised: We then average the embeddings of all subtokens to retrieve an embedding for the node name.

Original: Finally, we concatenate the learned type representation r ∗ ( v ) , computed as discussed earlier, with the node name representation, and pass it through a linear layer to obtain the initial representations for each node in the graph.
Revised: Finally, we concatenate the learned type representation r ∗ ( v ) , computed as discussed earlier, with the node name representation, and pass it through a linear layer to obtain the initial representations for each node in the graph.

Original: Programs Graphs for V AR N AMING
Revised: Programs Graphs for V AR N AMING

Original: Given a program and an existing variable v , we build a program graph as discussed above and then replace the variable name in all corresponding variable tokens by a special <SLOT> token.
Revised: Given a program and an existing variable v , we build a program graph as discussed above and then replace the variable name in all corresponding variable tokens by a special <SLOT> token.

Original: To predict a name, we use the initial node labels computed as the concatenation of learnable token embeddings and type embeddings as discussed above, run
Revised: To predict a name, we use the initial node labels computed as the concatenation of learnable token embeddings and type embeddings as discussed above, run

Original: GGNN propagation for 8 time steps 1 and then compute a variable usage representation by averaging the representations for all <SLOT> tokens.
Revised: GGNN propagation for 8 time steps 1 and then compute a variable usage representation by averaging the representations for all <SLOT> tokens.

Original: This representation is then used as the initial state of a one-layer GRU, which predicts the target name as a sequence of subtokens ( e.g. , the name inputStreamBuffer is treated as the sequence [ input , stream , buffer ]).
Revised: This representation is then used as the initial state of a one-layer GRU, which predicts the target name as a sequence of subtokens ( e.g. , the name inputStreamBuffer is treated as the sequence [ input , stream , buffer ]).

Original: We train this graph2seq architecture using a maximum likelihood objective.
Revised: We train this graph2seq architecture using a maximum likelihood objective.

Original: In section 5, we report the accuracy for predicting the exact nameand the F1 score for predicting its subtokens.
Revised: In section 5, we report the accuracy for predicting the exact name

Original: Program Graphs for V AR M ISUSE
Revised: Program Graphs for V AR M ISUSE

Original: To model V AR M ISUSE with program graphs we need to modify the graph.
Revised: To model V AR M ISUSE with program graphs we need to modify the graph.

Original: First, to compute a context representation c ( t ) for a slot t where we want to predict the used variable, we insert a new node v <SLOT> at the position of t , corresponding to a “hole” at this point, and connect it to the remaining graph using all applicable edges that do not depend on the chosen variable at the slot ( i.e. , everything but LastUse , LastWrite , and LastLexicalUse edges).
Revised: First, to compute a context representation c ( t ) for a slot t where we want to predict the used variable, we insert a new node v <SLOT> at the position of t , corresponding to a “hole” at this point, and connect it to the remaining graph using all applicable edges that do not depend on the chosen variable at the slot ( i.e. , everything but LastUse , LastWrite , and LastLexicalUse edges).

Original: Then, to compute the usage representation u ( t, v ) ofof each candidate variable v at the target slot, we insert a “candidate” node v t,v for all v in V t , and connect it to the graph by inserting the LastUse ,
Revised: Then, to compute the usage representation u ( t, v ) of each candidate variable v at the target slot, we insert a “candidate” node v t,v for all v in V t , and connect it to the graph by inserting the LastUse ,

Original: LastWrite and LastLexicalUse edges that would be used if the variable were to be used at this slot.
Revised: LastWrite and LastLexicalUse edges that would be used if the variable were to be used at this slot.

Original: Each of these candidate nodes represents the speculative placement of the variable within the scope.
Revised: Each of these candidate nodes represents the speculative placement of the variable within the scope.

Original: Using the initial node representations, concatenated with an extra bit that is set to one for the candidate nodes v t,v , we run GGNN propagation for 8 time steps.
Revised: Using the initial node representations, concatenated with an extra bit that is set to one for the candidate nodes v t,v , we run GGNN propagation for 8 time steps.

Original: The context and usage representation are then the ﬁnal node states of the nodes, i.e. , c ( t )
Revised: The context and usage representation are then the ﬁnal node states of the nodes, i.e. , c ( t )

Original: =
Revised: =

Original: h ( v <SLOT> ) and u ( t, v ) =
Revised: h ( v <SLOT> ) and u ( t, v ) =

Original: h ( v t,v ) .
Revised: h ( v t,v ) .

Original: Finally, the correct variable usage at the location is computed as arg max v  c ( t ) T u ( t, v ) .
Revised: Finally, the correct variable usage at the location is computed as arg max v W [ c ( t ) , u ( t, v )] where W is a linear layer that uses the concatenation of c ( t ) and u ( t, v ) .

Original: We train using maximum likelihood.
Revised: We train using a max-margin objective.

Original: 4. I MPLEMENTATION
Revised: 4. I MPLEMENTATION

Original: Using GGNNs for sets of large, diverse graphs requires some engineering effort, as efﬁcient batching is hard in the presence of diverse shapes.
Revised: Using GGNNs for sets of large, diverse graphs requires some engineering effort, as efﬁcient batching is hard in the presence of diverse shapes.

Original: An important observation is that large graphs are normally very sparse, and thus a representation of edges as an adjacency list would usually be advantageous to reduce memory consumption.
Revised: An important observation is that large graphs are normally very sparse, and thus a representation of edges as an adjacency list would usually be advantageous to reduce memory consumption.

Original: In our case, this can be easily implemented using a sparse tensor representation, allowing large batch sizes that exploit the parallelism of modern GPUs efﬁciently.
Revised: In our case, this can be easily implemented using a sparse tensor representation, allowing large batch sizes that exploit the parallelism of modern GPUs efﬁciently.

Original: A second key insight is to represent a batch of graphs as one large graph with many disconnected components.
Revised: A second key insight is to represent a batch of graphs as one large graph with many disconnected components.

Original: This just requires appropriate pre-processing to make node identities unique.
Revised: This just requires appropriate pre-processing to make node identities unique.

Original: As this makes batch construction somewhat CPU-intensive, we found it useful to prepare minibatches on a separate thread.
Revised: As this makes batch construction somewhat CPU-intensive, we found it useful to prepare minibatches on a separate thread.

Original: Our TensorFlow (Abadi et al., 2016) implementation scales to 55 graphs per second during training and 219 graphs per second during test-time using a single NVidia GeForceGTX Titan
Revised: Our TensorFlow (Abadi et al., 2016) implementation scales to 55 graphs per second during training and 219 graphs per second during test-time using a single NVidia GeForce

Original:  X with graphs having on average 2,228 (median 936) nodes and 8,350 (median 3,274) edges and  GGNN unrolling iterations, all 16 edge types (forward and backward edges for8 original edge types) and the size of the hidden layer set to 64.
Revised: GTX Titan X with graphs having on average 2,228 (median 936) nodes and 8,350 (median 3,274) edges and 8 GGNN unrolling iterations, all 20 edge types (forward and backward edges for original edge types) and the size of the hidden layer set to 64.

Original: The number of types of edges in the GGNN contribute proportionally to the running time.
Revised: The number of types of edges in the GGNN contribute proportionally to the running time.

Original: For example, a GGNN run for our ablation study using only the two most common edge types ( NextToken , Child ) achieves 105 graphs/second during training and 419 graphs/second at test time with the same hyperparameters.
Revised: For example, a GGNN run for our ablation study using only the two most common edge types ( NextToken , Child ) achievesgraphs/second during training and 419 graphs/second at test time with the same hyperparameters.

Original: We intend to release our (generic)sparse GGNN implementation soon.
Revised: Our (generic) implementation of GGNN is available at https://github.com/Microsoft/ gated-graph-neural-network-samples , using a simpler demonstration task.

Original:  E VALUATION
Revised:  E VALUATION

Original: Dataset We collected a dataset for the V AR M ISUSE task from open source C # projects on GitHub.
Revised: Dataset We collected a dataset for the V AR M ISUSE task from open source C # projects on GitHub.

Original: To select projects, we picked the top-starred (non-fork) projects in GitHub.
Revised: To select projects, we picked the top-starred (non-fork) projects in GitHub.

Original: We then ﬁltered out projects that we could not (easily) compile in full using Roslyn 2 , as we require a compilation to extract precise type information for the code (including those types present in external libraries).
Revised: We then ﬁltered out projects that we could not (easily) compile in full using Roslyn 2 , as we require a compilation to extract precise type information for the code (including those types present in external libraries).

Original: Our ﬁnal dataset contains 29 projects from a diverse set of domains (compilers, databases, ...) with about 2.9 million non-empty lines of code.
Revised: Our ﬁnal dataset contains 29 projects from a diverse set of domains (compilers, databases, ...) with about 2.9 million non-empty lines of code.

Original: A full table is shown in Appendix D, and all data is available online at [anonymized]. [Table] [Table]
Revised: A full table is shown in Appendix D.

Original: For the task of detecting variable misuses [Table] we collect data from all projects by selecting all variable usage locations, ﬁltering out variable declarations, where at least one other type-compatible replacement variable is in scope.
Revised: For the task of detecting variable misuses, we collect data from all projects by selecting all variable usage locations, ﬁltering out variable declarations, where at least one other type-compatible replacement variable is in scope.

Original: The task is then to infer the correct variable that originally existed in that location.
Revised: The task is then to infer the correct variable that originally existed in that location.

Original: Thus, by construction there is at least one type-correct replacement variable, i.e. picking it would notnot raise an error during type checking.
Revised: Thus, by construction there is at least one type-correct replacement variable, i.e. picking it would not raise an error during type checking.

Original: In our test datasets, at each slot there are on average 3.8 type-correct alternative variables (median 3, σ = 2 . 6 ).
Revised: In our test datasets, at each slot there are on average 3.8 type-correct alternative variables (median 3, σ = 2 . 6 ).

Original: From our dataset, we selected two projects as our development set.
Revised: From our dataset, we selected two projects as our development set.

Original: From the rest of the projects, we selected three projects for U NSEEN P
Revised: From the rest of the projects, we selected three projects for U NSEEN P

Original: ROJ T EST to allow testing on projects with completely unknown structure and types.
Revised: ROJ T EST to allow testing on projects with completely unknown structure and types.

Original: We split the remaining 23 projects into train/validation/test sets in the proportion 60-10-30, splitting along ﬁles ( i.e. , all examples from one source ﬁle are in the same set).
Revised: We split the remaining 23 projects into train/validation/test sets in the proportion 60-10-30, splitting along ﬁles ( i.e. , all examples from one source ﬁle are in the same set).

Original: We call the test set obtained like this S EEN P ROJ T EST .
Revised: We call the test set obtained like this S EEN P ROJ T EST .

Original: Baselines For V AR M ISUSE , we consider two bidirectional RNN-based baselines.
Revised: Baselines For V AR M ISUSE , we consider two bidirectional RNN-based baselines.

Original: The local model (L OC ) is a simple two-layer bidirectional GRU run over the tokens before and after the target location.
Revised: The local model (L OC ) is a simple two-layer bidirectional GRU run over the tokens before and after the target location.

Original: For this baseline, c ( t ) is set to the slot representation computed by the RNN, and the usage context of each variable u ( t, v ) is the embedding of the name and type of the variable, computed in the same way as the initial node labels in the GGNN.
Revised: For this baseline, c ( t ) is set to the slot representation computed by the RNN, and the usage context of each variable u ( t, v ) is the embedding of the name and type of the variable, computed in the same way as the initial node labels in the GGNN.

Original: This baseline allows us to evaluate how important the usage context information is for this task.
Revised: This baseline allows us to evaluate how important the usage context information is for this task.

Original: The ﬂat dataﬂow model (A VG B I RNN) is an extension to
Revised: The ﬂat dataﬂow model (A VG B I RNN) is an extension to

Original: L OC , where the usage representation u ( t, v ) is computed using another two-layer bidirectional RNN run over the tokens before/after each usage, and then averaging over the computed representations at the variable token v .
Revised: L OC , where the usage representation u ( t, v ) is computed using another two-layer bidirectional RNN run over the tokens before/after each usage, and then averaging over the computed representations at the variable token v .

Original: The local context, c ( t ) , is identical to L OC .
Revised: The local context, c ( t ) , is identical to L OC .

Original: A VG B I RNN is a signiﬁcantly stronger baseline that already takes some structural information into account, as the averaging over all variables usages helps with long-range dependencies.
Revised: A VG B I RNN is a signiﬁcantly stronger baseline that already takes some structural information into account, as the averaging over all variables usages helps with long-range dependencies.

Original: As in GGNN, both models pick the variable that maximizes c ( t ) T u ( t, v ) .
Revised: Both models pick the variable that maximizes c ( t ) T u ( t, v ) .

Original: For V AR N AMING , we replace L OC by A VG LBL, which uses a log-bilinear model for 4 left andright context tokens of each variable usage, and then averages over these context representations (this corresponds to the model in Allamanis et al. (2015)).
Revised: For V AR N AMING , we replace L OC by A VG LBL, which uses a log-bilinear model for 4 left andright context tokens of each variable usage, and then averages over these context representations (this corresponds to the model in Allamanis et al. (2015)).

Original: We also test A VG B I RNN on V AR N AMING , which essentially replaces the log-bilinear context model by a bidirectional RNN.
Revised: We also test A VG B I RNN on V AR N AMING , which essentially replaces the log-bilinear context model by a bidirectional RNN.

Original: 5. Q UANTITATIVE E VALUATION
Revised: 5. Q UANTITATIVE E VALUATION

Original: Table 1 shows the evaluation results of the models for both tasks.
Revised: Table 1 shows the evaluation results of the models for both tasks.

Original: As L OC captures very little information, it performs relatively badly.
Revised: As L OC captures very little information, it performs relatively badly.

Original: A VG LBL and A VG B I RNN, which capture information from many variable usage sites, but do not explicitly encode the rich structure of the problem, still lag behind the GGNN by a wide margin.
Revised: A VG LBL and A VG B I RNN, which capture information from many variable usage sites, but do not explicitly encode the rich structure of the problem, still lag behind the GGNN by a wide margin.

Original: The performance difference is larger for V AR M ISUSE , since the structure and the semantics of code are far more important within this setting.
Revised: The performance difference is larger for V AR M ISUSE , since the structure and the semantics of code are far more important within this setting.

Original: Generalization to new projects Generalizing across a diverse set of source code projects with different domains is an important challenge in machine learning.
Revised: Generalization to new projects Generalizing across a diverse set of source code projects with different domains is an important challenge in machine learning.

Original: We repeat the evaluation using the
Revised: We repeat the evaluation using the

Revised: [Table]

Revised: [Table]

Original: U NSEEN P
Revised: U NSEEN P

Original: ROJ T EST set stemming from projects that have no ﬁles in the training set.
Revised: ROJ T EST set stemming from projects that have no ﬁles in the training set.

Original: The right side of
Revised: The right side of

Original: Table 1 shows that our models still achieve good performance, although it is slightly lower compared to S EEN P ROJ T EST .
Revised: Table 1 shows that our models still achieve good performance, although it is slightly lower compared to S EEN P ROJ T EST .

Original: This is expected since the type lattice is mostly unknown in U NSEEN P
Revised: This is expected since the type lattice is mostly unknown in U NSEEN P

Original: ROJ T EST .
Revised: ROJ T EST .

Original: We believe that some of the most important issues when transferring to new domains is the fact that projects have signiﬁcantly different type hierarchies and that the vocabulary used ( e.g. within tokens) is very different from the training projects.
Revised: We believe that some of the most important issues when transferring to new domains is the fact that projects have signiﬁcantly different type hierarchies and that the vocabulary used ( e.g. within tokens) is very different from the training projects.

Original: Ablation Study To study the effect of some of the design choices for our models, we have run some additional experiments and show their results in Table 2.
Revised: Ablation Study To study the effect of some of the design choices for our models, we have run some additional experiments and show their results in Table 2.

Original: First, we varied the edges used in the program graph.
Revised: First, we varied the edges used in the program graph.

Original: We ﬁnd that restricting the model to syntactic information has a large impact on performance on both tasks, whereas restricting it to semantic edges seems to mostly impact performance on V AR M ISUSE .
Revised: We ﬁnd that restricting the model to syntactic information has a large impact on performance on both tasks, whereas restricting it to semantic edges seems to mostly impact performance on V AR M ISUSE .

Original: Similarly, the ComputedFrom , FormalArgName and ReturnsTo edges give a small boost on V AR M ISUSE , but greatly improve performance on V AR N AMING .
Revised: Similarly, the ComputedFrom , FormalArgName and ReturnsTo edges give a small boost on V AR M ISUSE , but greatly improve performance on V AR N AMING .

Original: As evidenced by the experiments with the node label representation, syntax node and token names seem to matter little for V AR M ISUSE , but naturally have a great impact on V AR N AMING .
Revised: As evidenced by the experiments with the node label representation, syntax node and token names seem to matter little for V AR M ISUSE , but naturally have a great impact on V AR N AMING .

Original: 5. Q UALITATIVE E VALUATION
Revised: 5. Q UALITATIVE E VALUATION

Original: Figure 3 illustrates a prediction that GGNN makes on a sample test snippet.
Revised: Figure 3 illustrates the predictions that GGNN makes on a sample test snippet.

Original: The snippet recursively searches for the global directives ﬁle by gradually descending into the root folder.
Revised: The snippet recursively searches for the global directives ﬁle by gradually descending into the root folder.

Original: Reasoning about the correct variable usages is hard, even for humans, but the GGNN correctly predicts the variable usages an all locations except two (slot 1 and 8).
Revised: Reasoning about the correct variable usages is hard, even for humans, but the GGNN correctly predicts the variable usages an all locations except two (slot 1 and 8).

Original: As a software engineer is writing the code it is imaginable that she may make a mistake misusing one variable in the place of another.
Revised: As a software engineer is writing the code it is imaginable that she may make a mistake misusing one variable in the place of another.

Original: Since all variables are string variables, no type errors will be raised.
Revised: Since all variables are string variables, no type errors will be raised.

Original: As the probabilities in Fig.
Revised: As the probabilities in Fig.

Original: 3 suggest most potential variable misuses can be ﬂagged by the model yielding valuable warnings to software engineers.
Revised: 3 suggest most potential variable misuses can be ﬂagged by the model yielding valuable warnings to software engineers.

Original: Additional samples with comments can be found in Appendix B.
Revised: Additional samples with comments can be found in Appendix B.

Original: Furthermore, Appendix C shows samples of pairs of code snippets that share similar representations as computed by the cosine similarity of the usage representation u ( t, v ) of GGNN.
Revised: Furthermore, Appendix C shows samples of pairs of code snippets that share similar representations as computed by the cosine similarity of the usage representation u ( t, v ) of GGNN.

Original: The reader can notice that the network learns to group variable usages that share semantic similarities together.
Revised: The reader can notice that the network learns to group variable usages that share semantic similarities together.

Original: For example, checking for null before the use of a variable yields similar distributed representations across code segments (Sample 1 in Appendix C). 
Revised: For example, checking for null before the use of a variable yields similar distributed representations across code segments (Sample 1 in Appendix C). [Table] [Table] [Figure]

Original: 5. D ISCOVERED V ARIABLE M ISUSE B UGS
Revised: 5. D ISCOVERED V ARIABLE M ISUSE B UGS

Original: We have used our V AR M ISUSE model to identify likely locations of bugs in RavenDB (a document database) and Roslyn (Microsoft’s C # compiler framework).
Revised: We have used our V AR M ISUSE model to identify likely locations of bugs in RavenDB (a document database) and Roslyn (Microsoft’s C # compiler framework).

Original: For this, we manually reviewed a sample of the top 500 locations in both projects where our model was most conﬁdent about a choosing a variable differing from the ground truth, and found three bugs in each of the projects.
Revised: For this, we manually reviewed a sample of the top 500 locations in both projects where our model was most conﬁdent about a choosing a variable differing from the ground truth, and found three bugs in each of the projects.

Original: Figs.
Revised: Figs.

Original: 1,4,5 show the issues discovered in RavenDB.
Revised: 1,4,5 show the issues discovered in RavenDB.

Original: The bug in Fig. 1 was possibly caused by copy-pasting, and cannot be easily caught by traditional methods.
Revised: The bug in Fig. 1 was possibly caused by copy-pasting, and cannot be easily caught by traditional methods.

Original: A compiler will not warn about unused variables (since first is used) and virtually nobody would write a test testing another test.
Revised: A compiler will not warn about unused variables (since first is used) and virtually nobody would write a test testing another test.

Original: Fig.
Revised: Fig.

Original: 4 shows an issue that, although not critical, can lead to increased memory consumption.
Revised: 4 shows an issue that, although not critical, can lead to increased memory consumption.

Revised: if (IsValidBackup(backupFilename

Revised: [Equation]

Original: Fig.
Revised: = false [Equation] { output [Equation]

Revised: throw new InvalidOperationException(

Revised: [Equation]

Original:  5 shows another issue arising from a non-informative error message.
Revised: like [Equation] Fig. 5 shows another issue arising from a non-informative error message.

Original: We privately reported three additional bugs to the Roslyn developers, who have ﬁxed the issues in the meantime (cf. https://github.com/dotnet/roslyn/pull/23437 , which does not break anonymity of this submission).
Revised: We privately reported three additional bugs to the Roslyn developers, who have ﬁxed the issues in the meantime (cf. https://github.com/dotnet/roslyn/pull/23437 ).

Original: One of the reported bugs could cause a crash in VisualStudio when using certain Roslyn features.
Revised: One of the reported bugs could cause a crash in Visual Studio when using certain Roslyn features.

Original: Finding these issues in widely released and tested code suggests that our model can be useful during the software development process, complementing classic program analysis tools.
Revised: Finding these issues in widely released and tested code suggests that our model can be useful during the software development process, complementing classic program analysis tools.

Original: For example, one usage scenario would be to guide the code reviewing process to locations a V AR M ISUSE model has identiﬁed as unusual, or use it as a prior to focus testing or expensive code analysis efforts.[Table] [Table] null
Revised: For example, one usage scenario would be to guide the code reviewing process to locations a V AR M ISUSE model has identiﬁed as unusual, or use it as a prior to focus testing or expensive code analysis efforts.

Original: [Table]

Original: [Table]

Original: (File

Original: [Table]

Original: Exists

Original: [Table]

Original: path

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Figure]

Original:  D ISCUSSION & C ONCLUSIONS
Revised:  D ISCUSSION & C ONCLUSIONS

Original: Although source code is well understood and studied within other disciplines such as programming language research, it is a relatively new domain for deep learning.
Revised: Although source code is well understood and studied within other disciplines such as programming language research, it is a relatively new domain for deep learning.

Original: It presents novel opportunities compared to textual or perceptual data, as its (local) semantics are well-deﬁned and rich additional information can be extracted using well-known, efﬁcient program analyses.
Revised: It presents novel opportunities compared to textual or perceptual data, as its (local) semantics are well-deﬁned and rich additional information can be extracted using well-known, efﬁcient program analyses.

Original: On the other hand, integrating this wealth of structured information poses an interesting challenge.
Revised: On the other hand, integrating this wealth of structured information poses an interesting challenge.

Original: Our V AR M ISUSE task exposes these opportunities, going beyond simpler tasks such as code completion.
Revised: Our V AR M ISUSE task exposes these opportunities, going beyond simpler tasks such as code completion.

Original: We consider it as a ﬁrst proxy for the core challenge of learning the meaning of source code, as it requires to probabilistically reﬁne standard information included in type systems.if (IsValidBackup(backupFilename) = = false) { output("Error: [Equation]
Revised: We consider it as a ﬁrst proxy for the core challenge of learning the meaning of source code, as it requires to probabilistically reﬁne standard information included in type systems.

Original: + backupLocation

Original: [Equation]

Original: doesn’t look like a valid backup

Original: [Equation]

Original: ; throw new InvalidOperationException( backupLocation

Original: [Equation]

Original: like a valid

Original: [Equation]

Original: A P ERFORMANCE C URVES
Revised: A P ERFORMANCE C URVES

Original: Figure 6 show the ROC and precision-recall curves for the GGNN model.
Revised: Figure 6 show the ROC and precision-recall curves for the GGNN model.

Original: As the reader may observe, setting a false positive rate to 10% we get a true positive rate 4 of 73% for the S EEN P ROJ T EST and 69% for the unseen test.
Revised: As the reader may observe, setting a false positive rate to 10% we get a true positive rate 4 of 73% for the S EEN P ROJ T EST and 69% for the unseen test.

Original: This suggests that this model can be practically used at a high precision setting with acceptable performance.
Revised: This suggests that this model can be practically used at a high precision setting with acceptable performance.

Original: B V AR M ISUSE P REDICTION S AMPLES
Revised: B V AR M ISUSE P REDICTION S AMPLES

Original: Below we list a set of samples from our S EEN P ROJ T EST projects with comments about the model performance.
Revised: Below we list a set of samples from our S EEN P ROJ T EST projects with comments about the model performance.

Original: Code comments and formatting may have been altered for typesetting reasons.
Revised: Code comments and formatting may have been altered for typesetting reasons.

Original: The ground truth choice is underlined.
Revised: The ground truth choice is underlined.

Original: Samplefor (var port = #1 ; # < #3 ; #4 ++) { if (!activePorts.Contains( #5 )) return #6 ; } # startingFrom : 97%, endingAt : 3% # port : 100%, startingFrom : 0%, endingAt : 0% # endingAt : 100%, startingFrom : 0%, port : 0% # port : 100%, startingFrom : 0%, endingAt : 0% # port : 100%, startingFrom : 0%, endingAt [Table] [Table] port : 100%, startingFrom 
Revised: Samplefor (var port = #1 ; # < #3 ; #4 ++) { if (!activePorts.Contains( #5 )) return #6 ; } # startingFrom : 97%, endingAt : 3% # port : 100%, startingFrom : 0%, endingAt : 0% # endingAt : 100%, startingFrom : 0%, port : 0% # port : 100%, startingFrom : 0%, endingAt : 0% # port : 100%, startingFrom : 0%, endingAt : 0% # port : 100%, startingFrom : 0%, endingAt : 0% (cid:46)

Original: [Table]

Original: 0%, endingAt

Original: [Table]

Original: [Table]

Original: The model correctly predicts all variables in the loop.
Revised: The model correctly predicts all variables in the loop.

Original: Samplevar path =
Revised: Samplevar path =

Original: CreateFileName( #1 ); bitmap.
Revised: CreateFileName( #1 ); bitmap.

Original: Save( # , ImageFormat.
Revised: Save( #2 , ImageFormat.

Original: Png); return #3 ; # name : 86%, DIR_PATH : 14% # path : 90%, name : 8%, DIR_PATH : 2% # path : 76%, name : 16%, DIR_PATH : 8% (cid:46) String variables are not confused their semantic role is inferred correctly
Revised: Png); return #3 ; # name : 86%, DIR_PATH : 14% # path : 90%, name : 8%, DIR_PATH : 2% # path : 76%, name : 16%, DIR_PATH : 8% (cid:46) String variables are not confused their semantic role is inferred correctly.

Original: Sample[global::System.
Revised: Sample[global::System.

Original: Diagnostics.
Revised: Diagnostics.

Original: DebuggerNonUserCodeAttribute]publicvoidMergeFrom(pb::CodedInputStreaminput){ uinttag;while((tag=input.
Revised: DebuggerNonUserCodeAttribute]publicvoidMergeFrom(pb::CodedInputStreaminput){ uinttag;while((tag=input.

Original: ReadTag())!=0){ switch(tag){ default: input
Revised: ReadTag())!=0){ switch(tag){ default: input.

Revised: SkipLastField();break; case 10: {

Original: ;break;
Revised: ); break; } } } } #

Original: Payload : 66%, payload_ : 44% (cid:46)
Revised: Payload : 66%, payload_ : 44% (cid:46)

Original: The model is commonly confused by aliases,
Revised: The model is commonly confused by aliases,

Original: i
Revised: i

Original: e. variables that point to the same location in memory.
Revised: e. variables that point to the same location in memory.

Original: In this sample, either choice would have yielded identical behavior.
Revised: In this sample, either choice would have yielded identical behavior.

Original: Samplepublic override bool IsDisposed { get{ lock ( #1 ) { return #2 ; } } } # _gate : 99%, _observers : 1% # _isDisposed : 90%, _isStopped : 8%, HasObservers : 2% (cid:46)
Revised: Samplepublic override bool IsDisposed { get{ lock ( #1 ) { return #2 ; } } } # _gate : 99%, _observers : 1% # _isDisposed : 90%, _isStopped : 8%, HasObservers : 2% (cid:46)

Original: The ReturnsTo edge can help predict variables that otherwise would have been impossible.
Revised: The ReturnsTo edge can help predict variables that otherwise would have been impossible.

Original: Sample/// <summary> /// Notifies all subscribed observers about the exception.
Revised: Sample/// <summary> /// Notifies all subscribed observers about the exception.

Original: /// </summary> /// <param name="error">The exception to send to all observers.</param> public override void OnError(Exception error)
Revised: /// </summary> /// <param name="error">The exception to send to all observers.</param> public override void OnError(Exception error)

Original: { if ( # == null) throw new ArgumentNullException(nameof( #2 )); var os = default(IObserver<T>[]); lock ( #3 ) {
Revised: { if ( # == null) throw new ArgumentNullException(nameof( #2 )); var os = default(IObserver<T>[]); lock ( #3 ) {

Original: CheckDisposed(); if (! #4 ) { os = _observers.
Revised: CheckDisposed(); if (! #4 ) { os = _observers.

Original: Data; _observers =
Revised: Data; _observers =

Original: ImmutableList<IObserver<T>>.Empty; # = true; # = #7 ; } } if (os !=
Revised: ImmutableList<IObserver<T>>.Empty; # = true; # = #7 ; } } if (os !=

Original: null) { foreach (var o in os) { o.OnError( #8 ); } } 
Revised: null) { foreach (var o in os) { o.OnError( #8 ); } } } # error : exception

Original: } # error : 93%, _exception : 7% #  error : 98%, _exception : 2% # _gate : 100%, _observers : 0% # _isStopped : 86%, _isDisposed : 13%, HasObservers : 1% # _isStopped : 91%, _isDisposed : 9%, HasObservers :0% # _exception : 100%, error : 0% # error : 98%, _exception : 2% # _exception : 99%, error : 1% (cid:46)
Revised:  error : 98%, _exception : 2% # _gate : observers isStopped _isDisposed %, HasObservers :

Revised: isStopped : 91%, _isDisposed : 9%,

Revised: HasObservers

Original: The model predicts the correct variables from all slots apart from the last.
Revised: _exception : , error : 0% # error 98%, _exception  _exception :

Revised: 99%, error :

Revised: % (cid:46)

Revised: The model predicts the correct variables from all slots apart from the last.

Original: Reasoning about the last one, requires interprocedural understanding of the code across the class ﬁle.
Revised: Reasoning about the last one, requires interprocedural understanding of the code across the class ﬁle.

Original: Sampleprivate bool BecomingCommand(object message) { if (ReceiveCommand( #1 ) return true; if ( #2 .ToString() =
Revised: Sampleprivate bool BecomingCommand(object message) { if (ReceiveCommand( #1 ) return true; if ( #2 .ToString() =

Original: =
Revised: =

Original: #3 ) #4 .Tell( #5 ); else return false; return true; } # message : 100%, Response : 0%, Message : 0% # message : 100%, Response : 0%, Message : 0% #
Revised: #3 ) #4 .Tell( #5 ); else return false; return true; } # message : 100%, Response : 0%, Message : 0% # message : 100%, Response : 0%, Message : 0% #

Original: Response : 91%, Message : 9% #
Revised: Response : 91%, Message : 9% #

Original: Probe : 98%, AskedForDelete : 2% #
Revised: Probe : 98%, AskedForDelete : 2% #

Original: Response : 98%, Message : 2% (cid:46)
Revised: Response : 98%, Message : 2% (cid:46)

Original: The model predicts correctly all usages except from the one in slot #3.
Revised: The model predicts correctly all usages except from the one in slot #3.

Original: Reasoning about this snippet requires additional semantic information about the intent of the code.
Revised: Reasoning about this snippet requires additional semantic information about the intent of the code.

Original: Samplevar response = ResultsFilter(typeof(TResponse), #1 , #2 , request); # httpMethod : 99%, absoluteUrl : 1%, UserName : 0%, UserAgent : 0% # absoluteUrl : 99%, httpMethod : 1%, UserName : 0%, UserAgent : 0% (cid:46)
Revised: Samplevar response = ResultsFilter(typeof(TResponse), #1 , #2 , request); # httpMethod : 99%, absoluteUrl : 1%, UserName : 0%, UserAgent : 0% # absoluteUrl : 99%, httpMethod : 1%, UserName : 0%, UserAgent : 0% (cid:46)

Original: The model knows about selecting the correct string parameters because it matches them to the formal parameter names.
Revised: The model knows about selecting the correct string parameters because it matches them to the formal parameter names.

Original: Sampleif ( # >= #2 ) throw new InvalidOperationException(Strings_Core.
Revised: Sampleif ( # >= #2 ) throw new InvalidOperationException(Strings_Core.

Original: FAILED_CLOCK_MONITORING)
Revised: FAILED_CLOCK_MONITORING)

Original: ; # n : 100%, MAXERROR : 0%, SYNC_MAXRETRIES : 0% #
Revised: ; # n : 100%, MAXERROR : 0%, SYNC_MAXRETRIES : 0% #

Original: MAXERROR : 62%, SYNC_MAXRETRIES : 22%, n : 16% (cid:46)
Revised: MAXERROR : 62%, SYNC_MAXRETRIES : 22%, n : 16% (cid:46)

Original: It is hard for the model to reason about conditionals, especially with rare constants as in slot #2.
Revised: It is hard for the model to reason about conditionals, especially with rare constants as in slot #2.

Original: C N EAREST N EIGHBOR OF GGNN U SAGE R EPRESENTATIONS
Revised: C N EAREST N EIGHBOR OF GGNN U SAGE R EPRESENTATIONS

Original: Here we show pairs of nearest neighbors based on the cosine similarity of the learned representations u ( t, v ) .
Revised: Here we show pairs of nearest neighbors based on the cosine similarity of the learned representations u ( t, v ) .

Original: Each slot t is marked in dark blue and all usages of v are marked in yellow ( i.e. variableName ).
Revised: Each slot t is marked in dark blue and all usages of v are marked in yellow ( i.e. variableName ).

Original: This is a 
Revised: This is a set of hand-picked examples showing good and bad examples. A brief description follows after each pair.

Original: Sample... public void MoveShapeUp(BaseShape shape ) { if ( shape !=
Revised: Sample... public void MoveShapeUp(BaseShape shape ) { if ( shape !=

Original: null) { for(int i=0; i < Shapes.
Revised: null) { for(int i=0; i < Shapes.

Original: Count -1; i++){ if (Shapes[i] == shape ){
Revised: Count -1; i++){ if (Shapes[i] == shape ){

Original: Shapes.
Revised: Shapes.

Original: Move(i, ++i); return; } } } } ...
Revised: Move(i, ++i); return; } } } } ...

Original: ...lock(lockObject) { if ( unobservableExceptionHanler !
Revised: ...lock(lockObject) { if ( unobservableExceptionHanler !

Original: = null) return false; unobservableExceptionHanler = handler; }...
Revised: = null) return false; unobservableExceptionHanler = handler; }...

Original: (cid:46) Slots that are checked for null-ness have similar representations.
Revised: (cid:46) Slots that are checked for null-ness have similar representations.

Original: Sample... public IActorRef ResolveActorRef(ActorPath actorPath ){ if(HasAddress( actorPath .Address)) return _local.
Revised: Sample... public IActorRef ResolveActorRef(ActorPath actorPath ){ if(HasAddress( actorPath .Address)) return _local.

Original: ResolveActorRef(RootGuardian, actorPath .ElementsWithUid); ... ...
Revised: ResolveActorRef(RootGuardian, actorPath .ElementsWithUid); ... ...

Original: ActorPath actorPath ; if (TryParseCachedPath(path, out actorPath)) { if (HasAddress( actorPath .Address))
Revised: ActorPath actorPath ; if (TryParseCachedPath(path, out actorPath)) { if (HasAddress( actorPath .Address))

Original: { if ( actorPath .ToStringWithoutAddress().Equals("/")) return RootGuarding; ...
Revised: { if ( actorPath .ToStringWithoutAddress().Equals("/")) return RootGuarding; ...

Original: (cid:46) Slots that follow similar API protocols have similar representations.
Revised: (cid:46) Slots that follow similar API protocols have similar representations.

Original: Note that the function
Revised: Note that the function

Original: HasAddress is a local function, seen only in the testset.
Revised: HasAddress is a local function, seen only in the testset.

Original: Sample... foreach(var filter in configuration.
Revised: Sample... foreach(var filter in configuration.

Original: Filters){
Revised: Filters){

Revised: GlobalJobFilter.

Original: .Filters.
Revised: Filters.

Original: Add filter ); }.
Revised: Add( filter ); }...

Original: public void 
Revised: ...public void Count_ReturnsNumberOfElements ){ _collection. Add Assert

Revised: Equal collection.

Revised: (cid:46)

Original: Adding elements to a collection-like object yields similar representations.
Revised: Adding elements to a collection-like object yields similar representations.

Original: D D ATASET
Revised: D D ATASET

Original: The collected dataset and its characteristics are listed in Table 3.
Revised: The collected dataset and its characteristics are listed in Table 4.

Original: The full dataset as a set of projects and its parsed JSON will become available online.
Revised: The full dataset as a set of projects and its parsed JSON will become available online.

Original: [Table]
Revised: [Table]

Original: ffc4912f 3.6k 6.1k
Revised: ffc4912f 3.6k 6.1k

Original: Background Job Processing Library
Revised: Background Job Processing Library

Original: Humanizer † cc11a77e 2.4k 4.4k
Revised: Humanizer † cc11a77e 2.4k 4.4k

Original: String Manipulation and Formatting
Revised: String Manipulation and Formatting

Original: Lean † f574bfd  26.4k 28.3k
Revised: Lean † f574bfd  26.4k 28.3k

Original: Algorithmic Trading Engine Nancy 72e1f  7.5k 15.
Revised: Algorithmic Trading Engine Nancy 72e1f  7.5k 15.

Original: HTTP Service Framework Newtonsoft.
Revised: HTTP Service Framework Newtonsoft.

Original: Json 6057d9b  14.9k 16.1k
Revised: Json 6057d9b  14.9k 16.1k

Original: JSON Library Ninject 7006297f 0.7k 2.1k
Revised: JSON Library Ninject 7006297f 0.7k 2.1k

Original: Code Injection Library NLog 643e326a 8.3k 11.0k
Revised: Code Injection Library NLog 643e326a 8.3k 11.0k

Original: Logging Library Opserver 51b032e  3.7k 4.5k
Revised: Logging Library Opserver 51b032e  3.7k 4.5k

Original: Monitoring System OptiKey 7d35c  6.1k 3.9k
Revised: Monitoring System OptiKey 7d35c  6.1k 3.9k

Original: Assistive On-Screen Keyboard orleans e0d6a  30.7k 35.6k
Revised: Assistive On-Screen Keyboard orleans e0d6a  30.7k 35.6k

Original: Distributed Virtual Actor Model Polly 0afdbc  3.8k 9.1k
Revised: Distributed Virtual Actor Model Polly 0afdbc  3.8k 9.1k

Original: Resilience & Transient Fault Handling Library quartznet b33e6f  9.6k 9.8k
Revised: Resilience & Transient Fault Handling Library quartznet b33e6f  9.6k 9.8k

Original: Scheduler ravendb Dev  78.0k 82.7k
Revised: Scheduler ravendb Dev  78.0k 82.7k

Original: Document Database RestSharp 70de357b 4.0k 4.5k
Revised: Document Database RestSharp 70de357b 4.0k 4.5k

Original: REST and HTTP API Client Library Rx.
Revised: REST and HTTP API Client Library Rx.

Original: NET 2d146fe  14.0k 21.9k
Revised: NET 2d146fe  14.0k 21.9k

Original: Reactive Language Extensions scriptcs f3cc8bcb 2.7k 4.3k
Revised: Reactive Language Extensions scriptcs f3cc8bcb 2.7k 4.3k

Original: C# Text Editor ServiceStack 6d59da  38.0k 46.2k
Revised: C# Text Editor ServiceStack 6d59da  38.0k 46.2k

Original: Web Framework ShareX 718dd  22.3k 18.1k
Revised: Web Framework ShareX 718dd  22.3k 18.1k

Original: Sharing Application SignalR fa88089e
Revised: Sharing Application SignalR fa88089e

Original: 6.5k 10.5k
Revised: 6.5k 10.5k

Original: Push Notiﬁcation Framework Wox cdaf  2.0k 
Revised: Push Notiﬁcation Framework Wox cdaf  2.0k 2.1k

Revised: Application Launcher