Original: G RAPH C ODE BERT: P RE - TRAINING C ODE R EPRESENTATIONS WITH D ATA F LOW
Revised: G RAPH C ODE BERT: P RE - TRAINING C ODE R EPRESENTATIONS WITH D ATA F LOW

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Pre-trained models for programming language have achieved dramatic empirical improvements on a variety of code-related tasks such as code search, code completion, code summarization, etc.
Revised: Pre-trained models for programming language have achieved dramatic empirical improvements on a variety of code-related tasks such as code search, code completion, code summarization, etc.

Original: However, existing pre-trained models regard a code snippet as a sequence of tokens, while ignoring the inherent structure of code, which provides crucial code semantics and would enhance the code understanding process.
Revised: However, existing pre-trained models regard a code snippet as a sequence of tokens, while ignoring the inherent structure of code, which provides crucial code semantics and would enhance the code understanding process.

Original: We present GraphCodeBERT, a pre-trained model for programming language that considers the inherent structure of code.
Revised: We present GraphCodeBERT, a pre-trained model for programming language that considers the inherent structure of code.

Original: Instead of taking syntacticlevel structure of code like abstract syntax tree (AST), we use data ﬂow in the pre-training stage, which is a semantic-level structure of code that encodes the rela- tion of “where -the-value-comes- from” between variables
Revised: Instead of taking syntactic-level structure of code like abstract syntax tree (AST), we use data ﬂow in the pre-training stage, which is a semantic-level structure of code that encodes the relation of “wherethe-value-comes-from” between variables.

Original: Such a semantic- level structure is neat and does not bring an unnecessarily deep hierarchy of AST, the property of which makes the model more efﬁcient.
Revised: Such a semantic-level structure is less complex and does not bring an unnecessarily deep hierarchy of AST, the property of which makes the model more efﬁcient.

Original: We develop GraphCodeBERT based on Transformer.
Revised: We develop GraphCodeBERT based on Transformer.

Original: In addition to using the task of masked language modeling we introduce two structure-aware pre-training tasks.
Revised: In addition to using the task of masked language modeling, we introduce two structure-aware pre-training tasks.

Original: One is to predict code structure edges, and the other is to align representations between source code and code structure.
Revised: One is to predict code structure edges, and the other is to align representations between source code and code structure.

Original: We implement the model in an efﬁcient way with a graph-guided masked attention function to incorporate the code structure.
Revised: We implement the model in an efﬁcient way with a graph-guided masked attention function to incorporate the code structure.

Original: We evaluate our model on four tasks, including code search, clone detection, code translation and code reﬁnement.
Revised: We evaluate our model on four tasks, including code search, clone detection, code translation, and code reﬁnement.

Original: Results show that code structure and newly introduced pre-training tasks can improve GraphCodeBERT and achieves state-of-the-art performance on the four downstream tasks.
Revised: Results show that code structure and newly introduced pre-training tasks can improve GraphCodeBERT and achieves state-of-the-art performance on the four downstream tasks.

Original: We further show that the model prefers structure-level attentions over token-level attentions in the task of code search
Revised: We further show that the model prefers structure-level attentions over token-level attentions in the task of code search.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Pre-trained models such as ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2018) have led to strong improvement on numerous natural language processing (NLP) tasks.
Revised: Pre-trained models such as ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2018) have led to strong improvement on numerous natural language processing (NLP) tasks.

Original: These pre-trained models are ﬁrst pre-trained on a large unsupervised text corpus, and then ﬁne-tuned on downstream tasks.
Revised: These pre-trained models are ﬁrst pre-trained on a large unsupervised text corpus, and then ﬁne-tuned on downstream tasks.

Original: The success of pre-trained models in NLP also promotes the development of pre-trained models for programming language.
Revised: The success of pre-trained models in NLP also promotes the development of pre-trained models for programming language.

Original: Existing works (Kanade et al., 2019; Karampatsis &
Revised: Existing works (Kanade et al., 2019; Karampatsis &

Original: Sutton, 2020; Feng et al., 2020; Svyatkovskiy et al., 2020; Buratti et al., 2020) regard a source code as a sequence of tokens and pre-train models on source code to support code-related tasks such as code search, code completion, code summarization, etc.
Revised: Sutton, 2020; Feng et al., 2020; Svyatkovskiy et al., 2020; Buratti et al., 2020) regard a source code as a sequence of tokens and pre-train models on source code to support code-related tasks such as code search, code completion, code summarization, etc.

Original: However, previous works only utilize source code for pre-training, while ignoring the inherent structure of code.
Revised: However, previous works only utilize source code for pre-training, while ignoring the inherent structure of code.

Original: Such code structure provides useful semantic information of code, which would beneﬁt the code understanding process.
Revised: Such code structure provides useful semantic information of code, which would beneﬁt the code understanding process.

Original: Taking the expression v = max value − min value as an example, v is computed from max value and min value .
Revised: Taking the expression v = max value − min value as an example, v is computed from max value and min value .

Original: Programmers do not always follow the naming conventions so that it’s hard to understand the semantic of the variable v only from its name.
Revised: Programmers do not always follow the naming conventions so that it’s hard to understand the semantic of the variable v only from its name.

Original: The semantic structure of code provides a way to understand the semantic of the variable v by leveraging dependency relation between variables.
Revised: The semantic structure of code provides a way to understand the semantic of the variable v by leveraging dependency relation between variables.

Revised: Research

Original: In this work, we present GraphCodeBERT, a pre-trained model for programming language that considers the inherent structure of code.
Revised: In this work, we present GraphCodeBERT, a pre-trained model for programming language that considers the inherent structure of code.

Original: Instead of taking syntactic-level structure of code like abstract syntax tree (AST), we leverage semantic-level information of code, i.e. data ﬂow, for pretraining.
Revised: Instead of taking syntactic-level structure of code like abstract syntax tree (AST), we leverage semantic-level information of code, i.e. data ﬂow, for pretraining.

Original: Data ﬂow is a graph, in which nodes represent variables and edges represent the relation of “where-the-value-comes-from” between variables.
Revised: Data ﬂow is a graph, in which nodes represent variables and edges represent the relation of “where-the-value-comes-from” between variables.

Original: Compared with AST, data ﬂow is neat and does not bring an unnecessarily deep hierarchy, the property of which makes the model more efﬁcient.
Revised: Compared with AST, data ﬂow is less complex and does not bring an unnecessarily deep hierarchy, the property of which makes the model more efﬁcient.

Original: In order to learn code representation from source code and code structure, we introduce two new structure-aware pre-training tasks.
Revised: In order to learn code representation from source code and code structure, we introduce two new structure-aware pre-training tasks.

Original: One is data ﬂow edges prediction for learning representation from code structure, and the other is variable-alignment across source code and data ﬂow for aligning representation between source code and code structure.
Revised: One is data ﬂow edges prediction for learning representation from code structure, and the other is variable-alignment across source code and data ﬂow for aligning representation between source code and code structure.

Original: GraphCodeBERT is based on Transformer neural architecture (Vaswani et al., 2017) and we extend it by introducing a graph-guided masked attention function to incorporate the code structure.
Revised: GraphCodeBERT is based on Transformer neural architecture (Vaswani et al., 2017) and we extend it by introducing a graph-guided masked attention function to incorporate the code structure.

Original: We pre-train GraphCodeBERT on the CodeSearchNet dataset (Husain et al., 2019), which includes 2.4M functions of six programming languages paired with natural language documents.
Revised: We pre-train GraphCodeBERT on the CodeSearchNet dataset (Husain et al., 2019), which includes 2.3M functions of six programming languages paired with natural language documents.

Original: We evaluate the model on four downstream tasks: natural language code search, clone detection, code translation, and code reﬁnement.
Revised: We evaluate the model on four downstream tasks: natural language code search, clone detection, code translation, and code reﬁnement.

Original: Experiments show that our model achieves state-of-the-art performance on the four tasks.
Revised: Experiments show that our model achieves state-of-the-art performance on the four tasks.

Original: Further analysis shows that code structure and newly introduced pre-training tasks can improve GraphCodeBERT and the model has consistent preference for attending data ﬂow.
Revised: Further analysis shows that code structure and newly introduced pre-training tasks can improve GraphCodeBERT and the model has consistent preference for attending data ﬂow.

Original: In summary, the contributions of this paper are: (1) GraphCodeBERT is the ﬁrst pre-trained model that leverages semantic structure of code to learn code representation.
Revised: In summary, the contributions of this paper are: (1) GraphCodeBERT is the ﬁrst pre-trained model that leverages semantic structure of code to learn code representation.

Original: (2) We introduce two new structure-aware pre-training tasks for learning representation from source code and data ﬂow.
Revised: (2) We introduce two new structure-aware pre-training tasks for learning representation from source code and data ﬂow.

Original: GraphCodeBERT provides signiﬁcant improvement on four downstream tasks, i.e. code search, clone detection, code translation, and code reﬁnement.
Revised: GraphCodeBERT provides signiﬁcant improvement on four downstream tasks, i.e. code search, clone detection, code translation, and code reﬁnement.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Pre-Trained Models for Programming Languages Inspired by the big success of pre-training in
Revised: Pre-Trained Models for Programming Languages Inspired by the big success of pre-training in

Original: NLP (Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019; Raffel et al., 2019), pre-trained models for programming languages also promotes the development of code intelligence (Kanade et al., 2019;
Revised: NLP (Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019; Raffel et al., 2019), pre-trained models for programming languages also promotes the development of code intelligence (Kanade et al., 2019;

Original: Feng et al., 2020; Karampatsis & Sutton, 2020; Svyatkovskiy et al., 2020; Buratti et al., 2020).
Revised: Feng et al., 2020; Karampatsis & Sutton, 2020; Svyatkovskiy et al., 2020; Buratti et al., 2020).

Original: Kanade et al.
Revised: Kanade et al.

Original: pre-train a BERT model on a massive corpus of Python source codes by masked language modeling and next sentence prediction objectives.
Revised: pre-train a BERT model on a massive corpus of Python source codes by masked language modeling and next sentence prediction objectives.

Original: Feng et al.
Revised: Feng et al.

Original: (2020) propose CodeBERT, a bimodal pre-trained model for programming and natural languages by masked language modeling and replaced token detection to support text-code tasks such as code search.
Revised: (2020) propose CodeBERT, a bimodal pre-trained model for programming and natural languages by masked language modeling and replaced token detection to support text-code tasks such as code search.

Original: Karampatsis & Sutton (2020) pre-train contextual embeddings on a JavaScript corpus using the ELMo framework for program repair task.
Revised: Karampatsis & Sutton (2020) pre-train contextual embeddings on a JavaScript corpus using the ELMo framework for program repair task.

Original: Svyatkovskiy et al.
Revised: Svyatkovskiy et al.

Original: (2020) propose GPT-C, which is a variant of the GPT-2 trained from scratch on source code data to support generative tasks like code completion.
Revised: (2020) propose GPT-C, which is a variant of the GPT-2 trained from scratch on source code data to support generative tasks like code completion.

Original: Buratti et al.
Revised: Buratti et al.

Original: (2020) present C-BERT, a transformer-based language model pre-trained on a collection of repositories written in C language, and achieve high accuracy in the abstract syntax tree (AST) tagging task.
Revised: (2020) present C-BERT, a transformer-based language model pre-trained on a collection of repositories written in C language, and achieve high accuracy in the abstract syntax tree (AST) tagging task.

Original: Different with previous works, GraphCodeBERT is the ﬁrst pre-trained model that leverages code structure to learn code representation to improve code understanding.
Revised: Different with previous works, GraphCodeBERT is the ﬁrst pre-trained model that leverages code structure to learn code representation to improve code understanding.

Original: We further introduce a graphguided masked attention function to incorporate the code structure into Transformer and two new structure-aware pre-training tasks to learn representation from source code and code structure.
Revised: We further introduce a graphguided masked attention function to incorporate the code structure into Transformer and two new structure-aware pre-training tasks to learn representation from source code and code structure.

Original: Neural Networks with Code Structure
Revised: Neural Networks with Code Structure

Original: In recent years, some neural networks leveraging code structure such as AST have been proposed and achieved strong performance in code-related tasks like code completion (Li et al., 2017;
Revised: In recent years, some neural networks leveraging code structure such as AST have been proposed and achieved strong performance in code-related tasks like code completion (Li et al., 2017;

Original: Alon et al., 2019;
Revised: Alon et al., 2019;

Original: Kim et al., 2020), code generation (Rabinovich et al., 2017; Yin & Neubig, 2017; Brockschmidt et al., 2018), code clone detection (Wei & Li, 2017;
Revised: Kim et al., 2020), code generation (Rabinovich et al., 2017; Yin & Neubig, 2017; Brockschmidt et al., 2018), code clone detection (Wei & Li, 2017;

Original: Zhang et al., 2019; Wang et al., 2020), code summarization (Alon et al., 2018;
Revised: Zhang et al., 2019; Wang et al., 2020), code summarization (Alon et al., 2018;

Original: Hu et al., 2018) and so on (Nguyen & Nguyen, 2015; Allamanis et al., 2018; Hellendoorn et al., 2019).
Revised: Hu et al., 2018) and so on (Nguyen & Nguyen, 2015; Allamanis et al., 2018; Hellendoorn et al., 2019).

Original: Nguyen & Nguyen (2015) propose an AST-based language model to support the detection and suggestion of a syntactic template at the current editing location.
Revised: Nguyen & Nguyen (2015) propose an AST-based language model to support the detection and suggestion of a syntactic template at the current editing location.

Original: Allamanis et al.
Revised: Allamanis et al.

Original: (2018) use graphs to represent programs and graph neural network to reason over program structures.
Revised: (2018) use graphs to represent programs and graph neural network to reason over program structures.

Original: Hellendoorn et al.
Revised: Hellendoorn et al.

Original: (2019) propose two different architectures using a gated graph neural network and Transformers for combining local and global information to leverage richly structured representations of source code.
Revised: (2019) propose two different architectures using a gated graph neural network and Transformers for combining local and global information to leverage richly structured representations of source code.

Original: However, these works leverage code structure to learn models on speciﬁc tasks from scratch without using pre-trained models.
Revised: However, these works leverage code structure to learn models on speciﬁc tasks from scratch without using pre-trained models.

Original: In this work, we study how to leverage code structure for pre-training code representation.
Revised: In this work, we study how to leverage code structure for pre-training code representation.

Original:  D ATA F LOW
Revised:  D ATA F LOW

Original: In this section, we describe the basic concept and extraction of data ﬂow.
Revised: In this section, we describe the basic concept and extraction of data ﬂow.

Original: In next section, we will describe how to use data ﬂow for pre-training.
Revised: In next section, we will describe how to use data ﬂow for pre-training.

Original: Data ﬂow is a graph that represents dependency relation between variables, in which nodes represent variables and edges represent where the value of each variable comes from.
Revised: Data ﬂow is a graph that represents dependency relation between variables, in which nodes represent variables and edges represent where the value of each variable comes from.

Original: Unlike AST, data ﬂow is same under different abstract grammars for the same source code.
Revised: Unlike AST, data ﬂow is same under different abstract grammars for the same source code.

Original: Such code structure provides crucial code semantic information for code understanding.
Revised: Such code structure provides crucial code semantic information for code understanding.

Original: Taking v = max value − min value as an example, programmers do not always follow the naming conventions so that it is hard to understand the semantic of the variable.
Revised: Taking v = max value − min value as an example, programmers do not always follow the naming conventions so that it is hard to understand the semantic of the variable.

Original: Data ﬂow provides a way to understand the semantic of the variable v to some extent, i.e. the value of v comes from max value and min value in data ﬂow.
Revised: Data ﬂow provides a way to understand the semantic of the variable v to some extent, i.e. the value of v comes from max value and min value in data ﬂow.

Original: Besides, data ﬂow supports the model to consider long-range dependencies induced by using the same variable or function in distant locations.
Revised: Besides, data ﬂow supports the model to consider long-range dependencies induced by using the same variable or function in distant locations.

Original: Taking Figure 1 as an example, there are four variables with same name (i.e. x 3 , x 7 , x 9 and x 11 ) but with different semantic.
Revised: Taking Figure 1 as an example, there are four variables with same name (i.e. x 3 , x 7 , x 9 and x 11 ) but with different semantic.

Original: The graph in the ﬁgure shows dependency relation between these variables and supports x 11 to pay more attention to x 7 and x 9 instead of x 3 .
Revised: The graph in the ﬁgure shows dependency relation between these variables and supports x 11 to pay more attention to x 7 and x 9 instead of x 3 .

Original: Next, we describe how to extract data ﬂow from a source code.
Revised: Next, we describe how to extract data ﬂow from a source code.

Original: [Figure] (a, b): [Figure]
Revised: def max(a, b): x= ifb>a:x=

Original: = if b>a: x=b else:x=a returnx
Revised: b else:x=a returnx

Original: Source code
Revised: Source code

Original: Parse into AST
Revised: Parse into AST

Original: [Figure]
Revised: Compiler Tool

Original: Identify variable sequence [Figure] b [Figure]
Revised: Identify variable sequence 16 79   def max(a, b): x= ifb>a:x=

Original: x [Figure] if b>a: x
Revised: b else:x=a returnx

Original: [Figure] else:x=a returnx [Figure] Variable
Revised: Identify

Original: [Figure]
Revised: [Figure]

Original: a
Revised: in AST

Original: [Figure] b  a [Figure] x a x x {+
+}
Revised: Variable relation a b xb a b x a x x {+
+}

Original: [Figure]
Revised: [Figure]

Original: comes-from” between variables.

Original: Figure 1 shows the extraction of data ﬂow through a source code.
Revised: Figure 1 shows the extraction of data ﬂow through a source code.

Original: Given a source code C = { c 1 , c 2 , ..., c n } , we ﬁrst parse the code into an abstract syntax tree (AST) by a standard compiler tool 1 .
Revised: Given a source code C = { c 1 , c 2 , ..., c n } , we ﬁrst parse the code into an abstract syntax tree (AST) by a standard compiler tool 2 .

Original: The AST includes syntax information of the code and terminals (leaves) are used to identify the variable sequence, denoted as V
Revised: The AST includes syntax information of the code and terminals (leaves) are used to identify the variable sequence, denoted as V

Original: = { v 1 , v 2 , ..., v k } .
Revised: = { v 1 , v 2 , ..., v k } .

Original: We take each variable as a node of the graph and an direct edge ε = (cid:104) v i , v j (cid:105) from v i to v j refers that the value of j -th variable comes from i -th variable.
Revised: We take each variable as a node of the graph and an direct edge ε = (cid:104) v i , v j (cid:105) from v i to v j refers that the value of j -th variable comes from i -th variable.

Original: Taking x = expr as an example, edges from all variables in expr to x are added into the graph.
Revised: Taking x = expr as an example, edges from all variables in expr to x are added into the graph.

Original: We denote the set of directed edges as E = {
Revised: We denote the set of directed edges as E = {

Original: ε 1 , ε 2 , ..., ε l } and the graph G ( C ) =
Revised: ε 1 , ε 2 , ..., ε l } and the graph G ( C ) =

Original: ( V, E ) is data ﬂow used to represent dependency relation between variables of the source code C .
Revised: ( V, E ) is data ﬂow used to represent dependency relation between variables of the source code C .

Original:  G RAPH C ODE BERT
Revised:  G RAPH C ODE BERT

Original: In this section, we describe GraphCodeBERT, a graph-based pre-trained model based on Transformer for programming language.
Revised: In this section, we describe GraphCodeBERT, a graph-based pre-trained model based on Transformer for programming language.

Original: We introduce model architecture, graph-guided masked attention and pre-training tasks including standard masked language model and newly introduced ones.
Revised: We introduce model architecture, graph-guided masked attention and pre-training tasks including standard masked language model and newly introduced ones.

Original: More details about model pre-training setting are provided in the Appendix A. 
Revised: More details about model pre-training setting are provided in the Appendix A. [Figure]

Revised: GraphCodeBERT

Revised: [Figure]

Original: 4. M ODEL A RCHITECTURE
Revised: 4. M ODEL A RCHITECTURE

Original: Figure 2 shows the model architecture of GraphCodeBERT.
Revised: Figure 2 shows the model architecture of GraphCodeBERT.

Original: We follow BERT (Devlin et al., 2018) and use the multi-layer bidirectional Transformer (Vaswani et al., 2017) as the model backbone.
Revised: We follow BERT (Devlin et al., 2018) and use the multi-layer bidirectional Transformer (Vaswani et al., 2017) as the model backbone.

Original: Instead of only using source code, we also utilize paired comments to pre-train the model to support more code-related tasks involving natural language such as natural language code search (Feng et al., 2020).
Revised: Instead of only using source code, we also utilize paired comments to pre-train the model to support more code-related tasks involving natural language such as natural language code search (Feng et al., 2020).

Original: We further take data ﬂow, which is a graph, as a part of the input to the model.
Revised: We further take data ﬂow, which is a graph, as a part of the input to the model.

Original: Given a source code C = { c 1 , c 2 , ..., c n } with its comment W = { w 1 , w 2 , ..., w m } , we can obtain the corresponding data ﬂow G ( C ) =
Revised: Given a source code C = { c 1 , c 2 , ..., c n } with its comment W = { w 1 , w 2 , ..., w m } , we can obtain the corresponding data ﬂow G ( C ) =

Original: ( V, E ) as discussed in the Section 3, where V = { v 1 , v 2 , ..., v k } is a set of variables and E = { ε 1 , ε 2 , ..., ε l } is a set of direct edges that represent where the value of each variable comes from.
Revised: ( V, E ) as discussed in the Section 3, where V = { v 1 , v 2 , ..., v k } is a set of variables and E = { ε 1 , ε 2 , ..., ε l } is a set of direct edges that represent where the value of each variable comes from.

Original: We concatenate the comment, source code and the set of variables as the Variable
Revised: We concatenate the comment, source code and the set of variables as the sequence input X

Original: [Figure]

Original: … x=0 if b> a : x=b else

Original: [Figure]

Original: Code a 1 b 2 x 3 0 4 b 5 a 6 x 7 b 8 x 9 a 10 x

Original: [Figure]

Original: Data Flow [CLS] …

Original: x=0 if b> [MASK] : x=b else … [SEP] Return [MASK] value [SEP] a 1 b 2 x 3 0 4 b 5 a 6 x 7 b 8 x 9 a 10 x… L L L
Revised: =

Original: GraphCodeBERT

Original: [Figure]

Original: a 14 5  data flow edge prediction among variables  11 3 x x x variable-alignment across source code and

Original: [Figure]

Original: [Figure] [Figure] sequence input X = { [ CLS ] , W, [ SEP ] , C, [ SEP ] , V } , where [ CLS ] is a special token in front of three segments and [ SEP ] is a special symbol to split two kinds of data types.
Revised: [ CLS ] , W, [ SEP ] , C, [ SEP ] , V } , where [ CLS ] is a special token in front of three segments and [ SEP ] is a special symbol to split two kinds of data types.

Original: GraphCodeBERT takes the sequence X as the input and then converts the sequence into input vectors H 0 .
Revised: GraphCodeBERT takes the sequence X as the input and then converts the sequence into input vectors H 0 .

Original: For each token, its input vector is constructed by summing the corresponding token and position embeddings.
Revised: For each token, its input vector is constructed by summing the corresponding token and position embeddings.

Original: We use a special position embedding for all variables to indicate that they are nodes of data ﬂow.
Revised: We use a special position embedding for all variables to indicate that they are nodes of data ﬂow.

Original: The model applies N transformer layers over the input vectors to produce contextual representations H n = transformer n ( H n − 1 ) , n ∈ [1 , N ] .
Revised: The model applies N transformer layers over the input vectors to produce contextual representations H n = transformer n ( H n − 1 ) , n ∈ [1 , N ] .

Original: Each transformer layer contains an architecturally identical transformerblock that applies a multi-headed self-attention operation (Vaswani et al., 2017) followed by a feed forward layer over the input H n − 1 in the n -th layer. 
Revised: Each transformer layer contains an architecturally identical transformer that applies a multi-headed self-attention operation (Vaswani et al., 2017) followed by a feed forward layer over the input H n − 1 in the n -th layer. [Equation]

Revised: where MultiAttn is a multi-headed self-attention mechanism, FFN is a two layers feed forward network, and LN represents a layer normalization operation.

Original: For the n -th transformer layer, the output H n of a multi-headed self-attention is computed via:
Revised: For the n -th transformer layer, the output ˆ G n of a multi-headed self-attention is computed via:

Original: [Equation]
Revised: [Equation]

Original: where the previous layer’s output H n − 1 ∈ R | X |× d h is linearly projected to a triplet of queries, keys and values using model parameters W Qi , W Ki , W Vi ∈ R d h × d k , respectively.
Revised: where the previous layer’s output H n − 1 ∈ R | X |× d h is linearly projected to a triplet of queries, keys and values using model parameters W Qi , W Ki , W Vi ∈ R d h × d k , respectively.

Original: u is the number of heads, d k is the dimension of a head, and W On ∈ R d
Revised: u is the number of heads, d k is the dimension of a head, and W On ∈ R d

Original: h × d h is the model parameters.
Revised: h × d h is the model parameters.

Original: M ∈ R | X |×| X | is a mask matrix, where M ij is 0 if i -th token is allowed to attend j -th token otherwise −∞ .
Revised: M ∈ R | X |×| X | is a mask matrix, where M ij is 0 if i -th token is allowed to attend j -th token otherwise −∞ .

Original: 4. G RAPH - GUIDED M ASKED A TTENTION
Revised: 4. G RAPH - GUIDED M ASKED A TTENTION

Original: To incorporate the graph structure into Transformer, we deﬁne a graph-guided masked attention function to ﬁlter out irrelevant signals.
Revised: To incorporate the graph structure into Transformer, we deﬁne a graph-guided masked attention function to ﬁlter out irrelevant signals.

Original: The attention masking function could avoid the key k i attended by the query q j by adding the attention score q Tj k i an inﬁnitely negative value so that the attention weight becomes zero after using a softmax function.
Revised: The attention masking function could avoid the key k i attended by the query q j by adding the attention score q Tj k i an inﬁnitely negative value so that the attention weight becomes zero after using a softmax function.

Original: To represent dependency relation between variables, a node-query q v i is allowed to attend to a node-key k v j if there is a direct edge from the node v j to the node v i (i.e. (cid:104) v j , v i (cid:105) ∈ E ) or they are the same node (i.e. i = j ).
Revised: To represent dependency relation between variables, a node-query q v i is allowed to attend to a node-key k v j if there is a direct edge from the node v j to the node v i (i.e. (cid:104) v j , v i (cid:105) ∈ E ) or they are the same node (i.e. i = j ).

Original: Otherwise, the attention is masked by adding an inﬁnitely negative value into the attention score.
Revised: Otherwise, the attention is masked by adding an inﬁnitely negative value into the attention score.

Original: To represent the relation between source code tokens and nodes of the data ﬂow, we ﬁrst deﬁne a set E (cid:48) , where (cid:104) v i , c j (cid:105) / (cid:104)
Revised: To represent the relation between source code tokens and nodes of the data ﬂow, we ﬁrst deﬁne a set E (cid:48) , where (cid:104) v i , c j (cid:105) / (cid:104)

Original: c j , v i (cid:105)
Revised: c j , v i (cid:105)

Original: ∈ E (cid:48)
Revised: ∈ E (cid:48)

Original: if the variable v i is identiﬁed from the source code token c j .
Revised: if the variable v i is identiﬁed from the source code token c j .

Original: We then allow the node q v i and code k c
Revised: We then allow the node q v i and code k c

Original: j attend each other if and only if (cid:104) v i ,
Revised: j attend each other if and only if (cid:104) v i ,

Original: c j (cid:105) / (cid:104)
Revised: c j (cid:105) / (cid:104)

Original: c j , v i (cid:105)
Revised: c j , v i (cid:105)

Original: ∈ E (cid:48) .
Revised: ∈ E (cid:48) .

Original: More formally, we use the following graph-guided masked attention matrix as the mask matrix M in the equation 2: [Equation] otherwise
Revised: More formally, we use the following graph-guided masked attention matrix as the mask matrix M in the equation 4:

Original: [Equation]
Revised: [Equation]

Original: 4. P RE - TRAINING T ASKS
Revised: 4. P RE - TRAINING T ASKS

Original: We describe three pre-training tasks used for pre-training GraphCodeBERT in this section.
Revised: We describe three pre-training tasks used for pre-training GraphCodeBERT in this section.

Original: The ﬁrst task is masked language modeling (Devlin et al., 2018) for learning representation from the source code.
Revised: The ﬁrst task is masked language modeling (Devlin et al., 2018) for learning representation from the source code.

Original: The second task is data ﬂow edge prediction for learning representation from data ﬂow, where we ﬁrst mask some variables’ data ﬂow edges and then let GraphCodeBERT predict those edges.
Revised: The second task is data ﬂow edge prediction for learning representation from data ﬂow, where we ﬁrst mask some variables’ data ﬂow edges and then let GraphCodeBERT predict those edges.

Original: The last task is variable-alignment across source code and data ﬂow for aligning representation between source code and data ﬂow, which predicts where a variable is identiﬁed from.
Revised: The last task is variable-alignment across source code and data ﬂow for aligning representation between source code and data ﬂow, which predicts where a variable is identiﬁed from.

Original: Masked Language Modeling We follow Devlin et al.
Revised: Masked Language Modeling We follow Devlin et al.

Original: (2018) to apply masked language modeling (MLM) pre-training task.
Revised: (2018) to apply masked language modeling (MLM) pre-training task.

Original: Specially, we sample randomly 15% of the tokens from the source code and paired comment.
Revised: Specially, we sample randomly 15% of the tokens from the source code and paired comment.

Original: We replace them with a [MASK] token 80% of the time, with a random token 10% of the time, and leave them unchanged 10% of the time.
Revised: We replace them with a [MASK] token 80% of the time, with a random token 10% of the time, and leave them unchanged 10% of the time.

Original: The MLM objective is to predict original tokens of these sampled tokens, which has proven effective in previous works (Devlin et al., 2018;
Revised: The MLM objective is to predict original tokens of these sampled tokens, which has proven effective in previous works (Devlin et al., 2018;

Original: Liu et al., 2019;
Revised: Liu et al., 2019;

Original: Feng et al., 2020).
Revised: Feng et al., 2020).

Original: In particular, the model can leverage the comment context if the source code context is not sufﬁcient to infer the masked code token, encouraging the model to align the natural language and programming language representations.
Revised: In particular, the model can leverage the comment context if the source code context is not sufﬁcient to infer the masked code token, encouraging the model to align the natural language and programming language representations.

Original: Edge Prediction To learn representation from data ﬂow, we introduce a pre-training task of data ﬂow edges prediction.
Revised: Edge Prediction To learn representation from data ﬂow, we introduce a pre-training task of data ﬂow edges prediction.

Original: The motivation is to encourage the model to learn structure-aware representation that encodes the relation of “where-the-value-comes-from” for better code understanding.
Revised: The motivation is to encourage the model to learn structure-aware representation that encodes the relation of “where-the-value-comes-from” for better code understanding.

Original: Specially, we randomly sample 20% of nodes V s in data ﬂow, mask direct edges connecting these sampled nodes by add an inﬁnitely negative value in the mask matrix, and then predict these masked edges E mask .
Revised: Specially, we randomly sample 20% of nodes V s in data ﬂow, mask direct edges connecting these sampled nodes by add an inﬁnitely negative value in the mask matrix, and then predict these masked edges E mask .

Original: Taking the variable x 11 in Figure 2 for an example, we ﬁrst mask edges (cid:104) x 7 , x 11 (cid:105) and (cid:104) x 9 , x 11 (cid:105) in the graph and then let the model to predict these edges.
Revised: Taking the variable x 11 in Figure 2 for an example, we ﬁrst mask edges (cid:104) x 7 , x 11 (cid:105) and (cid:104) x 9 , x 11 (cid:105) in the graph and then let the model to predict these edges.

Original: Formally, the pre-training objective of the task is calculated as Equation 5, where E c =
Revised: Formally, the pre-training objective of the task is calculated as Equation 7, where E c =

Original: V s × V ∪ V × V s is a set of candidates for edge prediction, δ ( e ij ∈ E ) is 1 if (cid:104) v i , v j
Revised: V s × V ∪ V × V s is a set of candidates for edge prediction, δ ( e ij ∈ E ) is 1 if (cid:104) v i , v j

Original: (cid:105) ∈
Revised: (cid:105) ∈

Original: E otherwise 0, and the probability p e ij of existing an edge from i -th to j -th node is calculated by dot product following a sigmoid function using representations of two nodes from GraphCodeBERT.
Revised: E otherwise 0, and the probability p e ij of existing an edge from i -th to j -th node is calculated by dot product following a sigmoid function using representations of two nodes from GraphCodeBERT.

Revised: To balance positive-negative ratio of examples, we sample negative and positive samples with the same number for E c .

Original: loss EdgePred
Revised: loss EdgePred

Original: [Equation]
Revised: [Equation]

Original: Node Alignment To align representation between source code and data ﬂow, we introduce a pretraining task of node alignment across source code and data ﬂow, which is similar to data ﬂow edge prediction.
Revised: Node Alignment To align representation between source code and data ﬂow, we introduce a pretraining task of node alignment across source code and data ﬂow, which is similar to data ﬂow edge prediction.

Original: Instead of predicting edges between nodes, we predict edges between code tokens and nodes.
Revised: Instead of predicting edges between nodes, we predict edges between code tokens and nodes.

Original: The motivation is to encourage the model to align variables and source code according to data ﬂow.
Revised: The motivation is to encourage the model to align variables and source code according to data ﬂow.

Original: Taking Figure 3 for an example, we ﬁrst mask edges between the variable x 11 in data ﬂow and code tokens, and then predict which code token the variable x 11 in data ﬂow is identiﬁed from.
Revised: Taking Figure 3 for an example, we ﬁrst mask edges between the variable x 11 in data ﬂow and code tokens, and then predict which code token the variable x 11 in data ﬂow is identiﬁed from.

Original: As we can see, the model could predict that the variable x 11 is identiﬁed form the variable x in the expression “return x” according to data ﬂow information (i.e. the value of x 11 comes from x 7 or x 9 ).
Revised: As we can see, the model could predict that the variable x 11 is identiﬁed form the variable x in the expression “return x” according to data ﬂow information (i.e. the value of x 11 comes from x 7 or x 9 ).

Original: GraphCodeBERT
Revised: GraphCodeBERT

Original: [Figure]
Revised: [Figure]

Original: Specially, we randomly sample 20% nodes V (cid:48) s in the graph, mask edges between code tokens and sampled nodes, and then predict masked edges E (cid:48) mask .
Revised: Specially, we randomly sample 20% nodes V (cid:48) s in the graph, mask edges between code tokens and sampled nodes, and then predict masked edges E (cid:48) mask .

Original: The pre-training objective of this task is similar to Equation 5, where E (cid:48) c
Revised: The pre-training objective of this task is similar to Equation 7, where E (cid:48) c

Original: = V (cid:48) s × C is a set of candidates for node alignment.
Revised: = V (cid:48) s × C is a set of candidates for node alignment.

Revised: Similarly, we also sample negative and positive samples with the same number for E (cid:48) c .

Original: loss
Revised: loss

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our model on four downstream tasks, including code search, clone detection, code translation and code reﬁnement.
Revised: We evaluate our model on four downstream tasks, including code search, clone detection, code translation and code reﬁnement.

Original: Detailed experimental settings can be found in the Appendix.
Revised: Detailed experimental settings can be found in the Appendix.

Original: 5. N ATURAL L ANGUAGE C ODE S EARCH
Revised: 5. N ATURAL L ANGUAGE C ODE S EARCH

Original: Given a natural language as  input, the task aims to ﬁnd the most semantically related code from a collection of candidate codes.
Revised: Given a natural language as the input, the task aims to ﬁnd the most semantically related code from a collection of candidate codes.

Original: We conduct experiments on the CodeSearchNet dataset (Husain et al., 2019), which includes six programming languages.
Revised: We conduct experiments on the CodeSearchNet code corpus (Husain et al., 2019), which includes six programming languages.

Revised: Different from the dataset and the setting used in the Husain et al.

Revised: (2019), we ﬁlter low-quality queries by handcrafted rules and expandcandidates to the whole code corpus, which is closer to the real-life scenario.

Revised: More details about the ﬁltered dataset are provided in the Appendix B.

Original: We use Mean Reciprocal Rank (MRR) as our evaluation metric and report results of existing methods in the Table 1.
Revised: We use Mean Reciprocal Rank (MRR) as our evaluation metric and report results of existing methods in the Table 1.

Original: [Table]
Revised: [Table]

Original: The ﬁrst group calculates inner product of code and query encodings as relevance scores to rank candidate codes. Husain et al. (2019) implement four methods to obtain the encodings, including bag-of-words, convolutional neural network, bidirectionalrecurrent neural network, and multi-head attention. The second group is the results of pre-trained models.
Revised: We al bag bidirectional

Original: Roberta (Liu et al., 2019) is a pretrained model on text corpus with MLM learning objective, while RoBERTa (code) is pre-trained only on code.
Revised: Roberta (Liu et al., 2019) is a pre-trained model on text corpus with MLM learning objective, while RoBERTa (code) is pre-trained only on code.

Original: CodeBERT (Feng et al., 2020) is pre-trained on code-text pairs with MLM and replaced token detection learning objectives.
Revised: CodeBERT (Feng et al., 2020) is pre-trained on code-text pairs with MLM and replaced token detection learning objectives.

Original: As we can see, GraphCodeBERT that leverages code structure for pre-training brings a 1.4% gain of MRR, achieving the state-of-art performance.
Revised: As we can see, GraphCodeBERT that leverages code structure for pre-training brings a 2% gain of MRR, achieving the state-of-art performance.

Original: We also conducted t-test between our GraphCodeBERT and other baselines [Table] and the results show the improvementson 5 programming languages are signiﬁcant with p < 0 .
Revised: We also conducted t-test between our GraphCodeBERT and other baselines, and the results show the improvements are signiﬁcant with p < 0 .

Original: 5. C ODE C LONE D ETECTION
Revised: 5. C ODE C LONE D ETECTION

Original: [Table]

Original: < Code clones are multiple code fragments that output similar results when given the same input.
Revised: Code clones are multiple code fragments that output similar results when given the same input.

Original: The task aims to measure the similarity between two code fragments, which can help reduce the cost of software maintenance and prevent bugs.
Revised: The task aims to measure the similarity between two code fragments, which can help reduce the cost of software maintenance and prevent bugs.

Original: We conduct experiments on the BigCloneBench dataset (Svajlenko et al., 2014) and report results in the Table 2.
Revised: We conduct experiments on the BigCloneBench dataset (Svajlenko et al., 2014) and report results in the Table 2.

Original: Deckard (Jiang et al., 2007) is to compute vectors for structural information within ASTs and then a Locality Sensitive Hashing (LSH) (Datar et al., 2004) is used to cluster similar vectors for detection.
Revised: Deckard (Jiang et al., 2007) is to compute vectors for structural information within ASTs and then a Locality Sensitive Hashing (LSH) (Datar et al., 2004) is used to cluster similar vectors for detection.

Original: RtvNN (White et al., 2016) trains a recursive autoencoder to learn representations for AST.
Revised: RtvNN (White et al., 2016) trains a recursive autoencoder to learn representations for AST.

Original: CDLH (Wei & Li, 2017) learn representations of code fragments via ASTbased LSTM and hamming distance is used to optimize the distance between the vector representation of AST pairs. 
Revised: CDLH (Wei & Li, 2017) learn representations of code fragments via AST-based LSTM and hamming distance is used to optimize the distance between the vector representation of AST pairs. [Table]

Original: ASTNN Zhang et al.
Revised: ASTNN Zhang et al.

Original: (2019) uses RNNs to encode AST subtrees for statements, then feed the encodings of all statement trees into an RNN to learn representation for a program.
Revised: (2019) uses RNNs to encode AST subtrees for statements, then feed the encodings of all statement trees into an RNN to learn representation for a program.

Original: FAAST-GMN (Wang et al., 2020) uses GNNs over a ﬂow-augmented AST to leverages explicit control and data ﬂow information for code clone detection.
Revised: FA-AST-GMN (Wang et al., 2020) uses GNNs over a ﬂow-augmented AST to leverages explicit control and data ﬂow information for code clone detection.

Original: Results show that our GraphCodeBERT that leverages code structure information signiﬁcantly outperforms other methods with p < 0 . 01 , which demonstrates the effectiveness of our pre-trained model for the task of code clone detection.
Revised: Results show that our GraphCodeBERT that leverages code structure information signiﬁcantly outperforms other methods with p < 0 . 01 , which demonstrates the effectiveness of our pre-trained model for the task of code clone detection.

Original: 5. C ODE T RANSLATION
Revised: 5. C ODE T RANSLATION

Original: Code translation aims to migrate legacy software from one programming language in a platform to another.
Revised: Code translation aims to migrate legacy software from one programming language in a platform to another.

Original: Following Nguyen et al.
Revised: Following Nguyen et al.

Original: and
Revised: and

Original: Chen et al.
Revised: Chen et al.

Original: , we conduct experiments on a dataset crawled from the same several open-source projects as them and report results in the Table 3.
Revised: , we conduct experiments on a dataset crawled from the same several open-source projects as them and report results in the Table 3.

Original: The Naive method is directly copying the source code as the translation result.
Revised: The Naive method is directly copying the source code as the translation result.

Original: PBSMT is short for phrase-based statistical machine translation (Koehn et al., 2003), and has been exploited in previous works (Nguyen et al., 2013; Karaivanov et al., 2014).
Revised: PBSMT is short for phrase-based statistical machine translation (Koehn et al., 2003), and has been exploited in previous works (Nguyen et al., 2013; Karaivanov et al., 2014).

Original: As for the Transformer , we use the
Revised: As for the Transformer , we use the

Original: [Table]
Revised: [Table]

Original: outperforms other

Original: [Table]

Original: signiﬁcantly

Original: [Table]

Original: same number of layers and hidden size as pre-trained models.
Revised: same number of layers and hidden size as pre-trained models.

Original: To leverage the pretrained models for translation, we initialize the encoder with pre-trained models and randomly initialize parameters of the decoder and the source-to-target attention.
Revised: To leverage the pretrained models for translation, we initialize the encoder with pre-trained models and randomly initialize parameters of the decoder and the source-to-target attention.

Original: Results show that the models initialized with pretrained models (i.e the second group) signiﬁcantly outperform PBSMT and Transformer models.
Revised: Results show that the models initialized with pretrained models (i.e the second group) signiﬁcantly outperform PBSMT and Transformer models.

Original: Among them, GraphCodeBERT achieves state-of-art performance, which demonstrates the effectiveness of our model for code translation.
Revised: Among them, GraphCodeBERT achieves state-of-art performance, which demonstrates the effectiveness of our model for code translation.

Original: 5. C ODE R EFINEMENT
Revised: 5. C ODE R EFINEMENT

Original: Code reﬁnement aims to automatically ﬁx bugs in the code, which can contribute to reducing the cost of bug-ﬁxes.
Revised: Code reﬁnement aims to automatically ﬁx bugs in the code, which can contribute to reducing the cost of bug-ﬁxes.

Original: We use the dataset released by Tufano et al.
Revised: We use the dataset released by Tufano et al.

Original: and report results in the Table 4.
Revised: and report results in the Table 4.

Original: The Naive method directly copies the buggy code as the reﬁnement result.
Revised: The Naive method directly copies the buggy code as the reﬁnement result.

Original: For the Transformer , we use the same number of layers and hidden size as the pre-trained models.
Revised: For the Transformer , we use the same number of layers and hidden size as the pre-trained models.

Original: Same as the Section 5.3, we initialize the encoder with pre-trained models and randomly initialize parameters of the decoder
Revised: Same as the Section 5.3, we initialize the encoder with pre-trained models and randomly initialize parameters of the decoder

Original: [Table]
Revised: [Table]

Original: outperforms other models signiﬁcantly ( p < 0 . 01 ).

Original: and the source-to-target attention.
Revised: and the source-to-target attention.

Original: Then we use the training data to ﬁne-tune the whole model.
Revised: Then we use the training data to ﬁne-tune the whole model.

Original: In the table, we see that the Transformer signiﬁcantly outperforms LSTM .
Revised: In the table, we see that the Transformer signiﬁcantly outperforms LSTM .

Original: Results in the second group shows that pre-trained models outperform Transformer models further, and
Revised: Results in the second group shows that pre-trained models outperform Transformer models further, and

Original: GraphCodeBERT achieves better performance than other pre-trained models on both datasets, which shows leveraging code structure information are helpful to the task of code reﬁnement.
Revised: GraphCodeBERT achieves better performance than other pre-trained models on both datasets, which shows leveraging code structure information are helpful to the task of code reﬁnement.

Original: 5. M ODEL A NALYSIS Ablation Study
Revised: 5. M ODEL A NALYSIS Ablation Study

Original: We conduct ablation study on the task of natural language code search to understand various components in our approach impact overall performance.
Revised: We conduct ablation study on the task of natural language code search to understand various components in our approach impact overall performance.

Original: We remove two pre-training tasks and data ﬂow, respectively, to analyze their contribution.
Revised: We remove two pre-training tasks and data ﬂow, respectively, to analyze their contribution.

Original: Table 5 shows that the overall performance drops from 77.4% to 76.6% ∼ 76.8% when removing Node Alignment and Edge Prediction pre-training tasks, respectively, which reveals the importance of two structure-aware pre-training tasks.
Revised: Table 5 shows that the overall performance drops from 71.3% to 70.3% ∼ 70.7% when removing Node Alignment and Edge Prediction pre-training tasks, respectively, which reveals the importance of two structure-aware pre-training tasks.

Original: After ablating the data ﬂow totally, we can see that the performance drops from 77.4% to 76.0%, which means leveraging data ﬂow to learn code representation could improve GraphCodeBERT.
Revised: After ablating the data ﬂow totally, we can see that the performance drops from 71.3% to 69.3%, which means leveraging data ﬂow to learn code representation could improve GraphCodeBERT.

Original: [Table]
Revised: [Table]

Original: Node-vs. Token-level Attention Table 6 shows how frequently a special token [ CLS ] that is used to calculate probability of correct candidate attends to code tokens (Codes) and variables (Nodes).
Revised: Node-vs. Token-level Attention Table 6 shows how frequently a special token [ CLS ] that is used to calculate probability of correct candidate attends to code tokens (Codes) and variables (Nodes) [Table]

Original: We see that although the number of nodes account for 5% ∼ 20%, attentions over nodes overwhelm node/code ratio (around 10% to 32%) across all programming languages.
Revised: We see that although the number of nodes account for 5% ∼ 20%, attentions over nodes overwhelm node/code ratio (around 10% to 32%) across all programming languages.

Original: The results indicate that data ﬂow plays an important role in code understanding process and the model pays more attention to nodes in data ﬂow than code tokens
Revised: The results indicate that data ﬂow plays an important role in code understanding process and the model pays more attention to nodes in data ﬂow than code tokens

Original: [Table]
Revised: [Table]

Original: Comparison between AST and Data Flow Figure 4 shows MRR score with respect to input sequence length on the validation dataset of Ruby programming language for the task of code search.
Revised: Comparison between AST and Data Flow Figure 4 shows MRR score with respect to input sequence length on the validation dataset of Ruby programming language for the task of code search.

Original: AST Pre-order Traversal regards AST as a sequence by linearizing all AST nodes using pre-order traversal algorithm.
Revised: AST Pre-order Traversal regards AST as a sequence by linearizing all AST nodes using pre-order traversal algorithm.

Original: AST Subtree Masking regards AST as a tree and introduce subtree masking (Nguyen et al., 2019) for self-attention of the Transformer.
Revised: AST Subtree Masking regards AST as a tree and introduce subtree masking (Nguyen et al., 2019) for self-attention of the Transformer.

Original: In subtree masking, each 
Revised: In subtree masking, each node-query in AST attends only to its own subtree descendants, and each leaf-query only attends to leaves of AST. Transformer has a self-attention component with O ( n 2 ) time and memory complexity where n is the input sequence length, and thus is not efﬁcient to scale to long inputs.

Original: [Figure]
Revised: [Figure]

Original: leaves of AST.

Original: node-query in AST attends only to its own subtree descendants, and each leaf-query only attends to Transformer has a self-attention component with O ( n 2 ) time and memory complexity where n is the input sequence length, and thus is not efﬁcient to scale to long inputs.

Original: We observe that injecting AST even hurts the performance when the sequence length is short (e.g. shorter than 128), while GraphCodeBERT consistently brings performance boost on varying sequence length and obtains better MRR score than ASTbased methods.
Revised: We observe that injecting AST even hurts the performance when the sequence length is short (e.g. shorter than 128), while GraphCodeBERT consistently brings performance boost on varying sequence length and obtains better MRR score than AST-based methods.

Original: The main reason is that data ﬂow is neat and the number of nodes account for 5% ∼ 20% (see Table 6), which does not bring an unnecessarily deep hierarchy of AST and makes the model more accurate and efﬁcient.
Revised: The main reason is that data ﬂow is less complex and the number of nodes account for 5% ∼ 20% (see Table 6), which does not bring an unnecessarily deep hierarchy of AST and makes the model more accurate and efﬁcient.

Original: Case Study We also give a case study to demonstrate that data ﬂow would enhance the code understanding process.
Revised: Case Study We also give a case study to demonstrate that data ﬂow would enhance the code understanding process.

Original: Given a source code and a comment, we use GraphCodeBERT with and without data ﬂow to predict whether the comment correctly describes the source code.
Revised: Given a source code and a comment, we use GraphCodeBERT with and without data ﬂow to predict whether the comment correctly describes the source code.

Original: Results are given in Figure 5.
Revised: Results are given in Figure 5.

Original: We can see that both models make correct prediction in the original example, where the threshold is 0.5 (left panel).
Revised: We can see that both models make correct prediction in the original example, where the threshold is 0.5 (left panel).

Original: To study the code understanding ability of models, we change the source code (center panel) and the comment (right panel), respectively.
Revised: To study the code understanding ability of models, we change the source code (center panel) and the comment (right panel), respectively.

Original: Although we make a small change on the source code ( return a → return b ) and the comment ( sum value → mean value ), the semantic of the source code and the comment are completely different and corresponding gold labels change from 1 to 0.
Revised: Although we make a small change on the source code ( return a → return b ) and the comment ( sum value → mean value ), the semantic of the source code and the comment are completely different and corresponding gold labels change from 1 to 0.

Original: As we can see in the ﬁgure, GraphCodeBERT without using data ﬂow fails these tests and still outputs high probability for negative examples.
Revised: As we can see in the ﬁgure, GraphCodeBERT without using data ﬂow fails these tests and still outputs high probability for negative examples.

Original: After leveraging data ﬂow,
Revised: After leveraging data ﬂow,

Original: GraphCodeBERT better understands the semantic of source code and makes correct predictions on all tests, which demonstrates that data ﬂow could improve the code understanding ability of the model.
Revised: GraphCodeBERT better understands the semantic of source code and makes correct predictions on all tests, which demonstrates that data ﬂow could improve the code understanding ability of the model.

Original: Unchanged
Revised: Unchanged

Original: Code: 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 𝑎𝑎 → 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 𝑏𝑏
Revised: Code: 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 𝑎𝑎 → 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 𝑏𝑏

Original: NL: 𝑠𝑠𝑟𝑟𝑠𝑠 𝑣𝑣𝑎𝑎𝑣𝑣𝑟𝑟𝑟𝑟 → 𝑠𝑠𝑟𝑟𝑎𝑎𝑟𝑟 𝑣𝑣𝑎𝑎𝑣𝑣𝑟𝑟𝑟𝑟
Revised: NL: 𝑠𝑠𝑟𝑟𝑠𝑠 𝑣𝑣𝑎𝑎𝑣𝑣𝑟𝑟𝑟𝑟 → 𝑠𝑠𝑟𝑟𝑎𝑎𝑟𝑟 𝑣𝑣𝑎𝑎𝑣𝑣𝑟𝑟𝑟𝑟

Original: Input
Revised: Input

Original: Label
Revised: Label

Original: Prediction
Revised: Prediction

Original: GraphCodeBERT: 0.6563 (1)
Revised: GraphCodeBERT: 0.6563 (1)

Original: GraphCodeBERT: 0.8728 (1) (w/o Data Flow)
Revised: GraphCodeBERT: 0.8728 (1) (w/o Data Flow)

Original: GraphCodeBERT: 0.4615 (0)
Revised: GraphCodeBERT: 0.4615 (0)

Original: GraphCodeBERT: 0.8608 (1) (w/o Data Flow)
Revised: GraphCodeBERT: 0.8608 (1) (w/o Data Flow)

Original: GraphCodeBERT: 0.2884 (0)
Revised: GraphCodeBERT: 0.2884 (0)

Original: GraphCodeBERT: 0.9048 (1) (w/o Data Flow)
Revised: GraphCodeBERT: 0.9048 (1) (w/o Data Flow)

Original: NL: Return sum value of an array
Revised: NL: Return sum value of an array

Original: Code: import numpy as np def f(array): a=np.sum(array) b=np.mean(array) return a
Revised: Code: import numpy as np def f(array): a=np.sum(array) b=np.mean(array) return a

Original: NL: Return sum value of an array
Revised: NL: Return sum value of an array

Original: Code: import numpy as np def f(array): a=np.sum(array) b=np.mean(array) return b
Revised: Code: import numpy as np def f(array): a=np.sum(array) b=np.mean(array) return b

Original: NL: Return mean value of an array
Revised: NL: Return mean value of an array

Original: Code: import numpy as np def f(array): a=np.sum(array) b=np.mean(array) return a
Revised: Code: import numpy as np def f(array): a=np.sum(array) b=np.mean(array) return a

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we present GraphCodeBERT that leverages data ﬂow to learn code representation.
Revised: In this paper, we present GraphCodeBERT that leverages data ﬂow to learn code representation.

Original: To the best of our knowledge this is the ﬁrst pre-trained model that considers code structure for pre-training code representations.
Revised: To the best of our knowledge, this is the ﬁrst pre-trained model that considers code structure for pre-training code representations.

Original: We introduce two structure-aware pre-training tasks and show that GraphCodeBERT achieves state-of-the-art performance on four code-related downstream tasks, including code search, clone detection, code translation and code reﬁnement.
Revised: We introduce two structure-aware pre-training tasks and show that GraphCodeBERT achieves state-of-the-art performance on four code-related downstream tasks, including code search, clone detection, code translation and code reﬁnement.

Original: Further analysis shows that code structure and newly introduced pre-training tasks boost the performance.
Revised: Further analysis shows that code structure and newly introduced pre-training tasks boost the performance.

Original: Additionally, case study in the task of code search shows that applying data ﬂow in the pre-trained model improves code understanding.
Revised: Additionally, case study in the task of code search shows that applying data ﬂow in the pre-trained model improves code understanding.

Revised: A CKNOWLEDGMENTS

Revised: Daya Guo and Jian Yin are supported by the Research Foundation of Science and Technology Plan

Revised: Project in Guangdong Province (2017B030308007)

Original: A P RE - TRAINING D ETAILS
Revised: A P RE - TRAINING D ETAILS

Original: GraphCodeBERT includes 12 layers Transformer with 768 dimensional hidden states and 12 attention heads.
Revised: GraphCodeBERT includes 12 layers Transformer with 768 dimensional hidden states and 12 attention heads.

Original: For fair comparison, we use the same dataset as CodeBERT (Feng et al., 2020) to pretrain our model.
Revised: For fair comparison, we use the same dataset as CodeBERT (Feng et al., 2020) to pretrain our model.

Original: The dataset is the CodeSearchNet dataset [-
-] (Husain et al., 2019), which includes 2.4M functions with document pairs for six programming languages.
Revised: The dataset is the CodeSearchNet dataset [-
-]3 (Husain et al., 2019), which includes 2.3M functions with document pairs for six programming languages.

Original: We train the model on two DGXmachines, each having 16 NVIDIA Tesla V100 with 32GB memory.
Revised: We train the model on two DGXmachines, each having 16 NVIDIA Tesla V100 with 32GB memory.

Original: We set the max length of sequences and nodes as 512 and 128, respectively.
Revised: We set the max length of sequences and nodes as 512 and 128, respectively.

Original: We use the Adam optimizer to update model parameters with 1,024 batch size and 2e-4 learning rate.
Revised: We use the Adam optimizer to update model parameters with 1,024 batch size and 2e-4 learning rate.

Original: To accelerate the training process, we adopt the parameters of CodeBERT released by Feng et al.
Revised: To accelerate the training process, we adopt the parameters of CodeBERT released by Feng et al.

Original: (2020) to initialize the model.
Revised: (2020) to initialize the model.

Original: The model is trained with 200K batches and costs about 83 hours.
Revised: The model is trained with 200K batches and costs about 83 hours.

Original: At each iteration, we alternate EdgePred and NodeAlign objectives in combination with MLM to pre-train the model.
Revised: At each iteration, we alternate EdgePred and NodeAlign objectives in combination with MLM to pre-train the model.

Original: And we follow Lample & Conneau (2019) to sample each batch from the same programming language according to a multinomial distribution with probabilities { q i } i =1 ...N , where n i is number of examples for i -th programming language and α =0.7.
Revised: And we follow Lample & Conneau (2019) to sample each batch from the same programming language according to a multinomial distribution with probabilities { q i } i =1 ...N , where n i is number of examples for i -th programming language and α =0.7.

Original: Sampling with this distribution could alleviates the bias towards high-resource languages.
Revised: Sampling with this distribution could alleviates the bias towards high-resource languages.

Original: [Equation]
Revised: [Equation]

Original: B N ATURAL L ANGUAGE C ODE S EARCH
Revised: B N ATURAL L ANGUAGE C ODE S EARCH

Original: Given a natural language as  input, code search aims to ﬁnd the most semantically related code from a collection of candidate codes.
Revised: Given a natural language as the input, code search aims to ﬁnd the most semantically related code from a collection of candidate codes.

Original: We conduct experiments on the CodeSearchNet dataset (Husain et al., 2019), which includes six programming languages.
Revised: We conduct experiments on the CodeSearchNet code corpus (Husain et al., 2019) and follow Husain et al.

Original: We list data statistics about the CodeSearchNet corpus in Table 7.
Revised: (2019) to take the ﬁrst paragraph of the documentation as the query for the corresponding function. However, we observe that some queries contain content unrelated to the code, such as a link “http://...” that refers to external resources. Therefore, we ﬁlter following examples to improve the quality of the dataset. (1) Examples whose code could not be parsed into abstract syntax tree.

Original: For each query in the test data, the answer is retrieved from a set of 1,000 candidate codes.
Revised: (2) Examples whose query tokens number is shorter than 3 or larger than 256.

Original: We treat the task as a binary classiﬁcation to ﬁne-tune language-speciﬁc models for each programming language, where we use query, source code and data ﬂow as the input.
Revised: (3) Examples whose query contains special tokens such as “http://”. (4) Examples whose query is empty or not written in English.

Original: The probability of correct candidate is calculated by adding a linear layer followed by a sigmoid function on the representation of the special token [ CLS ] . In the ﬁne-turning step, we set the learning rate as 5e-5, the batch size as 128, the max sequence length as 256 and the max number of nodes as 64.
Revised: Different from the setting of Husain et al.

Original: We use the Adam optimizer to update model parameters and perform early stopping on the  development set.
Revised: (2019), the answer of each query is retrieved from the whole development and testing code corpus instead of 1,000 candidate codes. We list data statistics about the ﬁltered dataset in Table 7.

Original: [Table]
Revised: [Table]

Original: We give two cases of the GraphCodeBERT output for this task in Figure 6. In the ﬁrst example, the model successfully ﬁnds Python source code that correctly matches the sementic of the query “Scans through a string for substrings matched some patterns”.
Revised: We use GraphCodeBERT to separately encode query and source code with data ﬂow, and calculate inner product of their representations of the special token [ CLS ] as relevance scores to rank candidate codes.

Original: The source code ﬁnds all substrings by calling re.findall () build-in fucntion. In the second case, the query is “Combing the individual byte arrays into one array”, and  the  model searches a source code from Java candidate codes. As we can see, the source code concatenates multiple arrays into one array by calling System.arraycopy () build-in fucntion.
Revised: In the ﬁne-turning step, we set the learning rate as 2e-5, the batch size as 32, the max sequence length of queries and codes as 128 and 256, and the max number of nodes as 64. We use the Adam optimizer to update model parameters and perform early stopping on the development set.

Original: C C ODE C LONE D ETECTION
Revised: C C ODE C LONE D ETECTION

Original: Code clone detection aims to measure the similarity between two code fragments.
Revised: Code clone detection aims to measure the similarity between two code fragments.

Original: We use BigCloneBench dataset (Svajlenko et al., 2014), which contains over 6,000,000 true clone pairs and[Figure] 260,000 false clone pairs from 10 different functionalities.
Revised: We use BigCloneBench dataset (Svajlenko et al., 2014), which contains over 6,000,000 true clone pairs and 260,000 false clone pairs from 10 different functionalities.

Original: We follow the settings in Wei & Li (2017), discarding code fragments without any tagged true and false clone pairs and using 9,134 remaining code fragments.
Revised: We follow the settings in Wei & Li (2017), discarding code fragments without any tagged true and false clone pairs and using 9,134 remaining code fragments.

Original: Finally, the dataset provided by Wang et al.
Revised: Finally, the dataset provided by Wang et al.

Original: (2020) includes 901,724/416,328/416, examples for training/validation/testing.
Revised: (2020) includes 901,724/416,328/416, examples for training/validation/testing.

Original: We treat the task as a binary classiﬁcation to ﬁne-tune GraphCodeBERT, where we use source code and data ﬂow as the input.
Revised: We treat the task as a binary classiﬁcation to ﬁne-tune GraphCodeBERT, where we use source code and data ﬂow as the input.

Original: The probability of true clone is calculated by dot product from the representation of [ CLS ] .
Revised: The probability of true clone is calculated by dot product from the representation of [ CLS ] .

Original: In the ﬁne-turning step, we set the learning rate as 5e-5, the batch size as 128, the max sequence length as 256 the max number of nodes as 64.
Revised: In the ﬁne-turning step, we set the learning rate as 2e-5, the batch size as 16, the max sequence length as 512 the max number of nodes as 128.

Original: We use the Adam optimizer to update model parameters and tune hyper-parameters and perform early stopping on the development set.
Revised: We use the Adam optimizer to update model parameters and tune hyper-parameters and perform early stopping on the development set.

Original: We give a case of the GraphCodeBERT output for this task in Figure 7.
Revised: We give a case of the GraphCodeBERT output for this task in Figure 6.

Original: In this example, two Java source codes both download content from a given URL and convert the type of the content into string type.
Revised: In this example, two Java source codes both download content from a given URL and convert the type of the content into string type.

Original: Therefore, two codes are semantically similar since they output similar results when given the same input.
Revised: Therefore, two codes are semantically similar since they output similar results when given the same input.

Original: As we can see, our model gives a high score for this case and the pair is classiﬁed as true clone pair.[Figure]
Revised: As we can see, our model gives a high score for this case and the pair is classiﬁed as true clone pair.

Original: D C ODE T RANSLATION
Revised: D C ODE T RANSLATION

Original: Code translation aims to migrate legacy software from one programming language in a platform to another.
Revised: Code translation aims to migrate legacy software from one programming language in a platform to another.

Original: We conduct experiments on a dataset crawled from the same several open-source projects as
Revised: We conduct experiments on a dataset crawled from the same several open-source projects as

Original: Nguyen et al.
Revised: Nguyen et al.

Original: and
Revised: and

Original: Chen et al.
Revised: Chen et al.

Original: (2018), i.e. Lucene 3 , POI 4 , JGit 5 and Antlr 6 .
Revised: (2018), i.e. Lucene 4 , POI 5 , JGit 6 and Antlr 7 .

Original: We do not use
Revised: We do not use

Original: Itext 7 and JTS 8 as they do because of the license problem.
Revised: Itext 8 and JTS 9 as they do because of the license problem.

Original: Those projects have both Java and C# implementation.
Revised: Those projects have both Java and C# implementation.

Original: We pair the methods in the two languages based on their ﬁle names and method names.
Revised: We pair the methods in the two languages based on their ﬁle names and method names.

Original: After removing duplication and methods with null function body, the total number of method pairs is 11,800, and we split 500 pairs from them as the development set and another 1,000 pairs for  test.
Revised: After removing duplication and methods with null function body, the total number of method pairs is 11,800, and we split 500 pairs from them as the development set and another 1,000 pairs for [Figure] test.

Original: To demonstrate the effectiveness of GraphCodeBERT on the task of code translation, we adopt various pre-trained models as encoders and stay hyperparameters consistent.
Revised: To demonstrate the effectiveness of GraphCodeBERT on the task of code translation, we adopt various pre-trained models as encoders and stay hyperparameters consistent.

Original: We set the learning rate as 1e-4, the batch size as 64, the max sequence length as 512 and the max number of nodes as 128.
Revised: We set the learning rate as 1e-4, the batch size as 32, the max sequence length as 256 and the max number of nodes as 64.

Original: We use the Adam optimizer to update model parameters and tune hyper-parameters and perform early stopping on the development set.
Revised: We use the Adam optimizer to update model parameters and tune hyper-parameters and perform early stopping on the development set.

Original: We give a case of the GraphCodeBERT output for this task in Figure 8.
Revised: We give a case of the GraphCodeBERT output for this task in Figure 7.

Original: In this example, the model successfully translates a piece of Java code into its C# version.
Revised: In this example, the model successfully translates a piece of Java code into its C# version.

Original: The differences include the type name (from “boolean” to “bool”) and the usage of getting a string value of a bool variable (from “String.valueOf(b)” to “b.ToString()”).
Revised: The differences include the type name (from “boolean” to “bool”) and the usage of getting a string value of a bool variable (from “String.valueOf(b)” to “b.ToString()”).

Original: E C ODE R EFINEMENT
Revised: E C ODE R EFINEMENT

Original: Code reﬁnement aims to automatically ﬁx bugs in the code.
Revised: Code reﬁnement aims to automatically ﬁx bugs in the code.

Original: We use the dataset released by Tufano et al.
Revised: We use the dataset released by Tufano et al.

Original: The source is buggy Java functions while the target is the according ﬁxed ones.
Revised: The source is buggy Java functions while the target is the according ﬁxed ones.

Original: Almost all the names of variables and custom methods are normalized.
Revised: Almost all the names of variables and custom methods are normalized.

Original: The dataset contains two subsets based on the code length.
Revised: The dataset contains two subsets based on the code length.

Original: For the small dataset, the numbers of training, development and test samples are 46,680, 5,835 and 5,835.
Revised: For the small dataset, the numbers of training, development and test samples are 46,680, 5,835 and 5,835.

Original: For the medium dataset, the numbers are 52,364, 6,545 and 6,545.
Revised: For the medium dataset, the numbers are 52,364, 6,545 and 6,545.

Original: We also use the sequence-to-sequence Transformer model to conduct the experiments.
Revised: We also use the sequence-to-sequence Transformer model to conduct the experiments.

Original: In the ﬁne-tuning step, we adopt various pre-trained models as encoders.
Revised: In the ﬁne-tuning step, we adopt various pre-trained models as encoders.

Original: We set the learning rate as 1e-4, the batch size as 64, the max sequence length as 256 and the max number of nodes as 64.
Revised: We set the learning rate as 1e-4, the batch size as 32, the max sequence length as 256 and the max number of nodes as 64.

Original: We use the
Revised: We use the

Original: Adam optimizer to update model parameters and perform early stopping on the development set.
Revised: Adam optimizer to update model parameters and perform early stopping on the development set.

Original: We give two cases of the GraphCodeBERT output for this task in Figure 9.
Revised: We give two cases of the GraphCodeBERT output for this task in Figure 8.

Original: In the ﬁrst example, the model successfully ﬁxes the operation bug (from “*” to “+”) to match the function name “add”.
Revised: In the ﬁrst example, the model successfully ﬁxes the operation bug (from “*” to “+”) to match the function name “add”.

Original: In the second case, the source function and type names are normalized.
Revised: In the second case, the source function and type names are normalized.

Original: The return type of this function is “void” but the buggy code gives a return value.
Revised: The return type of this function is “void” but the buggy code gives a return value.

Original: Our model successfully removes the “return” word so that the return type of the function matches its declaration.[Equation] C
Revised: Our model successfully removes the “return” word so that the return type of the function matches its declaration.

Original: [Equation]
Revised: [Equation]

Original: S TUDY F. N ATURAL L ANGUAGE C ODE S EARCH
Revised: F. N ATURAL L ANGUAGE C ODE S EARCH

Original: We give a case study to illustrate retrieved results by GraphCodeBERT on the natural language code search task, with a comparison to CodeBERT and RoBERTa (code) models.
Revised: We give a case study to illustrate retrieved results by GraphCodeBERT on the natural language code search task, with a comparison to CodeBERT and RoBERTa (code) models.

Original: Two examples are given in Figure 10 and we can see that GraphCodeBERT successfully retrieves correct source codes for given queries on both examples.
Revised: Two examples are given in Figure 9 and we can see that GraphCodeBERT successfully retrieves correct source codes for given queries on both examples.

Original: As we can see in the ﬁrst case, incorporating data ﬂow will help
Revised: As we can see in the ﬁrst case, incorporating data ﬂow will help

Original: Graph-CodeBERT better understand the complicated expression “[(k, v) for k, v in self.items() if v is not self.
Revised: Graph-CodeBERT better understand the complicated expression “[(k, v) for k, v in self.items() if v is not self.

Original: EMPTY]” by leveraging dependency relation among variables in data ﬂow graph.
Revised: EMPTY]” by leveraging dependency relation among variables in data ﬂow graph.

Original: In the second case, the terminology “%Y-%m-%d” in Python program language is a format of date time.
Revised: In the second case, the terminology “%Y-%m-%d” in Python program language is a format of date time.

Original: GraphCodeBERT and CodeBERT both successfully search the correct function.
Revised: GraphCodeBERT and CodeBERT both successfully search the correct function.

Original: Compared with
Revised: Compared with

Original: RoBERTa (code), the second case shows that utilizing natural language descriptions for pre-training helps models do better semantic matching between source codes and queries on the code search task
Revised: RoBERTa (code), the second case shows that utilizing natural language descriptions for pre-training helps models do better semantic matching between source codes and queries on the code search task

Original: [Figure]
Revised: [Figure]

Original: F. C ODE C LONE D ETECTION
Revised: F. C ODE C LONE D ETECTION

Original: We give a case study to compare GraphCodeBERT with CodeBERT and RoBERTa (code) models on code clone detection task.
Revised: We give a case study to compare GraphCodeBERT with CodeBERT and RoBERTa (code) models on code clone detection task.

Original: An example is shown in Figure 11.
Revised: An example is shown in Figure 10.

Original: The ﬁrst source code is to return the HTML content from a given URL, while the second source code is to return the last line from a ﬁxed URL “http://kmttg.googlecode.com/svn/trunk/version”.
Revised: The ﬁrst source code is to return the HTML content from a given URL, while the second source code is to return the last line from a ﬁxed URL “http://kmttg.googlecode.com/svn/trunk/version”.

Original: Their semantics are not similar due to their different outputs.
Revised: Their semantics are not similar due to their different outputs.

Original: Data ﬂow could help GraphCodeBERT better understand that the return value “pageHTML” in ﬁrst source code comes from “pageHTML.append(line); pageHTML.append(“ \ r \ n”);” instead of “bufferedWriter.write(pageHTML.toString());” and the return value “version” in the second source code comes from “version = inputLine” or “version = null;”.
Revised: Data ﬂow could help GraphCodeBERT better understand that the return value “pageHTML” in ﬁrst source code comes from “pageHTML.append(line); pageHTML.append(“ \ r \ n”);” instead of “bufferedWriter.write(pageHTML.toString());” and the return value “version” in the second source code comes from “version = inputLine” or “version = null;”.

Original: Although two source codes are highly overlapped (marked in yellow), GraphCodeBERT successfully predict the gold label compared with other models without data ﬂow.
Revised: Although two source codes are highly overlapped marked in yellow , GraphCodeBERT successfully predict the gold label compared with other models without data ﬂow.

Original: [Figure]
Revised: [Figure]

Original: F. C ODE T RANSLATION AND C ODE R EFINEMENT
Revised: F. C ODE T RANSLATION AND C ODE R EFINEMENT

Original: We give a case study to compare GraphCodeBERT with Transformer without using data ﬂow on code generation tasks, including code translation and code reﬁnement.
Revised: We give a case study to compare GraphCodeBERT with Transformer without using data ﬂow on code generation tasks, including code translation and code reﬁnement.

Original: We list three cases in Table 8 and Table 9, respectively.
Revised: We list three cases in Table 8 and Table [Figure] respectively.

Original: [src] represents the source input, [ref] represents the reference, [sys] represents Transformer without data ﬂow and [ours] represents GraphCodeBERT.
Revised: [src] represents the source input, [ref] represents the reference, [sys] represents Transformer without data ﬂow and [ours] represents GraphCodeBERT.

Original: We can see that the
Revised: We can see that the

Original: Transformer ([sys]) baseline makes several mistakes, including repeating tokens, logic errors and syntax errors, while GraphCodeBERT ([ours]) as a encoder could improve the generation.
Revised: Transformer ([sys]) baseline makes several mistakes, including repeating tokens, logic errors and syntax errors [Figure] while GraphCodeBERT ([ours]) as a encoder could improve the generation.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: G E RROR A NALYSIS
Revised: G E RROR A NALYSIS

Original: We also conduct error analysis and summary two main classes of errors for both code understanding and generation tasks.
Revised: We also conduct error analysis and summary two main classes of errors for both code understanding and generation tasks.

Original: Figure 12 gives three error cases of GraphCodeBERT on the natural language code search task.
Revised: Figure 11 gives three error cases of GraphCodeBERT on the natural language code search task.

Original: We observe that GraphCodeBERR mainly fails to retrieve those source code that involves functions of the library like “tf” (Tensorﬂow) in the ﬁrst case and “ GoogleCloudStorageHook” in the second case.
Revised: We observe that GraphCodeBERR mainly fails to retrieve those source code that involves functions of the library like “tf” (Tensorﬂow) in the ﬁrst case and “ GoogleCloudStorageHook” in the second case.

Original: It’s difﬁcult for GraphCodeBERR to understand meanings of APIs like “tf.io.read ﬁle” and “tf.image.decode image” without relevant information.
Revised: It’s difﬁcult for GraphCodeBERR to understand meanings of APIs like “tf.io.read ﬁle” and “tf.image.decode image” without relevant information.

Original: A potential direction to mitigate the problem is to incorporate deﬁnitions of the library.
Revised: A potential direction to mitigate the problem is to incorporate deﬁnitions of the library.

Original: The other major problem is that there are some terminologies like “unistr” in the query (corresponding to “decode(‘utf-8’)” in Python code) in third case.
Revised: The other major problem is that there are some terminologies like “unistr” in the query (corresponding to “decode(‘utf-8’)” in Python code) in third case.

Original: Incorporating more text-code pairs for pre-training might alleviate this problem.
Revised: Incorporating more text-code pairs for pre-training might alleviate this problem.

Original: As for the code generation task, Table 10 shows two cases of GraphCodeBERT on the code translation task.
Revised: As for the code generation task, Table 10 shows two cases of GraphCodeBERT on the code translation task.

Original: We ﬁnd that the major problems include semantic errors like identiﬁers from nowhere in the ﬁrst case and syntax errors like missing a “ } ” symbol before “return n” in the second case.
Revised: We ﬁnd that the major problems include semantic errors like identiﬁers from nowhere in the ﬁrst case and syntax errors like missing a “ } ” symbol before “return n” in the second case.

Original: This problem might be mitigated by incorporating a dedicated decoder that takes into account grammar of programming languages and different generation paradigm like generating a sequence of production rules (Yin & Neubig, 2017; Guo et al., 2018; 2019) in a context-free grammar manner.
Revised: This problem might be mitigated by incorporating a dedicated decoder that takes into account grammar of programming languages and different generation paradigm like generating a sequence of production rules (Yin & Neubig, 2017; Guo et al., 2018; 2019) in a context-free grammar manner.

Original: [Figure]
Revised: [Figure]

Original: Case

Original: : semantic error – identiﬁers from nowhere

Original: [Figure]
Revised: [Figure]

Original: [src] public String toString
Revised: [src] public String toString

Original: [Table]
Revised: [Table]