Original: T HE E FFECTIVENESS OF P RE -T RAINED C ODE E MBEDDINGS
Revised: T HE E FFECTIVENESS OF P RE -T RAINED C ODE E MBEDDINGS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Word embeddings are widely used in machine learning based natural language processing systems.
Revised: Word embeddings are widely used in machine learning based natural language processing systems.

Original: It is common to use pre-trained word embeddings which provide beneﬁts such as reduced training time and improved overall performance.
Revised: It is common to use pre-trained word embeddings which provide beneﬁts such as reduced training time and improved overall performance.

Original: There has been a recent interest in applying natural language processing techniques to programming languages.
Revised: There has been a recent interest in applying natural language processing techniques to programming languages.

Original: However, none of this recent work uses pretrained embeddings on code tokens.
Revised: However, none of this recent work uses pretrained embeddings on code tokens.

Original: Usingan extreme summarization  task, we show that using pre-trained embeddings on code tokens provides the same beneﬁts as it does to natural languages, achieving: over 1.8x speedup, 5% relative performance improvement, and resistance to over-ﬁtting.
Revised: Using extreme summarization as the downstream task, we show that using pre-trained embeddings on code tokens provides the same beneﬁts as it does to natural languages, achieving: over 1.9x speedup, 5% improvement in test loss, 4% improvement in F1 scores, and resistance to over-ﬁtting.

Original: We also show that the choice ofprogramming language used for the embeddings does not have to match that of the task to achieve these beneﬁts.
Revised: We also show that the choice of language used for the embeddings does not have to match that of the task to achieve these beneﬁts and that even embeddings pre-trained on human languages provide these beneﬁts to programming languages.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: One of the initial steps in a machine learning natural language processing (NLP) pipeline is converting the one-hot encoded R V tokens into dense R D representations, with V being the size of the vocabulary, D the embedding dimensions and V << D .
Revised: One of the initial steps in a machine learning natural language processing (NLP) pipeline is converting the one-hot encoded R V tokens into dense R D representations, with V being the size of the vocabulary, D the embedding dimensions and V << D .

Original: This conversion is usually done with a single layer neural network, commonly called an embedding layer.
Revised: This conversion is usually done with a single layer neural network, commonly called an embedding layer.

Original: The parameters of the embedding layer can either be initialized randomly or initialized via  parameters obtained from a model such as word2vec (Mikolov et al., 2013b;a), GloVe (Pennington et al., 2014) or a language model (McCann et al., 2017; Howard & Ruder, 2018; Peters et al., 2018).
Revised: The parameters of the embedding layer can either be initialized randomly or initialized via ”pretrained” parameters obtained from a model such as word2vec (Mikolov et al., 2013b;a), GloVe (Pennington et al., 2014) or a language model (McCann et al., 2017; Howard & Ruder, 2018; Peters et al., 2018).

Original: It is common to use pre-trained parameters (most frequently the GloVe embeddings), which act as a form of transfer learning (Mou et al., 2016) similar to that of using pre-trained parameters for the convolutional kernels in a machine learning computer vision task (Huh et al., 2016; Kornblith et al., 2018).
Revised: It is common to use pre-trained parameters (most frequently the GloVe embeddings), which act as a form of transfer learning (Mou et al., 2016) similar to that of using pre-trained parameters for the convolutional kernels in a machine learning computer vision task (Huh et al., 2016; Kornblith et al., 2018).

Original: These parameters of the embedding layer are then ﬁne-tuned whilst training on the desired downstream task.
Revised: These parameters in the embedding layer are then ﬁne-tuned whilst training on the desired downstream task.

Original: The use of these pre-trained embeddings over random initialization allows machine learning models to: train faster, achieve improved overall performance (Kim, 2014), increase the stability of their training, and reduce the amount of over-ﬁtting (Mou et al., 2016).
Revised: The use of these pre-trained embeddings over random initialization allows machine learning models to: train faster, achieve improved overall performance (Kim, 2014), increase the stability of their training, and reduce the amount of over-ﬁtting (Mou et al., 2016).

Original: Recently there has been an increased interest in applying NLP techniques to programming languages and software engineering applications (Vechev & Yahav, 2016; Allamanis et al., 2017a), the most common of which involves predicting the names of methods or variables using surrounding source code (Raychev et al., 2014; 2015; Allamanis et al., 2015; 2017b; Alon et al., 2018b;a).
Revised: Recently there has been an increased interest in applying NLP techniques to programming languages and software engineering applications (Vechev & Yahav, 2016; Allamanis et al., 2017a), the most common of which involves predicting the names of methods or variables using surrounding source code (Raychev et al., 2014; 2015; Allamanis et al., 2015; 2017b; Alon et al., 2018b;a).

Original: Remarkably, none of this work takes advantage of pre-trained embeddings created on source code, which we dub pre-trained code embeddings.
Revised: Remarkably, none of this work takes advantage of pre-trained embeddings created on source code.

Original: From the example below in table 1, we can see how semantic knowledge (provided by the pre-trained code embeddings) of the method body would help us predict the method name, i.e. knowing how pi and radius are used to calculate an area and how height and width are used to calculate an aspect ratio.
Revised: From the example below in table 1, we can see how semantic knowledge (provided by the pre-trained code embeddings) of the method body would help us predict the method name, i.e. knowing how pi and radius are used to calculate an area and how height and width are used to calculate an aspect ratio.

Original: This semantic knowledge is available to us as even though computers do not need to understand the semantic meaning of a method or variable name, they are mainly chosen to be understood by other human programmers (Hindle et al., 2012).

Original: float getSurfaceArea (int radius) { return  * Math
Revised: float getSurfaceArea (int radius) { return 4 * Math.

Original: PI * radius * radius; } float getAspectRatio int height, int width) { return height / width; }
Revised: PI * radius * radius; } float getAspectRatio (int height, int width) { return height / width; }

Revised: This semantic knowledge is available to us as even though computers do not need to understand the semantic meaning of a method or variable name, they are mainly chosen to be understood by other human programmers (Hindle et al., 2012).

Original: In this paper, we detail experiments using pre-trained code embeddings on the  task of predicting a method name from a method body.
Revised: In this paper, we detail experiments using pre-trained code embeddings on the downstream task of predicting a method name from a method body.

Original: This task is known as extreme summarization (Allamanis et al., 2016) as a method name can be thought of as a summary of the method body.
Revised: This task is known as extreme summarization (Allamanis et al., 2016b) as a method name can be thought of as a summary of the method body.

Original: Our experiments are focused on answering the following research questions:
Revised: Our experiments are focused on answering the following research questions:

Original: Do pre-trained code embeddings reduce training time?
Revised: Do pre-trained code embeddings reduce training time?

Original: Do pre-trained code embeddings improve performance?
Revised: Do pre-trained code embeddings improve performance?

Original: Do pre-trained code embeddings increase stability of training?
Revised: Do pre-trained code embeddings increase stability of training?

Original: Do pre-trained code embeddings reduce over-ﬁtting?
Revised: Do pre-trained code embeddings reduce over-ﬁtting?

Original: How does the choice of corpora used for the pre-trained code embeddings affect all of the above?
Revised: How does the choice of corpora used for the pre-trained code embeddings affect all of the above?

Original: To answer RQ5, we gather a corpus of C, Java and Python code and train embeddings for each corpus separately.
Revised: To answer RQ5, we gather a corpus of C, Java and Python code and train embeddings for each corpus separately, as well as comparing them with embeddings trained on natural language We

Original: We then test each of these on the same  extreme summarization dataset, which is in Java.
Revised: then test each of these on the same downstream task, extreme summarization, which is in Java.

Original: We also release the pre-trained code embeddings.
Revised: We also release the pre-trained code embeddings.

Original: As far as we are aware, this is the ﬁrst study on the effectiveness of pre-trained code embeddings applied to an extreme (code) summarization task.
Revised: As far as we are aware, this is the ﬁrst study on the effectiveness of pre-trained code embeddings applied to an extreme (code) summarization task.

Original:  M ODELS 2. L ANGUAGE M ODEL
Revised:  M ODELS 2. L ANGUAGE M ODEL

Original: We train our embeddings using a language model (LM).
Revised: We train our embeddings using a language model (LM).

Original: We use a LM over the word2vec or GloVe models as LMs have shown to capture long-term dependences (Linzen et al., 2016) and hierarchical relations (Gulordava et al., 2018).
Revised: We choose a LM over the word2vec or GloVe models as LMs have shown to capture long-term dependences (Linzen et al., 2016) and hierarchical relations (Gulordava et al., 2018).

Original: We believe both of these properties are essential for predicting a method name from method body.
Revised: We believe both of these properties are essential for predicting a method name from method body.

Original: The long-term dependencies are required due to the average length of the method body over 72 tokens 2 .
Revised: The long-term dependencies are required due to the average length of the method body over 72 tokens 2 .

Original: The hierarchical relations are needed due to the way data ﬂows through variables within the method body, starting from the method argument(s)
Revised: The hierarchical relations are needed due to the way data ﬂows through variables within the method body, starting from the method argument(s)

Original: (at the top of the hierarchy) to the return value(s) (at the bottom of the hierarchy).
Revised: (at the top of the hierarchy) to the return value(s) (at the bottom of the hierarchy).

Original: A language model is a probability distribution over sequences of tokens.
Revised: A language model is a probability distribution over sequences of tokens.

Original: Each token, x , is represented by a one-hot vector x ∈ R V , with V being the size of the vocabulary.
Revised: Each token, x , is represented by a one-hot vector x ∈ R V , with V being the size of the vocabulary.

Original: The probability given to a sequence of tokens x 1 , ..., x T can be calculated as:
Revised: The probability given to a sequence of tokens x 1 , ..., x T can be calculated as:

Original: [Equation]
Revised: [Equation]

Original: We model this probability distribution with a recurrent neural network trained to predict the next token in a sequence of given tokens.
Revised: We model this probability distribution with a recurrent neural network trained to predict the next token in a sequence of given tokens.

Original: Speciﬁcally, we use the AWD-LSTM-LM model (Merity et al., 2017; 2018) due to its performance at modeling natural languages and open source implementation.
Revised: Speciﬁcally, we use the AWD-LSTM-LM model (Merity et al., 2017; 2018) due to its performance at modeling natural languages and open source implementation.

Original: 2. C ONVOLUTIONAL A TTENTION M ODEL
Revised: 2. C ONVOLUTIONAL A TTENTION M ODEL

Original: The extreme summarization task uses the Copy Convolutional Attention Model from Allamanis et al.
Revised: The extreme summarization task uses the Copy Convolutional Attention Model from Allamanis et al.

Original:  Brieﬂy, the model takes a series of code tokens from the method body, c , as input and outputs the code tokens that form the method name, m .
Revised: (2016a). Brieﬂy, the model takes a series of code tokens from the method body, c , as input and outputs the code tokens that form the method name, m .

Original: It generates the method name one token at a time, using a recurrent hidden state, h t , provided by a Gated Recurrent Unit (GRU) (Cho et al., 2014) and a series of convolutional ﬁlters over the embeddings of the tokens c , which produce attention (Bahdanau et al., 2014) features, L feat .
Revised: It generates the method name one token at a time, using a recurrent hidden state, h t , provided by a Gated Recurrent Unit (GRU) (Cho et al., 2014) and a series of convolutional ﬁlters over the embeddings of the tokens c , which produce attention (Bahdanau et al., 2014) features, L feat .

Original: It also has a mechanism to directly copy tokens from the body to the output.
Revised: It also has a mechanism to directly copy tokens from the body to the output.

Original: This model was chosen as it is the state-of-the-art on the extreme summarization dataset used and provided a clear improvement over baseline models.
Revised: This model was chosen as it is the state-of-the-art on the extreme summarization dataset used and provided a clear improvement over baseline models.

Original: It also has an open source implementation.
Revised: It also has an open source implementation.

Original:  E XPERIMENTAL S ETUP 3. E MBEDDING D ATASET
Revised:  E XPERIMENTAL S ETUP 3. E MBEDDING D ATASET

Original: The dataset used for the pre-trained code embeddings was gathered from GitHub.
Revised: The dataset used for the pre-trained code embeddings was gathered from GitHub.

Original: To ensure the quality of the data we only used projects with over 10,000 stars and manually checked each project’s suitability, i.e. did not use projects which were tutorials or guides.
Revised: To ensure the quality of the data we only used projects with over 10,000 stars and manually checked each project’s suitability, i.e. did not use projects which were tutorials or guides.

Original: After scraping the appropriate projects for each of the three languages (C, Java and Python) we tokenized each, converting each token to lowercase as well as splitting each token into subtokens on camelCase and snake case , e.g. getSurfaceArea becomes get , surface and area .
Revised: After scraping the appropriate projects for each of the three languages (C, Java and Python) we tokenized each, converting each token to lowercase as well as splitting each token into subtokens on camelCase and snake case , e.g. getSurfaceArea becomes get , surface and area .

Original: This was done to match the tokenization of the extreme summarization dataset.
Revised: This was done to match the tokenization of the extreme summarization dataset.

Revised: There are approximately 100 million tokens across 20 million lines of code for each language.

Original: Each of the embeddings has their own distinct vocabulary, e.g. not all tokens that appear in the C corpus appear in the 
Revised: Each of the embeddings has their own distinct vocabulary, e.g. not all tokens that appear in the C corpus appear in the Java corpus.

Original: Java corpus.
Revised: For the natural language embeddings we used the WikiText-103 dataset Merity et al. (2016), as it contains a comparable 103 million tokens.

Original: 3. L ANGUAGE M ODEL
Revised: 3. L ANGUAGE M ODEL

Original: The AWD-LSTM-LM model was trained with all default parameters from the open source implementation, with the exception of: the embedding dimension changed to 128 and the hidden dimension changed to 512.
Revised: The AWD-LSTM-LM model was trained with all default parameters from the open source implementation, with the exception of: the embedding dimension changed to 128 and the hidden dimension changed to 512.

Original: The embedding dimension was changed to match that of the original Copy
Revised: The embedding dimension was changed to match that of the original Copy

Original: Convolutional Attention Model, and the hidden dimension was changed to ﬁt in GPU memory.
Revised: Convolutional Attention Model, and the hidden dimension was changed to ﬁt in GPU memory.

Original: Tokens that were not in the most 150,000 common or did not appear at least 3 times were converted into an <unk> token. The model was trained until the validation loss did not decrease for 5 epochs.
Revised: Tokens that were not in the most 150,000 common or did not appear at least 3 times were converted into an <unk> token and the model was trained until the validation loss did not decrease forepochs.

Original: 3. E XTREME S UMMARIZATION D ATASET
Revised: 3. E XTREME S UMMARIZATION D ATASET

Original: The extreme summarization task dataset is detailed in Allamanis et al.
Revised: The extreme summarization task dataset is detailed in Allamanis et al.

Original:  Brieﬂy, it consists of 10 Java projects selected for their quality and diversity in application.
Revised: (2016a). Brieﬂy, it consists of 10 Java projects selected for their quality and diversity in application.

Original: For each project, all full
Revised: For each project, all full

Original: Java methods are extracted with the method body used as the input and the method name used as the target.
Revised: Java methods are extracted with the method body used as the input and the method name used as the target.

Original: Each project has their own vocabulary, e.g. tokens that appear in one project may not appear in any others.
Revised: Each project has their own vocabulary, e.g. tokens that appear in one project may not appear in any others.

Original: All tokens are formatted the same as the embedding dataset to ensure maximum vocabulary overlap between each pre-trained embeddings and each Java project.
Revised: All tokens are formatted the same as the embedding dataset to ensure maximum vocabulary overlap between each pre-trained embeddings and each Java project.

Original: See table 2 for the names and short descriptions of each of the Java projects.

Original: 3. C ONVOLUTIONAL A TTENTION M ODEL
Revised: 3. C ONVOLUTIONAL A TTENTION M ODEL

Original: The Copy Convolutional Attention Model was trained with all default parameters from the open source implementation, and was trained for 25 epochs.
Revised: The Copy Convolutional Attention Model was trained with all default parameters from the open source implementation, and was trained for 25 epochs.

Original: The model was trained on each project separately and was run 5 times on each project for each of the embeddings.
Revised: The model was trained on each project separately and was run 5 times on each project for each of the embeddings.

Original: The results were then averaged together for each project.
Revised: The results were then averaged together for each project.

Original:  R ESULTS
Revised:  R ESULTS

Revised: Table 2 shows the rank 1 F1 scores achieved for each of the embeddings.

Revised: On average, we achieve a 4% relative improvement in F1 scores for each of the embeddings.

Revised: [Table]

Original: Figure 1 shows the validation losses achieved on all 10 Java projects.
Revised: Figure 1 shows the validation losses achieved on all 10 Java projects.

Original: Randomly initialized embeddings are shown in red, Java embeddings in orange, C in blue and Python in green.
Revised: Randomly initialized embeddings are shown in purple, Java embeddings in green, C in blue, Python in red and English in orange.

Original: It can be seen that for most projects the pre-trained embeddings train faster, achieve overall performance, are more stable and over-ﬁt less.
Revised: It can be seen that for most projects the pre-trained embeddings train faster, achieve lower losses, are more stable and over-ﬁt less.

Original: Table 2 shows overlap, speedup and improvement  for each project-embedding combination compared to random embeddings in terms of their validation losses during training.
Revised: Table 3 shows overlap, speedup and improvement in test loss for each project-embedding combination compared to random embeddings.

Original: Overlap is the percentage of tokens in the project vocabulary that also appear in the embedding vocabulary.
Revised: Overlap is the percentage of tokens in the project vocabulary that also appear in the embedding vocabulary.

Original: Speedup is calculated as:
Revised: Speedup is calculated as:

Original: S = N r /N e
Revised: [Equation]

Original: N r is the number of epochs taken by the random embedding to reach its best validation loss and N e is the number of epochs taken by a non-random embedding to reach that same validation loss.
Revised: N r is the number of epochs taken by the random embedding to reach its best validation loss and N e is the number of epochs taken by a non-random embedding to reach that same validation loss.

Original: Improvement is calculated as:
Revised: Improvement is calculated as:

Original: I =
Revised: I =

Original: [Equation] r /
Revised: L r /L e

Original: [Equation] e L r is the minimum validation loss achieved using random embeddings and L e is the minimum validation loss achieved using a non-random embedding.
Revised: L r is the test loss achieved using random embeddings and L e is the test loss achieved using a nonrandom embedding.

Original: An I of 1.05 would indicate a 5% relative performance improvement  over random embeddings.
Revised: An I of 1.05 would indicate a 5% relative performance improvement in test loss over random embeddings.

Original: [Table]
Revised: [Table]

Original: Speedup is relative speedup of convergence compared to Improvement rel improvement
Revised: [Figure]

Original: We notice that some projects, particularly the elasticsearch project (ﬁgure 1b), did not achieve any beneﬁts from using the pre-trained embeddings and in fact experienced a slow down when using the C embeddings.
Revised: We notice that some projects, particularly the elasticsearch project (ﬁgure 1b), did not achieve any beneﬁts from using the pre-trained embeddings and in fact experienced a slow down when using the C embeddings.

Original: To explore this further, we plotted speedup and improvement against overlap in ﬁgure 2. 
Revised: To explore this further, we plotted speedup and improvement against overlap in ﬁgure 2. [Figure]

Original: We measure the Pearson correlation coefﬁcient of each, receiving a coefﬁcient of 0.72 for speedup and 0.73 for improvement, a medium to strong positive correlation for each.
Revised: We measure the Pearson correlation coefﬁcient of each, receiving a coefﬁcient of 0.73 for speedup and 0.73 for improvement, a medium to strong positive correlation for each.

Original: This implies that using more of the pre-trainedcode embeddings provides more of both the speedup and improvement  beneﬁts.
Revised: This implies that using more of the pre-trained embeddings provides more of both the speedup and improvement in test loss beneﬁts.

Original: Table 3 shows the overlap, speedup and improvement averaged over all projects.
Revised: Table 4 shows the overlap, speedup and improvement averaged over all projects.

Revised: From this, we can see that we achieve around 1.9x speedup, along with a 5% relative improvement in test loss.

Revised: [Table]

Original: Intuitively, it would make sense that the Java embeddings give the best results as the summarization task is also in Java.
Revised: Intuitively, it would make sense that the Java embeddings give the best results as the summarization task is also in Java.

Original: We see this is not the case and the Java embeddings have the same speedup and improvement as the Python embeddings and only a small speedup improvement over the C embeddings, even though the average overlap using the Java embeddings is higher.
Revised: We see this is not the case, and the Java embeddings have the same speedup and improvement as the Python embeddings and only a small speedup improvement over the C embeddings, even though the average overlap using the Java embeddings is higher.

Original: [Table]
Revised: Most interesting is the fact that the English embeddings achieve the comparable speedup and performance improvement, even though they have only been trained on human languages.

Original: [Figure] [Figure] One potential reason for this is that even though C, Java and Python are syntactically different, the extreme summarization task does not require much of this syntactic information.
Revised: One potential reason for the similar performance between the embeddings that are trained on programming languages is that even though C, Java and Python are syntactically different, the extreme summarization task does not require much of this syntactic information.

Original: Consider the examples in table 4, which are Python versions of the Java examples in table 1.
Revised: Consider the examples in table 5, which are Python versions of the Java examples in table 1.

Original: Although the syntax has changed (dynamic typing, no braces or semicolons, etc.)
Revised: Although the syntax has changed (dynamic typing, no braces or semicolons, etc.)

Original: the available semantic information from the method body has not.
Revised: the available semantic information from the method body has not.

Original: This would imply that the language of the dataset used to pre-train code embeddings does not matter as much as the quality of the dataset with regards to sensible method and variable names.
Revised: This would imply that the language of the dataset used to pre-train code embeddings does not matter as much as the quality of the dataset with regards to sensible method and variable names.

Revised: This is further solidiﬁed in the fact that the English embeddings, which have only been trained on human languages, achieved similar performance compared to programming languages.

Original: def get surface area (radius): return 4 * math.pi * radius * radius def get aspect ratio (height, width): return height / width
Revised: def get surface area (radius): return 4 * math.pi * radius * radius def get aspect ratio (height, width): return height / width

Original: [Table] [Table]
Revised: Python versions of the Java examples from table

Original: We also look at the amount of over-ﬁtting on each project.
Revised: We also look at the amount of over-ﬁtting on each project.

Original: From ﬁgures 1c, 1d, 1f, 1g and 1i we can see how the random embeddings show a large amount of over-ﬁtting compared to the pre-trained code embeddings.
Revised: From ﬁgures 1c, 1d, 1f, 1g and 1i we can see how the random embeddings show a large amount of over-ﬁtting compared to the pre-trained code embeddings.

Original: We measure how much a project over-ﬁts as:
Revised: We measure how much a project over-ﬁts as:

Original: [Equation]
Revised: [Equation]

Original: L b is the best validation loss achieved and L f is the ﬁnal validation loss achieved.
Revised: L b is the best validation loss achieved and L f is the ﬁnal validation loss achieved.

Original: We dub this term the over-ﬁt factor, where an O = 1 would imply the ﬁnal loss is equal to the lowest loss and thus has not over-ﬁt at all (this could also mean the model is still converging, however from ﬁgure 1 we can see all project-embedding combinations converge before 25 epochs).
Revised: We dub this term the over-ﬁt factor, where an O = 1 would imply the ﬁnal loss is equal to the lowest loss and thus has not over-ﬁt at all (this could also mean the model is still converging, however from ﬁgure 1 we can see all project-embedding combinations converge before 25 epochs).

Original: Table 5 shows the over-ﬁt factors for each project-embedding combination.
Revised: Table 6 shows the over-ﬁt factors for each project-embedding combination.

Original: We can see that the random embeddings show the worst performance on every project.
Revised: We can see that the random embeddings show the worst performance on every project.

Original: Interestingly, there appears to be no correlation between the overlap and the amount of over-ﬁtting.
Revised: Interestingly, there appears to be no correlation between the overlap and the amount of over-ﬁtting.

Original: Table 6 shows the over-ﬁt factor averaged across all projects for each embedding.
Revised: Table 7 shows the over-ﬁt factor averaged across all projects for each embedding.

Original: Again, the results for each of the pre-trained embeddings are similar, showing that the language of the dataset used to train the embeddings does not have a signiﬁcant impact on performance. 
Revised: Again, the results for each of the pre-trained embeddings are similar, showing that the language of the dataset used to train the embeddings does not have a signiﬁcant impact on performance. [Table] [Table]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Language models have been used as a form of transfer learning in natural language processing applications with great success (McCann et al., 2017; Howard & Ruder, 2018; Peters et al., 2018).
Revised: Language models have been used as a form of transfer learning in natural language processing applications with great success (McCann et al., 2017; Howard & Ruder, 2018; Peters et al., 2018).

Original: There has also been recent work on the further analysis of language models (Merity et al., 2018; 2017) and how well they assist in transfer learning (Mou et al., 2016).
Revised: There has also been recent work on the further analysis of language models (Merity et al., 2018; 2017) and how well they assist in transfer learning (Mou et al., 2016).

Original: The use of probabilistic models for source code originated from Hindle et al.
Revised: The use of probabilistic models for source code originated from Hindle et al.

Original: From that, work on language models of code began on both the token level (Nguyen et al., 2013; Tu et al.,2014) and syntax level (Maddison & Tarlow, 2014).
Revised: From that, work on language models of code began on both the token level (Nguyen et al., 2013; Tu et al.,

Original: Predicting variable and method names has become a common task for machine learning applications in recent years.
Revised: Predicting variable and method names has become a common task for machine learning applications in recent years.

Original: Initial work was on the token level (Raychev et al., 2014; 2015; Allamanis et al., 2015) but it is beginning to become more common to represent programs as graphs using their abstract syntax tree (Allamanis et al., 2017b; Alon et al., 2018a;b).
Revised: Initial work was on the token level (Raychev et al., 2014; 2015; Allamanis et al., 2015) but it is beginning to become more common to represent programs as graphs using their abstract syntax tree (Allamanis et al., 2017b; Alon et al., 2018a;b).

Original:  D ISCUSSION & C ONCLUSIONS
Revised:  D ISCUSSION & C ONCLUSIONS

Original: We refer back to our research questions.
Revised: We refer back to our research questions.

Original: Do pre-trained code embeddings reduce training time?
Revised: Do pre-trained code embeddings reduce training time?

Original: Yes, tables2 and 3  show we get an average of 1.93x speedup.
Revised: Yes, tables 3 and 4 show we get an average of 1.93x speedup.

Original: This is correlated with the amount of overlap between the task vocabulary and the embedding vocabulary, shown in ﬁgure 2a.
Revised: This is correlated with the amount of overlap between the task vocabulary and the embedding vocabulary, shown in ﬁgure 2a.

Original: Do pre-trained code embeddings improve performance?
Revised: Do pre-trained code embeddings improve performance?

Original: Yes, tables2 and 3  show we get an average of 5% relative validation loss improvement.
Revised: Yes, tables 3 and 4 show we get an average of 5% relative validation loss improvement.

Original: Again, this is correlated with the amount of overlap between the vocabularies, shown in ﬁgure 2b.
Revised: Again, this is correlated with the amount of overlap between the vocabularies, shown in ﬁgure 2b.

Original: Do pre-trained code embeddings increase stability of training?
Revised: Do pre-trained code embeddings increase stability of training?

Original: Although this is difﬁcult to quantify due to how over-ﬁtting interacts with the variance of the validation loss curves, from ﬁgures 1a and 1c we can see a clear increase in the variance of the validation loss curves using random embeddings compared to those using pre-trained embeddings.
Revised: Although this is difﬁcult to quantify due to how over-ﬁtting interacts with the variance of the validation loss curves, from ﬁgures 1a and 1c we can see a clear increase in the variance of the validation loss curves using random embeddings compared to those using pre-trained embeddings.

Original: Do pre-trained code embeddings reduce over-ﬁtting?
Revised: Do pre-trained code embeddings reduce over-ﬁtting?

Original: Yes, tables5 and 6  show that the random embeddings over-ﬁt more than the pre-trained embeddings on every project.
Revised: Yes, tables 6 and 7 show that the random embeddings over-ﬁt more than the pre-trained embeddings on every project.

Original: However, this does not seem to have a correlation with the amount of vocabulary overlap and further work is needed to determine the cause of this.
Revised: However, this does not seem to have a correlation with the amount of vocabulary overlap and further work is needed to determine the cause of this.

Original: How does the choice of corpora used for the pre-trained code embeddings affect all of the above?
Revised: How does the choice of corpora used for the pre-trained code embeddings affect all of the above?

Original: Yes, but the amount is negligible.

Original: Intuitively, it would seem the best pre-trained embeddings would be those that are trained on the same language as that of the downstream task, but this is not the case.
Revised: Intuitively, it would seem the best pre-trained embeddings would be those that are trained on the same language as that of the downstream task, but this is not the case.

Original: We hypothesize through the examples shown in tables 1 and 4 that the differing syntax between the languages is not as important as sensible semantic method and variable names within the dataset.
Revised: We hypothesize through the examples shown in tables 1 and 5 that the differing syntax between the languages is not as important as sensible semantic method and variable names within the dataset.

Revised: This semantic information is also contained in human languages, which explains why the English embeddings also receive comparable performance