CODE 2 SEQ : G ENERATING S EQUENCES FROM S TRUCTURED R EPRESENTATIONS OF C ODE
A BSTRACT
The ability to generate natural language sequences from source code snippets has a variety of applications such as code summarization, documentation, and retrieval.
Sequence-to-sequence (seq2seq) models, adopted from neural machine translation (NMT), have achieved state-of-the-art performance on these tasks by treating source code as a sequence of tokens.
We present CODE 2 SEQ : an alternative approach that leverages the syntactic structure of programming languages to better encode source code.
Our model represents a code snippet as the set of compositional paths in its abstract syntax tree (AST) and uses attention to select the relevant paths while decoding.
We demonstrate the effectiveness of our approach for two tasks, two programming languages and four datasets of up to M exam ples
Our model signiﬁcantly outperforms previous models that were speciﬁcally designed for programming languages as well as state-of-the-art NMT models.
An interactive online demo of our model is available at http://code2seq.org .
I NTRODUCTION
Modeling the relation between source code and natural language can be used for automatic code summarization (Allamanis et al., 2016), documentation (Iyer et al., 2016), retrieval (Allamanis et al., 2015b), and even generation (Balog et al., 2016; Rabinovich et al., 2017; Yin and Neubig, 2017;
Devlin et al., 2017; Murali et al., 2017).
In this work, we consider the general problem of generating a natural language sequence from a given snippet of source code.
A direct approach is to frame the problem as a machine translation problem, where the source sentence is the sequence of tokens in the code and the target sentence is a corresponding natural language sequence.
This approach allows one to apply state-of-the-art neural machine translation (NMT) models from the sequence-to-sequence (seq2seq) paradigm (Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2014; Luong et al., 2015; Vaswani et al., 2017), yielding state-ofthe-art performance on various code captioning and documentation benchmarks (Iyer et al., 2016;
Allamanis et al., 2016; Loyola et al., 2017) despite having extremely long source sequences.
We present an alternative approach for encoding source code that leverages the syntactic structure of programming languages: CODE 2 SEQ .
Speciﬁcally, we represent a given code snippet as a set of compositional paths over its abstract syntax tree (AST), where each path is compressed to a ﬁxed-length vector using LSTMs (Hochreiter and Schmidhuber, 1997).
During decoding, CODE 2 SEQ attends over a different weighted sum of the path-vectors to produce each output token, much like
NMT models attend over token representations in the source sentence.
We show the effectiveness of our code2seq model on two tasks: (1) code summarization (Figure 1a), where we predict a Java method’s name given its body, and (2) code captioning (Figure 1b), where we predict a natural language sentence that describes a given C# snippet.
On both tasks, our
CODE 2 SEQ model outperforms models that were explicitly designed for code, such as the model of
Allamanis et al.
(2016) and CodeNN (Iyer et al., 2016), as well as state-of-the-art NMT models Code summarization in Java: Code captioning in C#:
[Figure]
(Luong et al., 2015; Vaswani et al., 2017).
To examine the importance of each component of the model, we conduct a thorough ablation study.
In particular, we show the importance of structural encoding of code, by showing how our model yields a signiﬁcant improvement over an ablation that uses only token-level information without syntactic paths.
To the best of our knowledge, this is the ﬁrst work to directly use paths in the abstract syntax tree for end-to-end generation of sequences.
R EPRESENTING C ODE AS AST P ATHS
An Abstract Syntax Tree (AST) is a tree which uniquely represents a source code snippet in a given language and grammar.
The leaves of the tree are called terminals , and usually refer to userdeﬁned values which represent identiﬁers and names from the code.
The non-leaf nodes are called nonterminals and represent a restricted set of structures in the language, e.g., loops, expressions, and variable declarations.
For example, Figure 2c shows a partial AST for the code snippet of Figure 2a.
Names (such as num ) and types (such as int ) are represented as values of terminals; syntactic structures such as variable declaration ( VarDec ) and a do-while loop ( DoStmt ) are represented as nonterminals.
Given the AST of a code snippet, we consider all pairwise paths between terminals, and represent them as sequences of terminal and nonterminal nodes.
We then use these paths with their terminals’ values to represent the code snippet itself.
For example, consider the two Java methods of Figure 2.
Both of these methods count occurrences of a character in a string.
They have exactly the same functionality, although a different implementation, and therefore different surface forms.
Encoding these snippets of code as sequences of tokens might overlook the recurring patterns that suggest the common method name.
However, a structural observation reveals syntactic paths that are common to both methods, and differ only in a single node of a Do -while statement versus a For statement.
This example shows the effectiveness of a syntactic encoding of code.
Such an encoder can generalize much better to unseen examples because the AST normalizes a lot of the surface form variance.
Since our encoding is compositional, the encoder can generalize even if the paths are not identical (e.g., a For node in one path and a While in the other).
Since a code snippet can contain an arbitrary number of such paths, we sample k paths as the representation of the code snippet.
To avoid bias, k new paths are sampled afresh in every training iteration.
In Section 5 we show that this runtime-sampling provides regularization and improves results compared to sampling the same k paths for each example in advance.
Formally, we use C to denote a given snippet of code.
Every training iteration k pairs of terminals are uniformly sampled from within the AST of C .
Each pair of terminals (cid:0) v i 1 , v il i (cid:1) implies a single path between them: v i 1 v i 2 ...v il i .
Finally, the input code example is represented as a set of these k random AST paths: (cid:8)(cid:0) v 1 1 v 1 2 ...v 1 l 1 (cid:1) , ..., (cid:0) v k 1 v k 2 ...v k l k (cid:1)(cid:9) , where l j is the length of the j th path.
int countOccurrences(String str, char ch) { int num = 0; int index =
-1; do { index = str.indexOf(ch, index + 1); if (index >= 0) { num++; } } while (index >= 0); return num; } (
[Figure]
int countOccurrences(String source, char value) { int count = 0; for ( int i = 0; i < source.length(); i++) { if (source.charAt(i) =
= value) { count++; } }
[Figure]
count; }
[Figure]
M ODEL A RCHITECTURE
Our model follows the standard encoder-decoder architecture for NMT (Section 3.1), with the signiﬁcant difference that the encoder does not read the input as a ﬂat sequence of tokens .
Instead, the encoder creates a vector representation for each AST path separately (Section 3.2).
The decoder then attends over the encoded AST paths (rather than the encoded tokens) while generating the target sequence.
An illustration of our model is shown in Figure 3.
3. E NCODER -D ECODER F RAMEWORK
Contemporary NMT models are largely based on an encoder-decoder architecture (Cho et al., 2014;
Sutskever et al., 2014; Luong et al., 2015; Bahdanau et al., 2014), where the encoder maps an input sequence of tokens x = ( x 1 , ..., x n ) to a sequence of continuous representations
z = ( z 1 , ..., z n ) .
Given z , the decoder then generates a sequence of output tokens y
=
( y 1 , ..., y m ) one token at a time, hence modeling the conditional probability: p ( y 1 , ..., y m | x 1 , ..., x n ) .
At each decoding step, the probability of the next target token depends on the previously generated token, and can therefore be factorized as:
[Equation]
In attention-based models, at each time step t in the decoding phase, a context vector c t is computed by attending over the elements in z using the decoding state h t , typically computed by an LSTM.
[Equation]
The context vector c t and the decoding state h t are then combined to predict the current target token y t .
Previous work differs in the way the context vector is computed and in the way it is combined with the current decoding state.
A standard approach (Luong et al., 2015) is to pass c t and h t through a multi-layer perceptron (MLP) and then predict the probability of the next token using softmax:
[Equation]
3. AST E NCODER
Given a set of AST paths ( x 1 , ..., x k ) , our goal is to create a vector representation
z i for each path x i = v
i 1 v i 2 ...v il i .
We represent each path separately using a bi-directional LSTM to encode the path, and sub-token embeddings to capture the compositional nature of the terminals’ values (the tokens).
Path Representation
Each AST path is composed of nodes and their child indices from a limited vocabulary of up to 364 symbols.
We represent each node using a learned embedding matrix E nodes and then encode the entire sequence using the ﬁnal states of a bi-directional LSTM:
[Equation]
Token Representation
The ﬁrst and last node of an AST path are terminals whose values are tokens in the code.
Following Allamanis et al.
(2015a; 2016), we split code tokens into sub tokens; for example, a token with the value ArrayList , will be decomposed into Array and List .
This is somewhat analogous to byte-pair encoding in NMT (Sennrich et al., 2016), although in the case of programming languages, coding conventions such as camel notation provide us with an explicit partition of each token.
Speciﬁcally, we use a learned embedding matrix E subtokens to represent each subtoken, and then sum the subtoken vectors to represent the full token:
[Equation]
The LSTM decoder may also predict subtokens at each step (e.g. when generating method names), although the decoder’s subtoken embedding matrix will be different.
Combined Representation To represent the entire path x = v 1 ...v l , we concatenate the path’s representation with each of the token representation of each terminal node, and apply a fullyconnected layer:
[Equation]
where value is the mapping of a terminal node to its associated value, and W in is a (2 d path + 2 d token )
× d hidden matrix.
Decoder Start State To provide the decoder with an initial state, we average the combined representations of all the k paths in the given example:
[Equation]
Unlike typical encoder-decoder models, the order of the input random paths is not taken into account.
Each path is encoded separately and the combined representations are aggregated with mean pooling to initialize the decoder’s state.
This represents the given source code as a set of random paths.
Attention Finally, the decoder generates the output sequence while attending over all of the combined representations z 1 , ...
z k , similarly to the way that seq2seq models attend over the source symbols.
The attention mechanism is used to dynamically select the distribution over these k combined representations while decoding, just like an NMT model would attend over the source tokens.
E XPERIMENTS
We evaluate our model on two code-to-sequence tasks: summarization (Section 4.1), in which we predict Java methods’ names from their bodies, and captioning (Section 4.2), where we generate natural language descriptions of C# code snippets. Although out of the focus of this work, in Section 4.
we show that we generate Javadocs more accurately than an existing work.
We thus demonstrate that our approach can produce both method names and natural language outputs, and can encode a code snippet in any language for which an AST can be constructed (i.e., a parser exists).
Setup The values of all of the parameters are initialized using the initialization heuristic of Glorot and Bengio (2010).
We optimize the cross-entropy loss (Rubinstein, 1999; 2001) with a Nesterov momentum (Nesterov, 1983) of 0 .
95 and an initial learning rate of 0 .
01 , decayed by a factor of 0 .every epoch.
For the Code Summarization task, we apply dropout (Srivastava et al., 2014) of 0 .on
the input vectors x j , and 0 . 7 for the Code Captioning task because of the smaller number of examples in the C# dataset. We apply a recurrent dropout of 0 .
5 on the LSTM that encodes the AST paths.
We used d tokens =
d nodes = d hidden =
d target = 128 .
For the Code Summarization task, each LSTM that encodes the AST paths had 128 units and the decoder LSTM had 320 cells. For the Code Captioning task, to support the longer target sequences, each encoder LSTM had 256 units and the decoder was of size 512 . Choice of k
We experimented with different values of k , the number of sampled paths from each example (which we set to 200 in the ﬁnal models). Lower values than k = 100 showed worse results, and increasing to k > 300 did not seem to consistently improve. Practically, k = 200 was found to be a reasonable sweet spot between capturing enough information while keeping training feasible in the GPU’s memory. Additionally, since the average number of paths in our Java-large training set is 220 paths per example.
For some large methods, a number as high as 200 is beneﬁcial.
4. C ODE S UMMARIZATION
In this task, we predict a Java method’s name given its body.
As was previously observed (Allamanis et al., 2016; Alon et al., 2018a), this is a good benchmark because a method name in open-source
Java projects tends to be succinct and precise, and a method body is often a complete logical unit.
We predict the target method name as a sequence of sub-tokens, e.g., setMaxConnectionsPerServer is predicted as the sequence “set max connections per server”.
The target sequence length is about 3 on average.
We adopt the measure used by Allamanis et al.
and
Alon et al.
(2018a), who measured precision, recall, and F1 score over the target sequence, case insensitive.
Data We experiment with this task across three datsets.
In these datasets, we always train across multiple projects and predict on distinct projects:
Java-small – Contains 11 relatively large Java projects, which were originally used for 11 distinct models for training and predicting within the scope of the
same project (Allamanis et al., 2016).
We use the same data, but train and predict across projects : we took 9 projects for training, 1 project for validation and 1 project as our test set.
This dataset contains about 700 K examples. [Table]
-med – A new dataset of the 1000 top-starred Java projects from GitHub.
We randomly select 800 projects for training, 100 for validation and 100 for testing.
This dataset contains about 4M examples and we make it publicly available.
Java-large – A new dataset of the 9500 top-starred Java projects from GitHub that were created since January 2007.
We randomly select 9000 projects for training, 250 for validation and 300 for testing.
This dataset contains about 16 M examples and we make it publicly available.
More statistics of our datasets can be found in Appendix A.
Baselines We re-trained all of the baselines on all of the datasets of this task using the original implementations of the authors.
We compare CODE 2 SEQ to the following baselines: Allamanis et al.
(2016) who used a convolutional attention network to predict method names, syntactic paths with
Conditional Random Fields (CRFs) (Alon et al., 2018b), and code2vec (Alon et al., 2018a).
In addition, we compared to three NMT baselines that read the input source code as a stream of tokens: a 2 -layer bidirectional encoder-decoder LSTMs (split tokens and full tokens) with global attention (Luong et al., 2015), and the Transformer (Vaswani et al., 2017) which achieved state-of-the-art results for translation tasks.
Our model is incomparable to the model of Allamanis et al.
(2018) because they targeted a different task of predicting variable names, and are unable to generate target sequences.
We put signiﬁcant effort to strengthen the NMT baselines in order to provide a fair comparison: (1) we split tokens to subtokens, as in our model (e.g., HashSet → Hash Set ) - this was shown to improve the results by about 10 F1 points (Table 1), (2) we deliberately kept the original casing of the source tokens since we found it to improve their results, and (3) during inference, we replaced generated UNK tokens with the source tokens that were given the highest attention.
For the 2-layer
BiLSTM we used embeddings of size 512 , each of the encoder and decoder had 512 cells, and the default hyperparameters of OpenNMT (Klein et al., 2017).
For the Transformer, we used their original hyperparameters (Vaswani et al., 2017).
This resulted in a Transformer model with 169 M parameters, a BiLSTM model with 134 M parameters, while our code2seq model had only 37M.
Performance Table 1 shows the results for the code summarization task.
Our model signiﬁcantly outperforms the baselines in both precision and recall across all the three datasets, demonstrating that there is added value in leveraging ASTs to encode source code.
Our model improves over the best baselines, BiLSTM with split tokens, by between 4 to 8 F1 points on all benchmarks.
The BiLSTM with split tokens consistently achieved about 10 F1 score more than BiLSTM with full tokens, and for this reason we included only a split token Transformer baseline.
Our model outperforms
ConvAttention (Allamanis et al., 2016), which was designed speciﬁcally for this task, and outperforms Paths+CRFs (Alon et al., 2018b) which used syntactic features.
An additional comparison to code2vec on the code2vec dataset can be found in Appendix A. Examples for predictions made by our model and each of the baselines can be found in Appendix C and at http://code2seq.org .
Data Efﬁciency ConvAttention (Allamanis et al., 2016) performed even better than the Transformer on the Java-small dataset, but could not scale and leverage the larger datasets.
Paths+CRFs showed very low results on the Java-small dataset, which is expected due to the sparse nature of their paths and the CRF model.
When comparing our model with the best among the baselines (BiLSTM with split tokens), we see that our model achieves a relative improvement of 7 . 3% on Java-large, but as the dataset goes smaller – the larger the relative difference becomes: 13% on Java-med and 22% on
Java-small; when comparing to the Transformer: a relative improvement of 23% on Java-large and 37% on Java-small.
These results show the data efﬁciency of our architecture: while the data-hungry
NMT baselines require large datasets, our model can leverage both small and large datasets.
Sensitivity to input length We examined how the performance of each model changes as the size of the test method grows.
As shown in Figure 4, our model is superior to all examined baselines across all code lengths.
All models give their best results for short snippets of code, i.e., belowlines of code.
As the size of the input code increases, all examined models show a natural descent, and show stable results for lengths of 9 and above.
[Figure] F [Figure] 4. C ODE C APTIONING
For this task we consider predicting a full natural language sentence given a short C# code snippet.
We used the dataset of CodeNN (Iyer et al., 2016), which consists of 66,015 pairs of questions and answers from StackOverﬂow.
They used a semi-supervised classiﬁer to ﬁlter irrelevant examples and asked human annotators to provide two additional titles for the examples in the test set, making a total of three reference titles for each code snippet.
The target sequence length in this task is about 10 on average.
This dataset is especially challenging as it is orders of magnitude smaller than the code summarization datasets.
Additionally, StackOverﬂow code snippets are typically short, incomplete at times, and aim to provide an answer to a very speciﬁc question.
We evaluated using
BLEU score with smoothing, using the same evaluation scripts as Iyer et al.
Baselines We present results compared to CodeNN, 2-layer bidirectional LSTMs with attention, and the Transformer.
As before, we provide a fair comparison by splitting tokens to subtokens, and replacing UNK during inference.
We also include numbers from baselines used by Iyer et al.
Results Table 2 summarizes the results for the code captioning task.
Our model achieves a BLEU score of 23 . 04 , which improves by 2 .
51 points (12.2% relative) over CodeNN, who introduced this dataset, and over all the other baselines including BiLSTMs and the Transformer, which achieved slightly lower results than CodeNN.
Examples for predictions made by our model and each of the baselines can be found in Appendix F.
These results show that when the training examples are short and incomplete code snippets, our model generalizes better to unseen examples than a shallow textual token-level approach, thanks to its syntactic observation of the data.
4. C ODE D OCUMENTATION
Although the task of generating code documentation is outside of the focus of this work, we performed an additional comparison to Hu et al.
They train a standard seq2seq model by using the linearized AST as the source sequence and a Javadoc natural language sentence as the target
[Table] [Table] sequence. While they originally report BLEU score of 38 .
17 , we computed their BLEU score using prediction logs were provided us by the authors and found BLEU score of 8 . 97 , which we ﬁnd more realistic.
Training our model on the same dataset as them, matching LSTM sizes
[Table]
and using the same script on our predictions yields a BLEU score of 14 . 53 , which is a 62% relative gain over the model of Hu et al.
This shows that our structural approach represents code better than linearizing the AST and learning it as a sequence.
A BLATION S TUDY
To better understand the importance of different components of our model, we conducted an extensive ablation study.
We vary our model in different ways and measure the change in performance.
These experiments were performed for the code summarization task, on the validation set of the
Java-med dataset.
We examine several alternative designs:
No AST nodes – instead of encoding an AST path using an LSTM, take only the ﬁrst and last terminal values for constructing an input vector 2.
No decoder – no sequential decoding; instead, predict the target sequence as a single symbol using a single softmax layer.
No token splitting – no subtoken encoding; instead, embed the full token.
No tokens – use only the AST nodes without using the values associated with the terminals.
No attention – decode the target sequence given the initial decoder state, without attention.
No random – no re-sampling of k paths in each iteration; instead, sample in advance and use the same k paths for each example throughout the whole training process
Table 3 shows the results of these alternatives.
As seen, not encoding AST nodes resulted in a degradation especially in the precision: a decrease of 5 .
16 compared to 4 . 30 for the recall.
It is quite surprising that this ablation was still better than the baselines (Table 1): for example, the Transformer can implicitly capture pairs of tokens using its self attention.
However, not all tokens are AST leaves .
By focusing on AST leaves, we are increasing the focus on named tokens, and effectively ignoring functional tokens like brackets, parentheses, semicolons, etc.
Transformers can (in theory) capture the same signal, but perhaps they require signiﬁcantly more layers or a different optimization to actually learn to focus on those particular elements.
The AST gives us this information for free without having to spend more transformer layers just to learn it.
Additionally, for practical reasons we limited the length of the paths to 9 .
This leads to pairs of leaves that are close in the AST, but not necessarily close in the sequence.
In contrast, the Transformer’s attention is effectively skewed towards sequential proximity because of the positional embeddings.
Using a single prediction with no decoder reduces recall by more than a third.
This shows that the method name prediction task should be addressed as a sequential prediction, despite the methods’ relatively short names.
Using no token splitting or no tokens at all drastically hurt the results, showing the signiﬁcance of encoding both subtokens and syntactic paths.
Despite the low results of no tokens , it is still surprising that the model can achieve around half the score of the full model, as using no tokens is equivalent to reasoning about code which has no identiﬁer names, types, APIs, and constant values and can be very difﬁcult even for a human.
The no attention experiment shows the contribution of attention in our model, which is very close in its relative value to the contribution of attention in seq2seq models (Luong et al., 2015; Bahdanau et al., 2014).
The no random experiment shows the positive contribution of sampling k different paths afresh on every training iteration, instead of using the same sample of paths from each example during the entire training.
This approach provides data-level regularization that gives additional improvement to a model that is very powerful already.
Another visualization can be found in Appendix D.
R ELATED W ORK
The growing availability of open source repositories creates new opportunities for using machine learning to process source code en masse.
Several papers model code as a sequence of tokens (Iyer et al., 2016; Allamanis et al., 2016; Loyola et al., 2017), characters (Bielik et al., 2017), and API calls (Raychev et al., 2014).
While sometimes obtaining satisfying results, these models treat code as a sequence rather than a tree.
This forces these techniques to implicitly re-learn the (predeﬁned) syntax of the programming language, wasting resources and reducing accuracy.
Code representation models that use syntactic information have usually been evaluated on relatively easier tasks, which mainly focus on “ﬁlling the blanks” in a given program (Alon et al., 2018b;
Bielik et al., 2016; Raychev et al., 2016; 2015; Allamanis et al., 2018) or semantic classiﬁcation of code snippets (Alon et al., 2018a).
Moreover, none of the models that use syntactic relations are compositional, and therefore the number of possible syntactic relations is ﬁxed either before or after training, and often consumes a lot of memory.
The syntactic paths of Alon et al.
(2018b;a) are represented monolithically , and are therefore limited to only a subset of the paths that were observed enough times during training.
As a result, they cannot represent unseen relations .
In contrast, by representing AST paths node-by-node using LSTMs, our model can represent and use any syntactic path in any unseen example.
Further, our model decodes the output sequence step-by-step while attending over the input paths, and can thus generate unseen sequences, compared to code2vec (Alon et al., 2018a) which has a closed vocabulary.
Oda et al.
(2015) were the ﬁrst to generate sequences by leveraging the syntax of code.
They performed a line-by-line statistical machine translation (SMT) to translate Python code to pseudocode.
Our tasks are different, and we cannot assume an alignment between elements in the input and the output; our tasks take a whole code snippet as their input, and produce a much shorter sequence as output.
Additionally, a conceptual advantage of our model over line-by-line translation is that our model can capture multiline patterns in the source code.
These multiline patterns are often very useful for the model and get the most attention (Figure 1a).
A recent work (Hu et al., 2018) generates comments from code.
There is a conceptual difference between our approaches: Hu et al.
(2018) linearize the AST, and then pass it on to a standard seq2seq model.
We present a new model, in which the encoder already assumes that the input is tree-structured.
When training our model on their dataset, we improve over their BLEU score by 62% .
Allamanis et al.
(2018) represent code with Gated Graph Neural Networks.
Nodes in the graph represent identiﬁers, and edges represent syntactic and semantic relations in the code such as “ComputedFrom” and “LastWrite”.
The kinds of edges (features) are designed for the semantics of a speciﬁc programming language, for a speciﬁc task, and require an expert to think of and implement.
In contrast, our model has minimal assumptions on the input language and is general enough not to require neither expert semantic knowledge nor the manual design of features.
Our model can therefore be easily implemented for various input languages.
Bastings et al.
(2017) used graphconvolutional networks for machine translation of natural languages.
Piech et al.
(2015) encoded code using Tree-RNNs to propagate feedback on student code; Chen et al.
(2018) used Tree-RNNs for a tree-to-tree translation of code into another programming language.
C ONCLUSION
We presented a novel code-to-sequence model which considers the unique syntactic structure of source code with a sequential modeling of natural language.
The core idea is to sample paths in the
Abstract Syntax Tree of a code snippet, encode those paths with an LSTM and attend to them while generating the target sequence.
We demonstrate our approach by using it to predict method names across three datasets of varying sizes, predict natural language captions given partial and short code snippets, in two programming languages.
Our model performs signiﬁcantly better than previous programming-language-oriented works and state of the art NMT models applied in our settings.
We believe that the principles presented in this paper can serve as a basis for a wide range of tasks which involve source code and natural language, and can be extended to other kinds of generated outputs.
To this end, we make all our code, datasets, and trained models publicly available.
A CKNOWLEDGMENTS
We would like to thank Guy Waldman for developing the code2seq website ( http://code2seq. org ), Srinivasan Iyer for the guidance in using his C# dataset, Miltiadis Allamanis, Yoav Goldberg,
Charles Sutton and the anonymous reviewers for their fruitful comments and suggestions.
The research leading to these results has received funding from the European Union’s Seventh
Framework Programme (FP7) under grant agreement no. 615688-ERC- COG-PRIME.
A A DDITIONAL EVALUATION
Comparison to code2vec on their dataset We perform an additional comparison to code2vec (Alon et al., 2018a) on their proposed dataset.
As shown in Table 4, although their high F1 score on their dataset, our model achieves even higher precision, recall and F1 on their dataset.
We believe that the reason for the lower results of code2vec on our datasets is that their dataset is split to train/validation/test by ﬁle , while in our datasets we always split by project .
In their dataset, a ﬁle can be in the training set, while another ﬁle from the same project can be in the test set.
This makes their dataset signiﬁcantly easier, because method names “leak” to other ﬁles in the same project, and there are often duplicates in different ﬁles of the same project.
This is consistent with Allamanis et al.
(2018) who found that splitting by ﬁle makes the dataset easier than by project.
We decided to take the stricter approach, and not to use their dataset (even though our model achieves better results on it), in order to make all of our comparisons on split-by-project datasets.
Data statistics Table 5 shows some statistics of our used datasets.
B C ODE CAPTIONING EXAMPLES
Figure 6 contains examples from our test set for the code captioning task in C#, along with the prediction of our model and each of the baselines.
Figure 5 shows a timestep-by-timestep example, with the symbol decoded at each timestep and the top-attended path at that step.
The width of the path is proportional to the attention it was given by the model (because of space limitations we included only the top-attended path for each decoding step, but hundreds of paths are attended at each step).
C C ODE SUMMARIZATION EXAMPLES
Figure 8 contains examples from our test set for the code summarization task in C#, along with the prediction of our model and each of the baselines.
The presented predictions are made by models that were trained on the same Java-large dataset.
D C ODE C APTIONING RESULTS
Figure 9 shows a bar chart of the BLEU score of our model and the baselines, in the code captioning task (predicting natural language descriptions for C# code snippets).
The numbers are the same as in Table 2.
E C ODE S UMMARIZATION RESULTS
Figure 10 shows a bar chart of the F1 score of our model and the baselines, in the code summarization task (predicting method names in Java).
The numbers are the F1 columns from Table 1.
F A BLATION STUDY RESULTS
Figure 11 shows a bar chart of the relative decrease in precision and recall for each of the ablations described in Section 5 and presented in Table 3.
[Figure]
node
[Figure]
[Figure]
TreeView myTreeView = new TreeView(); myTreeView.
Nodes.
Clear(); foreach(stringparentText in xml.parent) {
TreeNode parent = new TreeNode(); parent.
Text = parentText; myTreeView.
Nodes.
Add(treeNodeDivisions)
; foreach (string childText in xml.child) {
TreeNode child = new TreeNode(); child.
Text = childText; parent.
Nodes.
Add(child); }
[Table]
How can TreeView TreeView a TreeView nodes from XML parentText string to a treeview node from a TreeView parentText of a tree treeNodeDivisions from to child childText
XML node of MDI child childText created in a tree nodes in
[Table]
How to set the name of a tabPage progragmatically
[Table]
2015) how to get data from xml ﬁle in c#
[Table]
how to add child nodes to treeview
[Table]
how to add child node in treeview in c #
[Table]
How to get all child nodes in TreeView ?
[Table]
add a child node to a treeview in c #
[Table]
How into string based on an ﬁrstRow a companyName ﬁrstRow ?
How to
[Table]
Facebook C# SDK Get Current User
[Table]
how can i get the value of a string?
[Table]
) how to get the value of a cell in excel using c #
[Table]
getting the ﬁrst row in excel
[Table]
how do I get the value of an xml ﬁle in c # ? code2seq
(this work) get the value of a column in excel using c # static void Main(string[] args) { // Create an instance of Bytescout.
PDFRenderer. //
RasterRenderer object and register it.
RasterRenderer renderer = new RasterRenderer(); renderer.
RegistrationName =
"demo"; renderer.
RegistrationKey = "demo"; // Load PDF document.
renderer.
LoadDocumentFromFile("multipage.pdf");for(inti=0;i<renderer.
GetPageCount();i++) {//RenderfirstpageofthedocumenttoBMPimage file.
renderer.
RenderPageToFile(i, RasterOutputFormat.
BMP, "image" + i + ".bmp"); } //
Open the first output file in default image viewer.
System.
Diagnostics.
Process.
Start("image0.bmp"); }
[Table]
MOSES † (Koehn et al., 2007)
How to add RasterRenderer renderer
RasterRenderer renderer in a string in RegistrationName renderer Registration
Key renderer LoadDocumentFromFile in C #
How to a renderer Is a RenderPageToFile renderer in a string to BMP RasterOutputFormat each in C #
IR †
Select TOP 5 * from SomeTable, using Dataview.
RowFilter?
SUM-NN † (Rush et al., 2015)
how do i create a text ﬁle in c# 2-layer BiLSTM (split tokens)
how do i create a pdf ﬁle in c # ?
Transformer (split tokens) how to merge two pdf ﬁles ?
CodeNN † (Iyer et al., 2016)
How to get the value of an array in C # ?
code2seq
(this work) get the image from a pdf ﬁle in c # void Main() { string text =
File.
ReadAllText(@"T:\File1.txt"); int num = 0; text = (Regex.
Replace(text, "map", delegate(Match m) { return "map" + num++; }));
File
WriteAllText(@"T:\File ", text); }
Model
Prediction
MOSES † (Koehn et al., 2007)
How to File then How to HTML ?
C # How to
Write to
IR †
C# remove extra carriage returns from Stream
SUM-NN † (Rush et al., 2015)
how do i create a text ﬁle in c# 2-layer BiLSTM (split tokens) how to read a text ﬁle from a text ﬁle
Transformer (split tokens) how to write a . txt ﬁle in c #
CodeNN † (Iyer et al., 2016) how to read a text ﬁle in c # ? code2seq
(this work) replace a string in a text ﬁle void ______(Counter childCounter, String request, String requestId, long duration, boolean systemError, int responseSize) { // si je suis le counter fils du counter du contexte parent // comme sql pour http alors on ajoute la requˆete fille if (parentContext !=
null && parentCounter.getName() .equals(parentContext.getParentCounter().getChildCounterName
())) { childHits++;childDurationsSum += (int) duration; } // pour drill-down on conserve pour chaque requˆete m`ere, les requˆetes //
filles appel´ees
et le nombre d'ex´ecutions pour chacune if (parentContext == null) { addChildRequestForDrillDown(requestId); } else { parentContext.addChildRequestForDrillDown(requestId); } }
Model
Prediction
ConvAttention (Allamanis et al., 2016) add
Paths+CRFs (Alon et al., 2018b) call code2vec (Alon et al., 2018a) log response 2-layer BiLSTM (full tokens) handle request 2-layer BiLSTM (split tokens) report child request
Transformer (split tokens) add child
Gold: add child request code2seq
(this work) add child request public static int ______(int value) { return value <= 0 ?
1 : value >= 0x40000000 ?
0x40000000 : 1 << (32 - Integer.numberOfLeadingZeros(value - 1)); }
Model
Prediction
ConvAttention (Allamanis et al., 2016) get
Paths+CRFs (Alon et al., 2018b) test bit inolz code2vec
(Alon et al., 2018a) multiply 2-layer BiLSTM (full tokens) next power of two 2-layer BiLSTM (split tokens) { (replaced UNK)
Transformer (split tokens) get bit length
Gold: ﬁnd next positive power of two code2seq (this work) get power of two
BigInteger ______(int bitlength, BigInteger e, BigInteger sqrdBound) { for (int i = 0; i !
= 5 * bitlength; i++) {
BigInteger p = new BigInteger(bitlength, 1, param.getRandom()); if (p.mod(e).equals(ONE)) { continue; }if (p.multiply(p).compareTo(sqrdBound) < 0) { continue; }if (!isProbablePrime(p)) { continue; }if (!e.gcd(p.subtract(ONE)).equals(ONE))
{ continue; }return p; }throw new IllegalStateException("unable to generate prime number..
...for RSA key"); }
Model
Prediction
ConvAttention (Allamanis et al., 2016) test
Paths+CRFs (Alon et al., 2018b) i code2vec (Alon et al., 2018a) to big integer 2-layer BiLSTM (full tokens) generate prime 2-layer BiLSTM (split tokens) generate prime number
Transformer (split tokens) generate
Gold: choose random prime code2seq (this work) generate prime number public boolean ______(Set<String> set, String value) { for (String entry : set) { if (entry.equalsIgnoreCase(value)) { return true; } } return false; }
Model
Prediction
ConvAttention (Allamanis et al., 2016) is
Paths+CRFs (Alon et al., 2018b) equals code2vec (Alon et al., 2018a) contains ignore case 2-layer BiLSTM (full tokens) contains ignore case 2-layer BiLSTM
(split tokens) contains
Transformer (split tokens) contains
Gold: contains ignore case code2seq
(this work) contains ignore case
Java examples from our test set for the code summarization task, along with the prediction of our model and each of the baselines.
[Figure]
MOSES † (Koehn et al., 2007)
IR †
SUM-NN † (Rush et al., 2015) 2-layer BiLSTM (split tokens)
Transformer (split tokens)
CodeNN † (Iyer et al., 2016)
[Figure]
[Figure]
No AST nodes
No decoder
No token-split
No tokens
No attention
No random
[Figure]
Relative
Decrease (%)
Precision
Recall