L EARNING TO C OMPLETE C ODE WITH S KETCHES
A BSTRACT
Code completion is usually cast as a language modelling problem, i.e. , continuing an input in a left-to-right fashion.
However, in practice, some parts of the completion ( e.g. , string literals) may be very hard to predict, whereas subsequent parts directly follow from the context.
To handle this, we instead consider the scenario of generating code completions with “holes” inserted in places where a model is uncertain.
We develop G RAMMFORMER , a Transformer-based model that guides code generation by the programming language grammar, and compare it to a variety of more standard sequence models.
We train the models on code completion for C# and Python given partial code context.
To evaluate models, we consider both R OUGE as well as a new metric
R EGEX A CC that measures success of generating completions matching long outputs with as few holes as possible.
In our experiments, G RAMMFORMER generates 10-50% more accurate completions compared to traditional generative models and 37-50% longer sketches compared to sketch-generating baselines trained with similar techniques.
I NTRODUCTION
Recent high-capacity language models (LM) have shown that machine learning models are able to generate coherent, realistic text, but it is often hard to guide them towards a speciﬁc goal, especially when describing the intent is complex or more costly than manually generating the target output.
One such scenario are LMs of source code (LMC).
Since Hindle et al.
(2012) increasingly sophisticated LMCs have been built, including transformer-based ones, such as those of Svyatkovskiy et al.
Feng et al.
; Chen et al.
(2021) and various similar unpublished models such as TabNine and SourceAI.
These models generate full sequences of code tokens left-to-right with any preﬁx acting as the (partial) user intent.
While LMs generate realistic-looking outputs, they are known to occasionally “hallucinate” (Puduppully et al., 2019; Malmi et al., 2019; Maynez et al., 2020; Liu et al., 2021), i.e. generate plausible but incorrect content.
This is particularly problematic in generating source code, where small mistakes can lead to erroneous code that is very hard to debug or introduces vulnerabilities (Pearce et al., 2021).
In this work, we investigate models that can decline to make predictions in places where there is high uncertainty ( e.g. , where the user should choose a name), but continue generating around these “holes”.
For example, in Fig.
1(left) a developer has typed some code and is about to type the next line.
A likely completion is to consume more command line arguments, but their name is unclear
[Figure]
[Figure]
sys 2 target =
sys.argv[1]
I
Ground-Truth : ID = sys.argv[2]
Suggested Code Completions :
L → R target = target.replace("\\", "/")
L → R + ⦸ target =
L → R + (cid:4) print(target)
Copilot (No suggestion)
G [Figure] (cid:4) =
sys.argv[2] However, creating such models is not trivial. A simple ﬁrst attempt may be to use a standard LMC, but output a “hole token” (cid:4) whenever the model is uncertain about the next output token. However, continuing after the “ (cid:4) ” then becomes infeasible, as the LMC was not trained on such data.
Hence, a suitable training dataset and objective need to devised.
As no large datasets with holes exist, we instead choose to use a reinforcement learning approach in which our reward function encourages the model to make “long” predictions with as few “ (cid:4) ” tokens as possible, but to avoid making incorrect predictions. We found that standard left-to-right sequence models perform poorly on this task. Hence, we developed G RAMMFORMER , a model that construct suggestions by generating a (partial) syntax tree, but which has the option of leaving non-terminals in its output. Contributions (1) We present G RAMMFORMER , a transformer-based model that generates code based on the programming language grammar and can predict hole tokens rather than output it is uncertain about.
(2) We develop R EGEX A CC , a metric that evaluates the quality of predictions with holes.
(3) We evaluate G RAMMFORMER on Python and C
# code and show that G RAMMFORMER makes longer and more precise statement-level sketch completions compared to baselines.
M ETHOD
Our aim is to predict code completions as sketches , a mix of actual tokens and “holes” (cid:4) , which are meant to signify that the model is unable to make a useful prediction within the given context and further user input is required.
Formally, we consider models that take a context sequence x of tokens as input and have to produce an output sequence y ; intuitively, x is what the user typed so far, and y is the suggestion presented to the user.
In our setting, y is a sketch , a mix of tokens from the programming language and the special token (cid:4) signifying a “hole” that could be ﬁlled by an arbitrary sequence of tokens.
For example, t = foo( (cid:4) ) is a sketch corresponding to assigning the return value of function foo to variable t , but leaves the arguments of the function call undetermined.
Metric A good sketch is one that (a) can be completed into the correct output and (b) is as precise as possible.
To measure how successful a method is in doing so, we deﬁne a new metric R EGEX A CC .
For (a), we use toRegex ( ˆ y ) to turn a predicted code sketch ˆ y into a regular expression by replacing all holes with the wildcard matching any non-empty sequence (“ .+ ” in Perl Compatible Regular
Expression syntax).
If the regex matches the ground truth, matches( ⋅ , ⋅ ) returns a score ofotherwise it returns 0.
To implement (b), we scale this result by the proportion of terminal tokens predicted, by deﬁning nTokens ( ˆ y ) as the function that returns the number of non-hole symbols in ˆ y .
More formally, assume an output sketch ˆ y and a ground-truth sequence y ∗ , where y ∗ does not contain any (cid:4) tokens.
R EGEX
A CC is then deﬁned as
[Equation]
Beyond R EGEX A CC , we also consider ROUGE (Lin, 2004), since a sketch can be thought as a form of a “summary” of the target text.
For this, we use a helper function E RASE H OLES ( ˆ y )
that simply x ( 0 ) :
r = ⟨ Expr ⟩ i ( 0 ) =
x ( 1 ) : r = ⟨ Expr ⟩ * ⟨ ParenthesizedExpr ⟩ i ( 1 ) =
x ( 2 ) : r = ⟨ Expr ⟩ * ( ⟨ Expr ⟩ ) i ( 2 ) =
x ( 3 ) :
r = ⟨ Expr ⟩ * ( ⟨ Expr ⟩ - ⟨ Expr ⟩ ) i ( 3 )
=x ( 4 ) : r = ⟨ Expr ⟩ * ( ⟨ Expr ⟩ - ⟨ Identiﬁer ⟩ ( ⟨ ArgList ⟩ ) )
i ( 4 ) =
x ( 5 ) : r = ⟨ Expr ⟩ * ( ⟨ Expr ⟩ - foo ( ⟨ ArgList ⟩ ) )
i ( 5 ) =
x ( 6 ) : r = ⟨ Expr ⟩ * ( ⟨ Expr ⟩ - foo ( ⟨ Identifer ⟩ ) )
i ( 6 ) =
x ( 7 ) : r = ⟨ Expr ⟩ * ( ⟨ Expr ⟩ - foo ( args ) )
i ( 7 ) =
x ( 8 ) : r = ⟨ Identiﬁer ⟩ * ( ⟨ Expr ⟩ - foo ( args ) )
i ( 8 ) =
x ( 9 ) : r = x * ( ⟨ Expr ⟩ - foo ( args ) )
i ( 9 ) = ⦸ selected (underlined) non-terminal.
Fig. 5 and Fig. 6 in Appx.
A show real example generation sequences from our datasets.
drops all (cid:4) {+tokens, and then consider R OUGE F
 ( E RASE H OLES ( ˆ+} y ) , y ∗ ) . R OUGE is more lenient to errors than R EGEX A CC and gives partial credit to non-matching but plausible sketches. 2. L INEAR C ODE S KETCH G ENERATION
First, we consider the idea of generating code sketches using a standard generative model for language.
To this end, we simply extend the vocabulary with the special “ (cid:4) ” token.
An obvious problem is that while we have plenty of training data for a standard generative model, we do not have training data for outputs y that contain the (cid:4) token.
Consequently, we cannot train the model in a fully supervised fashion, and instead turn to reinforcement learning.
Concretely, we devise a reward function r ( ⋅ ) that averages R EGEX A CC and R OUGE , i.e. for a predicted output sketch ˆ y and a ground truth output (without (cid:4) tokens)
y ∗ , we deﬁne
[Equation]
Using the combination of R OUGE (which does not consider holes) and
R EGEX
A CC is crucial here, as R OUGE is much “smoother” compared to R EGEX A CC , which is 0 for all but very few predictions, allowing us to measure partial improvement.
We use our reward function from Eq.
1 to evaluate the quality of the output of the full model and compute a loss.
Inspired by Paulus et al.
(2017) we use self-critical policy gradient training (Rennie et al., 2017) and for a prediction ˆ
y we minimise
[Equation]
Here, ˜ r ( x ) is the reward achieved by the prediction from the snapshots of the model that achieved the best score so far and L gen is the loss of the generative model.
Intuitively, this objective rewards models that improve upon the previous best policy with respect to r .
To model this in practice, we use a standard encoder/decoder Transformer model Vaswani et al.
(2017); Radford et al.
(2019), “translating” the context x into the output y using separate encoder and decoder models.
We additionally also consider the language modelling case, i.e. , a model that conditioned on x predicts token y 0 , conditioned on x , y 0 predicts token y 1 , etc. .
Pretraining In practice, we found that directly training a sequence model to maximise Eq. 1, is very slow and does not converge to a useful model.
Instead, we heuristically generate a dataset suitable for supervised pretraining.
We replace random AST non-terminals of the target output by (cid:4) and generate target sequences.
These contain terminals and zero or more (cid:4) .
We then pretrain the model on this dataset to convergence, and then ﬁne-tune it using the reward of Eq. 1.
2. G RAMMAR -B ASED C ODE S KETCH G ENERATION
In experiments, we found the simple extended sequence model from above to not perform well, in particular, (cid:4) tokens would not replace semantically meaningful subsequences ( e.g. “ szconv. (cid:4) ) ”
Algorithm for t = 0 , 1 , 2 , ... do i ( t ) ∼ P s ( i ∣ x ( t ) , N ( x ( t ) ))
▷ sample non-terminal position from N ( x ( t ) ) to expand if i ( t ) =
⦸ then ▷ if x ( t ) does not contain non-terminals or none was selected by P s break ▷ stop generation u [Equation] t
[Equation] [Equation] t [Equation]
▷ sample expansion of non-terminal at position i ( t ) x
[Equation]
t [Equation] t ) ∶∶
[Equation] ( t ) >
i ( t )
▷ create x ( t + 1 ) by replacing non-terminal at i ( t ) by u ( t ) ⊚ i ( t ) return N ON T ERMINALS T O H OLES ( x ( t ) )
▷ convert remaining non-terminals to holes and return does not contain a left parenthesis and requires the user to ﬁll it in.). To resolve this, we developed G RAMMFORMER , a grammar-guided model. It generates code by following the structure of the context-free grammar (CFG) deﬁning the programming language syntax, iteratively expanding nonterminal symbols.
Crucially, it can choose to not expand some non-terminal symbols, which can then be presented as (cid:4) to users.
In traditional grammar-based generation of text (Cohen et al., 2012) or code (Maddison & Tarlow, 2014; Yin & Neubig, 2017; Allamanis & Sutton, 2014; Bielik et al., 2016), the CFG is followed by sequentially expanding the left-most, bottom-most non-terminal symbol, using one of the production rules of the grammar.
G RAMMFORMER changes this and instead selects the non-terminal symbol to expand, if any. An example generation is shown in Fig. Probabilistic Model A CFG is deﬁned as a tuple ( Σ , N , S, R ) where Σ is a set of terminal symbols, N is a set of non-terminal symbols, S ∈ N is the root symbol and R is a set of production rules. We denote non-terminals as ⟨ NonTerminalName ⟩ .
G RAMMFORMER can be viewed as a sequenceto-sequence model transforming x = x 0 , x 1 , ..., x n into a new sequence in which one non-terminal symbol x i has been replaced by a new sequence of new symbols, according to a production rule of the grammar.
Examples of such sequences and rewrites are shown in Fig.
G RAMMFORMER does this rewriting in two steps.
First, a non-terminal selector model P s selects a non-terminal in x to expand and then the non-terminal expansion model P e determines how to expand it.
To deﬁne P s , let N ( x ) =
{ i ∣ x i ∈ N } ∪ { ⦸ } denote the set of non-terminal positions in x and a special “stop expansion” ⦸ symbol.
Conditioned on x , P s produces a probability distribution over N ( x ) .
In turn, P e is conditioned on x and a position i ∈ N ( x ) and models a probability distribution over expansion sequences u ∈ ( Σ ∪ N ) ∗ .
Note that factorising G RAMMFORMER into two models P s and P e is an important modelling decision: how to best expand a non-terminal is entirely separated from predicting whether a hole should be introduced.
These two concepts are intermixed in standard (sequence) decoders.
In practice, we deﬁne both models using neural architectures with partially shared parameters, as discussed below.
Alg. 1 shows a high-level description of G RAMMFORMER , in which P s and P e are used repeatedly to select and expand non-terminals (not necessarily the left-most one), until none are left or the P s indicates that expansion should stop.
Here, N ON T ERMINALS T O H OLES ( ⋅ ) replaces all remaining non-terminal symbols with a hole (cid:4) .
Note that G RAMMFORMER is not context-free, taking into account the whole input sequence when expanding a non-terminal.
Second, in contrast to many grammar-based methods (Yin & Neubig, 2017; Bielik et al., 2016), any non-terminal can be expanded at each step.
Finally, P e is not directly constrained to follow the production rule set R , but can generate any sequence.
In practice, it learns to follow to the rules of R from the data, but this ﬂexibility is important for handling string literals and argument tuples of variable length.
Neural Model To implement P s and P e , we use a shared encoder module that computes a representation of the input sequence x = x 0 , . . .
, x n as vectors e 0 , . . .
, e n , e i ∈ R D , where D is a hyperparameter.
Our encoder module is a Transformer (Vaswani et al., 2017), given the impressive results of transformer-based models in NLP and code (Feng et al., 2020).
Other architectures (RNNs, 1D-CNNs, Transformer variants) would be suitable, but we leave their study for future work.
P s is implemented similar to a pointer network on top of this encoder module, i.e.
[Equation]
where f is a learnable feed-forward neural network.
For our purposes, we deﬁne e ⦸ as the representation of the special start symbol [CLS] used in our Transformer encoder.
The expansion model P e follows a standard autoregressive decoder formulation, i.e.
[Equation]
We implement P dec as a (causal) relational Transformer decoder, similar to Wang et al.
Relational transformers augment the attention mechanism by incorporating predeﬁned relationships among elements; attention scores are then biased by learnable weights for each relation.
In G RAMMFORMER , we only use a single relation, connecting each token to the expanded non-terminal token x i , to help the model focus on the token it needs to generate an expansion for.
Objective Due to the lack of supervised data, we employ reinforcement learning to train G RAMMFORMER .
We use our reward function from Eq.
1 to evaluate the quality of the output of the full model.
We use self-critical policy gradient training as in Eq.
2 and minimise
L
[Equation]
[Equation]
Here, ˜ r ( x ) is the reward achieved by the snapshots of P s and P e that achieved the best score so far.
The rest of the objective follows the iterations of the loop in Alg. 1, where t is the iteration index, ˆ y is the predicted sketch, y ∗ is the ground-truth sequence of terminals, and I ( ⋅ ) is the indicator function.
Pretraining As in the sequence model, directly training with the RL objective Eq.
3 is computationally intensive due to the sampling requirement.
We again use a pretraining strategy.
First, we train P e to expand every non-terminal, independently of the expansion order learned by P s .
To do this, we use the input training examples and follow Alg. 1, but instead of sampling from P s ( ⋅ ) , we sample i ( t ) from a uniform distribution over the non-terminals in x ( t ) , ̃ N ( x ( t ) )
= { i ∣ x i ∈ N } .
This yields sequences of intermediate sketches x ( t ) for each example.
Furthermore, for each x ( t ) , we compute the ground-truth expansion ( u ( t ) ⊚ i ) ∗ for all non-terminals
i ∈ ̃
N ( x ( t ) ) .
We can then pretrain P e using the supervised objective
[Equation]
i.e. the negative log-likelihood of the correct expansion for all non-terminals in x ( t ) .
This computation is more computationally efﬁcient compared to the one in Eq.
3 since the cost of encoding x
( t ) is amortised across all potential expansions and no sampling is required.
Once P e is pretrained, we pretrain P s .
For this, we ﬁx the weights of the shared encoder module, and optimise only the remaining parameters of P s through Eq.
Once we have a pretrained both models, we then ﬁne-tune all model weights end-to-end, using Eq. 3.
Optimisation:
Grammar Flattening
Following the formal grammar of a programming language commonly introduces tedious expansions.
For example, the Python non-terminal ⟨ Call ⟩ is always expanded to ⟨ Expr ⟩ ( ⟨ ArgumentList ⟩ ) , and the C# non-terminal ⟨ NotEqualOp ⟩ is always expanded to the terminal !
= .
We “ﬂatten” the grammar by replacing non-terminals such as ⟨ Call ⟩ and ⟨ NotEqualOp ⟩ with all their possible expansions.
In Appx.
C we provide the list of the ﬂattened non-terminals.
Note that if we repeated this process for all non-terminals except from the starting symbol S , G RAMMFORMER would degenerate into a standard encoder-decoder model.
Beam Search At test time, we employ a two-step beam search, and replace sampling from P s and P e with their top- ν outputs, keeping a beam of size k .
First, for each x ( t ) in the beam, we compute P s and select the top- m non-terminal positions to expand.
For each of those m positions, we sample the top- n expansions from P e using a standard beam search.
We compute the likelihood of all k ⋅ n ⋅ m results, and then keep only the top- k .
This process (detailed in Appx.
E) is similar to a standard beam search but takes into account that two submodels are used.
Computational Cost
G RAMMFORMER ’s ability to predict sketches comes with additional computational cost compared to standard transformer encoder-decoders: at each iteration of the loop in Alg.
1 x ( t ) changes, P s and P e must be recomputed.
This means that the encoder-decoder runs once on each partial sequence, in contrast to left-to-right causal generation, in which intermediate results can be re-used.
Future work may consider selecting more than one element to expand from N ( x ( t ) ) at each step, reducing the expansion steps, similar to Welleck et al.
; Stern et al.
E VALUATION
To empirically evaluate our model’s ability to predict useful completions, we use R EGEX A CC and
R OUGE .
Note that we measure these on the generated sequence, i.e. we ignore the context tokens.
Datasets To collect a dataset, we clone all non-fork repositories with more than 20 stars on GitHub that have C# or Python as their top language.
Then, we deduplicate the corpus using the method of Allamanis (2019); Lopes et al.
Finally, we parse all ﬁles into a syntax tree using Treesitter, ignoring any ﬁles that cannot be parsed using the v0.19.0 grammar deﬁnitions.
Finally, we split the ﬁles into 70-10-20 train-validation-test.
To create (pre-)training examples, i.e. inputs to
Alg. 1, we search the syntax tree of each ﬁle and for each ⟨ SimpleStatement ⟩ non-terminal create an example.
The syntax tree rooted at the ⟨ SimpleStatement ⟩ non-terminal is then used to get the ground-truth expansions during pre-training and the ground-truth expansion y ∗ .
For our test set, we randomly sample a ⟨ SimpleStatement ⟩ non-terminal for each ﬁle to evaluate and obtain 318K (resp.
362K) examples for C# (resp.
Python).
For each example, x is the 200 terminal tokens before the ⟨ SimpleStatement ⟩ non-terminal.
More details about the dataset can be found in Appx.
B.
Baselines Since we are not aware of any prior model that targets code completion with sketches, we consider two Transformer-based baselines.
We consider both the sequence-to-sequence setting using separate encoder and decoder models (Vaswani et al., 2017) as well as the language modelling setting (where there is no distinction between encoder and decoder).
We refer to these as “ L → R ” and “ LM ”.
We use “ L → R ” to denote a standard Transformer encoder-decoder model (Vaswani et al., 2017) used in sequence-to-sequence tasks.
Additionally, we consider “ L → R + ⦸ ” and “ LM +⦸ ”, which are trained to stop generation by inserting a ﬁnal (cid:4) token that captures any sufﬁx.
Note that this models can only generate sketches that are preﬁxes of the target completion, i.e. it corresponds to a standard token-level generative model with a learnable stopping ability.
To train this model, we use self-critical policy gradient training (as in Eq. 2).
Model Training We provide the training details for all experiments.
Most of our models use a 6-layer Transformer as encoder and 6-layer Transformer as decoder, each with a hidden dimension of 768 and 12 attention heads, with the exception of the LM model (and its variations), which uses a single 12-layer Transformer, to match the number of parameters of the other models.
We set the intermediate dimension of each Transformer layer as 3072 and use 3 fully-connected layers with 3072, 768 and 1 hidden sizes as the feed-forward neural network f in the selector model P s .
The vocabulary is constructed using byte-pair encoding (Sennrich et al., 2015) and the vocabulary size is 25 000 .
We set max length of input and output sequences as 512 and 64, respectively.
We train the model with Adam optimiser using a learning rate of 2e-5 and batch size 4 096 .
We used automatic mix precision.
Training used 64 NVIDIA Tesla P100 with 16GB memory for 10 days.
For beam search we use k = 5 , n = 1 and m = ∞ , i.e. we consider all non-terminals in each x ( t ) .
We selected these during early experiments as a reasonable trade-off between speed and predictive performance.
Results Tbl.
1 shows the results for all considered models.
For both Python and C#, G RAMMFORMER outperforms the baseline methods in terms of R EGEX A CC , showing that the grammarbased generation can create better sketches compared to simpler methods.
Note that although L → R has a comparable or better R OUGE score, it does substantially worse than G RAMMFORMER with respect to R EGEX A CC , meaning that the predictions are “similar” but the sketches contain errors ( i.e. do not match the ground-truth).
This means that if a code completion system suggested the full output of L → R , the user would have to pause and correct the suggestion more frequently.
On the other hand, L → R + ⦸ improves over L → R in terms of R EGEX A CC but has a worse R OUGE and generates signiﬁcantly shorter suggestions (5.3 vs. 7.5 tokens-long for C#).
This is expected since
[Table] [Figure] L → R + ⦸ is trained to be more “conservative” ( i.e. avoid incorrect suggestions) but is also unable to introduce holes beyond the last generated token.
Finally, we can see that while G RAMMFORMER already performs well after our pre-training procedure, we can further improve its performance with our ﬁne-tuning technique.
We believe that this is because when P s and P e are trained jointly, they co-adapt: some of the capacity of the shared encoder module that is used to make predictions for hard-to-expand non-terminals is “freed” since P s learns to not expand them.
Fig.
3 shows how the length of the generated code sketch relates to the length of the ground truth expression.
While the differences between models are small for short target sequences, G RAMMFORMER generates substantially longer suggestions than other models when more complex suggestions are required.
In particular, the L → R + ⦸ model generates very short suggestions, as it is trained to stop generation whenever it reaches a point at which it is uncertain about the next token.
Fig.
4 in turn shows how often the suggested sketch was correct dependent on the length of the ground truth token sequence.
Here, L → R +⦸ does best because it generates the shortest ( i.e. , least determined) predictions, which is exactly the trade-off captured by our R EGEX A CC metric.
Of the models that generate longer suggestions, G RAMMFORMER clearly does best, with the improvement becoming more pronounced with the length of the target sequence.
Note that the performance of the models on C# is generally better compared to the performance in Python.
We believe that this has to do with the grammar of each language and the patterns it induces within the developer’s code.
Casalnuovo et al.
; Karampatsis et al.
(2020) have observed a similar phenomenon on the perplexity
Ablations Next, we look into ablations of G RAMMFORMER and reason about how its components perform.
To this end, Tbl. 2 shows the performance of different model variants on the C# dataset.
First, we analyse the effect of the selector model P s .
To this end, we consider two ablations.
The ﬁrst is the “random expansion” model, in which the non-terminal token to expand is sampled uniformly at random from the full set of non-terminal symbols, and which hence does not stop expansion as long as any holes are remaining.
This is effectively G RAMMFORMER after our pre-training procedure for P e .
This model achieves the best R OUGE score, but a relatively bad R EGEX A CC , as it is forced to generate a prediction even when it is very uncertain. The second ablation, a “ﬁxed threshold” [Figure] u r a c
[Figure]
[Table] model is similar to our ﬁrst ablation [Table] but stops expansion when the probability of the generated x ( t ) falls below a threshold.
We choose this threshold on the validation set.
This model makes shorter, but more accurate sketch predictions compared to the “random expansion” [Table] but is worse than G RAMMFORMER .
These two ablations demonstrate that our learned P s is useful.
Second, we consider the effect of using different reward functions r ( ⋅ ) in the training of G RAMMFORMER .
Concretely, whereas we use the mean of R EGEX A CC and R OUGE in G RAMMFORMER , we now consider using only a single of these two metrics.
As expected, the results on the corresponding metric improve, with a substantial cost in the other metric.
Concretely, using only R EGEX A CC leads to signiﬁcantly shorter predictions with a low R OUGE score.
We believe that this is because
R EGEX
A CC is a strict metric, returning
if the sketch does not match, which leads to sparse rewards and makes the resulting model more conservative at expanding non-terminals.
Finally, to evaluate the beneﬁt of the grammar-guided decoder, we consider a variant of G RAMMFORMER that does not allow the introduction of (cid:4) and instead has continue expansion until no non-terminals exist anymore.
This variant can be compared to L → R , which also cannot stop or introduce (cid:4) tokens.
Our ablation shows that there is substantial beneﬁt in using grammar-guided decoding, leading both to longer predictions as well as more correct ones.
3. Q UALITATIVE E VALUATION
Having observed the quantitative results, we now turn our attention to a qualitative look at the results and show some cherry-picked examples that illustrate desired and undesired behaviours of
G RAMMFORMER and the baselines, where we also include the suggestions of the GitHub Copilot system GitHub (2021).
Fig.
1 shows an example and eleven more are shown in Appx.
A. Fig.illustrates the importance of generating sketches instead of concrete sequences of terminal tokens: oftentimes, the code context does not provide sufﬁcient information about the user’s intent.
Sketchgenerating models can offer more informative suggestions given the partial intent.
Of course, G RAMMFORMER also makes mistakes.
For example, G RAMMFORMER and L → R + ⦸ are sometimes “too” conservative ( e.g. Fig.
15 in Appx.
A) generating holes where L → R generates fully concrete completions.
This suggests future research opportunities for better calibration of P s .
Finally, a pure language modelling approach to code completion will always be insufﬁcient.
For example, user-deﬁned types and rare APIs cannot be predicted by a language model, since‘’ the APIs cannot be known during training (Fig. 7 and Fig.
17 in Appx.
A).
Researching methods to scalably introduce information from static analyses and additional context may alleviate this.
R ELATED W ORK
One of the successful applications of LMCs is code completion (Svyatkovskiy et al., 2019; Karampatsis et al., 2020).
Transformer LMs have shown exceptional performance at the task being able to predict relatively long code sequences (Svyatkovskiy et al., 2020; Chen et al., 2021).
Grammarbased code completion and generation has been researched with neural (Maddison & Tarlow, 2014; Yin & Neubig, 2017; Kim et al., 2021) and non-neural models (Bielik et al., 2016), always expanding the left-most, bottom-most non-terminal.
In contrast to G RAMMFORMER s, all these models target the generation of complete code without the ability to create sketches.
R3NN (Parisotto et al., 2017) generates only complete programs of a simple string transformation DSL but expands the non-terminal with the highest conﬁdence, instead of the left-most, bottom-most one, similar to
G RAMMFORMER .
In contrast to the aforementioned models, G RAMMFORMER does not maintain an explicit tree representation but instead uses the sequences of leaves in the generation tree.
Sketch-like ideas appear in NLP such as the coarse-to-ﬁne semantic parsing of Dong & Lapata (2018) and chat-bots of Shum et al.
However, sketches are extracted deterministically to create a supervised dataset.
Similarly, SketchAdapt (Nye et al., 2019) uses a sequence model to generate sketches for small functional programs of a simple DSL towards speeding-up enumerative program synthesis from input-output examples.
SketchAdapt is also trained as a supervised sketch generator.
A supervised corpus is created by enumerating all possible sketches and selecting the one with the highest-probability and within a heuristically computed time budget.
In G RAMMFORMER domain, enumerating all sketches is computationally intractable due to the complexity of generalpurpose programming languages while no similar heuristic exists for code completion.
Recently, sequence generation approaches beyond the left-to-right paradigm have been proposed (Welleck et al., 2019; Stern et al., 2019;
Gu et al., 2019; Ford et al., 2018; Lee et al., 2018; Shah et al., 2018), usually by considering generation as an procedure that iteratively changes or extends a sequence.
These models often aim in speeding-up inference or allowing models to ﬁgure a better order for full sentence generation.
However, since these models focus on natural language and since its grammar is not deﬁned a priori, these methods do not follow a grammar that limits the space for sketch generation.
Additionally, these generate full utterances of text, rather than sketches.
A related concept is learning to abstain (Ziyin et al., 2019) where a model learns to predict a “don’t know”.
This resembles the stop symbol “ ⦸ ” with the difference that G RAMMFORMER employs RL to learn P s for a sequential problem rather than learning to abstain for a single-step classiﬁcation.
D ISCUSSION & C ONCLUSIONS
In this work, we presented G RAMMFORMER , a generative model of code that goes beyond standard left-to-right generation and is able to generate sketches, i.e. snippets of code with holes.
Designing generative machine learning models with such abilities is important towards facilitating better collaboration between machine learning models and their human users.
While we showed that G RAMMFORMER performs better than alternatives in sketch generation, there are still many research opportunities.
First, larger transformers will most probably yield better results, as shown in the literature.
Second, although we used R EGEX A CC as an evaluation metric, human studies to evaluate it are needed.
Such studies, similar to those in machine translation and summarization, can yield more informed r ( ⋅ ) and improved user experiences.
Second, although we focused on programming languages, modelling natural language also seems possible.
Finally, we treated programming languages as a sequence of terminals and non-terminals, ignoring the structure imposed by code’s semantics, e.g. data and control ﬂow.
Explicitly providing x‘the code’s structure, e.g. with relational transformers (Hellendoorn et al., 2019) may further improve G RAMMFORMER . A CKNOWLEDGMENTS
The authors would like to thank Alex Polozov for useful discussions. We also thank Patrick Fernandes, Szymon Malik and Guilherme Ilunga for working on earlier modeling ideas on sketch generation. Although those were unsuccessful, they provided the inspiration for this work
A G ENERATED S AMPLES
Fig.
5 and Fig.
6 show two examples from our dataset along with the ground-truth and the sequence of expansions performed by G RAMMFORMER .
Fig.
7-13 show example generations by G RAMMFORMER and the baseline models L → R and L → R + ⦸ .
The parentheses in red indicate the R EGEX A CC score for each suggestion.
For the L → R + ⦸ baseline the special non-terminal <suffix> is added to indicate that a hole is introduced at the end of the left-to-right generation.
Finally Fig.
15-17 show example generations where G RAMMFORMER make mistakes.
A discussion for each of those sample is found at the caption of each ﬁgure.
Context: using System.
Collections; using System.
Collections.
Generic; using UnityEngine; public class BottleFlipAcademy: Academy{ public float MaxDistance; public float MinScale; public bool IsRandomDirection; public override void AcademyReset(){
MaxDistance =
resetParameters["max_distance"]; <expression_statement>
Ground Truth:
MinScale = resetParameters["min_scale"];
Prediction:
MinScale =
resetParameters[<string_literal>];
Generation Process: <expression_statement> <left><assignment_operator> <right> ; <left> = <right> ; <identifier>
= <right> ; <identifier> = <expression> [ <argument> ] ; <identifier>
= <identifier>
[ <argument> ] ; <identifier> = resetParameters [ <argument> ] ; <identifier> = resetParameters [ <string_literal> ] ;
MinScale = resetParameters [ <string_literal> ] ; , instead of predicting a low-likelihood string literal.
B D ATASET S TATISTICS
Some statistics about the datasets used throughout this work are shown in Tbl.
C F LATTENED N ON -T ERMINALS
The non-terminals in Tbl.
4 are always expanded and are not considered as non-terminals.
Most of these non-terminals have always the same children (terminals or non-terminals), representing a single CFG rule.
By ﬂattening those non-terminals the depth of tree is reduced (and hence the number of loops needed in Alg. 1).
Context: import sys import os import platform if platform.system() =
=
"Linux": os.system('clear') elif platform.system() =
= "Windows": os.system('cls') target =
sys.argv[1] <expression_statement>
Ground Truth:
ID = sys.argv[2]
Prediction: <identifier> = sys.argv[2]
Generation Process: <left> = <right> <left> = <subscript> <left> = <value> [ <subscript> ] <identifier> = <value> [ <subscript> ] <identifier> = <attribute> [ <subscript> ] <identifier> = <object> .
<attribute> [ <subscript> ] <identifier>
= <object> .
<identifier> [ <subscript> ] <identifier>
= <object> .
argv [ <subscript> ] <identifier> = <object> .
argv [ <integer> ] <identifier> = <object> .
argv [ 2 ] <identifier> = <identifier> .
argv [ 2 ] <identifier> = sys .
argv [ 2 ]
Python
C#
Num Training Files/Trees
Num Validation Files/Trees
Num Test Files/Trees
[Table]
D U NDERSTANDING R EGEX A CC
Since R EGEX A CC is a new metric, we include two deterministic ways of introducing sketches in
Tbl. 5.
First, if all literals (strings, numeric) are replaced with a hole, we see that a high R EGEX A CC
Context: ...
ExchangeActivity =(ExchangeActivity) Singleton<
ActivitySys>.GetInstance().GetActivity(
COM_WEAL_TYPE.COM_WEAL_EXCHANGE, msg.stPkgData.stWealExchangeRes.dwWealID) ; if ( exchangeActivity != null ){ exchangeActivity.
IncreaseExchangeCount( (int)msg.stPkgData.stWealExchangeRes.bWealIdx, msg.stPkgData.stWealExchangeRes.dwDrawCnt);<expression_statement>
Ground Truth: exchangeActivity.
UpdateView();
Prediction: 𝑳 → 𝑹 :
Singleton<CUIManager>.GetInstance().CloseSendMsgAlert(); (0.000) 𝑳 → 𝑹 + ⦸ :
Singleton<<suffix> (0.000) 𝑳 → 𝑹 ∪ ∎ : exchangeActivity.<hole>(); (0.833) 𝑪𝒐𝑷𝒊𝒍𝒐𝒕 : } (0.000)
𝑮𝒓𝒂𝒎𝒎𝑭𝒐𝒓𝒎𝒆𝒓 : exchangeActivity.<identifier>(); (0.833) is achieved.
In contrast, replacing both identiﬁers and literals ( “just” parentheses,
we get an easy “lower-bound”.
how # — which is syntactically more verbose — achieves a better score, compared to Python.
In Tbl. 6, we show some example sketches and their associated R EGEX A CC score.
E B EAM SEARCH
Alg.
2 presents the beam search used in G RAMMFORMER .
Context: ...
[Test] public void CanPassTwoProviders( ){ // arrange var expectedLength = 100; var input1 = new TestSampleProvider(44100, 2, 50); var input2 = new TestSampleProvider(44100, 2, 50); var concatenator = new ConcatenatingSampleProvider(new[]{input1, input2}); var buffer = new float[2000]; var read = concatenator.
Read(buffer, 0, buffer.
Length);
Assert.
AreEqual(expectedLength, read, "read == expectedLength"); Assert.
AreEqual(49, buffer[49]); <expression_statement>
Ground Truth:
Assert.
AreEqual(0, buffer[50]);
Prediction: 𝑳 → 𝑹 : Assert.
AreEqual(50, buffer[50]); (0.000) 𝑳 → 𝑹 + ⦸ :
Assert.
AreEqual(<suffix> (0.333) 𝑳 → 𝑹 ∪ ∎ :
Assert.
AreEqual(expectedLength, read);
(0.00) 𝑪𝒐𝑷𝒊𝒍𝒐𝒕 :
Assert.
AreEqual(50, buffer[50]);
(0.000) 𝑮𝒓𝒂𝒎𝒎𝑭𝒐𝒓𝒎𝒆𝒓 :
Assert.
AreEqual(<argument>, buffer[50]); (0.917) b ← {( , false )} ▷ Initialize Beam (state, logprob, isDone) while ∃ ( x , p, isDone )
∈ b with isDone =
false do ▷ While beam contains incomplete generations
b ′ ← {} for ( x , p, isDone )
∈ b do ▷
For each sample in beam if isDone then ▷ If suggestion is complete b ′ ← b ′ ∪ {( x, p, isDone )}
▷ No operation, beam is complete continue for i ∈ T OP M
( P s ( i ∣ x , N ( x )))
do ▷ Get top- m non-terminal positions p s ← log P s ( i ∣ x , N ( x )) if i = ⦸ then
b ′ ← b ′ ∪ {( x, p + p s , true )} ▷ Stop Expansion else for y ∈ T OP N ( P e ( y ∣ x , i ) )
do ▷ Beam search on y yields n candidates p e ← P e ( y ∣ x , i ) b ′ ← b ′ ∪ {( x < i ∶∶ y ∶∶ x > i ) , p + p s + p e , false )}
▷ Expand x i b ← T OP K ( b ′ )
▷ Prune Candidates and keep top k return b
Context: ...
namespace Bug604053.Prueba{ public class Data{ public int M1{get; set;} public string M2{get; set;} public Data(int m1, string m2){M1 = m1; M2 = m2;} } [DataObject(true)] public class DataSource{ public Data[] Retrieve( ){
Data[] data = new Data[10]; for (int i = 0; i<10; i++){ <expression_statement>
Ground Truth: data[i] = new Data(i, i.ToString());
Prediction: 𝑳 → 𝑹 : data[i] = new Data( ); (0.000) 𝑳 → 𝑹 + ⦸ : data[i] = new Data( ); (0.000) 𝑳 → 𝑹 ∪ ∎ : data[i] = new Data( ); (0.000) 𝑪𝒐𝑷𝒊𝒍𝒐𝒕 : data[i] = new Data(i, "Data" + i); (0.000)
𝑮𝒓𝒂𝒎𝒎𝑭𝒐𝒓𝒎𝒆𝒓 : data[i] = new Data(i, <string_literal>); (0.706)
Context: # Provides a character-based width estimate when simple tags # such as <b> and <i> are present in a multi-line, # \"break\"-delimited, string.
Very approximate, but a useful # default.
def htmlWidth(sIn):iBr=indexOfBr(sIn) if(-1==iBr): s=sIn else: s=sIn[:iBr]<return_statement>
Ground Truth: return len(s)
Prediction: 𝑳 → 𝑹 : return len(s) (1.000) 𝑳 → 𝑹 + ⦸ : return <suffix> (0.200) 𝑳 → 𝑹 ∪ ∎ : return s (0.000)
𝑪𝒐𝑷𝒊𝒍𝒐𝒕 : s = s.replace("&nbsp;" , " ") (0.000)
𝑮𝒓𝒂𝒎𝒎𝑭𝒐𝒓𝒎𝒆𝒓 : return len(s) (1.000)
Context: #!/usr
/bin/env python from __future__ import print_function import argparse import os import subprocess import sys ap = argparse.
ArgumentParser() ap.add_argument("--release", action = "store_true") ap.add_argument("--prerelease", action = "store_true") <
expression_statement>
Ground Truth: ap.add_argument("--experimental", action = "store_true")
Prediction: 𝑳 → 𝑹 : args = ap.parse_args() (0.000) 𝑳 → 𝑹 + ⦸ : args = ap.parse_args() (0.000) 𝑳 → 𝑹 ∪ ∎ : args = ap.add_argument(<hole> (0.000) 𝑪𝒐𝑷𝒊𝒍𝒐𝒕 : ap.add_argument("--beta", action = "store_true") (0.000)
𝑮𝒓𝒂𝒎𝒎𝑭𝒐𝒓𝒎𝒆𝒓 : ap.add_argument(<string>, action = "store_true") (0.833)
Python block , tuple , and , or , + , - , * , / , & , || , // , % , @ , += , -= , *= , /= , //= , @= , &= , |= , call , keyword_argument , name , binary_operator , for_in_clause , unary_operator , ** , true , not_operator , none , false , boolean_operator , augumented_assignment , await , >> , pair , | , parameters , << , dictionary_comprehension , ellipsis , arguments , assignment , ^ , ~
C# block , tuple , and , or , + , - , * , / , & , || , // , % , @ , += , -= , * = , /= , //= , %= , @= , &= , |= , ** , >> , | , << , ^ , ~ , assignment_expression , invocation_expression , arguments , member_access_expression , try_statement , catch_clause , conditional_expression , =
= , array_type , rank , base_expression , conditional_access_expression , member_binding_expression , initializer , null_literal , > , element_access_expression , subscript , ??
, this_expression , implicit_array_creation_expression , cast_expression , !
= , variable_declaration , implicit_type , && , as_expression , as , < , local_declaration_statement , if_statement , >= , <= , throw_expression , default_expression , pattern , is_pattern_expression , binary_expression , bracketed_argument_list , name , object_creation_expression , await_expression , ,
Context: import sys import os import platform if platform.system() =
=
"Linux": os.system('clear') elif platform.system() =
= "Windows": os.system('cls') target =
sys.argv[1] <expression_statement>
Ground Truth:
ID = sys.argv[2]
Prediction: 𝑳 → 𝑹 : target =
target.replace("\\\\", "/") (0.000) 𝑳 → 𝑹 + ⦸ : target = <suffix> (0.000) 𝑳 → 𝑹 ∪ ∎ : print(target) (0.000) 𝑪𝒐𝑷𝒊𝒍𝒐𝒕 :
No Suggestion (0.000) 𝑮𝒓𝒂𝒎𝒎𝑭𝒐𝒓𝒎𝒆𝒓 : <identifier> = sys.argv[2] (0.875 )
C#
Python
Replace all literals with holes 0.
Replace all literals and identiﬁers with holes 0.
Ground-truth ap.add_argument("--experimental", action="store_true")
R EGEX A CC ap.add_argument( (cid:4) , action="store_true") 0.
ap.add_argument( (cid:4) , action= (cid:4) ) 0.
ap.add_argument( (cid:4) , (cid:4) ) 0.
ap.add_argument( (cid:4) , action="store_false") 0.
ap.add_argument( (cid:4) , required= (cid:4) ) 0.
[Figure]
[Figure]
void OnGUI(){ if(!props.wrongActionShowFrame)return; previousColor=GUI.color;GUI.color=
config.borderColor;
GUI.DrawTexture(left,config.texture);GUI.DrawTexture(right,config.texture); <expression_statement>
Ground Truth: GUI.DrawTexture(top, config.texture);
Prediction: 𝑳 → 𝑹 :
GUI.color = previousColor; (0.000) 𝑳 → 𝑹 + ⦸ : GUI.color = previousColor; (0.000) 𝑳
→ 𝑹 ∪ ∎ :
GUI.color = Color.white; (0.000) 𝑪𝒐𝑷𝒊𝒍𝒐𝒕 : GUI.DrawTexture( [Figure]
, config.texture); (1.000) 𝑮𝒓𝒂𝒎𝒎𝑭𝒐𝒓𝒎𝒆𝒓 :
[Figure]
[Figure] [Figure] [Figure]