SCELM O : S
OURCE C
ODE E MBEDDINGS
FROM L ANGUAGE M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Continuous embeddings of tokens in computer programs have been used to support a variety of software development tools, including readability, code search, and program repair.
Contextual embeddings are common in natural language processing but have not been previously applied in software engineering.
We introduce a new set of deep contextualized word representations for computer programs based on language models.
We train a set of embeddings using the ELMo (embeddings from language models) framework of Peters et al (2018).
We investigate whether these embeddings are effective when ﬁne-tuned for the downstream task of bug detection.
We show that even a low-dimensional embedding trained on a relatively small corpus of programs can improve a state-of-the-art machine learning system for bug detection
I NTRODUCTION
Learning rich representations for source code is an open problem that has the potential to enable software engineering and development tools.
Some work on machine learning for source code has used hand engineered features (Long &
Rinard, 2016, e.g.), but designing and implementing such features can be tedious and error-prone.
For this reason, other work considers the task of learning a representation of source code from data (Allamanis et al., 2018a).
Many models of source code are based on learned representations called embeddings, which transform words into a continuous vector space (Mikolov et al., 2013).
Currently in software engineering (SE) researchers have used static embeddings (Harer et al., 2018;
White et al., 2019; Pradel & Sen, 2018), which map a word to the same vector regardless of its context.
However, recent work in natural language processing (NLP) has found that contextual embeddings can lead to better performance (Peters et al., 2018;
Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019).
Contextualized embeddings assign a different vector to a word based on the context it is used.
For NLP this has the advantage that it can model phenomena like polysemy.
A natural question to ask is if these methods would also be beneﬁcial for learning better SE representations.
In this paper, we introduce a new set of contextual embeddings for source code.
Contextual embeddings have several potential modelling advantages that are speciﬁcally suited to modelling source code:
• Surrounding names contain important information about an identiﬁer.
For example, for a variable name, surrounding tokens might include functions that take that variable as an argument or assignments to the variable.
These tokens provide indirect information about possible values the variable could take, and so should affect its representation.
Even keywords can have very different meanings based on their context.
For instance, a private function is not the same as a private variable or a private class (in the case of Java / C++).
• Contextual embeddings assign a different representation to a variable each time it is used in the program.
By doing this, they can potentially capture how a variable’s value evolves through the program execution.
• Contextual embeddings enable the use of transfer learning.
Pre-training a large neural language model and querying it for contextualized representations while simultaneously ﬁne-tuning for the speciﬁc task is a very effective technique for supervised tasks for which there is a small amount of supervised data available.
As a result only a small model needs to be ﬁne-tuned atop the pre-trained model, without the need for task-speciﬁc architectures nor the need of training a large model for each task separately.
In this paper, we highlight the potential of contextual code embeddings for program repair.
Automatically ﬁnding bugs in code is an important open problem in SE.
Even simple bugs can be hard to spot and repair.
A promising approach to this end is name-based bug detection, introduced by DeepBugs (Pradel & Sen, 2018).
The current state-of-the-art in name-based bug detection relies on static representations from Word2Vec (Mikolov et al., 2013) to learn a classiﬁer that distinguishes correct from incorrect code for a speciﬁc bug pattern.
We introduce a new set of contextualized embeddings for code and explore its usefulness on the task of name-based bug detection.
Our method signiﬁcantly outperforms DeepBugs as well as other static representations methods on both the DeepBugs dataset as well as a new previously unused test set of JavaScript projects.
We release our implementation and representations as they could lead to improvements in a great variety of SE tasks.
R ELATED W ORK
Unsupervised static word embeddings have been extensively used to improve the accuracy of supervised tasks in NLP (Turian et al., 2010).
Notable examples of such methods are Word2Vec
(Mikolov et al., 2013) and GloVe (Pennington et al., 2014).
However, the above models learn only a single context-independent word representation.
To overcome this problem some models (Wieting et al., 2016; Bojanowski et al., 2017) enhance the representations with subword information, which can also somewhat deal with out-of-vocabulary words.
Another approach is to learn a different representation for every word sense (Neelakantan et al., 2014) but this requires knowing the set of word senses in advance.
More recent methods overcome the above issues by learning contextualized embeddings.
Melamud et al.
(2016) encode the context surrounding a pivot word using a bidirectional LSTM.
Peters et al.
(2018) use a deep bidirectional LSTM, learning word embeddings as functions of its internal states, calling the method Embeddings using Language Models (ELMo).
We discuss ELMo in detail in Section 3.
Devlin et al.
(2018) introduced bidirectional encoder representations from transformers (BERT).
This method learns pre-trained contextual embeddings by jointly conditioning on left and right context via an attention mechanism.
Program repair is an important task in software engineering and programming languages.
For a detailed review see
Monperrus (2018); Gazzola et al.
Many recent program repair methods are based on machine learning.
Yin et al.
(2018) learn to represent code edits using a gated graph neural network (GGNN) (Li et al., 2016).
Allamanis et al.
(2018b) learn to identify a particular class of bugs called variable misuse bugs, using a GGNN.
Chen et al.
(2019) introduce SequenceR which learns to transform buggy lines into ﬁxed ones via machine translation.
Our work is orthogonal to these approaches and can be used as input in other models.
Finally, our work is also related to code representation methods many of which have also been used in program repair.
Harer et al.
(2018) learn Word2Vec embeddings for C/C++ tokens to predict software vulnerabilities.
White et al.
(2019) learn Word2Vec embeddings for Java tokens and utilize them in program repair.
Alon et al.
learn code embeddings using abstract syntax tree paths.
A more detailed overview can be found in (Allamanis et al., 2018a; Chen & Monperrus, 2019).
E MBEDDINGS FROM L ANGUAGE M ODELS (ELM O )
ELMo (Peters et al., 2018) computes word embeddings from the hidden states of a language model.
Consequently, the embeddings of each token depend on its context of the input sequence, even out-of-vocabulary (OOV) tokens have effective input representations.
In this section, we brieﬂy describe the ELMo embeddings.
The ﬁrst step is that a neural language model is trained to maximize the likelihood of a training corpus.
The architecture used by ELMo a bidirectional LSTM with L layers and character convolutions in the input layer.
Let the input be a sequence of tokens ( t 1 , ...t N ) .
For each token t k , denote by x LM k the input representation from the character convolution.
Consequently, this representation passes through L layers of forward and backward LSTMs.
Then each layer j ∈ { 1 , ..., L } of the forward LSTM computes a hidden state −−→ h LMk,j , and likewise the hidden states of the backward LSTM are denoted by ←−− h LMk,j .
The parameters for the token representation and for the output softmax layer are tied for both directions, while different parameters are learned for each direction of the LSTMs.
After the language model has been trained, we can use it within another downstream task by combining the hidden states of the language model from each LSTM layer.
This process is called ELMo.
For each token t k of a sentence in the test set, the language model computes 2 L + 1 hidden states, one in each direction for each layer, and then the input layer.
To make the following more compact, we can write these as h LMk,
= x LMk for the input layer, and then h LMk,j = [ −−→ h LMk,j , ←−− h LMk,j ] for all of the other layers.
The set of these vectors is
[Equation]
To create the ﬁnal representation that is fed to downstream tasks, ELMo collapses the set of representations into a single vector E k for token t k .
A simplistic approach is to only select the top layer, so that E k = h LMk,L .
A more general one, which we use in this work, is to combine the layers via ﬁne-tuned task speciﬁc weights s =
( s 1 . . .
s L ) for every layer.
Then we can compute the embedding for token k as
[Equation]
where γ is an additional scalar parameter that scales the entire vector.
In our experiments we did not performed ﬁnetuning and thus used equal weights s j = 1 / ( L + 1) for each layer and γ = 1 .
However, our implementation also supports all the aforementioned ways of collapsing the set of representations.
A potential drawback of the method is that it still utilizes a softmax output layer with a ﬁxed vocabulary that does not scale effectively and it still predicts UNK for OOV tokens which may have a negative effect on the representations.
S OURCE C ODE ELM O
We describe Source Code ELMo (SCELMo), which trains ELMo on corpora of source code.
However, we note that normally ELMo models in other domains are able to effectively utilize much larger representations.
The code was tokenized using the esprima JavaScript tokenizer 1 .
For training the ELMo model we used a corpus of 150,
JavaScript Files (Raychev et al. 2016)
consisting of various open-source projects.
This corpus has previously been used on several tasks (Raychev et al., 2016;
Pradel & Sen, 2018; Bavishi et al., 2018).
We applied the patch released by Allamanis et al.
(2018a)
to ﬁlter out code duplication as this phenomenon was shown on this and other corpora to result in inﬂation of performance metrics.
This resulted in 64750 training ﬁles and 33229 validation ﬁles.
Since the validation set contains ﬁles from the same projects as the train the contained instances might be too similar and unrealistic overestimating.
To address this we also created a test set of 500 random JavaScript projects sampled from the top 20,000 open-source JavaScript projects as of May 2019.
The test corpus has not been previously utilized in previous work and is a better reﬂection of the performance of the learned bug detectors.
Lastly, it is important to know what the performance of the method will be if we do not have access to training data from the projects on which we would like to ﬁnd bugs.
This is common in practice for many real case scenarios.
For training the ELMo model, we use an embedding size of 100 features for each of the forward and backward LSTMs so that each layer sums up tofeatures.
C ONTEXTUAL E MBEDDINGS FOR P ROGRAM R EPAIR
In this section, we describe how contextual embeddings can be incorporated within a recent machine learning-based bug detection system, the DeepBugs system of Pradel & Sen (2018).
In the ﬁrst part of this section, we give background about the DeepBugs system, and then we describe how we incorporate SCELMo within DeepBugs.
DeepBugs treats the problem of ﬁnding a bug as a classiﬁcation problem.
The system considers a set of speciﬁc bug types, which are small mistakes that might be made in a program, such as swapping two arguments.
For each bug type, DeepBugs trains a binary classiﬁer that takes a program statement as input and predicts whether the statement contains that type of bug.
At test time, this classiﬁer can be run for every statement in the program to attempt to detect bugs.
In order to train the model both examples of correct and incorrect (buggy) code are necessary.
DeepBugs treats the existing code as correct and randomly mutates it to obtain buggy code.
To obtain training examples, we extract all expressions from the source code which are either the function calls with exactly two arguments and all binary expressions.
To create instances of buggy code we mutate each of the correct instances.
As such, arguments in function calls are swapped, the binary operator in binary expressions is replaced with another random one, and ﬁnally randomly either the left or the right operand is replaced by another random binary operand that appears in the same ﬁle.
Then the classiﬁcation task is a binary task to predict whether the instance is correct, i.e., it comes from the original code, or whether it is buggy, i.e. it was one of the randomly mutated examples.
The validation and test sets are mutated in the same way as the training set.
The split between correct and buggy instances has 50/50 class distribution as for each original code instance exactly one mutated buggy counterpart is created.
The architecture for the classiﬁer is a feedforward network with a single hidden layer of 200 dimensions with Relu activations and a sigmoid output layer.
For both the input and hidden layers a dropout of 0.2.
The network was trained in all experiments for 10 epochs with a batch size of 50 and the RMSProp optimizer.
We note that for maintaining a consistent comparison with DeepBugs we kept all the above parameters as well as the optimizer’s parameters ﬁxed to the values reported in Pradel & Sen (2018).
Tuning these parameters would probably result in at least a small performance increase for our method.
Listing 1: Swapped Arguments Bug
Listing 2: Incorrect Binary Operator 1 // Call to .length is missing.
2 if ( index < matrix ) { do_something(); 4 }
Listing 3: Incorrect Binary Operand
In our experiments, we consider three bug types that address a set of common programming mistakes: swapped arguments of function calls, using the wrong binary operator and using an incorrect binary operand in a binary expression.
The methodology can easily be applied to other bug types.
Figure 1 illustrates an example of each of the three bug types.
5. I NPUT TO THE C LASSIFIER
A key question is how a statement from the source code is converted into a feature vector that can be used within the classiﬁer.
DeepBugs uses a set of heuristics that, given a statement and a bug type, return a sequence of identiﬁers from the statement that are most likely to be relevant.
For instance, for the call to setTimeout in Listing 1 the following sequence of identiﬁers would be extracted:
[setTimeout, delay, function] .
A detailed description of the heuristics is available in Appendix A.
These heuristics result in a sequence of program identiﬁers.
These are converted to continuous vectors using word embeddings, concatenated, and this is the input to the classiﬁer.
DeepBugs uses Word2Vec embeddings trained on a corpus of code.
In our experiments, we train classiﬁers using three different types of word embeddings.
First, we kept the 10,000 most frequent identiﬁers/literals and assigned to each of them a random embedding of 200 features.
Second, to reproduce the results of Pradel & Sen (2018), we use the CBOW variant of Word2Vec to learn representations consisting of 200 features for the 10,000 most frequent identiﬁers/literals.
Finally, we train a FastText embeddings (Bojanowski et al., 2017) on the training set to learn identiﬁer embeddings that contain subword information.
The subwords used by FastText are all the character trigrams that appear in the training corpus.
Identiﬁers are therefore composed of multiple subwords.
To represent an identiﬁer, we sum the embeddings of each of its subwords and summing them up.
This allows the identiﬁer embeddings to contain information about the structure and morphology of identiﬁers.
This also allows the FastText embeddings, unlike the Word2Vec ones, to represent OOV words as a combination of character trigrams.
Note that DeepBugs can detect bugs only in statements that do not contain OOV (out-of-vocabulary) identiﬁers, because its Word2Vec embeddings cannot extract features for OOV names.
Instead our implementation does not skip such instances.
Since the original work discarded any instances that contain OOV identiﬁers we neither know how the method performs on such instances nor how often those appear in the utilized dataset of DeepBugs.
Moreover, DeepBugs supported only a speciﬁc subset of AST nodes and skipped the rest.
For example if a call’s argument is a complex expression consisting of other expressions then the call would be skipped.
However, we expanded the implementation to support all kinds of AST nodes and to not skip instances with nested expressions as discussed in Appendix A.
We note that we still skip an instance if one of its main parts (e.g., a function call’s argument) is a complex expression longer than 1,000 characters as such expressions might be overly long to reason about.
[Table] [Table]
C ONNECTING SCELM O TO THE B UG D ETECTOR
We investigated two variants of the bug detection model, which query SCELMo in different ways to get features for the classiﬁer.
The ﬁrst utilizes the heuristic of Section A to extract a small set of identiﬁers or literals that represent the code piece.
For example, for an incorrect binary operand instance we extract one identiﬁer or literal for the left and right operands respectively
[Table]
and we also extract its binary operator.
Then, those are concatenated to form a query to the network.
In the case of function calls we extract the identiﬁer corresponding to the name of the called function, one identiﬁer or literal for the ﬁrst and second argument respectively and an identiﬁers for the expression on which the function is called.
We also add the appropriate syntax tokens (a ’.’
if necessary, ’,’ between the two arguments, and left and right parentheses) to create a query that resembles a function call.
This baseline approach creates simplistic ﬁxed size queries for the network but does not utilize its full potential since the queries do not necessarily resemble actual code, nor correct code similar to the sequences in the training set for the embeddings.
We will refer to this baseline as
No-Context ELMo.
Our proposed method, we compute SCELMo embeddings to the language model all the tokens of the instances for which we need representations.
Valid instances are functions calls that contain exactly two arguments and binary expressions.
To create a ﬁxed-size representation we extract only the features corresponding a ﬁxed set of tokens.
Speciﬁcally, for functions calls we use the representations corresponding to the ﬁrst token of the expression on which the function is called, the function name, the ﬁrst token of the ﬁrst argument and the ﬁrst token of the second argument.
While, for binary expressions we use those of the ﬁrst token of the left operand, the binary operator, and the ﬁrst token of the right operand.
Since the representations contain contextual information, the returned vectors can capture information about the rest of the tokens in the code sequence.
R ESULTS
We next discuss the experiments we performed and their corresponding results.
We measured the performance of the three baselines as well as those of non-contextual ELMO and SCELMO.
Measuring the performance of non-contextual
ELMO allows us to evaluate how much improvement is due to speciﬁcs of the language model architecture, such as the character convolutional layer which can handle OOVs, and how much is due to the contextual information itself.
6. P ERFORMANCE ON V ALIDATION S ET
In our ﬁrst experiment we evaluate the performance of the methods in tasks where training data from the same projects are available.
The evaluation performed in this experiment gives a good estimation of how our method performs compared to the previous state-of-the-art technique of DeepBugs.
One main difference however is that the evaluation now also includes instances which contain OOV.
As a consequence the bug detections tasks are harder than those presented by Pradel & Sen (2018) as their evaluation does not include in both the training and validation set any instance for which an extracted identiﬁer is OOV.
Table 1 illustrates the performance of the baselines and our models.
As one would expect the FastText baseline improves over Word2Vec for all bug types due to the subword information.
Moreover, our model SCELMo massively outperforms all other methods.
Lastly, even no-context ELMo the heuristic version of SCELMo that does not utilize contextual information at test time outperforms the baseline methods showcasing how powerful the pretrained representations are.
6. I NCLUDING C OMPLEX E XPRESSIONS
In our next experiment we also included instances that contain elements that are complex or nested expressions.
For instance, in the original work if one the arguments of a function call or one of the operands of a binary expression is an expression consisting of other expressions then the instance would not be included in the dataset.
Several AST node
[Table] [Table]
types such as a NewExpression node or an ObjectExpression were not supported.
Figure 2 a few examples of instances that would be previously skipped 2 .
Such instances were skipped by Pradel & Sen (2018) and not included in their results.
We do note though that we still skip very long expressions that contain more than 1000 tokens
[Table]
First argument is binary expression 2 doComputation(x + find_min(components), callback); 1 //
Second argument is an unsupported node 2 factory.test(simulator, new Car(’Eagle’, ’Talon TSi’, 1993));
Similarly to the previous experiment SCELMo signiﬁcantly outperforms all other models.
This is evident in Table 2.
Lastly, we clarify that the results of this section should not be directly compared to those of the previous one as for this experiment the training set is also larger.
E XTERNAL T EST E VALUATION
The last experiment’s objective is to showcase how the various models would perform on unseen projects as this better illustrates the generalizability of the techniques.
The conﬁguration utilized is identical to that of the previous section.
By looking at Table 3 one can notice that the baselines have a major drop in performance.
This is a common ﬁnding in machine learning models of code, namely, that applying a trained model to a new software project is much more difﬁcult than to a new ﬁle in the same project.
In contrast, SCELMo offers up to 15% improvement in accuracy compared to Word2Vec baseline.
In fact, impressively enough SCELMo on the external test set is better than the evaluation set one of the baselines.
6. OOV S TATISTICS
In order to better understand the above results we measured the OOV rate of the basic elements of the code instances appearing in the dataset.
Here the OOV rate is calculated based on the vocabulary of 10000 entries utilized by the
Word2Vec and random baseline models.
These are illustrated in Tables 4 and 5.
We measured the OOV rates for both the version of the dataset used in Section 6.4, which we call Train and Validation, and that used in Section 6.2, which we call Extended Train and Extended Validation.
Tables 4 and 5 describe the OOV rates for different parts of the expression types that are considered by the DeepBugs bug detector.
A detailed description of the identiﬁers extraction heuristic can be found in Appendix A.
We ﬁrst focus on the swapped arguments bug pattern and consider all of the method call that have exactly two arguments.
Each method call contains the function name, a name of the ﬁrst argument, a name of the second argument, and a base object.
The base object is the identiﬁer that would be extracted from the expression (if such an expression exists) on which the function is called.
For instance, from the following expression: window.navigator.userAgent.indexOf(”Chrome”) , userAgent would be extracted as the base object.
Table 4 shows for each of the components how often they are OOV.
In the expanded version of the dataset if one of the arguments is a complex expression then it is converted into a name based on the heuristic described in Section A.
The resulting statistics contain valuable information as for instance, it is almost impossible for the Word2Vec baseline to reason about a swap arguments bug if the identiﬁers extracted for both arguments are OOV.
In a similar manner for the incorrect operand and operator bug patterns we consider all the binary operations.
Each binary expression consists of a left and right operand and a name is extracted for each of them.
For each operand we also measured the frequency with which the operand corresponds to certain common types such as identiﬁer, literal or a ThisExpression .
[Table] [Table]
I S NEURAL BUG - FINDING USEFUL IN PRACTICE ?
Although related work (Pradel & Sen, 2018; Allamanis et al., 2018b; Vasic et al., 2019) has shown that there is great potential for embedding based neural bug ﬁnders, the evaluation has mostly focused on synthetic bugs introduced by mutating the original code.
However, there is no strong indication that the synthetic bugs correlate to real ones, apart from a small study of the top 50 warnings for each bug type produced by DeepBugs.
A good example is the mutation operation utilized for the incorrect binary operator bug. A lot of the introduced bug instances could result in syntactic errors. This can potentially create a classiﬁer with a high bias towards correlating buggy code to syntactically incorrect code, thus hindering the model’s ability to generalize on real bugs. Ideally, in an industrial environment we would like the resulting models to achieve a false positive rate of less than 10 % (Sadowski et al., 2015).
Sadly, high true positive rates are not to be expected as well since static bug detectors were shown to be able to detect less than 5% of bugs
[Table]
[Table]
(Habib & Pradel, 2018) contained in the Defects4J corpus (Just et al., 2014) and less than 12% in a single-statement bugs corpus (Karampatsis & Sutton, 2019).
We note that in the second case the static analysis tool is given credit by reported any warning for the buggy line, so the actual percentage might lower than the reported one.
We next make a ﬁrst step on investigating the practical usefulness of our methods by applying the classiﬁers of the previous section on a small corpus of real JavaScript bugs.
However, we think that this is a very hard yet interesting problem that should be carefully examined in future work.
In order to mine a corpus of real bug changes we used the methodology described in (Karampatsis & Sutton, 2019).
We note that we adapted their implementation to utilize the Rhino JavaScript parser 3 .
Their methodology extracts bug ﬁxing commits and ﬁlters them to only keep those that contain small single-statement changes.
Finally, it classiﬁes each pair of modiﬁed statements by whether the ﬁt a set of mutation patterns.
The resulting dataset is shown in Table 6.
Upon acceptance of the paper we will release this dataset along with our implementation, the rest of data used, and the learned representations.
Finally, we queried the DeepBugs and SCELMo with each buggy instance as well as its ﬁxed variant and measured the percentage of correctly classiﬁed instances for each of the two categories.
We also ignored any instances for which the JavaScript parser utilized for both failed to extract an AST.
We classiﬁed as bugs any instances that were assigned a probability to be a bug > 75% .
In an actual system this threshold should ideally be tuned on a validation set.
Table 7 suggests that there might indeed be some potential for future practical applications of neural bug ﬁnding techniques.
Both are able to uncover some of the bugs.
However, the results also suggest that careful tuning of the predictions threshold might be necessary, especially if we take into account the industrial need to comply with a low false positive rate (FPR).
For instance, raising SCELMo’s prediction threshold to 80% for the swap arguments bug results in ﬁnding only 3.34% of the bugs but correctly classifying 100% of the repaired function calls, thus achieving 0.0% false positive rate.
Moreover, since SCELMo could not uncover any of the real binary operator bugs, future work could investigate the effect of utilizing different mutation strategies for the purpose of artiﬁcial bug-induction.
Future work could also investigate if ﬁne-tuning on small set of real bugs could result in more robust classiﬁers.
C ONCLUSION
We have presented SCELMo, which is to our knowledge the ﬁrst language-model based contextual embeddings for source code.
Contextual embeddings have many potential advantages for source code, because surrounding tokens can indirectly provide information about tokens, e.g. about likely values of variables.
We highlight the utility of SCELMo embeddings by using them within a recent state-of-the-art machine learning based bug detector.
The SCELMo embeddings yield a dramatic improvement in the synthetic bug detection performance benchmark, especially on lines of code that contain out-of-vocabulary tokens and complex expressions that can cause difﬁculty for the method.
We also showed and discussed the performance of the resulting bug detectors on a dataset of real bugs raising useful insights for future work.
A N AME E XTRACTION H EURISTIC
In order for DeepBugs to operate it is necessary to extract identiﬁers or literals for each expression part of the statement.
The bug detector for swapped arguments utilizes the following elements of the function call:
Base Object: The expression on which the function is called.
Callee: The called function.
Argument 1: The expression consisting the ﬁrst argument of the called function.
Argument 2: The expression consisting the ﬁrst argument of the called function.
Similarly the bug detectors for incorrect binary operators and operands utilize the following elements of the binary expression:
Binary Operator: The binary operator utilized in the expression.
Left Operand: The left operand of the binary expression.
Right Operand:
The right operand of the binary expression.
We next describe the extraction heuristic, which is shared by all the bug detectors.
The heuristic takes as input a node n representing an expression and returns name ( n ) based on the following rules:
• Identiﬁer: return its name.
• Literal: return its value.
• this expression: return this .
• Update expression with argument x : return name ( x ) .
• Member expression accessing a property p : return name ( p ) .
• Member expression accessing a property base [ p ] : return name ( base ) .
• Call expression base.callee ( ... ) : return name ( callee ) .
• Property node n : If n.key does not exist return name ( n.value ) .
If name ( n.key ) does not exist return name ( n.value ) .
Otherwise randomly return either name ( n.value ) or name () n.key ) .
• Binary expression with left operand l and right operand r : Run the heuristic on both l and r to retrieve name ( l ) and name ( r ) .
If name ( l ) does not exist return name ( r ) .
If name ( r ) does not exist return name ( l ) .
Otherwise randomly return either name ( l ) ir name ( r ) .
• Logical expression with left operand l and right operand r : Run the heuristic on both l and r to retrieve name ( l ) and name ( r ) .
If name ( l ) does not exist return name ( r ) .
If name ( r ) does not exist return name ( l ) .
Otherwise randomly return either name ( l ) ir name ( r ) .
• Assignment expression with left operand l and right operand r : Run the heuristic on both l and r to retrieve name ( l ) and name ( r ) .
If name ( l ) does not exist return name ( r ) .
If name ( r ) does not exist return name ( l ) .
Otherwise, randomly return either name ( l ) ir name ( r ) .
• Unary expression with argument u : Return name ( u ) .
• Array expression with elements l i : For all l i that name ( l i ) exists randomly choose one of them and return name ( l i ) .
• Conditional expression with operands c , l , and r : Randomly choose one out of c , l , r for which a name exists and return its name.
• Function expression: return function .
• Object expression: return { .
• New expression with a constructor function call c : return name ( c ) .
All random decisions follow a uniform distribution.