L EARNING TO I NFER R UN -T IME I NVARIANTS FROM S OURCE CODE
A BSTRACT
Source code is meant to be executed, as well as read.
Developers reason about its run-time properties by inferring “invariants", which constrain program behavior; but they rarely encode these explicitly, so machine-learning methods don’t have much aligned data to learn from.
We propose an approach that adapts cues within existing if-statements regarding explicit run-time expectations to generate aligned datasets of code and implicit invariants.
We also propose a contrastive loss to inhibit generation of illogical invariants.
Our model learns to infer a wide vocabulary of invariants for arbitrary code, which can be used to detect and repair real bugs.
This is complementary to trace-based methods, such as Daikon.
Our results conﬁrm that neural models can learn run-time expectations directly from code.
I NTRODUCTION
Software developers make many inferences while reading code, about its performance, correctness, and run-time behavior.
The latter is often summarized by program invariants – generic assumptions about code behavior, e.g. that a list index never escapes its bounds.
Automatically inferring these is hard: predicting run-time behavior is undecidable, thus sound analyses are limited to small programs such as simple loops (Sharma et al., 2013a;
Padhi et al., 2016).
Ernst et al.
(2007) pioneered learning them from highly informative execution trace data , but it requires access to realistic program inputs.
Developers also rarely “assert” invariants explicitly, so static learning data is limited.
Yet these obstacles may be largely artiﬁcial.
For one, practical programs rarely take on an exponential range of values, and developers reliably infer run-time constraints from a local context, using their past experience and cues from the code itself.
What’s more, developers do encode many runtime constraints in their code, in the form of if-statements.
These guard blocks of code that should only execute under a given condition, such as that a map contains an element, or a parameter is not null.
Interestingly, programmers rarely guard all possible conditions (e.g. nullity of all parameters), often rather relying on other (calling or called) methods to ensure them.
Our central claim is that the natural distribution of programs includes many groups of similar functions, some of which assert run-time assumptions explicitly, and with much detail, while others vary along these dimensions. If true, we can use explicit conditional checks that guard blocks in functions to teach our models about the implicit invariants of unguarded blocks in similar functions. Furthermore, we conjecture that in such comparable samples, the condition is both salient (since it is checked explicitly) and natural (since it is written by humans). Learning from such examples is thus a very appropriate training signal for inferring practically useful invariants.
int ix = getIndex(arr, el) [Figure] return arr[ix]
[Figure]
int ix = getIndex(arr, el); if (ix < 0 || ix >= arr.length) { return insert(arr, ix, el)
[Figure]
}else { [Figure] arr[ix]; }
[Figure]
ix = getIndex(arr
[Figure] el) [Figure]
arr
[Figure] Figure 1 shows how we exploit this symmetry between explicitly and implicitly guarded code: we remove explicit guards to generate the former.
By removing these one at the time and ﬁltering out implausible programs, we generate millions of realistic samples of functions and (now-)implicit runtime constraints.
Our model, B ODY G UARD , predicts a rich vocabulary of conditions about arbitrary code from new projects, and can be used to ﬁnd & ﬁx real missing-guard bugs with over 69% (repair) precision at 10% inspection cost.
It also predicts more than two-thirds of Daikon’s invariants that could previously only be inferred with run-time data, and some entirely new ones that can be validated automatically with trace data.
This is a signiﬁcant next step in learned static analysis, being the ﬁrst to reliably produce natural invariants from arbitrary code alone.
More generally, we show that learned models can implicitly represent behavioral semantics, just from code.
A PPROACH
We train our model using a new loss function on a large corpus of open source data and evaluate it on two downstream tasks.
For conciseness, we provide a high-level overview of our approach here.
2. D ATASETS
We generate samples following Figure 1 from top-starred Java projects from Github, which we split by organization into training (920 projects), held-out (19 projects), and test data (61 projects).
We extract all methods from each Java ﬁle and generate one sample for each (side-effect free) ifstatement by removing the guard and storing its condition.
As Figure 1 shows, this produces one or two equivalent code fragments for which the statement’s condition (or its negation) is a precondition.
The resultant sample contains the entire method (minus conditional check) as context, with the range of tokens where the invariant condition applies indicated.
Our model generates run-time conditions for an indicated range of code in a method.
We evaluate it in two settings.
First, we collect real missing if-guard bugs for our model to repair from 10K Java
Github projects, by parsing their ( ∼ 8M) commits for ones that a) introduced exactly one if-guard, and b) are described as bug-ﬁxing (Ray et al., 2016).
We ﬁnd ca.
3K of these.
Second, we use
Daikon (Ernst et al., 2007) to collect execution trace data from a smaller set of eight projects that we manually instrumented, to compare our predictions to both Daikon’s own and to traces directly.
This helps us both assess the validity of our invariants on real executions, and, more generally, understand the inference gap between static and dynamic information for source code analysis; i.e., is run-time data (when present) strictly more useful than code, or are the two information sources orthogonal?
Transformer layer onTargetFound() { … time = calculateTime(); action.update(-dx, -dy, time, interpolator); } time >
Masked Self-Attention
Feedforward NN Biased Input Attention
Transformer layer
Masked Self-Attention
Feedforward NN Biased Input Attention
Relational Self-Attention
Feedforward NN
Relational Self-Attention
Feedforward NN ⋮ ×
Relations: parent, computed-from, def-use, next-use, next-token
Transformer layer Transformer layer ⋮ × 2. M ODEL S ETUP B ODY G UARD uses a Transformer encoder-decoder model (Vaswani et al., 2017) to translate methods to invariants (see Figure 2).
Syntactic & semantic code metadata, e.g. the AST and data-ﬂow relations are known to help code understanding (Allamanis et al., 2018), thus we incorporate them using the relational attention mechanism from Hellendoorn et al. It adds a relation-speciﬁc bias term b rij into the query-key comparison of scaled dot-product attention
[Equation]
The (learned) bias is sensitive to known relations r between tokens i and j (if any, and summed if multiple), to allow the model to selectively sharpen (or dampen) the signiﬁcance of each relation. With 512-dimensional states, 8 heads, and 8 layers on both sides, our model has ∼ 67M parameters. We created our own program graph extractor for Java using the Eclipse JDT parser. In following with prior workAllamanis et al.
(2018), we extract ﬁve commonly used kinds of edges that represent different forms of relations in code. This includes lexical dependencies (“next-token” and its reverse, “prev-token”), syntactic (AST) relations (“parent”/’“child”), and various data-ﬂow relations (“lastuse”, “next-token”, and “computed-from”). As is common, for each of these relations we include a separate edge for its reverse, yielding 10 edge types total.
Finally, we use the same “leaves-only” representation as Hellendoorn et al. (2020) to limit our inputs to tokens only, by rerouting edges that connect non-terminal nodes to their representative syntax token (e.g. from an if-statement node to its “if” code token). We reuse this relational mechanism between the decoder and encoder, in order to produce only invariants for the relevant range of tokens (using a unary relation, i.e., “is part of range”) Ding et al.
2. D ECODING L OGICAL S TATEMENTS
In conditions, small syntactic differences lead to drastic changes in run-time behavior.
Removing esp.
if - else statements as we do naturally yields code fragments with very similar, but logically opposite (e.g. ‘ !
= null ’ vs. ‘ == null ’) conditions.
We supervise our model to encourage its representations for syntactically close but semantically opposite statements to be distinct by introducing a contrastive hinge loss term.
For every training sample, we also decode the invariant’s logical negation, but require it to have a higher entropy than the real target.
Concretely, given a statement inv with tokens t i and a negating function neg , we use the cross-entropy loss L CE :
[Equation]
(1) to compute the entropy distance w.r.t.
its negation:
[Equation]
2) in which (cid:15) is the minimum desired entropy “distance” in bits.
In this work, we set (cid:15) = 2 .
For this hinge-loss model , as we will call it in the rest of this paper, we train with a loss equal to L CE + L hinge .
A NALYSIS
We sample our two models’ held-out performance every 100,000 training steps, which produces the learning curves in Figure 3a.
The hinge loss model saturates slower, as it faces the more challenging [Figure] task of discriminating similar statements.
However, after ca.
one week of training, both models converge to approximately the same quality.
It speaks to the challenge of this strategically important task that the models reach just ∼ 30% accuracy, due in part to the diverse vocabulary of invariants.
Figure 3b shows the evaluation on the test data, namely precision/recall behavior of the two models when ranking their predictions (from beam search) by entropy.
Both models converge to (near) perfect precision at a commensurate expense of recall (breaking 80% precision around 20% recall).
The hinge-loss model has a decidedly better precision/recall tradeoff, as well as higher overall accuracy.
We manually analyzed a number of these functions and our models’ prediction on them, to determine the source of B ODY G UARD ’s inaccuracies.
In short, some of its mistakes are clearly a matter of modeling capacity: the vocabulary of invariants and code contexts is incredibly diverse, and our model ( < 100 M parameters) and dataset ( < 1 B tokens) may be too small to fully capture this distribution.
Secondly, the scope of our samples poses a substantial challenge: we focus on modeling at the function level only, as even large functions (spanning in the order of 500 tokens) pose a challenge to the Transformer-type models that we use in terms of data usage.
However, many invariants require inference at the level of ﬁles or even across function calls. Without that context, the task is inherently ambiguous for those samples. Further research on expanding our modeling horizon in software engineering is thus needed. 3. M ISSING I F D ETECTION
[Table]
We apply our models to predicting missing if-guards, ﬁrst given a localized buggy segment (top row of Table 1).
This most directly relates to our training signal, where we provided the location of the guarded code.
Our model achieves 29.3% accuracy (base model: 26.8%), with a favorable 69.1% precision at 10% – enough to ﬁx 215 out of 311 bugs once located.
It is likely beneﬁcial that this task is roughly as challenging as our test one: automatically synthesized training data is often overly easy compared to real tasks, which harms generalization (Hellendoorn et al., 2019b).
The missing condition in these samples is (arguably) the most salient invariant in the entire method, not just the indicated code block, which our model should be able to prioritize.
The next three rows of Table 1 show the results of running our invariant generator on every contiguous segment (up to 5 blocks) of code in each buggy method, ranking the top invariants across segments for inspection.
This is substantially harder than the previous task, reducing the overall accuracy threefold and roughly halving precision.
Nevertheless, that is still much better than might be expected if
B ODY G UARD had no location-sensitivity: we test over 30 blocks per method on average. [Figure]
3. V ALIDITY AND O VERLAP WITH D AIKON
Many of our emitted invariants are valid statements, even if not the real missing condition.
We analyze this validity on methods for which we have trace data from Daikon (Ernst et al., 2007): as Figure 4a shows, our tool retrieves more than half of Daikon’s invariants, and over two-thirds at 10% recall, from static code alone.
In addition, many of our pre- and post-condition 1 are not generated by Daikon (those out of its vocabulary, or with too few observations), even at a low entropy threshold.
We assess these using a simple logical engine that compares various kinds of our invariants (e.g. collection inclusion, array length, nullity) against the recorded traces.
We can check ca.
40% (12K) of our invariants in this way; on these, we ﬁnd that our invariants are valid ca.
60% of the time at full recall, and this ratio increases greatly as we sharpen the entropy threshold, to over 80%, at recall values under 10%, as shown in Figure 4b.
Many of our validated invariants were not produced by Daikon, implying that static and dynamic data are orthogonal for this task.
We manually analyzed 708 pre-conditions that B ODY G UARD generated at an entropy of ≤ 0.1; 540 of these could be checked with trace data, (449 valid, 91 invalid), and 122 of the remaining 168 were found valid on inspection.
In effect, over 80% of our invariants at this recall level (3.5%) are correct, and more than two-thirds of the invalid remainder could be ruled out using trace data, if available, leaving a false positive rate of just 6.5% (46/708) when execution data is available (while also adding about 200 valid invariants to Daikon’s predictions).
This conﬁrms that our tool is largely synergistic with dynamic, trace-based invariant generators.
C ONCLUSION
Formal invariant inference with oracles or solvers (Sharma et al., 2013a;b; Padhi et al., 2016; Pham et al., 2017) does not scale beyond simple arithmetic programs.
ML-based inference (Si et al., 2018;
Hellendoorn et al., 2019a; Brockschmidt et al., 2017) scales better but still usually predicts invariant validity rather than generates them.
In contrast, our work makes no assumptions about the code other than the availability of a parser, and is trained speciﬁcally to generate novel, likely invariants.
It succeeds thanks to the observation that typically used invariants are in a sense natural , like many other aspects of programs (Hindle et al., 2012; Barr et al., 2013; Tsimpourlas et al., 2020), and therefore predictable from code alone, intentionally standardized for ease of reading and writing Casalnuovo et al.
Previous approaches may be symbiotic with ours: trace data or
SMT solvers can be used to ﬁlter invalid invariants, and possibly guide our invariant generation.
Our approach can also support other code understanding tools that struggle to navigate an exponentially large program space.