S TATIC P REDICTION OF R UNTIME E RRORS BY L EARNING TO E XECUTE P ROGRAMS WITH E XTERNAL R ESOURCE D ESCRIPTIONS
David Bieber Google Research dbieber@google.com Rishab Goel Mila rgoel0112@gmail.com
Daniel Zheng Google Research danielzheng@google.com Hugo Larochelle Google Research hugolarochelle@google.com
Daniel Tarlow
Google Research dtarlow@google.com
A BSTRACT
The execution behavior of a program often depends on external resources, such as program inputs or file contents and so cannot be run in isolation.
Nevertheless software developers benefit from fast iteration loops where automated tools identify errors as early as possible, even before programs can be compiled and run
This presents an interesting machine learning challenge: can we predict runtime errors in a “static” setting, where program execution is not possible?
Here, we introduce a real-world dataset and task for predicting runtime errors, which we show is difficult for generic models like Transformers.
We approach this task by developing an interpreter-inspired architecture with an inductive bias towards mimicking program executions, which models exception handling and “learns to execute” descriptions of the contents of external resources.
Surprisingly, we show that the model can also predict the location of the error, despite being trained only on labels indicating the presence/absence and kind of error.
In total, we present a practical and difficult-yet-approachable challenge problem related to learning program execution and we demonstrate promising new capabilities of interpreterinspired machine learning models for code.
I NTRODUCTION
We investigate applying neural machine learning methods to the static analysis of source code for the prediction of runtime errors.
The execution behavior of a program is in general not fully defined by its source code in isolation, because programs often rely on external resources like inputs, the contents of files, or the network.
Nevertheless, software developers benefit from fast iteration loops where automated tools identify errors early, even when program execution is not yet an option.
Therefore we consider the following machine learning challenge: can we predict runtime errors in a “static” setting, where program execution is not possible?
Recent work has made considerable progress toward applying machine learning to learning for code tasks (Allamanis et al., 2018a).
Large language models show promise in source code generation, but have struggled on tasks that require reasoning about the execution behavior of programs (Chen et al., 2021a;
Austin et al., 2021), requiring detailed step-by-step supervision to make headway on the task (Anonymous, 2022).
We introduce the runtime error prediction task as a challenge problem, and in line with these findings, observe that it is a challenging task for generic models like Transformers.
To make progress on this challenging task, we identify a promising class of models from prior work, interpreter-inspired models, and we demonstrate they perform well on the task.
Instruction Pointer
Attention Graph Neural Network (IPA-GNN) (Bieber et al., 2020) models simulate the execution of a program, following its control flow structure, but operating in a continuous embedding space.
We make a number of improvements to IPA-GNN: scaling up to handle real-world code, adding the ability to “learn to execute” descriptions of the contents of external resources, and extending the architecture to model exception handling.
We perform evaluations comparing these interpreterinspired architectures against Transformer, LSTM, and GGNN baselines.
Results show that our combined improvements lead to a large increase in accuracy in predicting runtime errors and to interpretability that allows us to predict the location of errors even though the models are only trained on the presence or absence and class of error.
In total, we summarize our contributions as:
• We introduce the runtime error prediction task and Python Runtime Errors dataset, with runtime error labels for millions of competition Python programs.
• We demonstrate for the first time that IPA-GNN architectures are practical for processing real world programs, scaling them to a depth of 174 layers, and finding them to outperform generic models on the task.
• We demonstrate that external resource descriptions such as descriptions of the contents of stdin can be leveraged to improve performance on the task across all model architectures.
• We extend the IPA-GNN to model exception handling, resulting in the Exception IPAGNN, which we find can localize errors even when only trained on error presence and kind, not error location.
R ELATED W ORK
Execution behavior for identifying and localizing errors in source code Program analysis is a rich family of techniques for detecting defects in programs, including static analyses which are performed without executing code (Livshits & Lam, 2005; Xie & Aiken, 2006; Ayewah et al., 2008) and dynamic analyses which are performed at runtime (Cadar et al., 2008; Sen et al., 2005; Godefroid et al., 2005).
Linters and type checkers are popular error detection tools that use static analysis.
There has been considerable recent interest in applying machine learning to identifying and localizing errors in source code (Allamanis et al., 2018a).
Puri et al.
(2021) makes a large dataset of real world programs available, which we build on in constructing the Python Runtime Errors dataset.
Though dozens of tasks such as variable misuse detection and identifying “simple stupid bugs” (Chen et al., 2021b; Allamanis et al., 2018b; Karampatsis & Sutton, 2020; Hellendoorn et al., 2020) have been considered, most do not require reasoning about execution behavior, or else do so on synthetic data (Zaremba & Sutskever, 2014; Bieber et al., 2020), or using step-by-step trace supervision (Anonymous, 2022).
Chen et al.
(2021a)
and Austin et al.
(2021) find program execution to be a challenging task even for large-scale multiple billion parameter models.
In contrast with the tasks so far considered, the task we introduce requires reasoning about the execution behavior of real programs.
Several works including Vasic et al.
(2019) and Wang et al.
(2021) perform error localization in addition to identification, but they use localization supervision, whereas this work does not.
Interpreter-inspired models Several neural architectures draw inspiration from program interpreters (Bieber et al., 2020;
Graves et al., 2014; Boˇsnjak et al., 2017; Gaunt et al., 2017; Łukasz
Kaiser & Sutskever, 2016; Dehghani et al., 2019; Reed & de Freitas, 2016; Graves et al., 2016).
Our work is most similar to Bieber et al.
(2020) and Boˇsnjak et al.
(2017), focusing on how interpreters handle control flow and exception handling, rather than on memory allocation and function call stacks.
R UNTIME E RROR P REDICTION
The goal in the runtime error prediction task is to determine statically whether a program is liable to encounter a runtime error when it is run.
The programs will not be executable, as they depend on external resources which are not available.
Descriptions of the contents of these external resources are available, which makes reasoning about the execution behavior of the programs plausible, despite there being insufficient information available to perform the execution.
Examples of external resources include program inputs, file contents, and the network.
We treat the task as a classification task, with each error type as its own class, with “no error” as one additional class, and with each program having only a single target.
Python Runtime Errors We construct the Python Runtime Errors dataset from submissions to competitive programming problems from Project CodeNet (Puri et al., 2021), running each program on a sample input in a sandboxed environment to determine its target error class.
We describe the full dataset generation and filtering process in precise detail in Appendix A.
Our dataset contains 2.44 million syntactically valid Python 3 submissions with control flow graphs, each paired with one of 26 target classes.
The “no error” target is most common, accounting for 93.4% of examples.
For examples with one of the other 25 error classes, we additionally note the line number at which the error occurs, which we use as the ground truth for the unsupervised localization task of Section 5.3.
The dataset divides problems into train, validation, and test splits at a ratio of 80:10:10.
All submissions to the same problem become part of the same split, thereby reducing similarities between examples across splits that otherwise could arise from the presence of multiple similar submissions for the same problem.
Since there is a strong class imbalance in favor of the no error class, we also produce a balanced version of the test split by sampling the no error examples such that they comprise roughly 50% of the test split.
We use this balanced test split for all evaluations.
A PPROACH : IPA-GNN S AS R ELAXATIONS OF I NTERPRETERS
We make three modifications to the Instruction Pointer Attention Graph Neural Network (IPA-GNN) architecture.
These modifications scale the IPA-GNN to real-world code, allow it to incorporate external resource descriptions into its learned executions, and add support for modeling exception handling.
The IPA-GNN architecture is a continuous relaxation of the interpreter defined by the pseudocode in Algorithm 1, ignoring the magenta text, which we call Interpreter A.
We frame these modifications in relation to specific lines of the algorithm: scaling the IPA-GNN to real-world code (Section 4.1) and incorporating external resource descriptions (Section 4.2) both pertain to interpreting and executing statement x p at Line 3, and modeling exception handling adds the magenta text at lines 4-6 to yield Interpreter B (Section 4.3).
We showcase the behavior of both Interpreter A and B on a sample program in Figure 1, and illustrate an execution of the same program by a continuous relaxation of Interpreter B alongside it.
4. E XTENDING THE IPA-GNN TO R EAL - WORLD C ODE
Bieber et al.
(2020) interprets the IPA-GNN architecture as a message passing graph neural network operating on the statement-level control flow graph of the input program x .
Each node in the graph corresponds to a single statement in the program.
At each step t of the architecture, each node performs three steps: it executes the statement at that node (Line 3, Equation 2), computes a branch decision (Lines 7-8, Equation 4), and performs mean-field averaging over the resulting states and instruction pointers (Equations 5 and 6 in Bieber et al.
Unlike in Bieber et al.
(2020) where program statements are simple enough to be uniformly encoded as four-tuples, the programs in the Python Runtime Errors dataset consist of arbitrarily complex
Python statements authored by real programmers in a competition setting.
The language features n
S OURCE x = input() if x > 0: y = 4/3 * x else: y = abs(x) z
= y + sqrt(x) <exit> <raise>
[Figure]
S TD I NS TD I N D ESCRIPTION "
A SINGLE INTEGER -10..10"
[Figure]
t h A,B p
A p B h ˜ B p ˜ B {}  [10000000] { x: -3 }  [01000000] { x: -3 }  [00001000] { x: -3, y: 3 }  [00000100]
ValueError(lineno=6)  [00000001]
[Figure]
used are numerous and varied, and so the statement lengths vary substantially, with a mean statement length of 6.7 tokens; we report the full distribution of statement lengths in Figure 4. To compute per-statement embeddings Embed( x n ) for each statement x n in an arbitrary program x ’s statement-level control flow graph, we first apply either a local or global Transformer encoder to produce per-token embeddings, and then apply one of four pooling variants to a span of such embeddings.
In the local approach, we apply an attention mask to limit the embedding of a token in a statement to attending to other tokens in the same statement.
In the global approach, no such attention mask is applied, and so every token may attend to every other token in the program.
We consider four types of pooling in our hyperparameter search space: first , sum , mean , and max .
The resulting embedding is given by
[Equation]
First pooling takes the embedding of the first token in the span of node n .
Sum, mean, and max pooling apply their respective operations to the embeddings of all tokens in the span of node n .
Finally we find that real-world code requires up to 174 steps of the IPA-GNN.
To reduce the memory requirements, we apply rematerialization at each step of the model (Griewank & Walther, 2000;
Chen et al., 2016).
4. E XECUTING WITH R ESOURCE D ESCRIPTIONS
Each program x in the dataset is accompanied by a description of what values stdin may contain at runtime, which we tokenize to produce embedding d ( x ) .
Analogous to Algorithm 1 Line 1,
IPA-GNN architectures initialize with per-node hidden states h 0 , : = 0 and soft instruction pointer p 0 ,n = 1 { n = 0 } .
Following initialization, each step of an IPA-GNN begins by simulating execution (Line 3) of each non-terminal statement with non-zero probability under the soft instruction pointer to propose a new hidden state contribution
[Equation]
Here, magenta text shows our modification to the IPA-GNN architecture to incorporate external resource descriptions.
We consider both Feature-wise Linear Modulation (FiLM)
(Perez et al., 2017) and cross-attention (Lee et al., 2019) for the Modulate function, which we define in Appendix B.
This allows the IPA-GNN to execute differently at each step depending on what information the resource description provides, whether that be type information, value ranges, or candidate values.
We also consider one additional method that applies to any model: injecting the description as a docstring at the start of the program.
This method yields a new valid Python program, and so any model can accommodate it.
4. M ODELING E XCEPTION H ANDLING
The final modification we make to the IPA-GNN architecture is to model exception handling.
In
Algorithm 1, this corresponds to adding the magenta text to form Interpreter B, computing a raise decision (Lines 4-6, Equation 3).
With this modification, we call the architecture that results the “Exception IPA-GNN”.
Whereas execution always proceeds from statement to next statement in Interpreter A and in the
IPA-GNN, Interpreter B admits another behavior.
In Interpreter B and the Exception IPA-GNN, execution may proceed from any statement to a surrounding “except block”, if it is contained in a try/except frame, or else to a special global error node, which we denote n error .
In the sample execution in Figure 1c we see at step t = 4 the instruction pointer p B updates to n error = 8 .
We write that the IPA-GNN makes raise decisions as
[Equation]
The dense layer here has two outputs representing the case that an error is raised and that no error is raised.
Here r ( n ) denotes the node that statement n raises to in program x ; r ( n ) = n error if n is not contained in a try/except frame, and b t,n,n ′ denotes the probability assigned by the model to transitioning from executing n to n ′ .
Next the model makes soft branch decisions in an analogous manner; the dense layer for making branch decisions has distinct weights from the layer for making raise decisions.
[Equation]
The text in magenta corresponds to the “else” at Line 6.
The model has now assigned probability to up to three possible outcomes for each node: the probability that n raises an exception b t,n,r ( n ) , the probability that the true branch is followed b t,n,n 1 , and the probability that the false branch is followed b t,n,n 2 .
In the common case where a node is not a control node and has only one successor, rather than separate true and false branches, the probability of reaching that successor is simply 1 − b t,n,r ( n ) .
Finally, we assign each program a step limit T ( x ) using the same heuristic as Bieber et al.
(2020), detailed in Appendix C. After T ( x ) steps of the architecture, the model directly uses the probability mass at n exit and n error to predict whether the program raises an error, and if so it predicts the error type using the hidden state at the error node.
We write the modified IPA-GNN’s predictions as
P
[Equation]
no
[Equation]
p T
[Equation]
,n
[Equation] and [Equation] error , with
[Equation]
We train the model with a cross entropy loss on the class predictions, with “no error” treated as its own class.
4. U NSUPERVISED L OCALIZATION OF E RRORS
When we model exception handling explicitly in the IPA-GNN, the model makes soft decisions as to when to raise exceptions.
We can interpret these decisions as predictions of the location where a program might raise an error.
We can then evaluate how closely these location predictions match the true locations where exceptions are raised, despite never training the IPA-GNN with supervision that indicates error locations.
For programs that lack try/except frames, we compute the localization predictions of the model by summing, separately for each node, the contributions from that node to the exception node across all time steps.
This gives an estimate of exception provenance as
[Equation]
[Equation]
For programs with a try/except frame, we must trace the exception back to the statement that originally raised it.
We provide the calculation for this in Appendix D.
E XPERIMENTS
In our experiments we evaluate the following research questions:
RQ1: How does the adaptation of the IPA-GNN to real-world code compare against standard architectures like the GGNN, LSTM, and Transformer?
(Section 5.1) RQ2: What is the impact of including resource descriptions?
What methods for incorporating them work best?
(Section 5.2) RQ3: How interpretable are the latent instruction pointer quantities inside the Exception IPA-GNN for localizing where errors arise?
How does unsupervised localization with the Exception IPA-GNN compare to alternative unsupervised approaches to localization based on multiple instance learning and standard architectures?
(Section 5.3)
5. E VALUATION OF IPA-GNN A GAINST B ASELINES
We first compare the IPA-GNN architectures with Transformer (Vaswani et al., 2017), GGNN (Li et al., 2017), and LSTM (Hochreiter & Schmidhuber, 1997) baselines.
In all approaches, we use the 30,000 token vocabulary constructed in Appendix A, applying Byte-Pair Encoding (BPE) tokenization (Sennrich et al., 2016) to tokenize each program into a sequence of token indices.
The
Transformer operates on this sequence of token indices directly, with its final representation computed via mean pooling.
For all other models (GGNN, LSTM, IPA-GNN, and Exception IPA-GNN), the token indices are first combined via a masked (local) Transformer to produce per-node embeddings, and the model operates on these per-node embeddings, as in Section 4.1.
Following Bieber et al.
we encode programs for a GGNN using six edge types, and use a two-layer LSTM for both the LSTM baseline and for the RNN in all IPA-GNN variants.
For each approach, we perform an independent hyperparameter search using random search.
We list the hyperparameter space considered and model selection criteria in Appendix C.
The models are each trained to minimize a cross-entropy loss on the target class using stochastic gradient descent for up to 500,000 steps with a mini-batch size of 32.
In order to more closely match the target class distribution found in the balanced test set, we sample mini-batches such that the proportion of examples with target “no error” and those with an error target is 1:1 in expectation.
We evaluate the selected models on the balanced test set, and report the results in Table 1 (rows without check marks) [Table]
Weighted F1 score weights per-class F1 scores by class frequency, while weighted error F1 score restricts consideration to examples with a runtime error.
We perform additional evaluations using the same experimental setup but distinct initializations to compute measures of variance (Appendix E)
[Table]
[Table]
: The interpreter-inspired architectures show significant gains over the GGNN, Transformer, and baseline approaches on the runtime error prediction task.
We attribute this to the model’s inductive bias toward mimicking program execution
[Table]
[Table]
[Table]
I NCORPORATING R ESOURCE D ESCRIPTIONS
We next evaluate methods of incorporating resource descriptions into the models.
For each architecture we apply the docstring approach of processing resource descriptions from Section 4.2.
This completes a matrix of ablations, allowing us to distinguish the effects due to architecture change from the effect of the resource description.
We follow the same experimental setup as in Section 5.1, and show the results in Table 1 (compare rows with check marks to corresponding rows without).
We also consider the FiLM and cross-attention methods of incorporating resource descriptions into the IPA-GNN.
Following the same experimental setup again, we show the results of this experiment in Table 2.
Note that the best model overall by our model selection criteria on validation data was the IPA-GNN with cross-attention, though the Exception IPA-GNN performed better on the test split.
RQ2: Across all architectures it is clear that external resource descriptions are essential for improved performance on the runtime error prediction task.
On the IPA-GNN architectures, we see further improvements by considering architectures that incorporate the resource description directly into the execution step of the model, but these gains are inconsistent.
Critically, using any resource description method is better than none at all.
To understand how the resource descriptions lead to better performance, we compare in Figurethe instruction pointer values of two Exception IPA-GNN models on a single example (shown in
Table 8).
The model with the resource description recognizes that the input() calls will read input beyond the end of the stdin stream.
In contrast, the model without the resource description has less reason to suspect an error would be raised by those calls.
The descriptions of stdin in the
Python Runtime Errors dataset also frequently reveal type information, expected ranges for numeric values, and formatting details about the inputs.
We visualize additional examples in Appendix G.
5. I NTERPRETABILITY AND L OCALIZATION
We next investigate the Exception IPA-GNN model’s surprising ability to localize runtime errors without any localization supervision.
In unsupervised localization, the models predict the location of the error despite being trained only with supervision indicating error presence and kind.
Multiple Instance Learning Baselines
The unsupervised localization task may be viewed as multiple instance learning (MIL)
(Dietterich et al., 1997) by treating the subtask of predicting whether a particular line contains an error as an instance.
We thus consider as baselines two variations on the Transformer architecture, each using multiple instance learning.
The first is the “Local MIL
Transformer”, in which each statement in the program is encoded individually, as in the local node embeddings computation of Section 4.1.
The second is the “Global MIL Transformer”, in which all tokens in the program may attend to all other tokens in the Transformer encoder.
In both cases, the models make per-line predictions, aggregated to form an overall prediction (details in Appendix F).
Localization Experiment We use the same experimental protocol as in Section 5.1, and train each of the MIL Transformer and Exception IPA-GNN models.
As before, the models are trained only to minimize cross-entropy loss on predicting error kind and presence, receiving no supervision as to the location of the errors.
We report the localization results in Table 3, measuring the percent of test examples with errors for which the model correctly predicts the line number of the error.
RQ3: The Exception IPA-GNN’s unsupervised localization capabilities far
exceed that of the baseline approaches.
In Figure 2 we see the flow of instruction pointer mass during the execution of a sample program (Table 8) by two Exception IPA-GNN models, including the steps where the models
[Table]
raise probability mass to n error .
Tallying the contributions to n error from each node yields the exception provenance values in the right half of Table 8.
This shows the model’s internal state resembles plausible program executions and allows for unsupervised localization.
As a beneficial side-effect of learning plausible executions, the Exception IPA-GNN can localize the exceptions it predicts.
[Figure]
D ISCUSSION
In this work, we introduce the new task of predicting runtime errors in competitive programming problems and advance the capabilities of interpreter-inspired models.
We tackle the additional complexity of real-world code and demonstrate how natural language descriptions of external resources can be leveraged to reduce the ambiguity that arises in a static analysis setting.
We show that the resulting models outperform standard alternatives like Transformers and that the inductive bias built into these models allows for interesting interpretability in the context of unsupervised localization.
Though they perform best, current IPA-GNN models require taking many steps of execution, up to 174 on this dataset.
A future direction is to model more steps of program execution with each model step, to reduce the number of model steps necessary for long programs.
Extending the interpreterinspired models with additional interpreter features, or to support multi-file programs or programs with multiple user-defined functions is also an interesting avenue for future work.
Learning to understand programs remains a rich area of inquiry for machine learning research because of its complexity and the many interacting facets of code.
Learning to understand execution behavior is particularly challenging as programs grow in complexity, and as they depend on more external resources whose contents are not present in the code.
Our work presents a challenging problem and advances interpreter-inspired models, both of which we hope are ingredients towards making progress on these difficult and important problems.
A P YTHON R UNTIME E RROR D ATASET D ETAILS
We describe in detail the construction of the Python Runtime Error dataset from the submissions in Project CodeNet (Puri et al., 2021).
The Project CodeNet dataset contains over 14 million submissions to 4,053 distinct competitive programming problems, with the submissions spanning more than 50 programming languages.
We partition the problems into train, valid, and test splits at an 80:10:10 ratio.
By making all submissions to the same problem part of the same split we mitigate concerns about potential data leakage from similar submissions to the same problem.
We restrict our consideration to Python submissions, which account for 3,286,314 of the overall Project CodeNet submissions, with 3,119 of the problems receiving at least one submission in Python.
In preparing the dataset we execute approximately 3 million problems in a sandboxed environment to collect their runtime error information, we perform two stages of filtering on the dataset, syntactic and complexity filtering, and we construct a textual representation of the input space for each problem from the problem description.
A. S YNTACTIC F ILTERING
In this first phase of filtering, we remove submissions in Python 2 as well as those which fail to parse and run from our dataset.
We remove 76,888 programs because they are in Python 2, 59, programs because they contain syntax errors, 2,011 programs that result in runtime errors during parsing, and 6 additional programs for which the python-graphs library fails to construct a control flow graph.
A program may result in a runtime error during parsing if it contains return, break, continue keywords outside of an appropriate frame.
A. P ROGRAM E XECUTION
We attempt to run each submission in a sandboxed environment using the sample input provided in the Project CodeNet dataset.
The environment is a custom harness running on a Google Cloud
Platform (GCP) virtual environment.
This allows us to collect standard out and standard error, to monitor for timeouts, and to catch and serialize any Python exceptions raised during execution.
We restrict execution of each program to 1 second, marking any program exceeding this time as a timeout error.
If the program encounters a Python exception, we use the name of that exception as the target class for the program.
If an error type occurs only once in the dataset, we consider the target class to be Other.
Programs not exhibiting an error or timeout are given target class “no error”.
In addition to collecting the target class, we record for each runtime error the line number at which the error occurs.
We use these line numbers as the ground truth for the unsupervised error localization task considered in Section 5.3.
A. P ARSING I NPUT S PACE I NFORMATION FROM P ROBLEM S TATEMENTS
For each problem, we parse the problem statement to extract the input description and input constraints , if they exist.
Problem statements are written either in English or Japanese, and so we write our parser to support both languages.
When one or more of these two sections are present in the problem statement, we construct an input space description containing the contents of the present sections.
For the experiments that use input space information as a docstring, we prepend to each submission our the input space description for its corresponding problem.
Similarly the input space descriptions are used in the experiments that process input space information with either cross-attention or FiLM.
A. V OCABULARY C ONSTRUCTION AND C OMPLEXITY F ILTERING
All experiments use the same vocabulary and tokenization procedure.
For this, we select the standard Byte-Pair Encoding (BPE) tokenization procedure (Sennrich et al., 2016).
We construct the vocabulary using 1,000,000 submissions selected from the training split, along with the input space descriptions constructed for all problems in the train split.
We use a vocabulary size of 30,000.
We then apply size-based filtering, further restricting the set of programs considered.
First, the program length after tokenization is not to exceed 512 tokens, the number of nodes and edges in the control flow graph are each not to exceed 128, and the step limit T ( x ) for a program computed in Appendix C is not to exceed 174.
We select these numbers to trim the long tail of exceptionally long programs, and this filtering reduces the total number of acceptable programs by less than 1%.
To achieve consistent datasets comparable across all experiments, we use the longest form of each program (the program augmented with its input space information as a docstring), when computing the program sizes for size-based submission filtering.
We further impose the restriction that no user-defined functions (UDFs) are called in a submission; this further reduces the number of submissions by 682,220.
A user-defined function is a function defined in the submission source code, as opposed to being a built-in or imported from a third party module.
Extending the IPA-GNN models to submissions with UDFs called at most once is trivially achieved by replacing the program’s control flow graph with its interprocedural control flow graph (ICFG) (Nielson & Nielson, 1999).
We leave the investigation of modelling user-defined functions to further work.
A. F INAL D ATASET D ETAILS
After applying syntactic filtering (only keeping Python 3 programs that parse) and complexity filtering (eliminating long programs and programs that call user-defined functions), we are left with a dataset of 2,441,130 examples.
The division of these examples by split and by target class is given in Table 4.
Figure 3 shows the distribution of program lengths in lines represented in the completed dataset, with an average program length of 14.2 lines.
The average statement length is 6.7 tokens, with full distribution shown in Figure 4.
B I NPUT M ODULATION
We consider cross-attention (Lee et al., 2019) and Feature-wise Linear Modulation (FiLM)
(Perez et al., 2017) as the Modulate function in Section 4.2.
After embedding the node and the resource
[Table]
description we use cross-attention as follows to modulate the input.
MultiHead(Embed( x n ) , d ( x ) , h t − 1 ,n ) =
Concat(Concat( head 1 , ..., head h ) W O , Embed( x n )) (9) where head i = softmax (cid:32)
QK ′ √ d k (cid:33)
V (10)
Q = W Qi Concat( Embed ( x n ) , h t − 1 ,n ) (11)
K = W Ki d ( x ) (12)
V =
W Vi d ( x ) (13)
Here, W O ∈ R hd v × d model , W Qi ∈ R d k × ( d model + d Embed( xn ) ) , W Ki ∈ R
d k × d d ( x ) , and W Vi ∈
R d v × d d ( x ) are learnable parameters.
Similarly, for FiLM we modulate the input with the resource description as follows:
[Equation]
where β
= σ
[Equation]
[Equation]
[Equation]
where W γ ∈ R d
d ( x ) × ( d model + d Embed( xn ) ) , and W γ ∈ R d d ( x ) × ( d model + d Embed( xn ) ) are learnable parameters.
C H YPERPARAMETER S ELECTION
We select hyperparameters by performing a random search independently for each model architecture.
The hyperparameters considered by the search are listed in Table 6.
All architectures use a
Transformer encoder, and the Transformer sizes considered in the search are listed in Table 6 and defined further in Table 5.
[Table]
For the IPA-GNN and Exception IPA-GNN, the function T ( x ) represents the number of execution steps modeled for program x .
We reuse the definition of T ( x ) from Bieber et al.
(2020) as closely as possible, only modifying it to accept arbitrary Python programs, rather than being restricted to the subset of Python features considered in the dataset of the earlier work.
D L OCALIZATION BY M ODELING E XCEPTION H ANDLING
For programs that lack try/except frames, we compute the localization predictions of the Exception
IPA-GNN model by summing, separately for each node, the contributions from that node to the exception node across all time steps.
This gives an estimate of exception provenance as
[Equation]
( error at statement n )
[Equation]
For programs with a try/except frame, however, we must trace the exception back to the statement that originally raised it.
To do this, we keep track of the exception provenance at each node at each time step; when an exception raises, it becomes the exception provenance at the statement that it raises to, and when a statement with non-zero exception provenance executes without raising, it propagates its exception provenance to the next node unchanged.
Define v t,n,n ′ as the amount of “exception probability mass” at time step t at node n ′ attributable to an exception starting at node n .
Then we write v t,n,n ′ =
(cid:88)
[Equation]
[Table]
The first term propagates exception provenance across normal non-raising execution, while the second term introduces exception provenance when an exception is raised.
We then write precisely p ( error at statement n )
= v T ( x ) ,n,n error , (19) allowing the Exception IPA-GNN to make localization predictions for any program in the dataset.
E M ETRIC V ARIANCES
Under the experimental conditions of Section 5.1, we perform three additional training runs to calculate the variance for each metric for each baseline model, and for the Exception IPA-GNN model using the docstring strategy for processing resource descriptions.
For these new training runs, we use the hyperparameters obtained from model selection.
We vary the random seed between runs (0, 1, 2), thereby changing the initialization and dropout behavior of each model across runs.
We report the results in Table 7; ± values are one standard deviation.
[Table]
F M ULTIPLE I NSTANCE L EARNING
The Local Transformer and Global Transformer models each compute per-statement node embeddings Embed( x n ) given by Equation 1.
In the multiple instance learning setting, these are transformed into unnormalized per-statement class predictions
[Equation]
We consider three strategies for aggregating these per-statement predictions into an overall prediction for the task.
Under the logsumexp strategy, we treat ϕ as logits and write
[Equation]
where K is the set of error classes.
The max and mean strategies meanwhile follow Wang et al.
in asserting
[Equation]
compute the location probabilities as
[Equation] [Equation]
and compute the outputs as
[Equation]
respectively, where L denotes the number of statements in x .
As with all methods considered, the MIL models are trained to minimize the cross-entropy loss in target class prediction, but these methods still allow reading off predictions of p ( lineno ) .
G E XAMPLE VISUALIZATIONS
Additional randomly sampled examples from the Python Runtime Error dataset validation split are shown here.
As in Figure 2, prediction visualizations for these examples are shown for the selected
B ASELINE and D OCSTRING Exception IPA-GNN model variants.
In instruction pointer value heatmaps, the x-axis represents timesteps and the y-axis represents nodes, with the last two rows respectively representing the exit node n exit and the exception node n error .
Note that for loop statements are associated with two spans in the statement-level control flow graph, one for the construction of the iterator, and a second for assignment to the loop variable.
Hence we list two indexes for each for loop statement in these figures, and report the total error contribution for the line.
[Figure]
Input:
Input is given from Standard Input in the following format:
N a 1 a 2 ... a N Constraints:
All values in input are integers.
[Figure]
<= [Figure] a i <= [Figure]
S OURCE
B ASELINE Error contrib.
R.D. Error contrib
[Figure]
=
[Figure] [Figure]
(int,
[Figure]
[Figure] [Figure] res
[Figure]
for i in range(1, len(A)+1, 2):
[Figure]
+= A[i]
[Figure]
[Figure]
print(res)
[Figure]
NDEX E RROR , occuring on line 5 ( n = 5 ).
B ASELINE incorrectly predicts N O ERROR with confidence 0.808.
D OCSTRING correctly predicts I NDEX E RROR with confidence 0.693, but localizes to line 3 ( n = 2 ).
Both B ASELINE and D OCSTRING instruction pointer values start out sharp and become diffuse when reaching the for-loop.
The B ASELINE instruction pointer value ends with most probability mass at n exit .
The D OCSTRING instruction pointer value has a small amount of probability mass reaching n exit , with most probability mass ending at n error .
Finally, we show in tabular form the error localization predictions corresponding to Figure 2 of
Section 5.3.
S TD I N D ESCRIPTION
Input:
Input is given from Standard Input in the following format:
H N A 1 A 2 ...
A N Constraints: 1 <=
H <
= 10ˆ9 1 <= N <
= 10ˆ5 1 <
=
A i <= 10ˆ
All values in input are integers.
n
S OURCE
B ASELINE Error contrib.
R.D. Error contrib.
H,N,A = list(map(int, input().split())) 9.
1, for i in A[N]: 43.
83.0if H < = 0: 2.
2.8 break else:
H -= A[i] 6.
0.0if set(A): 0.
0.1print("Yes") 9.
0.7 else: print("No") 3.
B ASELINE
R ESOURCE D ESCRIPTION
S
I N D ESCRIPTION
Input:
Input is given from Standard Input in the following format Constraints:
S OURCE
B ASELINE
R.D.
a = str(input()) 16.
0.4q =
int(input()) 3.
0.3s =
input
.split( for i in range(q)] 0.
99.3 3, for i in range(q): 6.
0.0if int(s[i][0]) =
= 1 and len(a) > 1: 0.
0.0a = a[::-1] 0.
0.0elif int(s[i][0])== 2 and int(s[i][1])==1: 0.
0.0a=s[i][2]+a 0.
0.0 else: a=a+s[i][2] 0.
0.0print(a) 1.
S TD I N D ESCRIPTION
Input: n m d1 d2 ... dm Two integers n and m are given in the first line.
The available denominations are given in the second line.
Constraints: 1 <= n <= 50000 1 <= m <= 20 1 <= denomination <=
The denominations are all different and contain 1.
n
S OURCE
B ASELINE Error contrib.
R.D. Error contrib.
from itertools import combinations with replacement as C 40.
n, m = map(int, input().split()) 2.
coin = sorted(list(map(int, input().split()))) 7.
if n in coin: 2.
0.2print(1) 2.
1.5 else: end = n // coin[0] +0.
0.1b =
False 0.
0.3 7, for i in range(2, end): 2.
0.7 9, for tup in list(C(coin, i)): 3.
1.2if sum(tup) =
= n: 0.
0.0print(i) 0.
0.1b =
True 0.
0.9 break if b: break 0.
B ASELINE
R ESOURCE D ESCRIPTION
The B ASELINE instruction pointer value makes its largest probability mass contribution to n error at n = 0 and ends up with mass split between n exit and n error .
The D OCSTRING instruction pointer value accumulates little probability in n error and ends up with most probability mass in n exit .