Original: S TATIC P REDICTION OF R UNTIME E RRORS BY L EARNING TO E XECUTE P ROGRAMS WITH E XTERNAL R ESOURCE D ESCRIPTIONS
Revised: S TATIC P REDICTION OF R UNTIME E RRORS BY L EARNING TO E XECUTE P ROGRAMS WITH E XTERNAL R ESOURCE D ESCRIPTIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The execution behavior of a program often depends on external resources, such as program inputs or file contents, and so the program cannot be run in isolation.
Revised: The execution behavior of a program often depends on external resources, such as program inputs or file contents, and so the program cannot be run in isolation.

Original: Nevertheless, software developers benefit from fast iteration loops where automated tools identify errors as early as possible, even before programs can be compiled and run.
Revised: Nevertheless, software developers benefit from fast iteration loops where automated tools identify errors as early as possible, even before programs can be compiled and run.

Original: This presents an interesting machine learning challenge: can we predict runtime errors in a “static” setting, where program execution is not possible?
Revised: This presents an interesting machine learning challenge: can we predict runtime errors in a “static” setting, where program execution is not possible?

Original: Here, we introduce a competitive programming dataset and task for predicting runtime errors, which we show is difficult for generic models like Transformers.
Revised: Here, we introduce a competitive programming dataset and task for predicting runtime errors, which we show is difficult for generic models like Transformers.

Original: We approach this task by developing an interpreter-inspired architecture with an inductive bias towards mimicking program executions, which models exception handling and “learns to execute” descriptions of external resources.
Revised: We approach this task by developing an interpreter-inspired architecture with an inductive bias towards mimicking program executions, which models exception handling and “learns to execute” descriptions of external resources.

Original: Surprisingly, we show that the model can also predict the locations of errors, despite being trained only on labels indicating error presence or absence and kind.
Revised: Surprisingly, we show that the model can also predict the locations of errors, despite being trained only on labels indicating error presence or absence and kind.

Original: In total, we present a practical and difficult-yet-approachable challenge problem related to learning program execution behavior and we demonstrate promising new capabilities of interpreter-inspired machine learning models for code.
Revised: In total, we present a practical and difficult-yet-approachable challenge problem related to learning program execution behavior and we demonstrate promising new capabilities of interpreter-inspired machine learning models for code.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: We investigate applying neural machine learning methods to the static analysis of source code for early prediction of runtime errors.
Revised: We investigate applying neural machine learning methods to the static analysis of source code for early prediction of runtime errors.

Original: The execution behavior of a program is in general not fully defined by its source code in isolation, because programs often rely on external resources like inputs, the contents of files, or the network.
Revised: The execution behavior of a program is in general not fully defined by its source code in isolation, because programs often rely on external resources like inputs, the contents of files, or the network.

Original: Nevertheless, software developers benefit from fast iteration loops where automated tools identify errors early, even when program execution is not yet an option.
Revised: Nevertheless, software developers benefit from fast iteration loops where automated tools identify errors early, even when program execution is not yet an option.

Original: Therefore we consider the following machine learning challenge: can we predict runtime errors in a “static” setting, where program execution is not possible?
Revised: Therefore we consider the following machine learning challenge: can we predict runtime errors in a “static” setting, where program execution is not possible?

Original: This runtime error prediction task is well suited as a challenge problem because it is difficult-yetapproachable, has real-world value for software developers, requires novel modeling considerations that we hypothesize will be applicable to a range of learning for code tasks, and with this work, now has a suitable large dataset of complex human-authored code with error labels.
Revised: This runtime error prediction task is well suited as a challenge problem because it is difficult-yetapproachable, has real-world value for software developers, requires novel modeling considerations that we hypothesize will be applicable to a range of learning for code tasks, and with this work, now has a suitable large dataset of complex human-authored code with error labels.

Original: The task is to predict whether a program will exhibit a runtime error when it is run, and if so to determine the error; even when static analysis cannot provide guarantees of an error in the code, patterns learned from data may point to likely errors.
Revised: The task is to predict whether a program will exhibit a runtime error when it is run, and if so to determine the error; even when static analysis cannot provide guarantees of an error in the code, patterns learned from data may point to likely errors.

Original: Our dataset consists of 2.4 million Python 3 programs from
Revised: Our dataset consists of 2.4 million Python 3 programs from

Original: Project CodeNet (Puri et al., 2021) written by competitive programmers.
Revised: Project CodeNet (Puri et al., 2021) written by competitive programmers.

Original: We have run all programs in a sandboxed environment on sample inputs to determine their error classes, finding the programs exhibit 26 distinct error classes including “no error”.
Revised: We have run all programs in a sandboxed environment on sample inputs to determine their error classes, finding the programs exhibit 26 distinct error classes including “no error”.

Original: Each program relies on an external resource, the stdin input stream, and we pair the programs with a natural language description of the behavior of the stream.
Revised: Each program relies on an external resource, the stdin input stream, and we pair the programs with a natural language description of the behavior of the stream.

Original: We make the task and dataset, along with all models considered in this work, available for the research community to facilitate reproduction of this work and further research 1 .
Revised: We make the task and dataset, along with all models considered in this work, available for the research community to facilitate reproduction of this work and further research 1 .

Original: To make progress on this challenging task, we identify a promising class of models from prior work, interpreter-inspired models, and we demonstrate they perform well on the task.
Revised: To make progress on this challenging task, we identify a promising class of models from prior work, interpreter-inspired models, and we demonstrate they perform well on the task.

Original: Instruction Pointer
Revised: Instruction Pointer

Original: Attention Graph Neural Network (IPA-GNN) (Bieber et al., 2020) models simulate the execution of a program, following its control flow structure, but operating in a continuous embedding space.
Revised: Attention Graph Neural Network (IPA-GNN) (Bieber et al., 2020) models simulate the execution of a program, following its control flow structure, but operating in a continuous embedding space.

Original: We make a number of improvements to IPA-GNN: scaling up to handle complex programs requiring thousands of execution steps, adding the ability to “learn to execute” descriptions of external resources, and extending the architecture to model exception handling and recover error locations.
Revised: We make a number of improvements to IPA-GNN: scaling up to handle complex programs requiring thousands of execution steps, adding the ability to “learn to execute” descriptions of external resources, and extending the architecture to model exception handling and recover error locations.

Original: We evaluate these interpreter-inspired architectures against Transformer, LSTM, and GGNN neural baselines, and against pylint as a static analysis baseline.
Revised: We evaluate these interpreter-inspired architectures against Transformer, LSTM, and GGNN neural baselines, and against pylint as a static analysis baseline.

Original: Our combined improvements lead to increased accuracy in predicting runtime errors and to interpretability allowing for prediction of error locations even though the models are only trained on error presence and error class, not error location.
Revised: Our combined improvements lead to increased accuracy in predicting runtime errors and to interpretability allowing for prediction of error locations even though the models are only trained on error presence and error class, not error location.

Original: In total, we summarize our contributions as:
Revised: In total, we summarize our contributions as:

Original: • We introduce the runtime error prediction task and a large accompanying dataset, providing runtime error annotations for millions of competition Python programs.
Revised: • We introduce the runtime error prediction task and a large accompanying dataset, providing runtime error annotations for millions of competition Python programs.

Original: • We demonstrate that IPA-GNN architectures are practical for the complexity of real programs by scaling them to handle competition programs, and there we find they outperform generic models.
Revised: • We demonstrate that IPA-GNN architectures are practical for the complexity of real programs by scaling them to handle competition programs, and there we find they outperform generic models.

Original: • We demonstrate that external resource descriptions, such as Japanese or English descriptions of stdin, can be leveraged to improve performance on the task across all model architectures.
Revised: • We demonstrate that external resource descriptions, such as Japanese or English descriptions of stdin, can be leveraged to improve performance on the task across all model architectures.

Original: • We extend the IPA-GNN to model exception handling, resulting in the Exception IPA-GNN, which we find can localize errors even when only trained on error presence and kind, not error location.
Revised: • We extend the IPA-GNN to model exception handling, resulting in the Exception IPA-GNN, which we find can localize errors even when only trained on error presence and kind, not error location.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Program analysis Program analysis is a rich family of techniques for detecting defects in programs, including static analyses which are performed without executing code (Livshits and Lam, 2005; Xie and Aiken, 2006; Ayewah et al., 2008) and dynamic analyses which are performed at runtime (Cadar et al., 2008; Sen et al., 2005; Godefroid et al., 2005).
Revised: Program analysis Program analysis is a rich family of techniques for detecting defects in programs, including static analyses which are performed without executing code (Livshits and Lam, 2005; Xie and Aiken, 2006; Ayewah et al., 2008) and dynamic analyses which are performed at runtime (Cadar et al., 2008; Sen et al., 2005; Godefroid et al., 2005).

Original: Linters and type checkers are popular error detection tools that use static analysis.
Revised: Linters and type checkers are popular error detection tools that use static analysis.

Original: Static analysis (e.g. symbolic execution) does not typically use concrete inputs, while dynamic analysis requires concrete inputs and program execution.
Revised: Static analysis (e.g. symbolic execution) does not typically use concrete inputs, while dynamic analysis requires concrete inputs and program execution.

Original: Compared with traditional static analysis, our approach is more flexible in its input representation, using a general “resource description” abstraction, which can represent the entire spectrum from concrete inputs to input constraints to missing inputs.
Revised: Compared with traditional static analysis, our approach is more flexible in its input representation, using a general “resource description” abstraction, which can represent the entire spectrum from concrete inputs to input constraints to missing inputs.

Original: Execution-aware models Several neural architectures draw inspiration from program interpreters (Graves et al., 2014; Łukasz Kaiser and Sutskever, 2016; Reed and de Freitas, 2016; Graves et al., 2016; Boˇsnjak et al., 2017; Gaunt et al., 2017; Dehghani et al., 2019; Bieber et al., 2020).
Revised: Execution-aware models Several neural architectures draw inspiration from program interpreters (Graves et al., 2014; Łukasz Kaiser and Sutskever, 2016; Reed and de Freitas, 2016; Graves et al., 2016; Boˇsnjak et al., 2017; Gaunt et al., 2017; Dehghani et al., 2019; Bieber et al., 2020).

Original: Our work is most similar to Bieber et al.
Revised: Our work is most similar to Bieber et al.

Original: and Bo ˇ snjak et al.
Revised: and Bo ˇ snjak et al.

Original: (2017), focusing on how interpreters handle control flow and exception handling, rather than on memory allocation and function call stacks.
Revised: (2017), focusing on how interpreters handle control flow and exception handling, rather than on memory allocation and function call stacks.

Original: Other works use program execution data directly, training with textual representations of execution traces as inputs (Nye et al., 2021a;
Revised: Other works use program execution data directly, training with textual representations of execution traces as inputs (Nye et al., 2021a;

Original: Pei et al., 2021; Nye et al., 2021b) or performing execution during synthesis (Chen et al., 2019; Li et al., 2022; Shrivastava et al., 2021).
Revised: Pei et al., 2021; Nye et al., 2021b) or performing execution during synthesis (Chen et al., 2019; Li et al., 2022; Shrivastava et al., 2021).

Original: Compared with these our approach uses weaker supervision, using only runtime error labels for training.
Revised: Compared with these our approach uses weaker supervision, using only runtime error labels for training.

Original: Fault detection and localization datasets There has been considerable recent interest in applying machine learning to identifying and localizing faults in source code (Allamanis et al., 2018a).
Revised: Fault detection and localization datasets There has been considerable recent interest in applying machine learning to identifying and localizing faults in source code (Allamanis et al., 2018a).

Original: Puri et al.
Revised: Puri et al.

Original: (2021) makes a large dataset of real world programs available, which we build on in constructing our runtime errors dataset.
Revised: (2021) makes a large dataset of real world programs available, which we build on in constructing our runtime errors dataset.

Original: Our dataset (i) is large (it has millions of examples), (ii) exhibits many programming language features, (iii) is written by human authors, and (iv) has error labels from the execution behavior of programs.
Revised: Our dataset (i) is large (it has millions of examples), (ii) exhibits many programming language features, (iii) is written by human authors, and (iv) has error labels from the execution behavior of programs.

Original: Previous code datasets only exhibit a subset of these properties: large real-world and competition code datasets (Hendrycks et al., 2021; Li et al., 2022;
Revised: Previous code datasets only exhibit a subset of these properties: large real-world and competition code datasets (Hendrycks et al., 2021; Li et al., 2022;

Original: Kanade et al., 2020; Raychev et al., 2016; Husain et al., 2019; Puri et al., 2021) exhibit properties i, ii, and iii, but not iv, while learning to execute datasets (Zaremba and Sutskever, 2014; Bieber et al., 2020) exhibit property iv
Revised: Kanade et al., 2020; Raychev et al., 2016; Husain et al., 2019; Puri et al., 2021) exhibit properties i, ii, and iii, but not iv, while learning to execute datasets (Zaremba and Sutskever, 2014; Bieber et al., 2020) exhibit property iv

Original: but not i, ii, or iii.
Revised: but not i, ii, or iii.

Original: Recent program synthesis datasets (Chen et al., 2021;
Revised: Recent program synthesis datasets (Chen et al., 2021;

Original: Austin et al., 2021) exhibit ii and iii only.
Revised: Austin et al., 2021) exhibit ii and iii only.

Original: Other datasets obtain error labels by injecting synthetic errors (Allamanis et al., 2018b; Karampatsis and Sutton, 2020; Pradel and Sen, 2018)
Revised: Other datasets obtain error labels by injecting synthetic errors (Allamanis et al., 2018b; Karampatsis and Sutton, 2020; Pradel and Sen, 2018)

Original: (lacking the realism of iii) or from commit messages (Just et al., 2014; Dinella et al., 2020) (lacking i and iv).
Revised: (lacking the realism of iii) or from commit messages (Just et al., 2014; Dinella et al., 2020) (lacking i and iv).

Original: Fault localization approaches Fault localization approaches vary in (i) level of supervision – weak (error labels)
Revised: Fault localization approaches Fault localization approaches vary in (i) level of supervision – weak (error labels)

Original: (Li et al., 2019) vs strong (explicit location labels) (Lou et al., 2021;
Revised: (Li et al., 2019) vs strong (explicit location labels) (Lou et al., 2021;

Original: Zhang et al.,
Revised: Zhang et al.,

Original: [Table]
Revised: [Table]

Original: ARGET
Revised: ARGET

Original: [Table]
Revised: [Table]

Original: LASS
Revised: LASS

Original: T RAIN #
Revised: T RAIN #

Original: V ALID #
Revised: V ALID #

Original: [Table]
Revised: [Table]

Original: EST
Revised: EST

Original: [Table]
Revised: [Table]

Original:  R UNTIME E RROR P REDICTION
Revised:  R UNTIME E RROR P REDICTION

Original: Task
Revised: Task

Original: The goal of the runtime error prediction task is to determine statically whether a program is liable to encounter a runtime error when run, and if so, what error kind.
Revised: The goal of the runtime error prediction task is to determine statically whether a program is liable to encounter a runtime error when run, and if so, what error kind.

Original: The programs cannot be executed directly, as they lack unit tests and depend on external resources which are not available.
Revised: The programs cannot be executed directly, as they lack unit tests and depend on external resources which are not available.

Original: A textual description of the external resources, which may be the program’s inputs, a file’s contents, or network access, is provided.
Revised: A textual description of the external resources, which may be the program’s inputs, a file’s contents, or network access, is provided.

Original: This makes reasoning about the execution behavior of the programs plausible, even though actually performing the execution is not.
Revised: This makes reasoning about the execution behavior of the programs plausible, even though actually performing the execution is not.

Original: We treat this task as one-class classification, with each error type as its own class and with “no error” as an additional class.
Revised: We treat this task as one-class classification, with each error type as its own class and with “no error” as an additional class.

Original: Dataset We construct the runtime errors dataset using Python submissions to competitive programming problems from Project CodeNet (Puri et al., 2021).
Revised: Dataset We construct the runtime errors dataset using Python submissions to competitive programming problems from Project CodeNet (Puri et al., 2021).

Original: Beginning with the 3.28 million
Revised: Beginning with the 3.28 million

Original: Python submissions in Project CodeNet, we filter the submissions to keep only those written in
Revised: Python submissions in Project CodeNet, we filter the submissions to keep only those written in

Original: Python 3, which are syntactically valid, which do not make calls to user-defined functions, and which do not exceed a threshold length of 512 tokens once tokenized.
Revised: Python 3, which are syntactically valid, which do not make calls to user-defined functions, and which do not exceed a threshold length of 512 tokens once tokenized.

Original: By running each submission in a sandboxed environment, we identify its ground truth runtime error class.
Revised: By running each submission in a sandboxed environment, we identify its ground truth runtime error class.

Original: Each submission is associated with a competitive programming problem whose problem statement we parse to obtain a description in either English or Japanese of the inputs the program is liable to receive at runtime.
Revised: Each submission is associated with a competitive programming problem whose problem statement we parse to obtain a description in either English or Japanese of the inputs the program is liable to receive at runtime.

Original: This process results in a dataset of 2.44 million submissions, each paired with one of 26 target classes.
Revised: This process results in a dataset of 2.44 million submissions, each paired with one of 26 target classes.

Original: The “no error” target is most common, accounting for 93.4% of examples.
Revised: The “no error” target is most common, accounting for 93.4% of examples.

Original: For examples with one of the other 25 error classes, we additionally note the line number at which the error occurs.
Revised: For examples with one of the other 25 error classes, we additionally note the line number at which the error occurs.

Original: We divide the problems into train, validation, and test splits at a ratio of 80:10:10.
Revised: We divide the problems into train, validation, and test splits at a ratio of 80:10:10.

Original: All submissions to the same problem become part of the same split.
Revised: All submissions to the same problem become part of the same split.

Original: This reduces similarities between examples across splits that otherwise could arise from the presence of multiple similar submissions for the same problem.
Revised: This reduces similarities between examples across splits that otherwise could arise from the presence of multiple similar submissions for the same problem.

Original: Since there is a strong class imbalance in favor of the no error class, we also produce a balanced version of the test split by sampling the no error examples such that they comprise roughly 50% of the test split.
Revised: Since there is a strong class imbalance in favor of the no error class, we also produce a balanced version of the test split by sampling the no error examples such that they comprise roughly 50% of the test split.

Original: We use this balanced test split for all evaluations.
Revised: We use this balanced test split for all evaluations.

Original: We report the number of examples having each target class in each split in Table 1.
Revised: We report the number of examples having each target class in each split in Table 1.

Original: We describe the full dataset generation and filtering process in greater detail in Appendix A, and we evaluate the limitations of the dataset quantitatively in Appendix B.
Revised: We describe the full dataset generation and filtering process in greater detail in Appendix A, and we evaluate the limitations of the dataset quantitatively in Appendix B.

Original: While there are many datasets in the literature that test understanding of different aspects of code including bug-finding, we believe ours fills a gap: it is large-scale (millions of examples), it has real-world implications and presents a practical opportunity for improvement using ML-based approaches, and it tests a combination of statistical reasoning and reasoning about program execution.
Revised: While there are many datasets in the literature that test understanding of different aspects of code including bug-finding, we believe ours fills a gap: it is large-scale (millions of examples), it has real-world implications and presents a practical opportunity for improvement using ML-based approaches, and it tests a combination of statistical reasoning and reasoning about program execution.

Original:  A PPROACH : IPA-GNN S AS R ELAXATIONS OF I NTERPRETERS
Revised:  A PPROACH : IPA-GNN S AS R ELAXATIONS OF I NTERPRETERS

Original: We make three modifications to the Instruction Pointer Attention Graph Neural Network (IPA-GNN architecture.
Revised: We make three modifications to the Instruction Pointer Attention Graph Neural Network (IPA-GNN architecture.

Original: These modifications scale the IPA-GNN to complex code, allow it to incorporate external resource descriptions into its learned executions, and add support for modeling exception handling.
Revised: These modifications scale the IPA-GNN to complex code, allow it to incorporate external resource descriptions into its learned executions, and add support for modeling exception handling.

Original: The IPA-GNN architecture is a continuous relaxation of the standard interpreter ( I ) defined by the pseudocode in Algorithm 1, minus the magenta text.
Revised: The IPA-GNN architecture is a continuous relaxation of the standard interpreter ( I ) defined by the pseudocode in Algorithm 1, minus the magenta text.

Original: We frame these modifications in relation to specific lines of the algorithm: scaling the IPA-GNN to complex human-authored code (Section 4.1) and incorporating external resource descriptions (Section 4.2) both pertain to interpreting and executing statement x p at Line 3, and modeling exception handling adds the magenta text at lines 4-6 to yield a new interpreter ( I ′ )
Revised: We frame these modifications in relation to specific lines of the algorithm: scaling the IPA-GNN to complex human-authored code (Section 4.1) and incorporating external resource descriptions (Section 4.2) both pertain to interpreting and executing statement x p at Line 3, and modeling exception handling adds the magenta text at lines 4-6 to yield a new interpreter ( I ′ )

Original: (Section 4.3).
Revised: (Section 4.3).

Original: We showcase the behavior of both interpreters
Revised: We showcase the behavior of both interpreters

Original: I and I ′ on a sample program in Figure 1, and illustrate an execution of the same program by a continuous relaxation of interpreter I ′
Revised: I and I ′ on a sample program in Figure 1, and illustrate an execution of the same program by a continuous relaxation of interpreter I ′

Original: ( ˜ I ′ ) alongside it.
Revised: ( ˜ I ′ ) alongside it.

Original: 4. E XTENDING THE IPA-GNN TO R EAL P ROGRAMS
Revised: 4. E XTENDING THE IPA-GNN TO R EAL P ROGRAMS

Original: Bieber et al.
Revised: Bieber et al.

Original: (2020) interprets the IPA-GNN architecture as a message passing graph neural network operating on the statement-level control-flow graph of the input program x .
Revised: (2020) interprets the IPA-GNN architecture as a message passing graph neural network operating on the statement-level control-flow graph of the input program x .

Original: Each node in the graph corresponds to a single statement in the program.
Revised: Each node in the graph corresponds to a single statement in the program.

Original: At each step t of the architecture, each node performs three steps: it executes the statement at that node (Line 3, Equation 2), computes a branch decision (Lines 7-8, Equation 4), and performs mean-field averaging over the resulting states and instruction pointers (Equations 5 and 6 in Bieber et al.
Revised: At each step t of the architecture, each node performs three steps: it executes the statement at that node (Line 3, Equation 2), computes a branch decision (Lines 7-8, Equation 4), and performs mean-field averaging over the resulting states and instruction pointers (Equations 5 and 6 in Bieber et al.

Original: (2020); Appendix C).
Revised: (2020); Appendix C).

Original: Unlike in Bieber et al.
Revised: Unlike in Bieber et al.

Original: (2020) where program statements are simple enough to be uniformly encoded as four-tuples, the programs in our runtime errors dataset consist of arbitrarily complex Python n
Revised: (2020) where program statements are simple enough to be uniformly encoded as four-tuples, the programs in our runtime errors dataset consist of arbitrarily complex Python n

Original: S OURCE x
Revised: S OURCE x

Original: [Equation]
Revised: [Equation]

Original: input() if x > 0: y = 4/3 * x else:
Revised: input() if x > 0: y = 4/3 * x else:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: exit> <raise>
Revised: exit> <raise>

Original: [Figure]
Revised: [Figure]

Original: S TD I NS TD I N D ESCRIPTION "
Revised: S TD I NS TD I N D ESCRIPTION "

Original: A SINGLE INTEGER -10..10"
Revised: A SINGLE INTEGER -10..10"

Original: [Figure]
Revised: [Figure]

Original: t h A,B p
Revised: t h A,B p

Original: I p I ′ h ˜
Revised: I p I ′ h ˜

Original: I ′ p
Revised: I ′ p

Original: ˜ I ′ {}    { x: -3 }    { x: -3 }    { x: -3, y: 3 }
Revised: ˜ I ′ {}    { x: -3 }    { x: -3 }    { x: -3, y: 3 }

Original: ValueError: line
Revised: ValueError: line

Original: [Figure]
Revised: [Figure]

Original: statements authored by real programmers in a competition setting.
Revised: statements authored by real programmers in a competition setting.

Original: The language features used are numerous and varied, and so the statement lengths vary substantially, with a mean statement length of 6.7 tokens; we report the full distribution of statement lengths in Figure 4.
Revised: The language features used are numerous and varied, and so the statement lengths vary substantially, with a mean statement length of 6.7 tokens; we report the full distribution of statement lengths in Figure 4.

Original: The IPA-GNN architecture operates on a program x ’s statement-level control-flow graph, and so requires per-statement embeddings Embed( x n ) for each statement x n .
Revised: The IPA-GNN architecture operates on a program x ’s statement-level control-flow graph, and so requires per-statement embeddings Embed( x n ) for each statement x n .

Original: We first apply either a local or global Transformer encoder to produce per-token embeddings, and we subsequently apply one of four pooling variants to a span of such embeddings to produce a node embedding per statement in a program.
Revised: We first apply either a local or global Transformer encoder to produce per-token embeddings, and we subsequently apply one of four pooling variants to a span of such embeddings to produce a node embedding per statement in a program.

Original: In the local approach, we apply an attention mask to limit the embedding of a token in a statement to attending to other tokens in the same statement.
Revised: In the local approach, we apply an attention mask to limit the embedding of a token in a statement to attending to other tokens in the same statement.

Original: In the global approach, no such attention mask is applied, and so every token may attend to every other token in the program.
Revised: In the global approach, no such attention mask is applied, and so every token may attend to every other token in the program.

Original: We consider four types of pooling in our hyperparameter search space: first , sum , mean , and max .
Revised: We consider four types of pooling in our hyperparameter search space: first , sum , mean , and max .

Original: The resulting embedding is given by
Revised: The resulting embedding is given by

Original: [Equation]
Revised: [Equation]

Original: First pooling takes the embedding of the first token in the span of node n .
Revised: First pooling takes the embedding of the first token in the span of node n .

Original: Sum, mean, and max pooling apply their respective operations to the embeddings of all tokens in the span of node n .
Revised: Sum, mean, and max pooling apply their respective operations to the embeddings of all tokens in the span of node n .

Original: Finally we find that the programs in our dataset require as many as 174 steps of the IPA-GNN under the model’s heuristic for step limit T ( x ) (Appendix E).
Revised: Finally we find that the programs in our dataset require as many as 174 steps of the IPA-GNN under the model’s heuristic for step limit T ( x ) (Appendix E).

Original: To reduce the memory requirements, we apply rematerialization at each layer of the model (Griewank and Walther, 2000; Chen et al., 2016).
Revised: To reduce the memory requirements, we apply rematerialization at each layer of the model (Griewank and Walther, 2000; Chen et al., 2016).

Original: 4. E XECUTING WITH R ESOURCE D ESCRIPTIONS
Revised: 4. E XECUTING WITH R ESOURCE D ESCRIPTIONS

Original: In our dataset, each program x may be accompanied by a description of what values stdin may contain at runtime.
Revised: In our dataset, each program x may be accompanied by a description of what values stdin may contain at runtime.

Original: We convert this description into embedding d ( x ) ; the embeddings, vocabulary, and tokenizer used to produce d ( x ) are shared with those used to produce token embeddings from program source.
Revised: We convert this description into embedding d ( x ) ; the embeddings, vocabulary, and tokenizer used to produce d ( x ) are shared with those used to produce token embeddings from program source.

Original: Analogous to Line 1 of Algorithm 1, IPA-GNN architectures initialize with pernode hidden states h 0 , : = 0 and soft instruction pointer p 0 ,n = 1 { n = 0 } .
Revised: Analogous to Line 1 of Algorithm 1, IPA-GNN architectures initialize with pernode hidden states h 0 , : = 0 and soft instruction pointer p 0 ,n = 1 { n = 0 } .

Original: Here p t,n represents the probability under the model that at node n is executing at step t .
Revised: Here p t,n represents the probability under the model that at node n is executing at step t .

Original: Following initialization, each step of an IPA-GNN begins by simulating execution (Line 3) of each non-terminal statement with non-zero probability under the soft instruction pointer to propose a new hidden state contribution a (1) t,n = RNN( h t − 1 ,n , Modulate(Embed( x n ) , d ( x ) , h t − 1 ,n )) .
Revised: Following initialization, each step of an IPA-GNN begins by simulating execution (Line 3) of each non-terminal statement with non-zero probability under the soft instruction pointer to propose a new hidden state contribution a (1) t,n = RNN( h t − 1 ,n , Modulate(Embed( x n ) , d ( x ) , h t − 1 ,n )) .

Original: [Equation]
Revised: [Equation]

Original: The text in magenta shows our modification to the IPA-GNN architecture to incorporate external resource descriptions.
Revised: The text in magenta shows our modification to the IPA-GNN architecture to incorporate external resource descriptions.

Original: We consider both Feature-wise Linear Modulation (FiLM)
Revised: We consider both Feature-wise Linear Modulation (FiLM)

Original: (Perez et al., 2017) and cross-attention (Lee et al., 2019) for the Modulate function, which we define in Appendix D.
Revised: (Perez et al., 2017) and cross-attention (Lee et al., 2019) for the Modulate function, which we define in Appendix D.

Original: Modulation allows the IPA-GNN to execute differently at each step conditioned on the information in the resource description, whether that be type information, value ranges, or candidate values.
Revised: Modulation allows the IPA-GNN to execute differently at each step conditioned on the information in the resource description, whether that be type information, value ranges, or candidate values.

Original: We also consider an additional method: injecting the description as a docstring at the start of the program.
Revised: We also consider an additional method: injecting the description as a docstring at the start of the program.

Original: This method yields a new valid Python program, and so any model can accommodate it.
Revised: This method yields a new valid Python program, and so any model can accommodate it.

Original: 4. M ODELING E XCEPTION H ANDLING
Revised: 4. M ODELING E XCEPTION H ANDLING

Original: The final modification we make to the IPA-GNN architecture is to model exception handling.
Revised: The final modification we make to the IPA-GNN architecture is to model exception handling.

Original: In
Revised: In

Original: Algorithm 1, this corresponds to adding the magenta text to form interpreter I ′ , computing a raise decision (Lines 4-6, Equation 3).
Revised: Algorithm 1, this corresponds to adding the magenta text to form interpreter I ′ , computing a raise decision (Lines 4-6, Equation 3).

Original: We call the architecture that results the “Exception IPA-GNN”.
Revised: We call the architecture that results the “Exception IPA-GNN”.

Original: Whereas execution always proceeds from statement to next statement in interpreter
Revised: Whereas execution always proceeds from statement to next statement in interpreter

Original: I and in the IPAGNN, interpreter I ′ admits another behavior.
Revised: I and in the IPAGNN, interpreter I ′ admits another behavior.

Original: Under I ′ and the Exception IPA-GNN, execution may proceed from any statement to a surrounding “except block”, if it is contained in a try/except frame, or else to a special global error node, which we denote n error .
Revised: Under I ′ and the Exception IPA-GNN, execution may proceed from any statement to a surrounding “except block”, if it is contained in a try/except frame, or else to a special global error node, which we denote n error .

Original: In the sample execution in Figure 1c we see at step t = 4 the instruction pointer p B updates to n error = 8 .
Revised: In the sample execution in Figure 1c we see at step t = 4 the instruction pointer p B updates to n error = 8 .

Original: We write that the IPA-GNN makes raise decisions as
Revised: We write that the IPA-GNN makes raise decisions as

Original: [Equation]
Revised: [Equation]

Original: The dense layer here has two outputs representing the cases that an error is and is not raised.
Revised: The dense layer here has two outputs representing the cases that an error is and is not raised.

Original: Here r ( n ) denotes the node that statement n raises to; r ( n ) = n error if n is not contained in a try/except frame, and b t,n,n ′ denotes the probability under the model of execution transitioning from n to n ′ .
Revised: Here r ( n ) denotes the node that statement n raises to; r ( n ) = n error if n is not contained in a try/except frame, and b t,n,n ′ denotes the probability under the model of execution transitioning from n to n ′ .

Original: Next the model makes soft branch decisions in an analogous manner; the dense layer for making branch decisions has distinct weights from the layer for making raise decisions.
Revised: Next the model makes soft branch decisions in an analogous manner; the dense layer for making branch decisions has distinct weights from the layer for making raise decisions.

Original: [Equation]
Revised: [Equation]

Original: The text in magenta corresponds to the “else” at Line 6.
Revised: The text in magenta corresponds to the “else” at Line 6.

Original: The model has now assigned probability to up to three possible outcomes for each node: the probability that n raises an exception b t,n,r ( n ) , the probability that the true branch is followed b t,n,n 1 , and the probability that the false branch is followed b t,n,n 2 .
Revised: The model has now assigned probability to up to three possible outcomes for each node: the probability that n raises an exception b t,n,r ( n ) , the probability that the true branch is followed b t,n,n 1 , and the probability that the false branch is followed b t,n,n 2 .

Original: In the common case where a node is not a control node and has only a single successor, the probability of reaching that successor is simply 1 − b t,n,r ( n ) .
Revised: In the common case where a node is not a control node and has only a single successor, the probability of reaching that successor is simply 1 − b t,n,r ( n ) .

Original: Finally, we assign each program a step limit T ( x ) using the same heuristic as Bieber et al.
Revised: Finally, we assign each program a step limit T ( x ) using the same heuristic as Bieber et al.

Original: (2020), detailed in Appendix E. After T ( x ) steps of the architecture, the model directly uses the probability mass at n exit and n error to predict whether the program raises an error, and if so it predicts the error type using the hidden state at the error node.
Revised: (2020), detailed in Appendix E. After T ( x ) steps of the architecture, the model directly uses the probability mass at n exit and n error to predict whether the program raises an error, and if so it predicts the error type using the hidden state at the error node.

Original: We write the modified IPA-GNN’s predictions as
Revised: We write the modified IPA-GNN’s predictions as

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: no error
Revised: no error

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) ,n exit and P ( error )
Revised: ) ,n exit and P ( error )

Original: ∝ p T ( x ) ,n error , with
Revised: ∝ p T ( x ) ,n error , with

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We train with a cross-entropy loss on the class predictions, treating “no error” as its own class.
Revised: We train with a cross-entropy loss on the class predictions, treating “no error” as its own class.

Original: 4. U NSUPERVISED L OCALIZATION OF E RRORS
Revised: 4. U NSUPERVISED L OCALIZATION OF E RRORS

Original: Since the Exception IPA-GNN makes soft decisions as to when to raise an exception, we aggregate these soft decisions to obtain the model’s prediction for where a program raises an error.
Revised: Since the Exception IPA-GNN makes soft decisions as to when to raise an exception, we aggregate these soft decisions to obtain the model’s prediction for where a program raises an error.

Original: We use this to evaluate the model’s localization accuracy despite training without error locations as supervision.
Revised: We use this to evaluate the model’s localization accuracy despite training without error locations as supervision.

Original: For programs that lack try/except frames, we compute the localization predictions of the model by summing, separately for each node, the contributions from that node to the exception node across all time steps.
Revised: For programs that lack try/except frames, we compute the localization predictions of the model by summing, separately for each node, the contributions from that node to the exception node across all time steps.

Original: This gives an estimate of exception provenance as
Revised: This gives an estimate of exception provenance as

Original: [Equation]
Revised: [Equation]

Original: For programs with a try/except frame, we must trace the exception back to the statement that originally raised it.
Revised: For programs with a try/except frame, we must trace the exception back to the statement that originally raised it.

Original: To do this we calculate a recurrence as detailed in Appendix H.
Revised: To do this we calculate a recurrence as detailed in Appendix H.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In our experiments we evaluate the following research questions:
Revised: In our experiments we evaluate the following research questions:

Original: RQ1: How does the adaptation of the IPA-GNN to realistic code compare against existing static analysis and against standard architectures like GGNN, LSTM, and Transformer?
Revised: RQ1: How does the adaptation of the IPA-GNN to realistic code compare against existing static analysis and against standard architectures like GGNN, LSTM, and Transformer?

Original: (Section 5.1) RQ2: What is the impact of including resource descriptions?
Revised: (Section 5.1) RQ2: What is the impact of including resource descriptions?

Original: What methods for incorporating them work best?
Revised: What methods for incorporating them work best?

Original: (Section 5.2) RQ3: How interpretable are the soft instruction pointer values in the Exception IPA-GNN for localizing errors?
Revised: (Section 5.2) RQ3: How interpretable are the soft instruction pointer values in the Exception IPA-GNN for localizing errors?

Original: How does unsupervised localization with the Exception IPA-GNN compare to alternative unsupervised localization approaches based on multiple instance learning?
Revised: How does unsupervised localization with the Exception IPA-GNN compare to alternative unsupervised localization approaches based on multiple instance learning?

Original: (Section 5.3)
Revised: (Section 5.3)

Original: E VALUATION OF IPA-GNN A GAINST B ASELINES
Revised: E VALUATION OF IPA-GNN A GAINST B ASELINES

Original: We describe the experimental setup for our first experiment, comparing the IPA-GNN architectures with Transformer (Vaswani et al., 2017), GGNN (Li et al., 2017), and LSTM (Hochreiter and
Revised: We describe the experimental setup for our first experiment, comparing the IPA-GNN architectures with Transformer (Vaswani et al., 2017), GGNN (Li et al., 2017), and LSTM (Hochreiter and

Original: Schmidhuber, 1997) baselines.
Revised: Schmidhuber, 1997) baselines.

Original: In all approaches, we use the 30,000 token vocabulary constructed in Appendix A, applying Byte-Pair Encoding (BPE) tokenization (Sennrich et al., 2016) to tokenize each program into a sequence of token indices.
Revised: In all approaches, we use the 30,000 token vocabulary constructed in Appendix A, applying Byte-Pair Encoding (BPE) tokenization (Sennrich et al., 2016) to tokenize each program into a sequence of token indices.

Original: The Transformer operates on this sequence of token indices directly, with its final representation computed via mean pooling.
Revised: The Transformer operates on this sequence of token indices directly, with its final representation computed via mean pooling.

Original: For all other models (GGNN, LSTM, IPA-GNN, and Exception IPA-GNN), the token indices are first combined via a masked (local) Transformer to produce per-node embeddings, and the model operates on these pernode embeddings as in Section 4.1.
Revised: For all other models (GGNN, LSTM, IPA-GNN, and Exception IPA-GNN), the token indices are first combined via a masked (local) Transformer to produce per-node embeddings, and the model operates on these pernode embeddings as in Section 4.1.

Original: Following Bieber et al.
Revised: Following Bieber et al.

Original: we encode programs for a GGNN using six edge types, and use a two-layer LSTM for the LSTM baseline and in all IPA-GNN variants.
Revised: we encode programs for a GGNN using six edge types, and use a two-layer LSTM for the LSTM baseline and in all IPA-GNN variants.

Original: [Table]
Revised: [Table]

Original: In order to compare against the capabilities of a standard static analysis setup
Revised: In order to compare against the capabilities of a standard static analysis setup

Original: [Table]
Revised: [Table]

Original: we also consider a baseline based on pylint.
Revised: we also consider a baseline based on pylint.

Original: For this baseline, we map a subset of the findings that pylint can identify to runtime error classes that they can indicate.
Revised: For this baseline, we map a subset of the findings that pylint can identify to runtime error classes that they can indicate.

Original: The baseline predicts an error class if pylint identifies a corresponding finding.
Revised: The baseline predicts an error class if pylint identifies a corresponding finding.

Original: The purpose of this baseline is to consider a standard tool used by Python developers and see how it is performing on the task.
Revised: The purpose of this baseline is to consider a standard tool used by Python developers and see how it is performing on the task.

Original: We provide further details in Appendix G.
Revised: We provide further details in Appendix G.

Original: For each neural approach, we perform an independent hyperparameter search using random search.
Revised: For each neural approach, we perform an independent hyperparameter search using random search.

Original: We list the hyperparameter space considered and model selection criteria in
Revised: We list the hyperparameter space considered and model selection criteria in

Original: Appendix E.
Revised: Appendix E.

Original: The models are each trained to minimize a cross-entropy loss on the target class using stochastic gradient descent for up to 500,000 steps with a mini-batch size of 32.
Revised: The models are each trained to minimize a cross-entropy loss on the target class using stochastic gradient descent for up to 500,000 steps with a mini-batch size of 32.

Original: In order to more closely match the target class distribution found in the balanced test set, we sample mini-batches such that the proportion of examples with target “no error” and those with an error target is 1:1 in expectation.
Revised: In order to more closely match the target class distribution found in the balanced test set, we sample mini-batches such that the proportion of examples with target “no error” and those with an error target is 1:1 in expectation.

Original: We evaluate the selected models on the balanced test set, and report the results in Table 2a (see rows without check marks).
Revised: We evaluate the selected models on the balanced test set, and report the results in Table 2a (see rows without check marks; we consider resource descriptions in Section 5.2 for RQ2).

Original: Weighted F1 score (W. F1) performs a weighted average of the per-class F1 scores by class frequency, and weighted error F1 score (E. F1) does the same while restricting consideration to those examples with a runtime error.
Revised: Weighted F1 score (W. F1) performs a weighted average of the per-class F1 scores by class frequency, and weighted error F score (E. F1) does the same while restricting consideration to those examples with a runtime error.

Original: We perform additional evaluations using the same experimental setup but distinct initializations to compute measures of variance, which we detail in Appendix F.
Revised: We perform additional evaluations using the same experimental setup but distinct initializations to compute measures of variance, which we detail in Appendix F.

Original: RQ1: The interpreter-inspired architectures show significant gains over the pylint, LSTM, GGNN and Transformer baseline approaches on the runtime error prediction task.
Revised: RQ1: The interpreter-inspired architectures show significant gains over the pylint, LSTM, GGNN and Transformer baseline approaches on the runtime error prediction task.

Original: We observe that the pylint baseline can make incorrect predictions because it correctly identifies an issue in the code under analysis when that code does not result in a runtime error in our dataset; pylint’s lower performance on runtime error prediction is not evidence against pylint’s performance for its intended use cases.
Revised: We observe that the pylint baseline can make incorrect predictions because it correctly identifies an issue in the code under analysis when that code does not result in a runtime error in our dataset; pylint’s lower performance on runtime error prediction is not evidence against pylint’s performance for its intended use cases.

Original: We attribute the interpreter-inspired architectures’ relative success over other neural architectures to their inductive bias toward mimicking program execution.
Revised: We attribute the interpreter-inspired architectures’ relative success over other neural architectures to their inductive bias toward mimicking program execution.

Original: 5. I NCORPORATING R ESOURCE D ESCRIPTIONS
Revised: 5. I NCORPORATING R ESOURCE D ESCRIPTIONS

Original: We next evaluate methods of incorporating resource descriptions into the models.
Revised: We next evaluate methods of incorporating resource descriptions into the models.

Original: For each architecture we apply the docstring approach of processing resource descriptions of Section 4.2.
Revised: For each architecture we apply the docstring approach of processing resource descriptions of Section 4.2.

Original: This completes a matrix of ablations, allowing us to distinguish the effects due to architecture change from the effect of the resource description.
Revised: This completes a matrix of ablations, allowing us to distinguish the effects due to architecture change from the effect of the resource description.

Original: We follow the same experimental setup as in Section 5.1, and show the results again in Table 2a (compare rows with check marks to those without).
Revised: We follow the same experimental setup as in Section 5.1, and show the results again in Table 2a (compare rows with check marks to those without).

Original: We also consider the FiLM and cross-attention methods of incorporating resource descriptions into the IPA-GNN.
Revised: We also consider the FiLM and cross-attention methods of incorporating resource descriptions into the IPA-GNN.

Original: Following the same experimental setup again, we show the results of this experiment in Table 3.
Revised: Following the same experimental setup again, we show the results of this experiment in Table 3.

Original: Note that the best model overall by our model selection criteria on validation data was the IPA-GNN with cross-attention, though the Exception IPA-GNN performed better on test.
Revised: Note that the best model overall by our model selection criteria on validation data was the IPA-GNN with cross-attention, though the Exception IPA-GNN performed better on test.

Original: RQ2: Across all architectures, the results show external resource descriptions improve performance on the runtime error prediction task.
Revised: RQ2: Across all architectures, the results show external resource descriptions improve performance on the runtime error prediction task.

Original: On the IPA-GNN architectures, we see further improvements by considering architectures that incorporate the resource description directly into the execution step
Revised: On the IPA-GNN architectures, we see further improvements by considering architectures that incorporate the resource description directly into the execution step

Original: B ASELINE
Revised: B ASELINE

Original: D OCSTRING
Revised: D OCSTRING

Original: F I LM
Revised: F I LM

Original: C ROSS - ATTENTION M ODEL
Revised: C ROSS - ATTENTION M ODEL

Original: A CC .
Revised: A CC .

Original: W. F
Revised: W. F

Original: E. F
Revised: E. F

Original: A CC .
Revised: A CC .

Original: W. F
Revised: W. F

Original: E. F
Revised: E. F

Original: A CC .
Revised: A CC .

Original: W. F
Revised: W. F

Original: E. F
Revised: E. F

Original: A CC .
Revised: A CC .

Original: W. F
Revised: W. F

Original: E. F
Revised: E. F

Original: IPA-GNN 68.
Revised: IPA-GNN 68.

Original: 62.6 E. IPA-GNN 68.
Revised: 62.6 E. IPA-GNN 68.

Original: of the model
Revised: of the model

Original: , but these gains are inconsistent.
Revised: , but these gains are inconsistent.

Original: The pylint baseline is unable to incorporate resource descriptions.
Revised: The pylint baseline is unable to incorporate resource descriptions.

Original: Critically, using any resource description method is better than none at all.
Revised: Critically, using any resource description method is better than none at all.

Original: To understand how the resource descriptions lead to better performance, we compare in Figurethe instruction pointer values of two Exception IPA-GNN models on a single example (shown in
Revised: To understand how the resource descriptions lead to better performance, we compare in Figurethe instruction pointer values of two Exception IPA-GNN models on a single example (shown in

Original: Table 4).
Revised: Table 4).

Original: The model with the resource description recognizes that the input() calls will read input beyond the end of the stdin stream.
Revised: The model with the resource description recognizes that the input() calls will read input beyond the end of the stdin stream.

Original: In contrast, the model without the resource description has less reason to suspect an error would be raised by those calls.
Revised: In contrast, the model without the resource description has less reason to suspect an error would be raised by those calls.

Original: The descriptions of stdin in our runtime errors dataset also frequently reveal type information, expected ranges for numeric values, and formatting details about the inputs.
Revised: The descriptions of stdin in our runtime errors dataset also frequently reveal type information, expected ranges for numeric values, and formatting details about the inputs.

Original: We visualize additional examples in Appendix K.
Revised: We visualize additional examples in Appendix K.

Original: 5. I NTERPRETABILITY AND L OCALIZATION
Revised: 5. I NTERPRETABILITY AND L OCALIZATION

Original: We next investigate the behavior of the Exception IPA-GNN model, evaluating its ability to localize runtime errors without any localization supervision.
Revised: We next investigate the behavior of the Exception IPA-GNN model, evaluating its ability to localize runtime errors without any localization supervision.

Original: In unsupervised localization, the models predict the location of the error despite being trained only with error presence and kind supervision.
Revised: In unsupervised localization, the models predict the location of the error despite being trained only with error presence and kind supervision.

Original: Multiple Instance Learning Baselines Unsupervised localization may be viewed as multiple instance learning (MIL)
Revised: Multiple Instance Learning Baselines Unsupervised localization may be viewed as multiple instance learning (MIL)

Original: (Dietterich et al., 1997).
Revised: (Dietterich et al., 1997).

Original: Consider the subtask of predicting whether a particular line contains an error.
Revised: Consider the subtask of predicting whether a particular line contains an error.

Original: In an n -line program, there are n instances of this subtask.
Revised: In an n -line program, there are n instances of this subtask.

Original: The available supervision only indicates if any one of these subtasks has an error, but not which one.
Revised: The available supervision only indicates if any one of these subtasks has an error, but not which one.

Original: By viewing each instance as a bag of subtasks, we have cast the problem as MIL.
Revised: By viewing each instance as a bag of subtasks, we have cast the problem as MIL.

Original: Using this view, we introduce two variations on the Transformer architecture as multiple instance learning baselines.
Revised: Using this view, we introduce two variations on the Transformer architecture as multiple instance learning baselines.

Original: The first is the “Local MIL Transformer”, in which each statement in the program is encoded individually, as in the local node embeddings computation of Section 4.1.
Revised: The first is the “Local MIL Transformer”, in which each statement in the program is encoded individually, as in the local node embeddings computation of Section 4.1.

Original: The second is the “Global MIL Transformer”, in which all tokens in the program may attend to all other tokens in the Transformer encoder.
Revised: The second is the “Global MIL Transformer”, in which all tokens in the program may attend to all other tokens in the Transformer encoder.

Original: In both cases, the models make per-line predictions, which are aggregated to form an overall prediction as defined in Appendix I.
Revised: In both cases, the models make per-line predictions, which are aggregated to form an overall prediction as defined in Appendix I.

Original: [Figure]
Revised: B ASELINE

Revised: R ESOURCE D ESCRIPTION

Original: S TD I N D ESCRIPTION
Revised: S TD I N D ESCRIPTION

Original: Input: Input is given from Standard Input in the following format Constraints:
Revised: Input:Inputis given from Standard Input in the following format Constraints:

Original: Each character of S is A or B. |S| =n
Revised: Each character of S is A or B. |S| =n

Original: S OURCE
Revised: S OURCE

Original: B ASELINE 
Revised: B ASELINE Error contrib.

Original: R.D. 
Revised: R.D. Error contrib.

Original: a = str(input()) 16.
Revised: a = str(input()) 0.

Original: 0.4q = int(input()) 3.
Revised: 2.4q = int(input()) 0.

Original: 0.3s =
Revised: 97.3s =

Original: [input().split() for i in range(q)] 0.
Revised: [input().split() for i in range(q)] 14.

Original: 99.3 3, for i in range(q): 6.
Revised: 0.0 3, for i in range(q): 3.

Original: 0.0if int(s[i][0]) =
Revised: 0.0if int(s[i][0]) =

Original: = 1 and len(a) > 1: 0.
Revised: = 1 and len(a) > 1: 1.

Original: 0.0a = a[::-1] 0.
Revised: 0.0a = a[::-1] 0.

Original: 0.0elif int(s[i][0])== 2 and int(s[i][1])==1: 0.
Revised: 0.0elif int(s[i][0])== 2 and int(s[i][1])==1: 0.

Original: 0.0a=s[i][2]+a 0. 0.0 else: a=a+s[i][2] 0.
Revised: 0.0a=s[i][2]+a 0.1 else: 0.0a= a+s[i][2] 0.

Original: 0.0print(a) 1.
Revised: 0.0print(a) 3.

Original: Localization Experiment Using the same protocol as Section 5.1, we train each of the MIL Transformer and Exception IPA-GNN models.
Revised: Localization Experiment Using the same protocol as Section 5.1, we train each of the MIL Transformer and Exception IPA-GNN models.

Original: As before, the models are trained only to minimize crossentropy loss on predicting error kind and presence, receiving no error location supervision.
Revised: As before, the models are trained only to minimize crossentropy loss on predicting error kind and presence, receiving no error location supervision.

Original: We report the localization results in Table 2b.
Revised: We report the localization results in Table 2b.

Original: Localization accuracy (“L OCAL .”) measures the percent of the test examples with errors for which the model correctly predicts the error line number.
Revised: Localization accuracy (“L OCAL .”) measures the percent of the test examples with errors for which the model correctly predicts the error line number.

Original: RQ3: The Exception IPA-GNN’s unsupervised localization capabilities far exceed that of baseline approaches.
Revised: RQ3: The Exception IPA-GNN’s unsupervised localization capabilities far exceed that of baseline approaches.

Original: In Figure 2 we see the flow of instruction pointer mass during the execution of a sample program (Table 4) by two Exception IPA-GNN models, including the steps where the models raise probability mass to n error .
Revised: In Figure 2 we see the flow of instruction pointer mass during the execution of a sample program (Table 4) by two Exception IPA-GNN models, including the steps where the models raise probability mass to n error .

Original: Tallying the contributions to n error from each node yields the exception provenance values in the right half of Table 4.
Revised: Tallying the contributions to n error from each node yields the exception provenance values in the right half of Table 4.

Original: This shows how the model’s internal state resembles plausible program executions and allows for unsupervised localization.
Revised: This shows how the model’s internal state resembles plausible program executions and allows for unsupervised localization.

Original: As a beneficial side-effect of learning plausible executions, the Exception IPA-GNN can localize the exceptions it predicts.
Revised: As a beneficial side-effect of learning plausible executions, the Exception IPA-GNN can localize the exceptions it predicts.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this work, we introduce the task of predicting runtime errors in competitive programming problems and advance the capabilities of interpreter-inspired models.
Revised: In this work, we introduce the task of predicting runtime errors in competitive programming problems and advance the capabilities of interpreter-inspired models.

Original: Our models support the complexity of competition code and demonstrate that natural language descriptions of external resources can reduce the ambiguity that arises in a static analysis setting.
Revised: Our models support the complexity of competition code and demonstrate that natural language descriptions of external resources can reduce the ambiguity that arises in a static analysis setting.

Original: We show that the interpreter-inspired models outperform standard alternatives and that their inductive biases allow for interesting interpretability in the context of unsupervised localization.
Revised: We show that the interpreter-inspired models outperform standard alternatives and that their inductive biases allow for interesting interpretability in the context of unsupervised localization.

Original: Though they perform best, current IPA-GNN models require taking many steps of execution, up to 174 on this dataset.
Revised: Though they perform best, current IPA-GNN models require taking many steps of execution, up to 174 on this dataset.

Original: A future direction is to model multiple steps of program execution with a single model step, to reduce the number of model steps necessary for long programs.
Revised: A future direction is to model multiple steps of program execution with a single model step, to reduce the number of model steps necessary for long programs.

Original: Extending the interpreter-inspired models with additional interpreter features, or supporting multi-file programs or programs with multiple user-defined functions are also interesting avenues for future work.
Revised: Extending the interpreter-inspired models with additional interpreter features, or supporting multi-file programs or programs with multiple user-defined functions are also interesting avenues for future work.

Original: Learning to understand programs remains a rich area of inquiry for machine learning research because of its complexity and the many aspects of code.
Revised: Learning to understand programs remains a rich area of inquiry for machine learning research because of its complexity and the many aspects of code.

Original: Learning to understand execution behavior is particularly challenging as programs grow in complexity, and as they depend on more external resources whose contents are not present in the code.
Revised: Learning to understand execution behavior is particularly challenging as programs grow in complexity, and as they depend on more external resources whose contents are not present in the code.

Original: Our work presents a challenging problem and advances interpreter-inspired models, both of which we hope are ingredients towards making progress on these difficult and important problems.
Revised: Our work presents a challenging problem and advances interpreter-inspired models, both of which we hope are ingredients towards making progress on these difficult and important problems.

Original: A P YTHON R UNTIME E RROR D ATASET D ETAILS
Revised: A P YTHON R UNTIME E RROR D ATASET D ETAILS

Original: We describe in detail the construction of the Python Runtime Error dataset from the submissions in Project CodeNet (Puri et al., 2021).
Revised: We describe in detail the construction of the Python Runtime Error dataset from the submissions in Project CodeNet (Puri et al., 2021).

Original: The Project CodeNet dataset contains over 14 million submissions to 4,053 distinct competitive programming problems, with the submissions spanning more than 50 programming languages.
Revised: The Project CodeNet dataset contains over 14 million submissions to 4,053 distinct competitive programming problems, with the submissions spanning more than 50 programming languages.

Original: We partition the problems into train, valid, and test splits at an 80:10:10 ratio.
Revised: We partition the problems into train, valid, and test splits at an 80:10:10 ratio.

Original: By making all submissions to the same problem part of the same split we mitigate concerns about potential data leakage from similar submissions to the same problem.
Revised: By making all submissions to the same problem part of the same split we mitigate concerns about potential data leakage from similar submissions to the same problem.

Original: We restrict our consideration to Python submissions, which account for 3,286,314 of the overall Project CodeNet submissions, with 3,119 of the problems receiving at least one submission in Python.
Revised: We restrict our consideration to Python submissions, which account for 3,286,314 of the overall Project CodeNet submissions, with 3,119 of the problems receiving at least one submission in Python.

Original: In preparing the dataset we execute approximately 3 million problems in a sandboxed environment to collect their runtime error information, we perform two stages of filtering on the dataset, syntactic and complexity filtering, and we construct a textual representation of the input space for each problem from the problem description.
Revised: In preparing the dataset we execute approximately 3 million problems in a sandboxed environment to collect their runtime error information, we perform two stages of filtering on the dataset, syntactic and complexity filtering, and we construct a textual representation of the input space for each problem from the problem description.

Original: A. S YNTACTIC F ILTERING
Revised: A. S YNTACTIC F ILTERING

Original: In this first phase of filtering, we remove submissions in Python 2 as well as those which fail to parse and run from our dataset.
Revised: In this first phase of filtering, we remove submissions in Python 2 as well as those which fail to parse and run from our dataset.

Original: We remove 76,888 programs because they are in Python 2, 59, programs because they contain syntax errors that prohibit parsing, 2,011 programs that result in runtime errors during parsing, and 6 additional programs for which the python-graphs library fails to construct a control-flow graph.
Revised: We remove 76,888 programs because they are in Python 2, 59, programs because they contain syntax errors that prohibit parsing, 2,011 programs that result in runtime errors during parsing, and 6 additional programs for which the python-graphs library fails to construct a control-flow graph.

Original: A program may result in a runtime error during parsing if it contains return, break, continue keywords outside of an appropriate frame.
Revised: A program may result in a runtime error during parsing if it contains return, break, continue keywords outside of an appropriate frame.

Original: A. P ROGRAM E XECUTION
Revised: A. P ROGRAM E XECUTION

Original: We attempt to run each submission in a sandboxed environment using the sample input provided in the Project CodeNet dataset.
Revised: We attempt to run each submission in a sandboxed environment using the sample input provided in the Project CodeNet dataset.

Original: The environment is a custom harness running on a Google Cloud
Revised: The environment is a custom harness running on a Google Cloud

Original: Platform (GCP) virtual environment.
Revised: Platform (GCP) virtual environment.

Original: This allows us to collect standard out and standard error, to monitor for timeouts, and to catch and serialize any Python exceptions raised during execution.
Revised: This allows us to collect standard out and standard error, to monitor for timeouts, and to catch and serialize any Python exceptions raised during execution.

Original: We restrict execution of each program to 1 second, marking any program exceeding this time as a timeout error.
Revised: We restrict execution of each program to 1 second, marking any program exceeding this time as a timeout error.

Original: If the program encounters a Python exception, we use the name of that exception as the target class for the program.
Revised: If the program encounters a Python exception, we use the name of that exception as the target class for the program.

Original: If an error type occurs only once in the dataset, we consider the target class to be Other.
Revised: If an error type occurs only once in the dataset, we consider the target class to be Other.

Original: Programs not exhibiting an error or timeout are given target class “no error”.
Revised: Programs not exhibiting an error or timeout are given target class “no error”.

Original: In addition to collecting the target class, we record for each runtime error the line number at which the error occurs.
Revised: In addition to collecting the target class, we record for each runtime error the line number at which the error occurs.

Original: We use these line numbers as the ground truth for the unsupervised error localization task considered in Section 5.3.
Revised: We use these line numbers as the ground truth for the unsupervised error localization task considered in Section 5.3.

Original: A. E XTRACTING R ESOURCE D ESCRIPTIONS BY P ARSING P ROBLEM S TATEMENTS
Revised: A. E XTRACTING R ESOURCE D ESCRIPTIONS BY P ARSING P ROBLEM S TATEMENTS

Original: For each problem, we parse the problem statement to extract the input description and input constraints , if they exist.
Revised: For each problem, we parse the problem statement to extract the input description and input constraints , if they exist.

Original: These two sections of the problem statement together form the external resource description that accompanies that problem.
Revised: These two sections of the problem statement together form the external resource description that accompanies that problem.

Original: The problem statements in our dataset are each written either in English or Japanese, and so we write our parser to support both languages.
Revised: The problem statements in our dataset are each written either in English or Japanese, and so we write our parser to support both languages.

Original: When one or both of these sections are present in the problem statement, we construct the external resource description for the problem by concatenating together the headers and contents of the sections that are present.
Revised: When one or both of these sections are present in the problem statement, we construct the external resource description for the problem by concatenating together the headers and contents of the sections that are present.

Original: For the experiments that use the resource description as a docstring, we prepend to each submission a docstring containing the resource description for problem that goes with that submission.
Revised: For the experiments that use the resource description as a docstring, we prepend to each submission a docstring containing the resource description for problem that goes with that submission.

Original: Similarly these serve as the resource descriptions in the experiments that process resource descriptions via either cross-attention or FiLM.
Revised: Similarly these serve as the resource descriptions in the experiments that process resource descriptions via either cross-attention or FiLM.

Original: A. V OCABULARY C ONSTRUCTION AND C OMPLEXITY F ILTERING
Revised: A. V OCABULARY C ONSTRUCTION AND C OMPLEXITY F ILTERING

Original: All experiments use the same vocabulary and tokenization procedure.
Revised: All experiments use the same vocabulary and tokenization procedure.

Original: For this, we select the standard Byte-Pair Encoding (BPE) tokenization procedure (Sennrich et al., 2016).
Revised: For this, we select the standard Byte-Pair Encoding (BPE) tokenization procedure (Sennrich et al., 2016).

Original: We construct the vocabulary using 1,000,000 submissions selected from the training split, along with the input space descriptions constructed for all problems in the train split.
Revised: We construct the vocabulary using 1,000,000 submissions selected from the training split, along with the input space descriptions constructed for all problems in the train split.

Original: We use a vocabulary size of 30,000.
Revised: We use a vocabulary size of 30,000.

Original: [Figure]
Revised: [Figure]

Original: We then apply size-based filtering, further restricting the set of programs considered.
Revised: We then apply size-based filtering, further restricting the set of programs considered.

Original: First, the program length after tokenization is not to exceed 512 tokens, the number of nodes and edges in the control-flow graph are each not to exceed 128, and the step limit T ( x ) for a program computed in Appendix E is not to exceed 174.
Revised: First, the program length after tokenization is not to exceed 512 tokens, the number of nodes and edges in the control-flow graph are each not to exceed 128, and the step limit T ( x ) for a program computed in Appendix E is not to exceed 174.

Original: We select these numbers to trim the long tail of exceptionally long programs, and this filtering reduces the total number of acceptable programs by less than 1%.
Revised: We select these numbers to trim the long tail of exceptionally long programs, and this filtering reduces the total number of acceptable programs by less than 1%.

Original: To achieve consistent datasets comparable across all experiments, we use the longest form of each program (the program augmented with its input space information as a docstring) when computing the program sizes for size-based submission filtering.
Revised: To achieve consistent datasets comparable across all experiments, we use the longest form of each program (the program augmented with its input space information as a docstring) when computing the program sizes for size-based submission filtering.

Original: We further impose the restriction that no user-defined functions (UDFs) are called in a submission; this further reduces the number of submissions by 682,220.
Revised: We further impose the restriction that no user-defined functions (UDFs) are called in a submission; this further reduces the number of submissions by 682,220.

Original: A user-defined function is a function defined in the submission source code, as opposed to being a built-in or imported from a third party module.
Revised: A user-defined function is a function defined in the submission source code, as opposed to being a built-in or imported from a third party module.

Original: Extending the IPA-GNN models to submissions with UDFs called at most once is trivially achieved by replacing the program’s control-flow graph with its interprocedural control-flow graph (ICFG) (Nielson and Nielson, 1999).
Revised: Extending the IPA-GNN models to submissions with UDFs called at most once is trivially achieved by replacing the program’s control-flow graph with its interprocedural control-flow graph (ICFG) (Nielson and Nielson, 1999).

Original: We leave the investigation of modeling user-defined functions to further work.
Revised: We leave the investigation of modeling user-defined functions to further work.

Original: A. F INAL D ATASET D ETAILS
Revised: A. F INAL D ATASET D ETAILS

Original: After applying syntactic filtering (only keeping Python 3 programs that parse) and complexity filtering (eliminating long programs and programs that call user-defined functions), we are left with a dataset of 2,441,130 examples.
Revised: After applying syntactic filtering (only keeping Python 3 programs that parse) and complexity filtering (eliminating long programs and programs that call user-defined functions), we are left with a dataset of 2,441,130 examples.

Original: The division of these examples by split and by target class is given in Table 1.
Revised: The division of these examples by split and by target class is given in Table 1.

Original: Figure 3 shows the distribution of program lengths in lines represented in the completed dataset, with an average program length of 14.2 lines.
Revised: Figure 3 shows the distribution of program lengths in lines represented in the completed dataset, with an average program length of 14.2 lines.

Original: The average statement length is 6.7 tokens, with full distribution shown in Figure 4.
Revised: The average statement length is 6.7 tokens, with full distribution shown in Figure 4.

Original: A. D ATA L ICENSE
Revised: A. D ATA L ICENSE

Original: The Project CodeNet (Puri et al., 2021) data that we use is available under the CDLA Permissive v2.0 license, and we release our derived dataset under this same license.
Revised: The Project CodeNet (Puri et al., 2021) data that we use is available under the CDLA Permissive v2.0 license, and we release our derived dataset under this same license.

Original: B U NDER - APPROXIMATION OF E RROR L ABELS
Revised: B U NDER - APPROXIMATION OF E RROR L ABELS

Original: As described in Section 3, the ground truth error targets in our dataset are obtained by running each submission on only a single input.
Revised: As described in Section 3, the ground truth error targets in our dataset are obtained by running each submission on only a single input.

Original: We do this because we only have a single input available from the online judges with which to execute the programs.
Revised: We do this because we only have a single input available from the online judges with which to execute the programs.

Original: As a result, the error labels we obtain under-approximate the full set of errors liable to appear at runtime.
Revised: As a result, the error labels we obtain under-approximate the full set of errors liable to appear at runtime.

Original: Metadata obtained from (Puri et al., 2021) indicates whether each submission encountered a runtime error on a larger set of inputs, though it does not indicate the kind or location of these errors when they are present.
Revised: Metadata obtained from (Puri et al., 2021) indicates whether each submission encountered a runtime error on a larger set of inputs, though it does not indicate the kind or location of these errors when they are present.

Original: We use this metadata to determine the degree to which our labels are an under-approximation.
Revised: We use this metadata to determine the degree to which our labels are an under-approximation.

Original: We find that on the balanced test set there are 1,076 submissions (4%) which, per the metadata, encounter an error, but for which our evaluation finds no error.
Revised: We find that on the balanced test set there are 1,076 submissions (4%) which, per the metadata, encounter an error, but for which our evaluation finds no error.

Original: These are likely examples for which the program runs without error on the input we have, but for which the program fails on some additional unavailable input.
Revised: These are likely examples for which the program runs without error on the input we have, but for which the program fails on some additional unavailable input.

Original: We next measure generalization from the labels in our dataset to the labels suggested by the metadata without retraining.
Revised: We next measure generalization from the labels in our dataset to the labels suggested by the metadata without retraining.

Original: Since these labels are only binary indicators of error presence, we use our model to perform binary classification by summing the predicted probabilities of all error kinds.
Revised: Since these labels are only binary indicators of error presence, we use our model to perform binary classification by summing the predicted probabilities of all error kinds.

Original: The model predicts “no error” on 76.2% of the examples for which our dataset finds no error.
Revised: The model predicts “no error” on 76.2% of the examples for which our dataset finds no error.

Original: On the examples for which the metadata indicates no error, this drops to 75.9%, and on the examples for which the metadata indicates there is an error, this rises to 80.9%.
Revised: On the examples for which the metadata indicates no error, this drops to 75.9%, and on the examples for which the metadata indicates there is an error, this rises to 80.9%.

Original: These examples, where a single input detects no error but multiple inputs detect an error, are difficult for the model to classify.
Revised: These examples, where a single input detects no error but multiple inputs detect an error, are difficult for the model to classify.

Original: We hypothesize that the types of errors our labels omit systematically differ from those our labels include as an explanation for this 4.7% discrepancy.
Revised: We hypothesize that the types of errors our labels omit systematically differ from those our labels include as an explanation for this 4.7% discrepancy.

Original: C IPA-GNN A RCHITECTURE
Revised: C IPA-GNN A RCHITECTURE

Original: We provide a concise and precise definition of the IPA-GNN baseline architecture, following the notation of Bieber et al.
Revised: We provide a concise and precise definition of the IPA-GNN baseline architecture, following the notation of Bieber et al.

Original: The IPA-GNN operates on the statement-level control-flow graph of the input program x , maintaining per-node per-step hidden states h t,n and a soft instruction pointer p t,n .
Revised: The IPA-GNN operates on the statement-level control-flow graph of the input program x , maintaining per-node per-step hidden states h t,n and a soft instruction pointer p t,n .

Original: At each step t , each node x n participates in execution, branch prediction, and aggregation.
Revised: At each step t , each node x n participates in execution, branch prediction, and aggregation.

Original: First, the IPA-GNN models executing the statement at each node to produce per-node state proposals
Revised: First, the IPA-GNN models executing the statement at each node to produce per-node state proposals

Original: [Equation]
Revised: [Equation]

Original: Then, the model uses these to inform soft branch decisions at every control flow juncture, given as
Revised: Then, the model uses these to inform soft branch decisions at every control flow juncture, given as

Original: [Equation]
Revised: [Equation]

Original: 9) where { n 1 , n 2 } =
Revised: 9) where { n 1 , n 2 } =

Original: N out ( x n ) when | N out ( x n ) | = 2 .
Revised: N out ( x n ) when | N out ( x n ) | = 2 .

Original: When | N out ( x n ) | = 1 we have b t,n,n ′ = 1 for n ′ ∈ N out ( x n ) indicating straight-line code.
Revised: When | N out ( x n ) | = 1 we have b t,n,n ′ = 1 for n ′ ∈ N out ( x n ) indicating straight-line code.

Original: For all other n , n ′ , b t,n,n ′ = 0 .
Revised: For all other n , n ′ , b t,n,n ′ = 0 .

Original: The state proposals and branch decisions in turn feed into the computation of the new hidden states
Revised: The state proposals and branch decisions in turn feed into the computation of the new hidden states

Original: [Equation]
Revised: [Equation]

Original: and new instruction pointer values
Revised: and new instruction pointer values

Original: [Equation]
Revised: [Equation]

Original: The hidden state at final time step T ( x ) at the program’s exit node n exit , given by h T ( x ) ,n exit are used for downstream predictions.
Revised: The hidden state at final time step T ( x ) at the program’s exit node n exit , given by h T ( x ) ,n exit are used for downstream predictions.

Original: D I NPUT M ODULATION
Revised: D I NPUT M ODULATION

Original: In Section 4.2, we consider both cross-attention (Lee et al., 2019) and Feature-wise Linear Modulation (FiLM)
Revised: In Section 4.2, we consider both cross-attention (Lee et al., 2019) and Feature-wise Linear Modulation (FiLM)

Original: (Perez et al., 2017) as options for the Modulate function.
Revised: (Perez et al., 2017) as options for the Modulate function.

Original: We provide the definitions of these operations here.
Revised: We provide the definitions of these operations here.

Original: First, cross-attention modules the input as:
Revised: First, cross-attention modules the input as:

Original: MultiHead
Revised: MultiHead

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: = Concat(Concat(
Revised: = Concat(Concat(

Original: [Equation]
Revised: [Equation]

Original: 1 , ..., head h
Revised: 1 , ..., head h

Original: [Equation]
Revised: [Equation]

Original: O ,
Revised: O ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: n ))
Revised: n ))

Original: (12) where head i = softmax (cid:32) QK
Revised: (12) where head i = softmax (cid:32) QK

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Here, W O ∈ R hd v × d model , W Qi ∈ R d k × ( d model + d Embed( xn ) ) , W Ki ∈ R
Revised: Here, W O ∈ R hd v × d model , W Qi ∈ R d k × ( d model + d Embed( xn ) ) , W Ki ∈ R

Original: d k × d d ( x ) , and W
Revised: d k × d d ( x ) , and W

Original: Vi ∈ R d v × d d ( x ) are learnable parameters.
Revised: Vi ∈ R d v × d d ( x ) are learnable parameters.

Original: Similarly, for FiLM we modulate the input with the resource description as follows:
Revised: Similarly, for FiLM we modulate the input with the resource description as follows:

Original: [Equation]
Revised: [Equation]

Original: where β
Revised: where β

Original: = σ ( W β Concat( x n , h t − 1 ,n )
Revised: = σ ( W β Concat( x n , h t − 1 ,n )

Original: + b β ) , (18) γ = σ ( W γ Concat( x n , h t − 1 ,n ) + b γ ) , (19) where W γ ∈ R d
Revised: + b β ) , (18) γ = σ ( W γ Concat( x n , h t − 1 ,n ) + b γ ) , (19) where W γ ∈ R d

Original: d
Revised: d

Original: ( x ) × ( d model
Revised: ( x ) × ( d model

Original: + d Embed( xn ) ) ,
Revised: + d Embed( xn ) ) ,

Original: and W γ ∈ R d d ( x ) × ( d model + d Embed( xn ) ) are learnable parameters.
Revised: and W γ ∈ R d d ( x ) × ( d model + d Embed( xn ) ) are learnable parameters.

Original: E T RAINING D ETAILS
Revised: E T RAINING D ETAILS

Original: Hyperparameter selection We select hyperparameters by performing a random search independently for each model architecture.
Revised: Hyperparameter selection We select hyperparameters by performing a random search independently for each model architecture.

Original: The hyperparameters considered by the search are listed in
Revised: The hyperparameters considered by the search are listed in

Original: Table 6.
Revised: Table 6.

Original: All architectures use a Transformer encoder, and the Transformer sizes considered in the search are listed in Table 6 and defined further in Table 5.
Revised: All architectures use a Transformer encoder, and the Transformer sizes considered in the search are listed in Table 6 and defined further in Table 5.

Original: [Table]
Revised: [Table]

Original: Step limit For the IPA-GNN and Exception IPA-GNN, the function T ( x ) represents the number of execution steps modeled for program x .
Revised: Step limit For the IPA-GNN and Exception IPA-GNN, the function T ( x ) represents the number of execution steps modeled for program x .

Original: We reuse the definition of T ( x ) from Bieber et al.
Revised: We reuse the definition of T ( x ) from Bieber et al.

Original: (2020) as closely as possible, only modifying it to accept arbitrary Python programs, rather than being restricted to the subset of Python features considered in the dataset of the earlier work.
Revised: (2020) as closely as possible, only modifying it to accept arbitrary Python programs, rather than being restricted to the subset of Python features considered in the dataset of the earlier work.

Revised: The scatter plot in Figure ?? shows the relationship between T ( x ) and program length.

Original: Parameter counts We provide in Table 7 the total number of parameters in each model, for the best performing hyperparameters in each model class.
Revised: Parameter counts We provide in Table 7 the total number of parameters in each model, for the best performing hyperparameters in each model class.

Original: For all model classes, the maximum number of parameters considered is roughly equal (approximately 8.8 million)
Revised: For all model classes, the maximum number of parameters considered is roughly equal (approximately 8.8 million)

Original: [Table]
Revised: [Table]

Original: Compute usage and model speeds All models are trained on Google Cloud Platform using
Revised: Compute usage and model speeds All models are trained on Google Cloud Platform using

Original: TPUv2 accelerators.
Revised: TPUv2 accelerators.

Original: We use approximately one TPU-week of compute in training each IPA-GNN model.
Revised: We use approximately one TPU-week of compute in training each IPA-GNN model.

Original: At inference time, IPA-GNN compute is proportional to the number of model steps, whichis up to 174 for examples in our dataset. We measure the average inference time on the test set: 0.
Revised: At inference time, IPA-GNN compute is proportional to the number of model steps, which

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ODEL
Revised: ODEL

Original: [Table]
Revised: [Table]

Original: ARAMETER
Revised: ARAMETER

Original: [Table]
Revised: [Table]

Original: OUNT
Revised: OUNT

Original: T RAIN L ATENCY
Revised: [Table] RAIN [Table] ATENCY

Original: [Table]
Revised: [Table]

Original: NFERENCE L ATENCY
Revised: NFERENCE [Table] ATENCY

Original: [Table]
Revised: [Table]

Revised: is up to 174 for examples in our dataset.

Revised: We measure the average inference time on the test set: 0.

Original: seconds per batch of 32.
Revised: seconds per batch of 32.

Original: We also measure the training speed in seconds per step for each method, which we report in Table 7.
Revised: We also measure the training speed in seconds per step for each method, which we report in Table 7.

Original: We observe that the IPA-GNN train times are slower than those of the generic models, a drawback of the IPA-GNN model family in its current implementations.
Revised: We observe that the IPA-GNN train times are slower than those of the generic models, a drawback of the IPA-GNN model family in its current implementations.

Original: That said, we also note that the IPA-GNN models do not benefit from the same optimizations as basic implementations of the well known general purpose models (GGNN, Transformer, and LSTM), and with further optimizations the IPA-GNN performance can be improved.
Revised: That said, we also note that the IPA-GNN models do not benefit from the same optimizations as basic implementations of the well known general purpose models (GGNN, Transformer, and LSTM), and with further optimizations the IPA-GNN performance can be improved.

Original: F M ETRIC V ARIANCES
Revised: F M ETRIC V ARIANCES

Original: Under the experimental conditions of Section 5.1, we perform three additional training runs to calculate the variance for each metric for each baseline model, and for the Exception IPA-GNN model using the docstring strategy for processing resource descriptions.
Revised: Under the experimental conditions of Section 5.1, we perform three additional training runs to calculate the variance for each metric for each baseline model, and for the Exception IPA-GNN model using the docstring strategy for processing resource descriptions.

Original: For these new training runs, we use the hyperparameters obtained from model selection.
Revised: For these new training runs, we use the hyperparameters obtained from model selection.

Original: We vary the random seed between runs (0, 1, 2), thereby changing the initialization and dropout behavior of each model across runs.
Revised: We vary the random seed between runs (0, 1, 2), thereby changing the initialization and dropout behavior of each model across runs.

Original: We report the results in Table 8; ± values are one standard deviation. 
Revised: We report the results in Table 8; ± values are one standard deviation. [Figure] [Table]

Original: [Table]
Revised: [Table]

Original: G S TATIC A NALYSIS B ASELINE
Revised: G S TATIC A NALYSIS B ASELINE

Original: Our work builds towards a developer tool that predicts runtime errors in programs without running the program, treating the task as static analysis.
Revised: Our work builds towards a developer tool that predicts runtime errors in programs without running the program, treating the task as static analysis.

Original: Existing static analysis tools already inspect Python source code for possible issues, though they are not generally designed with runtime error prediction in mind.
Revised: Existing static analysis tools already inspect Python source code for possible issues, though they are not generally designed with runtime error prediction in mind.

Original: Among the most popular such tools are the linters pylint and flake8, the type analyzer pytype, and the formatter black.
Revised: Among the most popular such tools are the linters pylint and flake8, the type analyzer pytype, and the formatter black.

Original: We elect to compare against pylint as it is the most common of these tools and hence most representative of a modern developer workflow.
Revised: We elect to compare against pylint as it is the most common of these tools and hence most representative of a modern developer workflow.

Original: Additionally, a formatter is not well suited to the task of predicting errors, and type analysis benefits from type annotations which are rarely utilized in competition code.
Revised: Additionally, a formatter is not well suited to the task of predicting errors, and type analysis benefits from type annotations which are rarely utilized in competition code.

Original: In our comparison of machine learning methods against pylint (Section 5), we build a runtime error classifier based on pylint’s output.
Revised: In our comparison of machine learning methods against pylint (Section 5) [Table] we build a runtime error classifier based on pylint’s output.

Original: For each kind of error or warning that pylint can detect, we determine whether it is indicative a runtime error class.
Revised: For each kind of error or warning that pylint can detect, we determine whether it is indicative a runtime error class.

Original: For example, pylint’s error no-member (E1101) indicates the AttributeError runtime error.
Revised: For example, pylint’s error no-member (E1101) indicates the AttributeError runtime error.

Original: The pylint baseline predicts a runtime error class whenever pylint’s errors or warnings indicate that error class, and “no error” otherwise.
Revised: The pylint baseline predicts a runtime error class whenever pylint’s errors or warnings indicate that error class, and “no error” otherwise.

Original: Table 9 shows the mapping from pylint findings to runtime error.
Revised: Table 9 shows the mapping from pylint findings to runtime error.

Original: Only eleven of twenty-six the runtime error classes (those listed in Table 9, and “no error”) can be predicted by this baseline.
Revised: Only eleven of twenty-six the runtime error classes (those listed in Table 9, and “no error”) can be predicted by this baseline.

Original: Additionally, the presence of a pylint finding that corresponds to an error does not guarantee the error would actually be present when running the program; for example an[Table] undefined variable may appear on an unused control-flow path, benign at runtime.
Revised: Additionally, the presence of a pylint finding that corresponds to an error does not guarantee the error would actually be present when running the program; for example an undefined variable may appear on an unused control-flow path, benign at runtime.

Original: The results of this baseline are reported in Section 5.
Revised: The results of this baseline are reported in Section 5.

Original: H L OCALIZATION BY M ODELING E XCEPTION H ANDLING
Revised: H L OCALIZATION BY M ODELING E XCEPTION H ANDLING

Original: For programs that lack try/except frames, we compute the localization predictions of the Exception
Revised: For programs that lack try/except frames, we compute the localization predictions of the Exception

Original: IPA-GNN model by summing, separately for each node, the contributions from that node to the exception node across all time steps.
Revised: IPA-GNN model by summing, separately for each node, the contributions from that node to the exception node across all time steps.

Original: This gives an estimate of exception provenance as
Revised: This gives an estimate of exception provenance as

Revised: [Equation]

Revised: at statement

Original: [Equation]
Revised: [Equation]

Original: For programs with a try/except frame, however, we must trace the exception back to the statement that originally raised it.
Revised: For programs with a try/except frame, however, we must trace the exception back to the statement that originally raised it.

Original: To do this, we keep track of the exception provenance at each node at each time step; when an exception raises, it becomes the exception provenance at the statement that it raises to, and when a statement with non-zero exception provenance executes without raising, it propagates its exception provenance to the next node unchanged.
Revised: To do this, we keep track of the exception provenance at each node at each time step; when an exception raises, it becomes the exception provenance at the statement that it raises to, and when a statement with non-zero exception provenance executes without raising, it propagates its exception provenance to the next node unchanged.

Original: Define v t,n,n ′ as the amount of “exception probability mass” at time step t at node n ′ attributable to an exception starting at node n .
Revised: Define v t,n,n ′ as the amount of “exception probability mass” at time step t at node n ′ attributable to an exception starting at node n .

Original: Then we write 
Revised: Then we write v t

Original: [Equation]
Revised: [Equation]

Original:  The first term propagates exception provenance across normal non-raising execution, while the second term introduces exception provenance when an exception is raised.
Revised: n [Equation] [Equation] The first term propagates exception provenance across normal non-raising execution, while the second term introduces exception provenance when an exception is raised.

Original: We then write precisely
Revised: We then write precisely

Original: [Equation]
Revised: [Equation]

Original: error at statement 
Revised: error at statement n

Original: [Equation]
Revised: [Equation]

Revised: n,n error ,

Revised: [Equation]

Original: (22) allowing the Exception IPA-GNN to make localization predictions for any program in the dataset.
Revised: 22) allowing the Exception IPA-GNN to make localization predictions for any program in the dataset.

Original: I L OCALIZATION BY M ULTIPLE I NSTANCE L EARNING
Revised: I L OCALIZATION BY M ULTIPLE I NSTANCE L EARNING

Original: The Local Transformer and Global Transformer models each compute per-statement node embeddings Embed( x n ) given by Equation 1.
Revised: The Local Transformer and Global Transformer models each compute per-statement node embeddings Embed( x n ) given by Equation 1.

Original: In the multiple instance learning setting, these are transformed into unnormalized per-statement class predictions
Revised: In the multiple instance learning setting, these are transformed into unnormalized per-statement class predictions

Original: [Equation]
Revised: [Equation]

Original: We consider three strategies for aggregating these per-statement predictions into an overall prediction for the task.
Revised: We consider three strategies for aggregating these per-statement predictions into an overall prediction for the task.

Original: Under the logsumexp strategy, we treat ϕ as logits and write
Revised: Under the logsumexp strategy, we treat ϕ as logits and write

Original: [Equation]
Revised: [Equation]

Original: where K is the set of error classes.
Revised: where K is the set of error classes.

Original: The max and mean strategies meanwhile follow Wang et al.
Revised: The max and mean strategies meanwhile follow Wang et al.

Original: in asserting
Revised: in asserting

Original: [Equation]
Revised: [Equation]

Original:  compute the location probabilities as
Revised: 26) compute the location probabilities as

Original: [Equation]
Revised: [Equation]

Original: and compute the outputs as
Revised: and compute the outputs as

Original: [Equation]
Revised: [Equation]

Original: respectively, where L denotes the number of statements in x .
Revised: respectively, where L denotes the number of statements in x .

Original: As with all methods considered, the MIL models are trained to minimize the cross-entropy loss in target class prediction, but these methods still allow reading off predictions of p ( lineno ) .
Revised: As with all methods considered, the MIL models are trained to minimize the cross-entropy loss in target class prediction, but these methods still allow reading off predictions of p ( lineno ) .

Original: J B ROADER I MPACT
Revised: J B ROADER I MPACT

Original: Our work builds toward improvements to developer tools, suggesting the possibility of future tools that predict runtime errors in code even when that code lacks unit tests.
Revised: Our work builds toward improvements to developer tools, suggesting the possibility of future tools that predict runtime errors in code even when that code lacks unit tests.

Original: However, the false positive rate under the current best models present a challenge.
Revised: However, the false positive rate under the current best models present a challenge.

Original: A developer tool built using these models may present the developer with incorrect predictions.
Revised: A developer tool built using these models may present the developer with incorrect predictions.

Original: This could cause the developer to make mistakes, or to lose trust in the tooling, lowering productivity in the short term and making it harder to win back trust in the long term when tools are built upon higher quality models with fewer errors.
Revised: This could cause the developer to make mistakes, or to lose trust in the tooling, lowering productivity in the short term and making it harder to win back trust in the long term when tools are built upon higher quality models with fewer errors.

Original: We therefore recommend that tool developers use a combination of cautious judgement and data driven evaluations when deciding when to implement features that rely on models like the ones we present.
Revised: We therefore recommend that tool developers use a combination of cautious judgement and data driven evaluations when deciding when to implement features that rely on models like the ones we present.

Original: K E XAMPLE V ISUALIZATIONS
Revised: K E XAMPLE V ISUALIZATIONS

Original: We sample three examples at random from the Python Runtime Error dataset validation split, and visualize them here.
Revised: We sample three examples at random from the Python Runtime Error dataset validation split, and visualize them here.

Original: As in Figure 2, we show instruction pointer heatmaps for both the B ASELINE and D OCSTRING Exception IPA-GNN model variants.
Revised: As in Figure 2, we show instruction pointer heatmaps for both the B ASELINE and D OCSTRING Exception IPA-GNN model variants.

Original: In the heatmaps, the x-axis represents time steps and the y-axis represents nodes, with the last two rows representing the exit node n exit and the exception node n error .
Revised: In the heatmaps, the x-axis represents time steps and the y-axis represents nodes, with the last two rows representing the exit node n exit and the exception node n error .

Original: Note that for loop statements are associated with two spans in the statement-level control-flow graph, one for construction of the loop iterator, and a second for assignment to the loop variable.
Revised: Note that for loop statements are associated with two spans in the statement-level control-flow graph, one for construction of the loop iterator, and a second for assignment to the loop variable.

Original: Hence we list two indexes for each for loop statement in these figures, and report the total error contribution for the line.
Revised: Hence we list two indexes for each for loop statement in these figures, and report the total error contribution for the line.

Original: S TD I N D ESCRIPTION
Revised: [Figure]

Original: Input: Input is given from Standard Input in the following format:
Revised: Input:Inputis given from Standard Input in the following format:

Original: N a 1 a 2 ... a N Constraints:
Revised: Na 1 a 2 ... a N Constraints:

Original: All values in input are integers.
Revised: All values in input are integers.

Original: [Figure]
Revised: [Figure]

Original: <=
Revised: S

Original: [Figure]
Revised: [Figure]

Original: a i <=n S OURCE
Revised: N = int(input

Original: B ASELINE Error contrib.

Original: R.D. Error contrib

Original: [Figure]
Revised: [Figure]

Revised: A = list(map(int, input().split()))

Original: [Figure]
Revised: [Figure]

Original: for i in range(1, len(A)+1, 2):
Revised: for i in range(1, len(A)+1, 2):

Original: [Figure]
Revised: [Figure]

Revised: +

Original: (res)
Revised: [Figure] [i] [Figure] print(res)

Original: [Figure]
Revised: [Figure]

Original: S TD I N D ESCRIPTION
Revised: S TD I N D ESCRIPTION

Original: Input: Input is given from Standard Input in the following format:
Revised: Input:Inputis given from Standard Input in the following format:

Original: H N A 1 A 2 ...
Revised: H N A 1 A 2 ...

Original: A N Constraints: 1 <=
Revised: A N Constraints: 1 <=

Original: H <
Revised: H <

Original: = 10ˆ9 1 <= N <
Revised: = 10ˆ9 1 <= N <

Original: = 10ˆ5 1 <
Revised: = 10ˆ5 1 <

Original: =
Revised: =

Original: A i <= 10ˆ All values in input are integers.
Revised: A i <= 10ˆ4 All values in input are integers.

Original: n
Revised: n

Original: S OURCE
Revised: S OURCE

Original: B ASELINE Error contrib.
Revised: B ASELINE Error contrib.

Original: R.D. Error contrib.
Revised: R.D. Error contrib.

Original: H,N,A = list(map(int, input().split())) 9.
Revised: H,N,A = list(map(int, input().split())) 1.

Original:  1, for i in A[N]: 43.
Revised: 2.1 1, for i in A[N]: 9.

Original: 83.0if H < = 0: 2.
Revised: 34.0if H < = 0: 4.

Original: 2.8 break else:
Revised: 0.8 break else:

Original: H -= A[i] 6.
Revised: H -= A[i] 0.

Original: 0.0if set(A): 0.
Revised: 3.5if set(A): 3.

Original: 0.1print("Yes") 9.
Revised: 15.7print("Yes") 1.

Original: 0.7 else: print("No") 3.
Revised: 2.3 else: print("No") 2.

Revised: B ASELINE

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: S TD I N D ESCRIPTION
Revised: S TD I N D ESCRIPTION

Original: Input: n m d1 d2 ... dm Two integers n and m are given in the first line.
Revised: Input:nm d1d2... dm Two integers n and m are given in the first line.

Original: The available denominations are given in the second line.
Revised: The available denominations are given in the second line.

Original: Constraints: 1 <= n <= 50000 1 <= m <= 20 1 <= denomination <=[-
-]
Revised: Constraints: 1 <= n <= 50000 1 <= m <= 20 1 <= denomination <=[-
-]

Original: The denominations are all different and contain 1.
Revised: The denominations are all different and contain 1.

Original: n
Revised: n

Original: S OURCE
Revised: S OURCE

Original: B ASELINE Error contrib.
Revised: B ASELINE Error contrib.

Original: R.D. Error contrib.
Revised: R.D. Error contrib.

Original: from itertools import combinations with replacement as C 40.
Revised: from itertools import combinations with replacement as C 46.

Original: n, m = map(int, input().split()) 2.
Revised: n, m = map(int, input().split()) 10.

Original: coin = sorted(list(map(int, input().split()))) 7.
Revised: 0.0coin = sorted(list(map(int, input().split()))) 1.

Original: if n in coin: 2.
Revised: if n in coin: 1.

Original: 0.2print(1) 2.
Revised: 4.6print(1) 0.

Original: 1.5 else: end = n // coin[0] +0.
Revised: 0.0 else: end = n // coin[0] +0.

Original: 0.1b =
Revised: 0.4b =

Original: False 0.
Revised: False 0.

Original: 0.3 7, for i in range(2, end): 2.
Revised: 0.3 7, for i in range(2, end): 3.

Original: 0.7 9, for tup in list(C(coin, i)): 3.
Revised: 1.9 9, for tup in list(C(coin, i)): 3.

Original: 1.2if sum(tup) =
Revised: 0.0if sum(tup) =

Original: = n: 0.
Revised: = n: 0.

Original: 0.0print(i) 0.
Revised: 0.3print(i) 0.

Original: 0.1b =
Revised: 0.4b =

Original: True 0.
Revised: True 0.

Original: 0.9 break if b: break 0.
Revised: 2.4 break if b: break 0.

Original: B ASELINE
Revised: B ASELINE

Original: [Figure] [Figure] [Figure] [Figure]
Revised: R ESOURCE D ESCRIPTION confidence 0.64.

Original: The B ASELINE instruction pointer value makes its largest probability mass contribution to n error at n = 0 and ends up with mass split between n exit and n error . The D OCSTRING instruction pointer value  accumulates little probability in n error and{+probability contribution at n = 
 but+} ends up with most probability mass in n exit .
Revised: The B ASELINE instruction pointer value makes its largest probability mass contribution to n error at n = 0 and ends up with mass split between n exit and n error . The D OCSTRING instruction pointer value also accumulates most n error {+probability contribution at n = 
 but+} ends up with most probability mass in n exit .