D EEP D ATA F LOW A NALYSIS
Anonymous authors
Paper under double-blind review
A BSTRACT
Compiler architects increasingly look to machine learning when building heuristics for compiler optimization.
The promise of automatic heuristic design, freeing the compiler engineer from the complex interactions of program, architecture, and other optimizations, is alluring.
However, most machine learning methods cannot replicate even the simplest of the abstract interpretations of data ﬂow analysis that are critical to making good optimization decisions.
This must change for machine learning to become the dominant technology in compiler heuristics.
To this end, we propose P RO G RA ML – Program Graphs for Machine Learning – a language-independent, portable representation of whole-program semantics for deep learning.
To benchmark current and future learning techniques for compiler analyses we introduce an open dataset of 461k Intermediate Representation (IR) ﬁles for LLVM, covering ﬁve source programming languages, and 15.4M corresponding data ﬂow results.
We formulate data ﬂow analysis as an MPNN and show that, using P RO G RA ML, standard analyses can be learned, yielding improved performance on downstream compiler optimization tasks.
I NTRODUCTION
Compiler implementation is a complex and expensive activity (Cooper & Torczon, 2012).
For this reason, there has been signiﬁcant interest in using machine learning to automate various compiler tasks (Allamanis et al., 2018).
Most works have restricted their attention to selecting compiler heuristics or making optimization decisions (Ashouri et al., 2018; Wang & O’Boyle, 2018).
Whether learned or engineered by human experts, these decisions naturally require reasoning about the program and its behavior.
Human experts most often rely upon data ﬂow analyses (Kildall, 1973; Kam & Ullman, 1976).
These are algorithms on abstract interpretations of the program, propagating information of interest through the program’s control-ﬂow graph until a ﬁxed point is reached (Kam & Ullman, 1977).
Two examples out of many data ﬂow analyses are: liveness – determining when resources become dead (unused) and may be reclaimed; and available expressions – discovering which expressions have been computed on all paths to points in the program.
Prior machine learning works, on the other hand, have typically represented the entirety of the program’s behavior as a ﬁxed-length, statically computed feature vector (Ashouri et al., 2018).
Typical feature values might be the number of instructions in a loop or the dependency depth.
The weakness of these techniques is shown by the fact that they are trivially confused by the addition of dead code, which changes their feature vectors without changing the program’s behavior or its response to optimizations.
Such learning algorithms are unable to learn their own abstract interpretations of the program and so cannot avoid these pitfalls or more subtle versions thereof (Barchi et al., 2019).
Recently, there have been attempts to develop representations that allow ﬁner-grain program reasoning.
Many, however, are limited both by how inputs are represented as well as how inputs are processed.
Representations based on source code and its direct artifacts (e.g., AST) (Alon et al., 2018a;
Yin et al., 2018; Haj-Ali et al., 2020) put unnecessary emphasis on naming and stylistic choices that may not correlate with the functionality of the code (e.g., Fig. 2a).
Approaches based on intermediate representations (IR)
(Ben-Nun et al., 2018; Mirhoseini et al., 2017; Brauckmann et al., 2020) remove such noise but fail to capture information about the program that is important for analysis (e.g., Fig.
2b variables, Fig.
2c commutativity).
In both cases, models are expected to reason about the ﬂow of information in programs using representations that do not directly encode this information.
Clearly, a program representation is needed that enables machine learning algorithms to reason about the execution of a program by developing its own data ﬂow analyses.
[Figure]
Since current approaches are ill-suited to program-wide data ﬂow analysis, we propose overcoming their limitations by making the program’s control, data, and call dependencies a central part of the program’s representation and a primary consideration when processing it.
We achieve this by seeing the program as a graph in which individual statements are connected to other statements through relational dependencies.
Each statement in the program is understood only in the context of the statements interacting with it.
Through relational reasoning (Battaglia et al., 2018), a latent representation of each statement is learned that is a function of not just the statement itself, but also of the (latent) representations of its graph neighborhood.
Notably, this formulation has a striking similarity to the IRs used by compilers, and the iterative propagation of information resembles the transfer functions and meet operators in traditional data
ﬂow analyses (Kildall, 1973).
Recently proposed techniques for learning over graphs have shown promise in a number of domains (Schlichtkrull et al., 2018; Ziwei et al., 2020).
With a suitable representation and graph-based model, we extend these approaches to the domain of compiler analysis, enabling downstream tasks built on top of such graph models to natively incorporate reasoning about data ﬂow into their decision making.
This improves performance on downstream tasks without requiring additional features.
We make the following contributions:
• We propose a portable, language-independent graph representation of programs derived from compiler IRs.
P RO G RA ML is the ﬁrst representation to capture whole-program control-, data-, and call relations between instructions and operands as well as their order and data types.
P RO G RA ML is a compiler-agnostic design for use at all points in the optimization pipeline; we provide implementations for LLVM and XLA IRs.
• We introduce a benchmark dataset that poses a suite of established compiler analysis tasks as supervised machine learning problems.
D EEP D ATA F LOW comprises ﬁve tasks that require, in combination, the ability to model: control- and data-ﬂow, function boundaries, instruction types, and the type and order of operands over complex programs.
D EEPD ATA F LOW is constructed from 461k real-world program IRs covering a diverse range of domains and source languages, totaling 8.5 billion data ﬂow analysis classiﬁcation labels.
• We adapt Gated-Graph Neural Networks (GGNN) to the P RO G RA ML representation.
We show that, within a bounded problem size, our approach achieves ≥ 0 .
939 F 1 score on all analysis tasks, a signiﬁcant improvement over state-of-the-art representations.
In evaluating the limits of this approach we propose directions to better learn over programs.
R ELATED W ORK
Data ﬂow analysis is a long established area of work ﬁrmly embedded in modern compilers.
Despite its central role, there has been limited work in learning such analysis.
Bielik et al.
(2017) use ASTs and code synthesis to learn rule-sets for static analyses, some of which are dataﬂow-related.
Our approach does not require a program generator or a hand-crafted DSL for rules.
Shi et al.
(2020) and Wang & Su (2020) use dynamic information (e.g., register snapshots and traces) from instrumented binaries to embed an assembler graph representation.
We propose a static approach that does not need runtime features.
Si et al.
(2018) use a graph embedding of an SSA form to generate invariants.
The lack of phi nodes and function call/return edges means that the representation is not suitable for interprocedural analysis as it stands.
Kanade et al.
(2020) explore a large-scale, context-dependent vector embedding.
This is done at a token level, however, and is unsuited for dataﬂow analysis.
Prior work on learning over programs employed methods from Natural Language Processing that represented programs as a sequence of lexical tokens (Allamanis, 2016; Cummins et al., 2017a).
[Figure]
However, source-level representations are not suited for analyzing partially optimized compiler IRs as the input source cannot be recovered.
In program analysis it is critical to capture the structured nature of programs (Raychev et al., 2015; Allamanis et al., 2017; Alon et al., 2018b).
Thus, syntactic (tree-based) as well as semantic (graph-based) representations have been proposed (Allamanis et al., 2018; Brauckmann et al., 2020).
Dam et al.
(2018) annotate nodes in Abstract Syntax Trees (ASTs) with type information and employ Tree-Based LSTMs (Tai et al., 2015) for program defect prediction.
Both Raychev et al. (2015) and Alon et al.
(2018a;b) use path-based abstractions of the
AST as program representations, while Allamanis et al.
(2017) augment ASTs with a hand-crafted set of additional typed edges and use GGNNs (Li et al., 2015) to learn downstream tasks related to variable naming.
Another line of research considers modelling binary similarity via control-ﬂow graphs (CFGs) with an adaptation of GNNs called Graph Matching Networks (Li et al., 2019).
The history of IR-based graph representations for optimization goes back to Ferrante et al. (1987), who remove superﬂuous control-ﬂow edges to ease optimization with a compact graph representation.
A more contemporary precursor to our approach is the ConteXtual Flow Graph (XFG) (BenNun et al., 2018), which combines control-ﬂow with data-ﬂow relations in order to learn unsupervised embeddings of LLVM-IR statements.
XFGs omit information that is critical to analysis including the notion of argument order, vertices for both variables and constants, and all control-ﬂow edges.
P RO G RA ML, in combining call-graphs (CG), control-ﬂow graphs, and data-ﬂow graphs (DFG), offers an IR-level program representation that is designed to be useful for a variety of purposes from speciﬁc program analyses to downstream optimization tasks.
Control and Data Flow
Graphs (CDFG) (Brauckmann et al., 2020) use graph vertices for statements and have bi-directional edges for control and data dependencies.
The CDFG uses only the instruction opcode to represent a statement, omitting operands, variables, data types, and constants.
This prohibits the reasoning about variables and expressions that are required for many data ﬂow analyses, including 3 out of the 5 benchmark tasks that we establish below. Mendis et al.
(2019) represent LLVM-IR using a graph that is specialized to a vectorization task.
They use unique edge types to differentiate the ﬁrst ﬁve operand positions and augment the graph structure with vectorization opportunities that they compute a priori.
Our approach is not specialized to a task, enabling such opportunities learned (e.g., subexpression detection), and uses a embedding weighting to differentiate edge positions without having to learn separate edge transfer weights for each.
Finally, an alternate approach is taken by
IR2Vec (Keerthy S et al., 2019), an LLVM-IR-speciﬁc representation that elegantly models part-ofstatements as relations.
However, in order to compute the values of the embeddings, IR2Vec requires access to the type of data ﬂow analyses that our approach learns from data alone.
A G RAPHICAL R EPRESENTATION FOR D EEP P ROGRAM A NALYSIS
This section presents P RO G RA ML, a novel IR-based program representation that closely matches the data structures used traditionally in inter-procedural data ﬂow analysis and can be processed natively by deep learning models.
We represent programs as directed multigraphs where instructions, variables, and constants are vertices, and relations between vertices are edges.
Edges are typed to differentiate control-, data-, and call-ﬂow.
Additionally, we augment edges with a local position attribute to encode the order of operands to instructions, and to differentiate between divergent branches in control-ﬂow. [Figure] [Figure] [Figure]
[Figure]
i i32 call i add i ret i i
[Figure] [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure]
We construct a P RO G RA ML graph G =
( V, E ) by traversing a compiler IR.
An initially empty graph G = ∅ is populated in three stages: control-ﬂow, data-ﬂow, and call-ﬂow, shown in Figure 3.
In practice the three stages of graph construction can be combined in a single O ( | V | + | E | ) pass.
(I) Control Flow We construct the full-ﬂow graph of an IR by inserting a vertex for each instruction and connecting control-ﬂow edges (Fig. 3a, 3b).
Control edges are augmented with a numeric position using an ascending sequence based on their order in the list of an instruction’s successors.
(II) Data Flow We introduce constant values and variables as graph vertices (Fig. 3c).
Dataﬂow edges are inserted to capture the relation from constants and variables to the instructions that use them as operands, and from instructions to produced variables.
As each unique variable and constant is a vertex, variables can be distinguished by their scope, and unlike the source-level representations of prior works, variables in different scopes map to distinct vertices and can thus be discerned.
Data edges have a position attribute that encodes the order of operands for instructions.
The latent representation of a statement (e.g., %1 = add i32 %0, 1 ) is thus a function of the vertex representing the instruction and the vertices of any operand variables or constants, modulated by their order in the list of operands.
(III) Call Flow Call edges capture the relation between an instruction that calls a function and the entry instruction of the called function (Fig. 3d).
Return call edges are added from each of the terminal instructions of a function to the calling statement.
Control edges do not span functions, such that an IR with functions F produces | F | disconnected subgraphs (the same is not true for data edges which may cross function boundaries, e.g., in the case of a global constant which is used across many parts of a program).
For IRs that support external linkage, an additional vertex is created representing an external call site and connected to all externally visible functions.
If a call site references a function not deﬁned in the current IR, a dummy function is created consisting of a single instruction vertex and connected through call edges to all call sites in the current IR. A unique dummy function is created for each externally deﬁned function.
G RAPH - BASED M ACHINE L EARNING FOR P ROGRAM A NALYSIS
We formulate our system in a Message Passing Neural Network (MPNN) framework (Gilmer et al., 2017).
Our design mimics the transfer functions and meet operators of classical iterative data ﬂow analysis (Kam & Ullman, 1977; Cooper et al., 2004), replacing the rule-based implementations with learnable analogues (message and update functions).
This single uniﬁed model can be specialized through training to solve a diverse set of problems without human intervention or algorithm design.
The P RO G RA ML model is an adaptation of GGNN (Li et al., 2015) that takes as input an attributed directed multigraph as presented in Section 3.
It consists of three logical phases: input encoding, message propagation and update, and result readout.
(I) Input Encoding
Starting from the augmented graph representation G =
( V, E ) , we capture the semantics of the program graph vertices by mapping every instruction, constant, and variable vertex v ∈ V to a vector representation h 0 v ∈ R d by lookup in a ﬁxed-size embedding table.
The mapping from vertex to learnable embedding vector f : v (cid:55)→ h 0 v must be deﬁned for each IR.
For LLVM-IR, we construct an embedding key from each vertex using the name of the instruction, e.g., store , and the data type for variables and constants, e.g., i32* (a pointer to a 32-bit integer).
In this manner we derive the set of unique embedding keys using the graph vertices of a training set of LLVM-IRs described in Section 5.1.
This deﬁnes the embedding table used for training and deployment.
An unknown element embedding is used during deployment to map embedding keys which were not observed in the training data.
Since composite types make the size of the vocabulary unbounded in principle, our data-driven approach trades a certain amount of semantic resolution against good coverage of the vocabulary by the available datasets (cf.
Table ?? ).
The embedding vectors are trained jointly with the rest of the model.
(II) Message Propagation
Each iteration step is divided into a message propagation followed by vertex state update.
Receiving messages M ( h t − 1 w , e wv ) are a function of neighboring states and the respective edge.
Messages are mean-aggregated over the neighborhood after transformation with a custom position-augmented transfer function that scales h w elementwise with a position-gating vector p ( e wv ) :
[Equation]
The position-gating p ( e wv ) =
2 σ ( W p emb( e wv ) + b p ) is implemented as a sigmoid-activated linear layer mapping from a constant sinusoidal position embedding (Vaswani et al., 2017; Gehring et al., 2017).
It enables the network to distinguish non-commutative operations such as division, and the branch type in diverging control-ﬂow.
In order to allow for reverse-propagation of information, which is necessary for backward compiler analyses, we add backward edges for each edge in the graph as separate edge-types.
In all our experiments, we employ Gated Recurrent Units (GRU) (Cho et al., 2014) as our update function.
Step (II) is iterated T times to extract vertex representations that are contextualized with respect to the given graph structure.
(III) Result Readout Data ﬂow analyses compute value sets composed of instructions or variables.
We support per-instruction and per-variable classiﬁcation tasks using a readout head on top of the iterated feature extraction, mapping, for each vertex, the extracted vertex features h Tv to probabilities R v ( h Tv , h 0 v ) :
[Equation]
where f ( · ) and g ( · ) are linear layers and σ ( · ) is the sigmoid activation function. [Figure]
D ATA F LOW E XPERIMENTS
Data ﬂow analysis is at the heart of modern compiler technology.
We pose a suite of data ﬂow analyses as supervised learning tasks to benchmark the representational power of machine learning approaches.
We selected a diverse set of tasks that capture a mixture of both forward and backward analyses, and control-, data-, and procedure-sensitive analyses.
Full details of the analyses are provided in Appendix A. These particular data ﬂow analyses can already be perfectly solved by nonML techniques.
Here, we use them to benchmark the capabilities of machine learning techniques.
5. T HE D EEP D ATA F LOW D ATASET
We assembled a 256M-line corpus of LLVM-IR ﬁles from a variety of sources and produced labeled datasets using ﬁve traditional data ﬂow analyses: control reachability, dominators, data dependencies, liveness, and subexpressions.
Each of the 15.4M analysis examples consists of an input graph in which a single vertex is annotated as the root node for analysis, and an output graph in which each vertex is annotated with a binary label corresponding to its value once the data ﬂow analysis has completed (Fig. 4).
A 3:1:1 ratio is used to divide the examples for the ﬁve problems into training, validation, and test instances.
For further details see Appendix B
. 5. M ODELS
We evaluate the effectiveness of our approach against two contrasting state-of-the-art approaches for learning over programs: one sequential model and one other graph model.
(I) Sequential Model inst2vec (Ben-Nun et al., 2018) sequentially processes the IR statements of a program to perform whole-program classiﬁcation.
An IR is tokenized and then mapped into a sequence of pre-trained 200 dimensional embedding vectors which are processed by an LSTM.
The ﬁnal state of the LSTM is fed through a two-layer fully connected neural network to produce a classiﬁcation of the full sequence.
We extend this approach by concatenating to the input sequence a one-hot token-selector to indicate the starting point for analysis.
Then, we feed the LSTM state through a fully connected layer after every token, producing a prediction for each instruction of the IR.
We use the same model parameters as in the original work.
(II) Graph Models
We use the model design outlined in Section 4 with two input representations: CDFG (Brauckmann et al., 2020), and P RO G RA ML.
For both approaches we use 32 dimensional embeddings initialized randomly, as in Brauckmann et al.
Input vertex-selectors , encoded as binary one-hot vectors, are used to mark the starting point for analyses and are concatenated to the initial embeddings.
For CDFG, we use the vocabulary described in Brauckmann et al.
For P RO G RA ML, we derive the vocabulary from the training set.
Message Passing Neural Networks typically use a small number of propagation steps out of practical consideration for time and space efﬁciency (Gilmer et al., 2017; Brauckmann et al., 2020).
In contrast, data ﬂow analyses iterate until a ﬁxed point is reached.
In this work we iterate for a ﬁxed number T of message passing steps and exclude from the training and validation sets graphs for which a traditional implementation of the analysis task requires greater than T iterations to solve.
[Table] We set T = 30 for training in all experiments and trained a model per task.
Once trained, we evaluate model inference using different T values to accommodate programs which required a greater number of steps to compute the ground truth.
See Appendix C.2 for training details.
5. E VALUATION
First, we evaluate the effectiveness of each vocabulary at representing unseen programs.
Then we evaluate model performance on increasingly large subsets of the D EEP D ATA F LOW (DDF) test sets.
Vocabulary Coverage Each of the three approaches uses a vocabulary to produce embeddings that describe the instructions and operands of a program.
inst2vec uses a vocabulary of 8,565 LLVM-IR statements (where a statement is an instruction and its operands) with identiﬁers and literals stripped.
CDFG uses the 75 LLVM instruction opcodes.
For P RO G RA ML we derive a vocabulary on a set of training graphs that represents instructions and data types separately.
Table 1 compares the coverage of these vocabularies as a percentage of the vertices in the test graphs that can be described using that vocabulary.
P RO G RA ML provides 2 .
1 × the coverage on unseen programs as state-of-the-art approaches, the best of which can represent fewer than half of the graph vertices of unseen programs.
DDF-30: Testing on Limited Problem Size We initially limit our testing to the subset of each task’s test set which can be solved using a traditional analysis implementation in ≤ 30 steps, denoted
DDF-30.
This matches the T = 30 message passing iterations used in computing the graph models’ ﬁnal states to ensure that a learned model, if it has successfully approximated the mechanism of an analysis, has sufﬁcient message passing iterations to solve each test input.
Table 2 summarizes the performance of inst2vec, CDFG, and P RO G RA ML.
The relational representation of our approach shows excellent performance across all of the tasks.
CDFG, which also captures control-ﬂow, achieves comparable performance on the R EACHABILITY and D OMINANCE tasks.
However, the lack of operand vertices, positional edges, and data types renders poor performance on the S UBEXPRESSIONS task.
Neither the CDFG or inst2vec representations enable per-variable classiﬁcation, so are incapable of the D ATA D EP and L IVENESS tasks.
To simplify comparison, we exclude these two tasks from inst2vec and CDFG aggregate scores.
In spite of this, P RO G RA ML correctly labels 4 .
50 × and 1 .
12 × more nodes than the state-of-the-art approaches.
The weakest P RO G RA ML performance is on the L IVENESS task.
When model performance is considered as a function of the number of training graphs, shown in Figure 5, we see that the performance of P RO G RA ML quickly converges towards near-perfect F 1 scores on a holdout validation set for all tasks except L IVENESS , where the model is still improving at the end of training.
This suggests estimating the transfer (message) and meet (update) operators of this backwards analysis poses a greater challenge for the network, and may beneﬁt from further training.
DDF-60: Generalizing to Larger Problems The DDF-30 set excludes 28.7% of D EEPD ATA F LOW graphs which require more than 30 steps to compute ground truth labels.
To test whether these learned models can generalize to solve larger problems, we used the models we trained at T = 30 but double the number of inference message passing steps to T = 60 and repeated the tests on all graphs which require ≤ 60 analysis steps (excluding 19.6%).
The results of this experiment, denoted DDF-60, are shown in Table 2.
We observe that performance is consistent on this larger problem set, demonstrating that P RO G RA ML models can generalize to problems larger than those they were trained on.
The results indicate that an approximate ﬁxed-point algorithm is learned by the model, a critical feature for enabling practical machine learning over programs.
[Table]
DDF: Scalability Challenges Finally, we test the analysis models that were trained for T =message passing iterations on all D EEP D ATA F LOW graphs, shown in Table 2 as DDF.
We use T = 200 inference message passing iterations to test the limits of stability and generalization of current graph neural networks.
9.6% of DDF graphs require more than 200 steps to compute.
Therefore, this experiment highlights two of the challenges in the formulation of data ﬂow analysis in an MPNN framework: ﬁrst, that using a ﬁxed number of message passing iterations across each and every edge leads to unnecessary work for problems that can be solved in fewer iterations or by propagating only along a dynamic subset of the edges at each timestep (the maximum number of steps required by a graph in DDF is 28,727).
Secondly, models that compute correct results for a graph when processed for an appropriate number of steps may prove unstable when processed for an excessively large number of steps.
In Table 2 we see substantial degradations of model performance in line with these two challenges.
D OMINANCE and L IVENESS show reductions in precision as the models over-approximate and have a large number of false positives.
R EACHABILITY and D ATA D EP , in contrast, show drops in recall as the ﬁxed T = 200 iterations is insufﬁcient to propagate the signal to the edges of large problems.
MPNNs do not scale in the way that we should like for large programs.
A part of this, we believe, is that using a generic MPNN system is wasteful.
Ordinary data ﬂow engines process nodes in a particular order (usually reverse post order) and are naturally able to identify that a ﬁxed point has been reached.
We believe that dynamically-sparse message passing strategies and an adaptive number of iterations could address these scalability challenges.
In the previous section we focus on data ﬂow analysis as a benchmark for the capabilities of machine learning for compiler analysis.
For the analyses considered, non-ML techniques achieve perfect scores.
In this section we apply P RO G RA ML to two downstream data ﬂow tasks for which non-ML techniques fail: predicting heterogeneous compute device mappings and algorithm classiﬁcation.
In both domains P RO G RA ML outperforms prior graph-based and sequence-based representations, reducing test error by 1 . 20 × and 1 . 35 × , respectively.
Finally, we ablate every component of our representation and summarize the contribution of each [Table] [Table] [Table]
[Table]
We apply our methodology to the challenging domain of heterogeneous compute device mapping. Given an OpenCL kernel and a choice of two devices to run it on (CPU or GPU) [Table]
the task is to predict the device which will provide the best performance.
This problem has received signiﬁcant prior attention
[Table]
with previous approaches using both hand-engineered features (Grewe et al., 2013) and sequential models (Cummins et al., 2017a; Ben-Nun et al., 2018).
We use the O PEN CL D EVMAP dataset (Cummins et al., 2017a)
[Table]
which provides 680 labeled CPU/GPU instances derived from
OpenCL kernels sourced from seven benchmark suites on two combinations of CPU/GPU hardware,
AMD and NVIDIA.
cf.
Appendix D.1 for details.
The performance of P RO G RA ML and baseline models is shown in Table 3.
As can be seen, P ROG RA ML outperforms prior works.
We set new state-of-the-art F 1 scores of 0.88 and 0.80.
6. A LGORITHM C LASSIFICATION
We apply our approach to the task of classifying algorithms from unlabeled implementations.
We use the Mou et al.
(2016) dataset. It contains implementations of 104 different algorithms that were submitted to a judge system. All samples were written by students in higher education. There are around 500 samples per algorithm.
We compile them with different combinations of optimization ﬂags to generate a dataset of overall 240k samples, as in Ben-Nun et al.
Approximately 10,000 ﬁles are held out each as development and test sets.
cf.
Appendix D.2 for details .
Table 4a compares the test error of our method against prior works.
Ablation Studies We ablate the P RO G RA ML representation in Table 4b.
Every component of our representation contributes positively to performance.
We note that structure alone ( No vocab ) is sufﬁcient to outperform prior work, suggesting that algorithm classiﬁcation is a problem that lends itself especially well to judging the power of the representation structure, since most algorithms are well-deﬁned independent of implementation details, such as data types.
However, the choice of vocabulary is important.
Replacing the P RO G RA ML vocabulary with that of a prior approach ( inst2vec vocab ) degrades performance.
The greatest contribution to the performance of P RO G RA ML on this task is data ﬂow edges.
Backward edges, which are required for reasoning about backward data ﬂow analyses, provide the second greatest contribution.
These results highlight the importance of data ﬂow analysis for improving program reasoning through machine learning.
C ONCLUSIONS
The evolution of ML for compilers requires more expressive representations.
We show that current techniques cannot reason about simple data ﬂows which are at the core of all compilers.
We present
P RO G RA ML, a graph-based representation for programs derived from compiler IRs that accurately captures the semantics of a program’s statements and the relations between them.
We are releasing the D EEP D ATA F LOW dataset as a community benchmark for evaluating approaches to learning over programs.
P RO G RA ML and D EEP D ATA F LOW open up new directions for research towards more ﬂexible and useful program analysis.
P RO G RA ML outperforms the state-of-the-art, but is limited by scalability issues imposed by MPNNs.
As future work, we will investigate how MPNNs could be improved to learn efﬁcient and stable ﬁxed-point algorithms, regardless of input graph size.
A D ATA F LOW D EFINITIONS
This section provides the deﬁnitions of the ﬁve analysis tasks used in this paper to evaluate the representational power of deep learning over programs.
We chose a diverse set of analysis tasks to capture a mixture of both forward and backward analyses, and control-, data-, and proceduresensitive analyses.
(I) R EACHABILITY : Reachable Instructions Control reachability is a fundamental compiler analysis which determines the set of points in a program that can be reached from a particular starting point.
Given succ ( n ) , which returns the control successors of an instruction n , the set of reachable instructions starting at root n can be found using forward analysis:
[Equation]
(II) D OMINANCE :
Instruction Dominance Instruction n dominates statement m if every control-ﬂow path the from the program entry n 0 to m passes through n .
Like reachability, this analysis only requires propagation of control-ﬂow, but unlike reachability, the set of dominator instructions are typically constructed through analysis of a program’s reverse control-ﬂow graph (Lengauer & Tarjan, 1979; Blazy et al., 2015):
[Equation]
Where pred ( n ) returns the control predecessors of instruction n .
We formulate the D OMINANCE problem as: Given a root instruction vertex n , label all vertices m where n ∈ Dom ( m ) .
III) D ATA D EP
: Data Dependencies The data dependencies of a variable v is the set of predecessor instructions that must be evaluated to produce v .
Computing data dependencies requires traversing the reverse data-ﬂow graph:
[Equation]
Where defs ( n ) returns the instructions that produce the operands of n .
(IV) L IVENESS Live-out variables A variable v is live-out of statement n if there exists some path from n to a statement that uses v , without redeﬁning it.
Given uses ( n ) , which returns the operand variables of n , and defs ( n ) , which returns deﬁned variables, the live-out variables can be computed forwards using:
[Equation]
V)
Global
Common Subexpressions
The identiﬁcation of common subexpressions is an important analysis for optimization.
For compiler IRs we deﬁne a subexpression as an instruction and its operands, ordered by either their position (for non-commutative operations), or lexicographically (for commutative operations).
We thus formulate the common subexpression problem as: Given an instruction (which forms part of a subexpression), label any other instructions in the program which compute the same subexpression.
This is an inter-procedural analysis, though operands must obey their scope.
Common subexpressions are typically identiﬁed using available expression analysis:
[Equation]
Where uses ( n ) return the expressions used by instruction n , and defs ( n ) returns the expressions deﬁned by n .
[Table]
[Table]
B D EEP D ATA F LOW D ATASET
The D EEP D ATA F LOW dataset comprises: 461k LLVM-IR ﬁles assembled from a range of sources,
P RO G RA ML representations of each of the IRs, and 15.4M sets of labeled graphs for the ﬁve data ﬂow analyses described in the previous section, totaling 8.5B classiﬁcation labels.
Programs We assembled a 256M-line corpus of real-world LLVM-IRs from a variety of sources, summarized in Table 5.
We selected popular open source software projects that cover a diverse range of domains and disciplines, augmented by uncategorized code mined from popular GitHub projects using the methodology described by Cummins et al.
(2017b).
Our corpus comprises ﬁve source languages (C, C++, Fortran, OpenCL, and Swift) covering a range of domains from functional to imperative, high-level to accelerators.
The software covers a broad range of disciplines from compilers and operating systems to traditional benchmarks, machine learning systems, and unclassiﬁed code downloaded from popular open source repositories.
P RO G RA ML Graphs We implemented P RO G RA ML construction as an llvm::ModulePass using LLVM version 10.0.0 and generated a graph representation of each of the LLVM-IRs.
P ROG RA ML construction takes an average of 10.72ms per ﬁle.
Our corpus of unlabeled graphs totals 268M vertices and 485M edges, with an average of 581 vertices and 1,051 edges per graph.
The maximum edge position is 355 (a large switch statement found in a TensorFlow compute kernel).
Data Flow Labels We produced labeled graph instances from the unlabeled corpus by computing ground truth labels for each of the analysis tasks described in Section A using a traditional analysis implementation.
For each of the ﬁve tasks, and for every unlabeled graph in the corpus, we produce n labeled graphs by selecting unique source vertices v 0 ∈ V , where n is proportional to the size of the graph: n = min (cid:18)(cid:24) | V | (cid:25) ,(cid:19)
Each example in the dataset consists of an input graph in which the source vertex is indicated using the vertex selector , and an output graph with the ground truth labels used for training or for evaluating the accuracy of model predictions.
For every example we produce, we also record the number of steps that the iterative analysis required to compute the labels.
We use this value to produce subsets of the dataset to test problems of different sizes, shown in Table 6.
[Figure]
We divided the datasets randomly using a 3:1:1 ratio for training, validation, and test instances.
The same random allocation of instances was used for each of the ﬁve tasks.
Where multiple examples were derived from a single IR, examples derived from the same IR were allocated to the same split.
As binary classiﬁcation tasks, data ﬂow analyses display strong class imbalances as only a small fraction of a program graph is typically relevant to computing the result set of an analysis.
On the
DDF test sets, an accuracy of 86.92% can be achieved by always predicting the negative class.
For this reason we report only binary precision, recall, and F1 scores with respect to the positive class when reporting model performance on D EEP D ATA F LOW tasks.
C D ATA F LOW E XPERIMENTS : S UPPLEMENTARY D ETAILS
This section provides additional details for the experiments presented in Section 5.
C.
[Figure] (I) Sequential Model
The inst2vec model consists of 619,650 trainable parameters in the conﬁguration outlined in Figure 6.
The model, implemented in TensorFlow, uses the same parameters as in the original work: 2 × 64 dimensional LSTM layers followed by a 64 dimensional dense layer
and the ﬁnal 2 dimensional output layer.
Sequences are padded and truncated to 5k tokens and processed in batches of 64.
(II) Graph Models For CDFG and P RO G RA ML approaches we use the same model architecture.
The model, implemented in PyTorch, consists of a customized GGNN with 87,070 trainable parameters.
Batches are implemented as disconnected graphs that are constructed to enable efﬁcient processing of graphs of differing size in parallel without padding.
We use a combined batch size of 10 , 000 vertices.
If a single graph contains more than 10 , 000 vertices, it is processed on its own.
[Table]
C. E XPERIMENTAL S ETUP
Training Details and Parameters All models were trained in an end-to-end fashion with the
Adam optimizer (Kingma & Ba, 2015) using the default conﬁguration and a learning rate of 1 · 10 −for the LSTMs and 2 . 5 · 10 − 4 for the GGNNs.
We trained the models on 1M training graphs, evaluating on a ﬁxed 10k validation set at 10k intervals for the ﬁrst 50k training graphs, and at 100k intervals thereafter.
The checkpoint with the greatest validation F 1 score is used for testing.
Runtimes All experiments were conducted on shared machines equipped with an NVIDIA GTX 1080 GPU, 32GB of RAM, mechanical hard drives, and server-grade Intel Xeon processors.
Figureprovides measurements of the average runtimes of each approach across the ﬁve DDF-30 tasks.
In our implementation, we ﬁnd training and testing to be I/O bound as programs are processed faster than loading many small ﬁles from disk.
In particular, CDFG performance suffers relative to P RO G RA ML as the conversion from P RO G RA ML to CDFG representations is performed ondemand.
For validation, inputs are loaded once into system memory and re-used, so the measured time provides a more accurate estimate of processing requirements.
D D OWNSTREAM T ASKS : S UPPLEMENTARY D ETAILS
This section provides additional details for the experiments present in Section 6.
D. H ETEROGENEOUS C OMPUTE D EVICE M APPING
Datasets The O PEN
CL D EV M AP dataset comprises 256 OpenCL kernels from two combinations of CPU/GPU pairs.
The AMD set uses an Intel Core i7-3820 CPU and AMD Tahiti 7970 GPU; the
NVIDIA set uses an Intel Core i7-3820 CPU and an NVIDIA GTX 970 GPU.
Each dataset consists of 680 labeled examples derived from the 256 unique kernels by varying dynamic inputs.
Models We compare P RO G RA ML with four approaches:
First, with a static baseline that selects the most-frequently optimal device for each dataset (CPU for AMD , GPU for NVIDIA ).
Second, with DeepTune (Cummins et al., 2017a), which is a sequential LSTM model at the OpenCL source level.
Third, to isolate the impact of transitioning from OpenCL source to LLVM-IR, we evaluate against a new DeepTune IR model, which adapts DeepTune to using tokenized sequences of
LLVM-IR as input instead of OpenCL tokens.
Finally, we compare against the state-of-the-art approach inst2vec (Ben-Nun et al., 2018), which replaces the OpenCL tokenizer with a sequence of 200-dimensional embeddings, pre-trained on a large corpus of LLVM-IR using a skip-gram model.
P RO G RA ML itself uses the GGNN adaptation as described in the paper.
We adapted the readout head to produce a single classiﬁcation label for each graph, rather than per-vertex classiﬁcations, by aggregating over the ﬁnal iterated vertex states.
We also included the available auxiliary input features of the D EV M AP dataset.
The auxiliary features are concatenated to the features extracted by the GGNN before classiﬁcation following the methodology of Cummins et al.
(2017a).
The experimental results in this section come from an earlier development iteration of P RO G RA ML which deviates from the method described in the main paper in the way in which it produces initial vertex embeddings.
Instead of deriving a textual representation of instructions and data types to produce a vocabulary, the vocabulary used for the D EV M AP experiment is that of inst2vec (Ben-Nun et al., 2018), where variables and constants are all represented by a single additional embedding vector.
The poor vocabulary coverage achieved by using inst2vec motivated us to provide the improved vocabulary implementation that we describe in the main paper (see Table 1).
Training Details and Parameters
All neural networks are regularized with dropout (Hinton et al., 2012) for generalization and Batch Normalization (Ioffe & Szegedy, 2015) in order to be uniformly applicable to vastly different scales of auxiliary input features.
We used 10 -fold cross-validation with rotating 80/10/10 splits by training on 80% of the data and selecting the model with the highest validation accuracy, setting aside 1 / 10 th of the training data to use for validation.
We trained each model for 300 epochs and selected the epoch with the greatest validation accuracy for testing.
Baseline models were trained with hyperparameters from the original works.
For the P RO G RA ML results we used 6 layers in the GGNN corresponding to 6 timesteps of message propagation, while sharing parameters between even and odd layers to introduce additional regularization of the weights.
We ran a sweep of basic hyperparameters which led us to use the pre-trained inst2vec statement embeddings (Ben-Nun et al., 2018) and to exclude the use of position representations.
Both of these hyperparameter choices help generalization by reducing the complexity of the model.
This is not surprising in light of the fact that the dataset only contains 680 samples derived from 256 unique programs.
P RO G RA ML was trained with the Adam optimizer with default parameters, a learning rate of 10 − 3 and a batch size of 18,000 nodes (resulting in ca.
12000 iteration steps of the optimizer).
For the P RO G RA ML result, we repeat the automated sweep for all hyperparameter conﬁgurations and picked the conﬁguration with the best average validation performance.
Performance on the unseen tenth of the data is reported.
D. A LGORITHM C LASSIFICATION
Dataset We use the POJ-104 dataset (Mou et al., 2016).
It contains implementations of 104 different algorithms that were submitted to a judge system.
All programs were written by students in higher education.
The dataset has around 500 samples per algorithm.
We compile them with different combinations of optimization ﬂags to generate a dataset of overall 240k samples, as in Ben-Nun et al.
Approximately 10,000 ﬁles are held out each as a development and test set.
Models We compare with tree-based convolutional neural networks (TBCNN) (Mou et al., 2016) and inst2vec (Ben-Nun et al., 2018).
We used author-provided parameters for the baseline models.
For P RO G RA ML we used 4 layers in the GGNN corresponding to 8 timesteps. To further test the expressive power of the graph-based representation against the tree-based (TBCNN) and sequential (inst2vec) prior work, we additionally compare against graph-based baselines based on XFG (BenNun et al., 2018).
To better understand the qualitative aspects of replacing a graph-based representation that captures program semantics like Contextual Flow Graphs (XFG) (Ben-Nun et al., 2018) with the more complete P RO G RA ML representation, we adapted a GGNN (Li et al., 2015) to directly predict algorithm classes from XFG representations of the programs.
In contrast to this, Ben-Nun et al.
(2018) used XFG only to generate statement contexts for use in skip-gram pre-training.
Here, we lift this graphical representation and make it accessible to a deep neural network directly, as opposed to the structure-less sequential approach in the original work (inst2vec).
Training Details and Parameters All models were trained with the AdamW (Loshchilov & Hutter, 2019) optimizer with learning rate 2 .
5 · 10 − 4 , β 1 = 0 .
9 , β 2 = 0 .
999 , ε = 10 − 8 for 80 epochs.
Dropout regularization is employed on the graph states with a rate of 0 . 2 .