Original: L EARNING TO I MPROVE C ODE E FFICIENCY
Revised: L EARNING TO I MPROVE C ODE E FFICIENCY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Improvements in the performance of computing systems, driven by Moore’s Law, have transformed society.
Revised: Improvements in the performance of computing systems, driven by Moore’s Law, have transformed society.

Original: As such hardware-driven gains slow down, it becomes even more important for software developers to focus on performance and efﬁciency during development.
Revised: As such hardware-driven gains slow down, it becomes even more important for software developers to focus on performance and efﬁciency during development.

Original: While several studies have demonstrated the potential from such improved code efﬁciency (e.g., 2x better generational improvements compared to hardware), unlocking these gains in practice has been challenging.
Revised: While several studies have demonstrated the potential from such improved code efﬁciency (e.g., 2x better generational improvements compared to hardware), unlocking these gains in practice has been challenging.

Original: Reasoning about algorithmic complexity and the interaction of coding patterns on hardware can be challenging for the average programmer, especially when combined with pragmatic constraints around development velocity and multi-person development.
Revised: Reasoning about algorithmic complexity and the interaction of coding patterns on hardware can be challenging for the average programmer, especially when combined with pragmatic constraints around development velocity and multi-person development.

Original: This paper seeks to address this problem.
Revised: This paper seeks to address this problem.

Original: We analyze a large competitive programming dataset from the Google Code Jam competition  and ﬁnd that efﬁcient code is indeed rare, with a 2x runtime difference between the median and the 90th percentile of solutions.
Revised: We analyze a large competitive programming dataset from the Google Code Jam competition (Google Code-Jam) and ﬁnd that efﬁcient code is indeed rare, with a 2x runtime difference between the median and the 90th percentile of solutions.

Original: We propose using machine learning to automatically provide prescriptive feedback in the form of hints, to guide programmers towards writing high-performance code.
Revised: We propose using machine learning to automatically provide prescriptive feedback in the form of hints, to guide programmers towards writing high-performance code.

Original: To automatically learn these hints from the dataset, we propose a novel discrete variational auto-encoder, where each discrete latent variable represents a different learned category of code-edit that increases performance.
Revised: To automatically learn these hints from the dataset, we propose a novel discrete variational auto-encoder, where each discrete latent variable represents a different learned category of code-edit that increases performance.

Original: We show that this method represents the multi-modal space of code efﬁciency edits better than a sequence-to-sequence baseline and generates a distribution of more efﬁcient solutions.
Revised: We show that this method represents the multi-modal space of code efﬁciency edits better than a sequence-to-sequence baseline and generates a distribution of more efﬁcient solutions.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The computational efﬁciency of code is often front-and-center in any computer science curriculum.
Revised: The computational efﬁciency of code is often front-and-center in any computer science curriculum.

Original: While there are many ways to solve a particular problem, there is often wide variance in the runtime of different implementations.
Revised: While there are many ways to solve a particular problem, there is often wide variance in the runtime of different implementations.

Original: This variance is often attributed to many different factors: the algorithmic complexity of the code in question, the data structures that are used, the libraries that are called, and lower-level execution effects like efﬁcient caching or memory usage.
Revised: This variance is often attributed to many different factors: the algorithmic complexity of the code in question, the data structures that are used, the libraries that are called, and lower-level execution effects like efﬁcient caching or memory usage.

Original: Similarly, computational efﬁciency is a critical component of professional software development.
Revised: Similarly, computational efﬁciency is a critical component of professional software development.

Original: The computing industry as a whole has relied on the automatic performance increases of Moore’s
Revised: The computing industry as a whole has relied on the automatic performance increases of Moore’s

Original: Law to scale massive warehouse computing systems to meet the internet requirements of the world.
Revised: Law to scale massive warehouse computing systems to meet the internet requirements of the world.

Original: As these automatic performance increases slow down, the burden of reducing computational cost and carbon footprint now falls on writing high-performance code (Patterson et al.
Revised: As these automatic performance increases slow down, the burden of reducing computational cost and carbon footprint now falls on writing high-performance code (Patterson et al.

Original: Writing efﬁcient code is challenging, even for experienced programmers, as it requires understanding computational complexity as well as the underlying hardware.
Revised: Writing efﬁcient code is challenging, even for experienced programmers, as it requires understanding computational complexity as well as the underlying hardware.

Original: Lower-level performance optimizations are therefore automated by compilers which automatically apply a small set of known, sound lowlevel program transformations to an already written program to increase its efﬁciency.
Revised: Lower-level performance optimizations are therefore automated by compilers which automatically apply a small set of known, sound lowlevel program transformations to an already written program to increase its efﬁciency.

Original: However, compilers and current tooling have more difﬁculty identifying higher-level optimizations, such as more efﬁcient algorithms for the same problem.
Revised: However, compilers and current tooling have more difﬁculty identifying higher-level optimizations, such as more efﬁcient algorithms for the same problem.

Original: So far, these types of optimizations could only be identiﬁed by humans.
Revised: So far, these types of optimizations could only be identiﬁed by humans.

Original: We hypothesize that machine learning can be used to guide humans towards such optimizations, by suggesting edits that optimize code efﬁciency.
Revised: We hypothesize that machine learning can be used to guide humans towards such optimizations, by suggesting edits that optimize code efﬁciency.

Original: To study this problem, we examine a competitive programming dataset where tens of thousands of developers have submitted answers to about 180 different questions.
Revised: To study this problem, we examine a competitive programming dataset where tens of thousands of developers have submitted answers to about 180 different questions.

Original: Studying these solutions, we ﬁnd wide variance in computational cost: the runtime difference between a median solution and the 90th percentile is over two-fold.
Revised: Studying these solutions, we ﬁnd wide variance in computational cost: the runtime difference between a median solution and the 90th percentile is over two-fold.

Original: The scarcity of high-performance solutions highlights the difﬁculty of our task.
Revised: The scarcity of high-performance solutions highlights the difﬁculty of our task.

Original: Therefore, we aim to provide prescriptive feedback to developers to guide them towards writing high-performance code.
Revised: Therefore, we aim to provide prescriptive feedback to developers to guide them towards writing high-performance code.

Original: We develop a framework to apply multiple categorical transformations to a single program using a novel discrete variational autoencoder where different vectors in the latent dictionary lead to different code transformations.
Revised: We develop a framework to apply multiple categorical transformations to a single program using a novel discrete variational autoencoder where different vectors in the latent dictionary lead to different code transformations.

Original: We ﬁnd that these learned categories are often consistent (e.g., a particular latent variable may control the data structure that is used for a particular problem or a for-loop vs. a while-loop), and that by applying these transformations to the program, we can move solutions into a more efﬁcient computational efﬁciency category vs. the code that the developer wrote.
Revised: We ﬁnd that these learned categories are often consistent (e.g., a particular latent variable may control the data structure that is used for a particular problem or a for-loop vs. a while-loop), and that by applying these transformations to the program, we can move solutions into a more efﬁcient computational efﬁciency category vs. the code that the developer wrote.

Original: This paper makes the following contributions:
Revised: This paper makes the following contributions:

Original: • We frame code efﬁciency optimization as a generative problem.
Revised: • We frame code efﬁciency optimization as a generative problem.

Original: • Using the Google Code Jam competitive programming dataset (Code-Jam), we analyze the distribution and characteristics of high-performance solutions.
Revised: • Using the Google Code Jam competitive programming dataset (Google Code-Jam), we analyze the distribution and characteristics of high-performance solutions.

Original: We ﬁnd that highperformance solutions are uncommon and consist of a combination of many distinct optimizations.
Revised: We ﬁnd that high-performance solutions are uncommon and consist of a combination of many distinct optimizations.

Original: We then derive a canonicalized program-edits dataset to train models to improve code efﬁciency.
Revised: We then derive a canonicalized program-edits dataset to train models to improve code efﬁciency.

Original: • We propose a novel discrete generative latent-variable model of program edits to model a distribution of fast programs, conditioned on their slower counterparts.
Revised: • We propose a novel discrete generative latent-variable model of program edits to model a distribution of fast programs, conditioned on their slower counterparts.

Original: We ﬁnd that this model outperforms a sequence-to-sequence baseline along three different axes: correctness, efﬁciency, and diversity.
Revised: We ﬁnd that this model outperforms a sequence-to-sequence baseline along three different axes: correctness, efﬁciency, and diversity.

Original: • We qualitatively demonstrate that the learned discrete latent variables represent different edits, and that the edits that are assigned to one latent variable are generally consistent.
Revised: • We qualitatively demonstrate that the learned discrete latent variables represent different edits, and that the edits that are assigned to one latent variable are generally consistent.

Original: As a side-effect, we learn an interpretable program embedding space.
Revised: As a side-effect, we learn an interpretable program embedding space.

Original: We believe that these results are a promising step towards automating the process of identifying and applying higher-level performance optimizations, which would fundamentally increase the capabilities of current developer tools while reducing the carbon footprint of computing.
Revised: We believe that these results are a promising step towards automating the process of identifying and applying higher-level performance optimizations, which would fundamentally increase the capabilities of current developer tools while reducing the carbon footprint of computing.

Original:  B ACKGROUND 2. P ROBLEM F ORMULATION
Revised:  B ACKGROUND 2. P ROBLEM F ORMULATION

Original: There are many different ways to implement a particular algorithm.
Revised: There are many different ways to implement a particular algorithm.

Original: For algorithms like matrix multiplication, small syntactic changes like loop reordering have a dramatic impact on execution cost (Leiserson et al.
Revised: For algorithms like matrix multiplication, small syntactic changes like loop reordering have a dramatic impact on execution cost (Leiserson et al.

Original: From low-level hardware effects like caching and branch prediction, to higher-level code choices like data structures, termination conditions, and loops - navigating the space of implementation options is a key element of software engineering.
Revised: From low-level hardware effects like caching and branch prediction, to higher-level code choices like data structures, termination conditions, and loops - navigating the space of implementation options is a key element of software engineering.

Original: We ﬁnd many of these performance archetypes when looking at competitive programming solutions.
Revised: We ﬁnd many of these performance archetypes when looking at competitive programming solutions.

Original: Figure 1 shows three example programs on the left and faster versions of those programs on the right.
Revised: Figure 1 shows three example programs on the left and faster versions of those programs on the right.

Original: The ﬁrst example showcases using a more efﬁcient datastructure (a heap), which then enables early termination of the main loop.
Revised: The ﬁrst example showcases using a more efﬁcient datastructure (a heap), which then enables early termination of the main loop.

Original: The second example highlights a performance bug, where fewer
Revised: The second example highlights a performance bug, where fewer

Original: API calls can accomplish the same task.
Revised: API calls can accomplish the same task.

Original: The third example highlights how using built-in libraries can be faster than writing bespoke implementations.
Revised: The third example highlights how using built-in libraries can be faster than writing bespoke implementations.

Original: These examples represent just three of the many discrete design choices that developers make while coding their solutions.
Revised: These examples represent just three of the many discrete design choices that developers make while coding their solutions.

Original: We hypothesize that these discrete choices can be learned, such that a generative model can suggest different code transformations that a developer could leverage to increase code efﬁciency.
Revised: We hypothesize that these discrete choices can be learned, such that a generative model can suggest different code transformations that a developer could leverage to increase code efﬁciency.

Original: 2. P ROGRAM E DIT D ATASET
Revised: 2. P ROGRAM E DIT D ATASET

Original: To study this problem, we use the dataset from the Google Code Jam international competitive programming competiton.
Revised: To study this problem, we use the dataset from the Google Code Jam international competitive programming competition (Google Code-Jam).

Original: Each question consists of a problem description, along with up to three test cases – inputs and desired outputs – of increasing complexity.
Revised: Each question consists of a problem description, along with up to three test cases – inputs and desired outputs – of increasing complexity.

Original: For each question, the dataset contains solutions from competition participants.
Revised: For each question, the dataset contains solutions from competition participants.

Original: If the submission passes each test, it is labeled as correct and annotated with run-time, otherwise it is marked as incorrect.
Revised: If the submission passes each test, it is labeled as correct and annotated with run-time, otherwise it is marked as incorrect.

Original: For our study, we focus on the solutions that are written in Python.
Revised: For our study, we focus on the solutions that are written in Python.

Original: As we aim to study execution complexity, we focus only on correct submissions and consider the run-time of the largest test case.
Revised: As we aim to study execution complexity, we focus only on correct submissions and consider the run-time of the largest test case.

Original: This distribution of run-times is shown in Figure 2.
Revised: This distribution of run-times is shown in Figure 2.

Original: Even for this constrained competitive programming task (with a natural focus on efﬁcient solutions),
Revised: Even for this constrained competitive programming task (with a natural focus on efﬁcient solutions),

Original: Figure 2(a) illustrates a wide distribution in run-time – supporting our hypothesis that writing efﬁcient
Revised: Figure 2(a) illustrates a wide distribution in run-time – supporting our hypothesis that writing efﬁcient

Original: (cid:3)(cid:3)(cid:87)(cid:3)(cid:32)(cid:3)(cid:76)(cid:81)(cid:87)(cid:11)(cid:76)(cid:81)(cid:83)(cid:88)(cid:87)(cid:11)(cid:12)(cid:12)(cid:3)(cid:3)(cid:73)(cid:82)(cid:85)(cid:3)(cid:76)(cid:3)(cid:76)(cid:81)(cid:3)(cid:85)(cid:68)(cid:81)(cid:74)(cid:72)(cid:11)(cid:20)(cid:15)(cid:3)(cid:87)(cid:14)(cid:20)(cid:12)(cid:29)(cid:3)(cid:3)(cid:3)(cid:3)(cid:81)(cid:15)(cid:3)(cid:69)(cid:3)(cid:32)(cid:3)(cid:62)(cid:76)(cid:81)(cid:87)(cid:11)(cid:86)(cid:12)(cid:3)(cid:73)(cid:82)(cid:85)(cid:3)(cid:86)(cid:3)(cid:76)(cid:81)(cid:3)(cid:76)(cid:81)(cid:83)(cid:88)(cid:87)(cid:11)(cid:12)(cid:17)(cid:86)(cid:83)(cid:79)(cid:76)(cid:87)(cid:11)(cid:5)(cid:3)(cid:5)(cid:12)(cid:64)(cid:3)(cid:3)(cid:3)(cid:3)(cid:75)(cid:82)(cid:88)(cid:86)(cid:72)(cid:86)(cid:3)(cid:32)(cid:3)(cid:62)(cid:76)(cid:81)(cid:87)(cid:11)(cid:86)(cid:12)(cid:3)(cid:73)(cid:82)(cid:85)(cid:3)(cid:86)(cid:3)(cid:76)(cid:81)(cid:3)(cid:76)(cid:81)(cid:83)(cid:88)(cid:87)(cid:11)(cid:12)(cid:17)(cid:86)(cid:83)(cid:79)(cid:76)(cid:87)(cid:11)(cid:5)(cid:3)(cid:5)(cid:12)(cid:64)(cid:3)(cid:3)(cid:3)(cid:3)(cid:75)(cid:82)(cid:88)(cid:86)(cid:72)(cid:86)(cid:17)(cid:86)(cid:82)(cid:85)(cid:87)(cid:11)(cid:12)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:85)(cid:72)(cid:86)(cid:88)(cid:79)(cid:87)(cid:3)(cid:32)(cid:3)(cid:19)(cid:3)(cid:3)(cid:3)(cid:3)(cid:73)(cid:82)(cid:85)(cid:3)(cid:75)(cid:3)(cid:76)(cid:81)(cid:3)(cid:75)(cid:82)(cid:88)(cid:86)(cid:72)(cid:86)(cid:29)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:76)(cid:73)(cid:3)(cid:69)(cid:3)(cid:33)(cid:32)(cid:3)(cid:75)(cid:29)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:85)(cid:72)(cid:86)(cid:88)(cid:79)(cid:87)(cid:3)(cid:14)(cid:32)(cid:3)(cid:20)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:69)(cid:3)(cid:16)(cid:32)(cid:3)(cid:75)(cid:3)(cid:3)(cid:3)(cid:3)(cid:83)(cid:85)(cid:76)(cid:81)(cid:87)(cid:11)(cid:5)(cid:38)(cid:68)(cid:86)(cid:72)(cid:3)(cid:6)(cid:94)(cid:96)(cid:29)(cid:3)(cid:94)(cid:96)(cid:5)(cid:17)(cid:73)(cid:82)(cid:85)(cid:80)(cid:68)(cid:87)(cid:11)(cid:76)(cid:15)(cid:3)(cid:85)(cid:72)(cid:86)(cid:88)(cid:79)(cid:87)(cid:12)(cid:12) (cid:76)(cid:80)(cid:83)(cid:82)(cid:85)(cid:87)(cid:3)(cid:75)(cid:72)(cid:68)(cid:83)(cid:84)
Revised: (cid:3)(cid:3)(cid:87)(cid:3)(cid:32)(cid:3)(cid:76)(cid:81)(cid:87)(cid:11)(cid:76)(cid:81)(cid:83)(cid:88)(cid:87)(cid:11)(cid:12)(cid:12)(cid:3)(cid:3)(cid:73)(cid:82)(cid:85)(cid:3)(cid:76)(cid:3)(cid:76)(cid:81)(cid:3)(cid:85)(cid:68)(cid:81)(cid:74)(cid:72)(cid:11)(cid:20)(cid:15)(cid:3)(cid:87)(cid:14)(cid:20)(cid:12)(cid:29)(cid:3)(cid:3)(cid:3)(cid:3)(cid:81)(cid:15)(cid:3)(cid:69)(cid:3)(cid:32)(cid:3)(cid:62)(cid:76)(cid:81)(cid:87)(cid:11)(cid:86)(cid:12)(cid:3)(cid:73)(cid:82)(cid:85)(cid:3)(cid:86)(cid:3)(cid:76)(cid:81)(cid:3)(cid:76)(cid:81)(cid:83)(cid:88)(cid:87)(cid:11)(cid:12)(cid:17)(cid:86)(cid:83)(cid:79)(cid:76)(cid:87)(cid:11)(cid:5)(cid:3)(cid:5)(cid:12)(cid:64)(cid:3)(cid:3)(cid:3)(cid:3)(cid:75)(cid:82)(cid:88)(cid:86)(cid:72)(cid:86)(cid:3)(cid:32)(cid:3)(cid:62)(cid:76)(cid:81)(cid:87)(cid:11)(cid:86)(cid:12)(cid:3)(cid:73)(cid:82)(cid:85)(cid:3)(cid:86)(cid:3)(cid:76)(cid:81)(cid:3)(cid:76)(cid:81)(cid:83)(cid:88)(cid:87)(cid:11)(cid:12)(cid:17)(cid:86)(cid:83)(cid:79)(cid:76)(cid:87)(cid:11)(cid:5)(cid:3)(cid:5)(cid:12)(cid:64)(cid:3)(cid:3)(cid:3)(cid:3)(cid:75)(cid:82)(cid:88)(cid:86)(cid:72)(cid:86)(cid:17)(cid:86)(cid:82)(cid:85)(cid:87)(cid:11)(cid:12)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:85)(cid:72)(cid:86)(cid:88)(cid:79)(cid:87)(cid:3)(cid:32)(cid:3)(cid:19)(cid:3)(cid:3)(cid:3)(cid:3)(cid:73)(cid:82)(cid:85)(cid:3)(cid:75)(cid:3)(cid:76)(cid:81)(cid:3)(cid:75)(cid:82)(cid:88)(cid:86)(cid:72)(cid:86)(cid:29)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:76)(cid:73)(cid:3)(cid:69)(cid:3)(cid:33)(cid:32)(cid:3)(cid:75)(cid:29)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:85)(cid:72)(cid:86)(cid:88)(cid:79)(cid:87)(cid:3)(cid:14)(cid:32)(cid:3)(cid:20)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:69)(cid:3)(cid:16)(cid:32)(cid:3)(cid:75)(cid:3)(cid:3)(cid:3)(cid:3)(cid:83)(cid:85)(cid:76)(cid:81)(cid:87)(cid:11)(cid:5)(cid:38)(cid:68)(cid:86)(cid:72)(cid:3)(cid:6)(cid:94)(cid:96)(cid:29)(cid:3)(cid:94)(cid:96)(cid:5)(cid:17)(cid:73)(cid:82)(cid:85)(cid:80)(cid:68)(cid:87)(cid:11)(cid:76)(cid:15)(cid:3)(cid:85)(cid:72)(cid:86)(cid:88)(cid:79)(cid:87)(cid:12)(cid:12) (cid:76)(cid:80)(cid:83)(cid:82)(cid:85)(cid:87)(cid:3)(cid:75)(cid:72)(cid:68)(cid:83)(cid:84)

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: code is a difﬁcult problem.
Revised: code is a difﬁcult problem.

Original: We observe solutions with run-times that are fractions of a second, as well as those that time-out after 5 seconds.
Revised: We observe solutions with run-times that are fractions of a second, as well as those that time-out after 5 seconds.

Original: High-performance solutions are rare.
Revised: High-performance solutions are rare.

Original: Additionally, Figure 2(b) indicates that the data is multi-modal with a wide textual distribution of solutions.
Revised: Additionally, Figure 2(b) indicates that the data is multi-modal with a wide textual distribution of solutions.

Original: Solutions are as short as a handful of lines of code and stretch to over a thousand.
Revised: Solutions are as short as a handful of lines of code and stretch to over a thousand.

Original: We aim to discover the common patterns in these solutions that lead to efﬁcient code, so that we can provide hints to programmers to improve performance.
Revised: We aim to discover the common patterns in these solutions that lead to efﬁcient code, so that we can provide hints to programmers to improve performance.

Original: We frame this task as a sequence-to-sequence problem: given an input code sequence, output a more efﬁcient version of the code.
Revised: We frame this task as a sequence-to-sequence problem: given an input code sequence, output a more efﬁcient version of the code.

Original: While sequence-to-sequence modeling is often posed as a one-to-one problem, our task is a one-to-many problem, as there are an overwhelmingly large number of such solutions to a single problem.
Revised: While sequence-to-sequence modeling is often posed as a one-to-one problem, our task is a one-to-many problem, as there are an overwhelmingly large number of such solutions to a single problem.

Original: Trivially, one could simply rename variables or change syntax.
Revised: Trivially, one could simply rename variables or change syntax.

Original: We would like to focus on substantive changes, so we deal with this in two ways: ﬁrst, we canonicalize the code submissions by renaming variables, function names, and strings with generic tokens.
Revised: We would like to focus on substantive changes, so we deal with this in two ways: ﬁrst, we canonicalize the code submissions by renaming variables, function names, and strings with generic tokens.

Original: Next, we construct a dataset of inputs and outputs by pairing programs according to their similarity.
Revised: Next, we construct a dataset of inputs and outputs by pairing programs according to their similarity.

Original: Speciﬁcally, we use the ROUGE-8 metric to score the similarity between all pairs of canonicalized submissions.
Revised: Speciﬁcally, we use the ROUGE-8 metric to score the similarity between all pairs of canonicalized submissions.

Original: Similar programs with a signiﬁcant run-time difference (we choose 1.2x speedup or larger) are added to the dataset.
Revised: Similar programs with a signiﬁcant run-time difference (we choose 1.2x speedup or larger) are added to the dataset.

Original: Figure 2(c) shows the relative runtime difference between the slow/fast pairs in the dataset.
Revised: Figure 2(c) shows the relative runtime difference between the slow/fast pairs in the dataset.

Original: Consistent with 2(a), we observe a broad distribution of run-time differentials.
Revised: Consistent with 2(a), we observe a broad distribution of run-time differentials.

Original: We ﬁnd that this canonicalization approach enables the model to learn on this dataset, but also means that we lose the ability to execute the edited program to evaluate run-time.
Revised: We ﬁnd that this canonicalization approach enables the model to learn on this dataset, but also means that we lose the ability to execute the edited program to evaluate run-time.

Original: However, even in canonical form, edits are qualitatively sufﬁcient to identify optimizations.
Revised: However, even in canonical form, edits are qualitatively sufﬁcient to identify optimizations.

Original: Section 4.3 describes the quantitative metrics that we use to further evaluate the quality of these edits.
Revised: Section 4.3 describes the quantitative metrics that we use to further evaluate the quality of these edits.

Original:  M ETHOD 3. M OTIVATION
Revised:  M ETHOD 3. M OTIVATION

Original: Our goal is to create a model that can condition on a given program and help developers identify more computationally efﬁcient variants of that program.
Revised: Our goal is to create a model that can condition on a given program and help developers identify more computationally efﬁcient variants of that program.

Original: There are three factors that we use to drive our modeling choices and evaluation.
Revised: There are three factors that we use to drive our modeling choices and evaluation.

Original: The resultant hints should be syntactically coherent.
Revised: The resultant hints should be syntactically coherent.

Original: As the model provides suggestions, the suggested changes do not need to be perfect, but still need to make sense.
Revised: As the model provides suggestions, the suggested changes do not need to be perfect, but still need to make sense.

Original: The changes suggested by the model should lead to more computationally efﬁcient code.
Revised: The changes suggested by the model should lead to more computationally efﬁcient code.

Original: To quantify efﬁciency, we rely on textual similarity to other known correct code in the data-set that is more efﬁcient.
Revised: To quantify efﬁciency, we rely on textual similarity to other known correct code in the data-set that is more efﬁcient.

Original: For a given source program, there are a multitude of discrete transformations that could lead to more efﬁcient code (that is, the problem is one-to-many).
Revised: For a given source program, there are a multitude of discrete transformations that could lead to more efﬁcient code (that is, the problem is one-to-many).

Original: For example, more efﬁcient code versions could change the data structures involved, wrap/unwrap code in functions, or adjust loop bounds.
Revised: For example, more efﬁcient code versions could change the data structures involved, wrap/unwrap code in functions, or adjust loop bounds.

Original: These edits are not known a priori, so the model should automatically learn which transformations increase efﬁciency, and then communicate those choices to developers.
Revised: These edits are not known a priori, so the model should automatically learn which transformations increase efﬁciency, and then communicate those choices to developers.

Original: Therefore, an ideal model would be able to generate many different distinct hints, so we consider code diversity as an auxiliary objective.
Revised: Therefore, an ideal model would be able to generate many different distinct hints, so we consider code diversity as an auxiliary objective.

Original: The criteria mentioned above (efﬁciency, correctness, diversity) inform our modeling choices.
Revised: The criteria mentioned above (efﬁciency, correctness, diversity) inform our modeling choices.

Original: In Section 4.3, we describe speciﬁc metrics to quantify each of these criteria.
Revised: In Section 4.3, we describe speciﬁc metrics to quantify each of these criteria.

Original: 3. N OTATION
Revised: 3. N OTATION

Original: We denote an input sequence x
Revised: We denote an input sequence x

Original: = [ x 1 , x 2 , ..., x T in ] and an output sequence y =
Revised: = [ x 1 , x 2 , ..., x T in ] and an output sequence y =

Original: [ y 1 , y 2 , ..., y T out ] , where x i , y j are tokens in a vocabulary V .
Revised: [ y 1 , y 2 , ..., y T out ] , where x i , y j are tokens in a vocabulary V .

Original: We will represent the encoder as f θ ( · ) and the decoder as g φ ( · ) , where θ and φ are the parameters of the networks.
Revised: We will represent the encoder as f θ ( · ) and the decoder as g φ ( · ) , where θ and φ are the parameters of the networks.

Original: As is standard in sequence modeling, we embed tokens using an embedding function e ( · ) and add positional encodings PE .
Revised: As is standard in sequence modeling, we embed tokens using an embedding function e ( · ) and add positional encodings PE .

Original: Unless it is informative to say so, we will assume that this is done implicitly.
Revised: Unless it is informative to say so, we will assume that this is done implicitly.

Original: 3. B ASELINE M ODEL : T RANSFORMER
Revised: 3. B ASELINE M ODEL : T RANSFORMER

Original: Prior work in program optimization often operates at the compiler or assembly-code level (Schkufza et al., 2013).
Revised: Prior work in program optimization often operates at the compiler or assembly-code level (Schkufza et al., 2013).

Original: As we aim to provide much higher-level natural language hints to edit source code instead, we use a state-of-the-art natural language model as a baseline (Vaswani et al., 2017).
Revised: As we aim to provide much higher-level natural language hints to edit source code instead, we use a state-of-the-art natural language model as a baseline (Vaswani et al., 2017).

Original: The baseline Transformer is trained in a sequence-to-sequence fashion, where given a pair of programs with runtimes r slow and r fast , we use the slower program as the input x , and the faster program as the output y .
Revised: The baseline Transformer is trained in a sequence-to-sequence fashion, where given a pair of programs with runtimes r slow and r fast , we use the slower program as the input x , and the faster program as the output y .

Original: We encode the input program using f θ ( x ) and decode the output program using g φ ( f θ ( x )) .
Revised: We encode the input program using f θ ( x ) and decode the output program using g φ ( f θ ( x )) .

Original: We train the network to minimize cross entropy loss.
Revised: We train the network to minimize cross entropy loss.

Original: To generate different programs given one input program, we simply randomly sample outputs.
Revised: To generate different programs given one input program, we simply randomly sample outputs.

Original: 3. D ISCRETE V ARIATIONAL A UTO -E NCODERS
Revised: 3. D ISCRETE V ARIATIONAL A UTO -E NCODERS

Original: As an alternative to a fully black-box solution, we propose solving the problem with a more explicit and interpretable model.
Revised: As an alternative to a fully black-box solution, we propose solving the problem with a more explicit and interpretable model.

Original: Since we observe in the dataset that there are many discrete categories of efﬁciency-improving code transformations, we propose using a discrete variational auto-encoder to learn these categories in an unsupervised way.
Revised: Since we observe in the dataset that there are many discrete categories of efﬁciency-improving code transformations, we propose using a discrete variational auto-encoder to learn these categories in an unsupervised way.

Original: Furthermore, we are interested in learning ﬁne-grained edits of code, where each category of learned edit ideally represents a single conceptual or localized change.
Revised: Furthermore, we are interested in learning ﬁne-grained edits of code, where each category of learned edit ideally represents a single conceptual or localized change.

Original: Towards that end, we encode x
Revised: Towards that end, we encode x

Original: [Figure]
Revised: [Figure]

Original: and y into latent vectors z fast and z slow , and take their difference z edit =
Revised: and y into latent vectors z fast and z slow , and take their difference z edit =

Original: z fast − z slow to capture the necessary information to edit x into y .
Revised: z fast − z slow to capture the necessary information to edit x into y .

Original: This two-tower encoding approach with tied encoders allows the model to learn a relative edit encoding, which we ﬁnd helps training stability.
Revised: This two-tower encoding approach with tied encoders allows the model to learn a relative edit encoding, which we ﬁnd helps training stability.

Original: We ﬁnd that code canonicalization aids in learning this relative encoding, as it removes semantic differences between the input and output programs.
Revised: We ﬁnd that code canonicalization aids in learning this relative encoding, as it removes semantic differences between the input and output programs.

Original: The embedding vectors z fast and z slow are computed using a multi-headed attention layer between the encoded outputs from f θ and a learned query vector c .
Revised: The embedding vectors z fast and z slow are computed using a multi-headed attention layer between the encoded outputs from f θ and a learned query vector c .

Original: This form of pooling has been found in other works as well (Lee et al., 2019).
Revised: This form of pooling has been found in other works as well (Lee et al., 2019).

Original: To capture the one-to-many relationship of code edits, and to further emphasize the idea of broad and interpretable edit categories, we choose a VQVAE to quantize z edit (Van Den Oord et al., 2017).
Revised: To capture the one-to-many relationship of code edits, and to further emphasize the idea of broad and interpretable edit categories, we choose a VQVAE to quantize z edit (Van Den Oord et al., 2017).

Original: This allows us to train a generative model of K discrete edit types.
Revised: This allows us to train a generative model of K discrete edit types.

Original: Each type, when applied to a piece of code, produces a different result.
Revised: Each type, when applied to a piece of code, produces a different result.

Original: This also has the advantage of preventing posterior collapse, a pervasive issue with variational auto-encoders, especially with respect to discrete sequence modeling.
Revised: This also has the advantage of preventing posterior collapse, a pervasive issue with variational auto-encoders, especially with respect to discrete sequence modeling.

Original: Speciﬁcally, we learn a dictionary of embedding vectors Z =
Revised: Speciﬁcally, we learn a dictionary of embedding vectors Z =

Original: [ z 1 , z 2 , . . .
Revised: [ z 1 , z 2 , . . .

Original: , z K ] .
Revised: , z K ] .

Original: When encoding the difference vector, we set z edit =
Revised: When encoding the difference vector, we set z edit =

Original: argmin k (cid:107)
Revised: argmin k (cid:107)

Original: z k − ( f θ ( y ) − f θ ( x ))
Revised: z k − ( f θ ( y ) − f θ ( x ))

Original: (cid:107) 2 .
Revised: (cid:107) 2 .

Original: Our Edit-VQVAE model is illustrated in Figure 3.4 and our forward and decoding procedure is outlined in Algorithms ?? and ?? in the Appendix.
Revised: Our Edit-VQVAE model is illustrated in Figure 3.4 and our forward and decoding procedure is outlined in Algorithms 1 and 2 in the Appendix.

Original: To support this model, we modify the decoder to take z edit as an auxiliary input.
Revised: To support this model, we modify the decoder to take z edit as an auxiliary input.

Original: The total input to the decoder is therefore the sum of the token embeddings, positional encodings, and difference vector:
Revised: The total input to the decoder is therefore the sum of the token embeddings, positional encodings, and difference vector:

Original: e ( x ) + PE + z edit .
Revised: e ( x ) + PE + z edit .

Original:  E VALUATION 4. M ETHODOLOGY
Revised:  E VALUATION 4. M ETHODOLOGY

Original: The primary utility of our model is to provide hints to programmers that identify improvement opportunities for their code.
Revised: The primary utility of our model is to provide hints to programmers that identify improvement opportunities for their code.

Original: This means that changes suggested by the model do not need to be perfect; they just need to point programmers in the right direction.
Revised: This means that changes suggested by the model do not need to be perfect; they just need to point programmers in the right direction.

Original: We evaluate these hints ﬁrst qualitatively in Section 4.2 then quantitatively in Section 4.3.
Revised: We evaluate these hints ﬁrst qualitatively in Section 4.2 then quantitatively in Section 4.3.

Original: Experimental setting.
Revised: Experimental setting.

Original: The program pair dataset for training and evaluating program optimization tasks is constructed from the raw dataset following the procedures described in Section 2.2.
Revised: The program pair dataset for training and evaluating program optimization tasks is constructed from the raw dataset following the procedures described in Section 2.2.

Original: After processing the raw program submission dataset into program pairs, we randomly split the program pairs into training, validation and testing set.
Revised: After processing the raw program submission dataset into program pairs, we randomly split the program pairs into training, validation and testing set.

Original: The training/validation sets contain 806 , 041 / 1 ,program pairs.
Revised: The training/validation sets contain 806 , 041 / 1 ,program pairs.

Original: The test set contains 100 input programs that are not visible during training.
Revised: The test set contains 100 input programs that are not visible during training.

Original: The program pair runtime distribution is illustrated in Figure 2(c).
Revised: The program pair runtime distribution is illustrated in Figure 2(c).

Original: We implemented all models in Jax (Bradbury et al., 2018), and trained the models using a peak learning rate of 0 .
Revised: We implemented all models in Jax (Bradbury et al., 2018), and trained the models using a peak learning rate of 0 .

Original: 01 with both warmup and decay schedules.
Revised: 01 with both warmup and decay schedules.

Original: All models were trained with a batch size of 16 for 100 epochs, using distributed data parallel training on 64 Google Cloud TPU cores and 16 host machines.
Revised: All models were trained with a batch size of 16 for 100 epochs, using distributed data parallel training on 64 Google Cloud TPU cores and 16 host machines.

Original: We performed rudimentary hyperparameter tuning for all models.
Revised: We performed rudimentary hyperparameter tuning for all models.

Original: For all
Revised: For all

Original: Transformer blocks, we set the dropout rate to p dropout = 0 .
Revised: Transformer blocks, we set the dropout rate to p dropout = 0 .

Original: 1 , the attention layer embedding size to d model = 128 , and the feed-forward inner layer size to d ff = 512 .
Revised: 1 , the attention layer embedding size to d model = 128 , and the feed-forward inner layer size to d ff = 512 .

Original: We use 6 Transformer layers, each with 8 attention heads, for both encoders and decoders.
Revised: We use 6 Transformer layers, each with 8 attention heads, for both encoders and decoders.

Original: The Edit-VQVAE has K = 64 latents.
Revised: The Edit-VQVAE has K = 64 latents.

Original: In the quantitative evaluation, we randomly sample N = 64 programs from each model for evaluation.
Revised: In the quantitative evaluation, we randomly sample N = 64 programs from each model for evaluation.

Original: The
Revised: The

Original: Edit-VQVAE generates 1 program per latent.
Revised: Edit-VQVAE generates 1 program per latent.

Original: 4. E DIT VQ-VAE Q UALITATIVE A NALYSIS
Revised: 4. E DIT VQ-VAE Q UALITATIVE A NALYSIS

Original: One of the main beneﬁts of a discrete latent variable model is an interpretable latent space that we can use to visualize the learned efﬁciency archetypes.
Revised: One of the main beneﬁts of a discrete latent variable model is an interpretable latent space that we can use to visualize the learned efﬁciency archetypes.

Original: After training and during generation, we remove the encoder and directly feed the canonicalized program into the decoder along with a categorical variable that selects one of 64 discrete latent edits.
Revised: After training and during generation, we remove the encoder and directly feed the canonicalized program into the decoder along with a categorical variable that selects one of 64 discrete latent edits.

Original: A real learned example of this is shown in Figure 4.
Revised: A real learned example of this is shown in Figure 4.

Original: The canonicalized program is edited by the decoder to increase performance (in this case the model suggests a heap).
Revised: The canonicalized program is edited by the decoder to increase performance (in this case the model suggests a heap).

Original: We ﬁnd that these edits expose the key optimization idea that can then be translated back into code by the programmer.
Revised: We ﬁnd that these edits expose the key optimization idea that can then be translated back into code by the programmer.

Original: We often ﬁnd that each of the discrete latents is responsible for a single syntactic change (adding a heap data structure, changing a for-loop to a while-loop and vice versa, adding functions, etc.) – and that we can take different transformations on the same program by varying the latent code provided to the VQ-VAE.
Revised: We often ﬁnd that each of the discrete latents is responsible for a single syntactic change (adding a heap data structure, changing a for-loop to a while-loop and vice versa, adding functions, etc.) – and that we can take different transformations on the same program by varying the latent code provided to the VQ-VAE.

Original: The model learns many different efﬁciency archetypes (Table 2), including those discussed in Section 2.
Revised: The model learns many different efﬁciency archetypes (Table 2), including those discussed in Section 2.

Original: We can visualize the effect that the transformations have on different program pairs through PCA decomposition as well (shown in the Appendix).
Revised: We can visualize the effect that the transformations have on different program pairs through PCA decomposition as well (shown in the Appendix).

Original: If the latent variable has learned to apply a similar edit to its input programs, one would expect that the slow programs would be collectively moved to a faster region of latent space.
Revised: If the latent variable has learned to apply a similar edit to its input programs, one would expect that the slow programs would be collectively moved to a faster region of latent space.

Original: We observe this behavior in Figure ?? .
Revised: We observe this behavior in Figure 5.

Original: 4. Q UANTITATIVE R ESULTS
Revised: 4. Q UANTITATIVE R ESULTS

Original: Beyond the qualitative analysis in Section 4.2, we quantitatively evaluate the suggestions provided by the Edit VQ-VAE vs. a vanilla sequence-to-sequence Transformer on three axes: correctness, efﬁciency, and diversity.
Revised: Beyond the qualitative analysis in Section 4.2, we quantitatively evaluate the suggestions provided by the Edit VQ-VAE vs. a vanilla sequence-to-sequence Transformer on three axes: correctness, efﬁciency, and diversity.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: We use syntactic program similarity as a tool to measure model performance.
Revised: We use syntactic program similarity as a tool to measure model performance.

Original: Program similarity is computed between the generated programs and a set of reference programs, where the reference programs are programs in the dataset that solve the same question more efﬁciently than the original input program.
Revised: Program similarity is computed between the generated programs and a set of reference programs, where the reference programs are programs in the dataset that solve the same question more efﬁciently than the original input program.

Original: For each input program x in the test set, we construct its reference program set as follows,
Revised: For each input program x in the test set, we construct its reference program set as follows,

Original: [Equation]
Revised: [Equation]

Original: where

Original: D q x ∈

Original: D

Original: is the

Original: subset

Original: of program submissions in the dataset

Original:  D that solves the same question q x as input program x , and t > 0 is a threshold for the program neighborhood.
Revised: where D q x ∈ D is the subset of program submissions in the dataset D that solves the same question q x as input program x , and t > 0 is a threshold for the program neighborhood.

Original: We propose the axis-speciﬁc metrics below, which are built upon conventional text similarity metrics such as the BLEU score (Papineni et al., 2002) and the ∆ BLEU score (Galley et al., 2015).
Revised: We propose the axis-speciﬁc metrics below, which are built upon conventional text similarity metrics such as the BLEU score (Papineni et al., 2002) and the ∆ BLEU score (Galley et al., 2015).

Original: • Correctness Delta, i.e
Revised: • Correctness Delta, i.e.

Original: . , similarity between the generated program and an existing solution to the same question as the input program.
Revised: , similarity between the generated program and an existing solution to the same question as the input program.

Original: We adopt the BLEU score here as a surrogate.
Revised: We adopt the BLEU score here as a surrogate.

Original: For any
Revised: For any

Original: [Table]
Revised: [Table]

Original: generated program ˆ y , the correctness score is deﬁned as,
Revised: generated program ˆ y , the correctness score is deﬁned as,

Original: [Equation]
Revised: [Equation]

Original: candidate
Revised: candidate

Original: [Equation]
Revised: [Equation]

Original: references
Revised: references

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: S C gives a score in [0 , 1] , a score of 1 indicates there is a perfect overlap between ˆ y and the reference programs in R ( x ) , while a score of 0 means there is no overlap.
Revised: S C gives a score in [0 , 1] , a score of 1 indicates there is a perfect overlap between ˆ y and the reference programs in R ( x ) , while a score of 0 means there is no overlap.

Original: A higher S c score indicates that the generated program is more similar to the reference programs, implying less programmer effort
Revised: A higher S c score indicates that the generated program is more similar to the reference programs, implying less programmer effort

Original: to
Revised: to

Original: translate the suggestion into a correct program.
Revised: translate the suggestion into a correct program.

Original: • Efﬁciency Delta, i.e. , similarity between the generated program and an existing more efﬁcient solution to the same question as the input program.
Revised: • Efﬁciency Delta, i.e. , similarity between the generated program and an existing more efﬁcient solution to the same question as the input program.

Original: There are two considerations: 1) how much more efﬁcient an alternative program is compared to the input, and 2) how different this alternative program is to the suggestion.
Revised: There are two considerations: 1) how much more efﬁcient an alternative program is compared to the input, and 2) how different this alternative program is to the suggestion.

Original: We adopt two metrics for efﬁciency, a hard metric using an exact match, and a soft metric using ∆ BLEU.
Revised: We adopt two metrics for efﬁciency, a hard metric using an exact match, and a soft metric using ∆ BLEU.

Original: – If the generated program ˆ y has an exact match with one of the reference programs in R ( x ) , we can directly use its runtime to compute the efﬁciency for ˆ y .
Revised: – If the generated program ˆ y has an exact match with one of the reference programs in R ( x ) , we can directly use its runtime to compute the efﬁciency for ˆ y .

Original: The exact match score is deﬁned as,
Revised: The exact match score is deﬁned as,

Original: [Equation]
Revised: [Equation]

Original: S EH gives a score in [0 , 1] , a score of 1 indicates an exact match with the most efﬁcient program in the reference set, while a score of 0 means no exact match.
Revised: S EH gives a score in [0 , 1] , a score of 1 indicates an exact match with the most efﬁcient program in the reference set, while a score of 0 means no exact match.

Original: – Requiring generated programs to be exactly the same as a reference program will undervalue good suggestions that are close to an efﬁcient solution, but not exactly the same.
Revised: – Requiring generated programs to be exactly the same as a reference program will undervalue good suggestions that are close to an efﬁcient solution, but not exactly the same.

Original: Therefore we also employ the ∆ BLEU score as a soft version for efﬁciency.
Revised: Therefore we also employ the ∆ BLEU score as a soft version for efﬁciency.

Original: ∆ BLEU is a variant of BLEU where each reference can be assigned with a unique weight indicating the quality of the reference.
Revised: ∆ BLEU is a variant of BLEU where each reference can be assigned with a unique weight indicating the quality of the reference.

Original: The score is deﬁned as,
Revised: The score is deﬁned as,

Original: [Equation]
Revised: [Equation]

Original: A reference program is assigned a higher weight if it is more efﬁcient.
Revised: A reference program is assigned a higher weight if it is more efﬁcient.

Original: S ES is in [0 , 1] , a score of 1 indicates a perfect overlap between ˆ y and a more efﬁcient reference program.
Revised: S ES is in [0 , 1] , a score of 1 indicates a perfect overlap between ˆ y and a more efﬁcient reference program.

Original: As with correctness, this approximates how much additional work would be required to transform the suggestion into an efﬁcient solution.
Revised: As with correctness, this approximates how much additional work would be required to transform the suggestion into an efﬁcient solution.

Original: • Diversity,
Revised: • Diversity,

Original: i
Revised: i

Original: e. , whether the model can generate diverse efﬁcient programs given one input program.
Revised: e. , whether the model can generate diverse efﬁcient programs given one input program.

Original: We deﬁne diversity as the model’s ability to recover the set of reference programs using the generated programs ˆ Y ( x ) = { ˆ y 1 , ˆ y 2 , · · · , ˆ y n } .
Revised: We deﬁne diversity as the model’s ability to recover the set of reference programs using the generated programs ˆ Y ( x ) = { ˆ y 1 , ˆ y 2 , · · · , ˆ y n } .

Original: The diversity score is deﬁned as,
Revised: The diversity score is deﬁned as,

Original: [Equation]
Revised: [Equation]

Original: S D is in [0 , 1] .
Revised: S D is in [0 , 1] .

Original: A score of 1 indicates perfect recovering of the reference set.
Revised: A score of 1 indicates perfect recovering of the reference set.

Original: Our results are summarized in Table 1.
Revised: Our results are summarized in Table 1.

Original: For metrics on correctness and efﬁciency, we show both average and maximum performance over the N samples.
Revised: For metrics on correctness and efﬁciency, we show both average and maximum performance over the N samples.

Original: We ﬁnd that Edit-VQVAE outperforms the sequence-to-sequence Transformer baseline by a large margin on all metrics.
Revised: We ﬁnd that Edit-VQVAE outperforms the sequence-to-sequence Transformer baseline by a large margin on all metrics.

Original: As a one-to-many model, the Edit-VQVAE is able to learn the program edit distribution from program pairs that share the same input programs in the training dataset.
Revised: As a one-to-many model, the Edit-VQVAE is able to learn the program edit distribution from program pairs that share the same input programs in the training dataset.

Original: Conversely, sequence-to-sequence models perform better when learning a one-to-one mapping with a single mode, but do not guarantee the quality of multiple random samples.
Revised: Conversely, sequence-to-sequence models perform better when learning a one-to-one mapping with a single mode, but do not guarantee the quality of multiple random samples.

Original: This is supported by our data showing that the gap between the Transformer baseline and the Edit-VQVAE is smaller on the maximum results than on the average results.
Revised: This is supported by our data showing that the gap between the Transformer baseline and the Edit-VQVAE is smaller on the maximum results than on the average results.

Original: The Transformer baseline is trained using all the program pairs in the same training set as the
Revised: The Transformer baseline is trained using all the program pairs in the same training set as the

Original: Edit-VQVAE.
Revised: Edit-VQVAE.

Original: What if the Transformer baseline is only trained on the program pairs with the largest relative performance?
Revised: What if the Transformer baseline is only trained on the program pairs with the largest relative performance?

Original: Would its maximum results outperform Edit-VQVAE?
Revised: Would its maximum results outperform Edit-VQVAE?

Original: Interestingly, the answer is no.
Revised: Interestingly, the answer is no.

Original: For program pairs that share the same input program in the training set, we only keep the pair with maximum runtime improvement.
Revised: For program pairs that share the same input program in the training set, we only keep the pair with maximum runtime improvement.

Original: This ﬁltered dataset is used to train another Transformer model, denoted as Transformer*.
Revised: This ﬁltered dataset is used to train another Transformer model, denoted as Transformer*.

Original: Transformer* signiﬁcantly outperforms Transformer on the maximum metrics, but it still underperforms Edit-VQVAE.
Revised: Transformer* signiﬁcantly outperforms Transformer on the maximum metrics, but it still underperforms Edit-VQVAE.

Original: To understand the effect of the design decisions in Edit-VQVAE, we perform an ablation study by removing the discrete and edit-based structure in the model.
Revised: To understand the effect of the design decisions in Edit-VQVAE, we perform an ablation study by removing the discrete and edit-based structure in the model.

Original: This results in two ablated models: 1) VQVAE without the edit structure, and 2) Edit-VAE.
Revised: This results in two ablated models: 1) VQVAE without the edit structure, and 2) Edit-VAE.

Original: In the VQVAE, the latent is obtained by directly feeding the concatenation of the program pairs into the Transformer encoder, leading to a model with 4 x parameters.
Revised: In the VQVAE, the latent is obtained by directly feeding the concatenation of the program pairs into the Transformer encoder, leading to a model with 4 x parameters.

Original: In the Edit-VAE, the discrete latent space is replaced by the original continuous Gaussian latent space.
Revised: In the Edit-VAE, the discrete latent space is replaced by the original continuous Gaussian latent space.

Original: Empirically, we found that it is difﬁcult for VQVAE to converge.
Revised: Empirically, we found that it is difﬁcult for VQVAE to converge.

Original: We hypothesize this is due to the fact that it is hard to control the scale of the output of a Transformer encoder block.
Revised: We hypothesize this is due to the fact that it is hard to control the scale of the output of a Transformer encoder block.

Original: The edit-based structure instead uses a relative encoding, which potentially controls the scale of the encoder output, and helps stabilize training.
Revised: The edit-based structure instead uses a relative encoding, which potentially controls the scale of the encoder output, and helps stabilize training.

Original: Additionally, we found that the Edit-VAE model suffers from posterior collapse and is not able to generate more than one program during test time.
Revised: Additionally, we found that the Edit-VAE model suffers from posterior collapse and is not able to generate more than one program during test time.

Original: Edit-VQVAE is more robust to posterior collapse due to a discrete latent space (Van Den Oord et al., 2017).
Revised: Edit-VQVAE is more robust to posterior collapse due to a discrete latent space (Van Den Oord et al., 2017).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Program Synthesis Program synthesis involves automatically writing code from a program speciﬁcation.
Revised: Program Synthesis Program synthesis involves automatically writing code from a program speciﬁcation.

Original: Many deep learning approaches have been proposed in recent years (Balog et al., 2016;
Revised: Many deep learning approaches have been proposed in recent years (Balog et al., 2016;

Original: Bunel et al., 2018; Devlin et al., 2017b; Kalyan et al., 2018;
Revised: Bunel et al., 2018; Devlin et al., 2017b; Kalyan et al., 2018;

Original: Devlin et al., 2017a; Lee et al., 2018;
Revised: Devlin et al., 2017a; Lee et al., 2018;

Original: Nye et al., 2019; Odena & Sutton, 2020; Parisotto et al., 2016).
Revised: Nye et al., 2019; Odena & Sutton, 2020; Parisotto et al., 2016).

Original: However, these works generally focused on writing code to satisfy a given speciﬁcation without regard for its efﬁciency.
Revised: However, these works generally focused on writing code to satisfy a given speciﬁcation without regard for its efﬁciency.

Original: There are neural (Zhao et al., 2018) and non-neural (Meng et al., 2011) models that edit code, but these are used for different applications such as ﬁxing errors (Yasunaga & Liang, 2020; Chen et al., 2021b).
Revised: There are neural (Zhao et al., 2018) and non-neural (Meng et al., 2011) models that edit code, but these are used for different applications such as ﬁxing errors (Yasunaga & Liang, 2020; Chen et al., 2021b).

Original: Language models and large-scale program synthesis Large language models (LLMs) built on
Revised: Language models and large-scale program synthesis Large language models (LLMs) built on

Original: Transformers (Vaswani et al., 2017) and trained with massive amounts of data have begun to yield signiﬁcant improvements in many natural language tasks (Chowdhery et al., 2022).
Revised: Transformers (Vaswani et al., 2017) and trained with massive amounts of data have begun to yield signiﬁcant improvements in many natural language tasks (Chowdhery et al., 2022).

Original: These breakthroughs have been ported over to program synthesis where LLMs have demonstrated an impressive ability to solve coding challenge problems (Austin et al., 2021; Chen et al., 2021a).
Revised: These breakthroughs have been ported over to program synthesis where LLMs have demonstrated an impressive ability to solve coding challenge problems (Austin et al., 2021; Chen et al., 2021a).

Original: These approaches have been concerned with generating a program from a high-level speciﬁcation, as opposed to editing an existing program to make it faster.
Revised: These approaches have been concerned with generating a program from a high-level speciﬁcation, as opposed to editing an existing program to make it faster.

Original: Other forms of optimization Program optimization has been applied to efﬁciency-related areas:
Revised: Other forms of optimization Program optimization has been applied to efﬁciency-related areas:

Original: Query optimization (e.g., using deep learning (Krishnan et al., 2018; Marcus et al., 2019)) seeks to optimize query execution plans for database systems.
Revised: Query optimization (e.g., using deep learning (Krishnan et al., 2018; Marcus et al., 2019)) seeks to optimize query execution plans for database systems.

Original: Stochastic superoptimization (Schkufza et al., 2013) uses random search to optimize x86 assembly code.
Revised: Stochastic superoptimization (Schkufza et al., 2013) uses random search to optimize x86 assembly code.

Original: Assembly and queries are both highly structured languages and the programs are loop-free and quite small.
Revised: Assembly and queries are both highly structured languages and the programs are loop-free and quite small.

Original: We aim to optimize higher-level source code, which translates to vastly larger assembly programs, using a generative approach.
Revised: We aim to optimize higher-level source code, which translates to vastly larger assembly programs, using a generative approach.

Original: Discrete representations Many real-world problems are discrete, and discrete representations can often be more composable and interpretable.
Revised: Discrete representations Many real-world problems are discrete, and discrete representations can often be more composable and interpretable.

Original: Learning discrete representations using generative modeling is challenging due to the lack of gradients, necessitating the use of gradient estimators like the Gumbel-softmax (Jang et al., 2016; Maddison et al., 2016) and REINFORCE (Williams, 1992).
Revised: Learning discrete representations using generative modeling is challenging due to the lack of gradients, necessitating the use of gradient estimators like the Gumbel-softmax (Jang et al., 2016; Maddison et al., 2016) and REINFORCE (Williams, 1992).

Original: The VQ-VAE (Van Den Oord et al., 2017) avoids this by using vector quantization and treating
Revised: The VQ-VAE (Van Den Oord et al., 2017) avoids this by using vector quantization and treating

Original: (a) A transformation that adds an early termination to a while loop.
Revised: (a) A transformation that adds an early termination to a while loop.

Original: We see that the model also applies other semantics-preserving transformations that can be safely ignored.
Revised: We see that the model also applies other semantics-preserving transformations that can be safely ignored.

Original: [Figure]
Revised: [Figure]

Original: A notable change this transformation identiﬁed
Revised: A notable change this transformation identiﬁed

Original: to replace
Revised: to replace

Original: which
Revised: which

Original: indeed
Revised: indeed

Original: lead
Revised: lead

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Table]
Revised: [Figure] [Table]

Original: [Figure] [Table] 
Revised: [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original: [Table] [Figure] 
Revised: [Table] [Figure] [Table]

Original: [Table] [Figure] [Table] [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: these latent codes as targets.
Revised: these latent codes as targets.

Original: It has shown tremendous success in generative modeling for images, and largely avoids the posterior collapse issue that is prevalent in continuous VAEs (Kingma &
Revised: It has shown tremendous success in generative modeling for images, and largely avoids the posterior collapse issue that is prevalent in continuous VAEs (Kingma &

Original: Welling, 2013).
Revised: Welling, 2013).

Original: We are therefore able to use VQ-VAEs with powerful Tranformer-based encoders and decoders, thus reaping the beneﬁts of both large language models and discrete representations.
Revised: We are therefore able to use VQ-VAEs with powerful Tranformer-based encoders and decoders, thus reaping the beneﬁts of both large language models and discrete representations.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we apply discrete categorical transformations to source code.
Revised: In this work, we apply discrete categorical transformations to source code.

Original: These transformations are aimed at increasing code efﬁciency, but could be useful for a variety of natural language tasks or discrete structures such as molecules or graphs.
Revised: These transformations are aimed at increasing code efﬁciency, but could be useful for a variety of natural language tasks or discrete structures such as molecules or graphs.

Original: We take one step towards learned models that improve code efﬁciency by demonstrating that discrete transformations can be learned from a supervised dataset – and that the VQ-VAE provides one architecture for doing so.
Revised: We take one step towards learned models that improve code efﬁciency by demonstrating that discrete transformations can be learned from a supervised dataset – and that the VQ-VAE provides one architecture for doing so.

Original: To move beyond this paper and enable automatic optimization, canonicalization is the ﬁrst challenge that needs to be solved.
Revised: To move beyond this paper and enable automatic optimization, canonicalization is the ﬁrst challenge that needs to be solved.

Original: There are many potential solutions, like using specialized pointer networks for variables or using a Transformer pre-trained on a very large code corpus.
Revised: There are many potential solutions, like using specialized pointer networks for variables or using a Transformer pre-trained on a very large code corpus.

Original: As we move towards building more ML models that can write code, it is critical that we also consider computational efﬁciency.
Revised: As we move towards building more ML models that can write code, it is critical that we also consider computational efﬁciency.

Original: Code that is run in datacenters across the world today contributes to not only cost, but also the carbon footprint of machines.
Revised: Code that is run in datacenters across the world today contributes to not only cost, but also the carbon footprint of machines.

Original: We believe that optimizing this footprint is an exciting application of deep learning.
Revised: We believe that optimizing this footprint is an exciting application of deep learning.

Original: A A PPENDIX 
Revised: A A PPENDIX A. E DIT -VQVAE P SEUDOCODE

Revised: Require: Input sequence x , output sequence y , encoder f θ ( · ) , decoder g φ ( · ) , embedding dictionary Z , positional encodings PE 1: z

Original: You may include other additional sections here.
Revised: 2: z 3: k argmin 5: return g ( x

Revised: PE

Revised: Require:

Revised: Input sequence x , output sequence y , encoder f θ ( · ) , decoder g φ ( · ) , embedding dictionary

Revised: Z , positional encodings PE

Revised: 5: return

Revised: A. A DDITIONAL F IGURES

Revised: [Figure]

Revised: [Table]

Revised: e

Revised: [Table]

Revised: a l

Revised: [Table]

Revised: import heapq

Revised: T = int

Revised: [Table]

Revised: input()) for i in range(1,

Revised: T + 1)

Revised: [Table]

Revised: n,

Revised: [Table]

Revised: =

Revised: [int(s)

Revised: [Table]

Revised: s in input

Revised: [Table]

Revised: split

Revised: [Table]

Revised: A =

Revised: [ int(N)

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: z e import heapq v = int( input()) for v in range(1, v + 1): (v, v) =

Revised: [int (v) for v in input().

Revised: split(‘ST

Revised: RING’)] v = [ int(v) for v in input().

Revised: split( ‘STRING ’)] f. heapify (v) v = 0 for v in range(v): v = heapq.

Revised: heappop (v) if vv >= 0: v -= v v += 1 else: break print( ‘STRING’ .format (v, v))

Revised: t

Revised: = int( input()) for i in range(1, t + 1):

Revised: N, B =

Revised: [int(s) for s in input().

Revised: split(" ") ] costs =

Revised: [int (cost) for cost in input().

Revised: split(" ") ] costs = sorted( costs) n_ houses =while n_ houses < len( costs): if B>= costs[n_ houses]:

Revised: B-= costs[n_ houses] n_ houses += else: break print("Case #{}: {} ".

Revised: format(i, n_ houses)) v = int( input()) fo r v in range(1, v + 1): ( v, v) =

Revised: [int (v) for v in input().

Revised: split( ‘STRING’ )] v = [ int(v) for v in input().

Revised: split( ‘STRING’ )] v = f(v) v = 0 while v < len(v): if v>=v[v ]: v-=v[v] v += else: break print( ‘STRING’ .format (v, v))

Revised: E d i t S ugg es t i on ( C a non i ca li z e d

Revised: P r og r a m ) def f() : for v in range(1, (int( input()) + 1)): (v, v) =

Revised: [int (v) for v in input().

Revised: split()] v = [ int(v) for v in input().

Revised: split()] v = v for v in f( range(len (v)))

Revised: : v = ((v // v[v] ) * v[v ]) f

Revised: STRING' + f(v))

Revised: + '

Revised: STRING')

Revised: + f(v)) ) if(f

Revised: == 'STRING ') :

Revised: f()

Revised: O r i g i n a l d ef f (v , v): v = (l e n(v ) - 1 ) wh i le ( v > 0): v =( ( v / / v [ v ] ) * v [ v ] ) r etu r n v d ef f () : fo r v i n r a nge

Revised: ( 1, ( int ( inp u t() ) + 1 )): (v , v) =

Revised: [ i nt ( v ) f

Revised: o r v in i npu t () .

Revised: s pli t ()] v =

Revised: [i n t(v ) fo r v i n i n put ( ).s

Revised: p lit ( )] v = f

Revised: ( v , v ) f

Revised: ( ( ((' S

Revised: TRI N G' + f( v ))

Revised: + 'S T

Revised: RIN G ')

Revised: + f( v )) )

Revised: i f(f =

Revised: = ' STR

Revised: I NG' ) : f( )

Revised: E d i t

Revised: # im p ort mat h de f f( v , v , v): v = v f or v in f( range(v)): v = (f . floor((v / v [ v ]))

Revised: * v [ v ]) re t urn v v = in t (in p ut()) fo r v i n r a nge(1, (v + 1)): ( v, v ) =

Revised: [int (v) for v in input().

Revised: split()] v = [ int ( v) for v in input().

Revised: split()]

Revised: f ((( ( '

Revised: ST R ING ' + f(v ))

Revised: + 'STRING ')

Revised: + f(f(v , v , v))))

Revised: E d i t # for v in range(int (input ())): (v, v) =

Revised: [int (v) for v in input().

Revised: split()] v = list(map

Revised: (int, input().

Revised: split())) v = 0 v.sort () for v in v: if(v <= v): v -= v v +=f('

Revised: STRING'.

Revised: format((v+ ), v))

Revised: O r

Revised: i g i n a l f or v in ran g e(i n t( i n put ( ))) : (v , v)

Revised: =

Revised: m a p(i n t, i npu t ().

Revised: s pli t ()) v = l

Revised: i st( m ap( i nt, inp u t() .

Revised: spl

Revised: i t() ) ) v .

Original: {+sor t () v = 
+}
Revised: {+sor t () v = 
+}

Revised: f o r v in v : i f(v <

Revised: = v ): v -= v v +=e lse : b rea k f ( f'S T

Revised: RIN G {(v + 1)} S TRI

Revised: N G{v } ')

Original: {+E d i t # v = in t (in p ut()) v = 
+}
Revised: {+E d i t # v = in t (in p ut()) v = 
+}

Revised: wh i le ( v > 0): v + = 1 ( v, v ) =

Revised: [int (v) for v in input().

Revised: split()] v = lis t (map (int, input().

Original: {+split())) v = 
+}
Revised: {+split())) v = 
+}

Revised: v.s o rt ( ) for v i n v: if ( v <= v): v -= v v +=f

Revised: (' S TRI

Revised: N G'.

Revised: format(v, v)) v - =

Revised: E d i t # for v in range(int (input ())) : (v , v) =

Revised: [ int(v) for v in input() .

Revised: split() ] v = list(map

Revised: (int , input() .

Revised: split())) v = 0 v.sort () for v in v : if(v <= v) : v -= v v +=f('

Revised: STRING’ .

Revised: format((v + 1) , v))

Revised: O r i g i n a l f or v in ran g e(i n t( i n put ( ))) : (v , v) =

Revised: [ int ( v) f or v in inp u t() .

Revised: s p lit ( ) ] v = li s t(m a p( i n t , inp u t() .

Revised: s p lit ( ))) v. s ort ( ) v = 0 fo r v i n v : i f(v < = v ) : v - = v v + =e lse : br

Revised: e ak f (f '

Revised: STR I NG{ ( v + 1)}

Revised: S TRI N G{ v } ')

Revised: E d i t # v = in t (in p ut()) fo

Revised: [Table]

Revised: v i n r a nge(v) :

Revised: ( v , v ) =

Revised: [ int(v) for v in input() .

Revised: split() ] v = l ist ( map (int , input() .

Revised: split())) v = 0 v .so r t () f or v in v : if ( v < = v) : v -= v v +=el

Revised: s e: b rea k f( ' STR

Revised: I NG' .

Revised: format((v + 1) , v) , flush =

Revised: True )

Revised: E d i t #