Original: S YNTHETIC D ATASETS FOR N EURAL P ROGRAM S YNTHESIS
Revised: S YNTHETIC D ATASETS FOR N EURAL P ROGRAM S YNTHESIS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The goal of program synthesis is to automatically generate programs in a particular language from corresponding speciﬁcations, e.g. input-output behavior.
Revised: The goal of program synthesis is to automatically generate programs in a particular language from corresponding speciﬁcations, e.g. input-output behavior.

Original: Many current approaches achieve impressive results after training on randomly generated
Revised: Many current approaches achieve impressive results after training on randomly generated

Original: I/O examples in limited domain-speciﬁc languages (DSLs), as with string transformations in RobustFill.
Revised: I/O examples in limited domain-speciﬁc languages (DSLs), as with string transformations in RobustFill.

Original: However, we empirically discover that applying test input generation techniques for languages with control ﬂow and rich input space causes deep networks to generalize poorly to certain data distributions; to correct this, we propose a new methodology for controlling and evaluating the bias of synthetic data distributions over both programs and speciﬁcations.
Revised: However, we empirically discover that applying test input generation techniques for languages with control ﬂow and rich input space causes deep networks to generalize poorly to certain data distributions; to correct this, we propose a new methodology for controlling and evaluating the bias of synthetic data distributions over both programs and speciﬁcations.

Original: We demonstrate, using the Karel DSL and a small Calculator DSL, that training deep networks on these distributions leads to improved cross-distribution generalization performance.
Revised: We demonstrate, using the Karel DSL and a small Calculator DSL, that training deep networks on these distributions leads to improved cross-distribution generalization performance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Program synthesis is one of the central problems in Artiﬁcial Intelligence studied from the early days (Manna & Waldinger, 1971; Waldinger & Lee, 1969) and has seen a lot of recent interest in the machine learning and programming languages community (Alur et al., 2013;
Revised: Program synthesis is one of the central problems in Artiﬁcial Intelligence studied from the early days (Manna & Waldinger, 1971; Waldinger & Lee, 1969) and has seen a lot of recent interest in the machine learning and programming languages community (Alur et al., 2013;

Original: Gulwani et al., 2012; 2015; Muggleton, 1991; Lin et al., 2014; Solar-Lezama, 2013).
Revised: Gulwani et al., 2012; 2015; Muggleton, 1991; Lin et al., 2014; Solar-Lezama, 2013).

Original: The recent neural approaches can be broadly classiﬁed into two categories: program induction and program synthesis .
Revised: The recent neural approaches can be broadly classiﬁed into two categories: program induction and program synthesis .

Original: Both approaches share the objective of learning program semantics but do so in different ways.
Revised: Both approaches share the objective of learning program semantics but do so in different ways.

Original: Program induction aims to embed the semantics of a particular algorithm into a differentiable model trained end-to-end, whereas the goal of program synthesis is for a model to learn the semantics of a domain-speciﬁc language (DSL) and produce programs deﬁned by corresponding speciﬁcations.
Revised: Program induction aims to embed the semantics of a particular algorithm into a differentiable model trained end-to-end, whereas the goal of program synthesis is for a model to learn the semantics of a domain-speciﬁc language (DSL) and produce programs deﬁned by corresponding speciﬁcations.

Original: Both problems necessitate large datasets, either of I/O pairs in the case of program induction, or programs with corresponding I/O pairs in the case of program synthesis.
Revised: Both problems necessitate large datasets, either of I/O pairs in the case of program induction, or programs with corresponding I/O pairs in the case of program synthesis.

Original: However, since large datasets for program induction and synthesis tasks do not exist, these approaches train models on large synthetically generated datasets.
Revised: However, since large datasets for program induction and synthesis tasks do not exist, these approaches train models on large synthetically generated datasets.

Original: Presumably, if a model can accurately predict arbitrary program outputs (for induction) or programs in the DSL (for synthesis) then it has likely learnt the correct algorithm or DSL semantics.
Revised: Presumably, if a model can accurately predict arbitrary program outputs (for induction) or programs in the DSL (for synthesis) then it has likely learnt the correct algorithm or DSL semantics.

Original: Although this approach has led to some impressive synthesis results in many domains, synthetically generating datasets that cover all DSL programs and the corresponding input space can be problematic, especially for more complex DSLs like Karel the Robot (Pattis, 1981) which includes complex control-ﬂow primitives (while loops and if conditionals) and operators.
Revised: Although this approach has led to some impressive synthesis results in many domains, synthetically generating datasets that cover all DSL programs and the corresponding input space can be problematic, especially for more complex DSLs like Karel the Robot (Pattis, 1981) which includes complex control-ﬂow primitives (while loops and if conditionals) and operators.

Original: Likewise, for induction tasks, the sampling procedure for program speciﬁcations may lead to undesirable biases in the training distribution that inhibit strong generalization.
Revised: Likewise, for induction tasks, the sampling procedure for program speciﬁcations may lead to undesirable biases in the training distribution that inhibit strong generalization.

Original: In this paper, we consider two problem settings.
Revised: In this paper, we consider two problem settings.

Original: The ﬁrst is the Karel domain and the recently proposed Karel synthesis model (Bunel et al., 2018).
Revised: The ﬁrst is the Karel domain and the recently proposed Karel synthesis model (Bunel et al., 2018).

Original: We identify many distributions of input examples and DSL programs for which the Karel synthesis model performs poorly.
Revised: We identify many distributions of input examples and DSL programs for which the Karel synthesis model performs poorly.

Original: The second problem setting is a program induction problem in which a model is trained to execute and predict the output of simple arithmetic expressions, which we denote the Calculator domain.
Revised: The second problem setting is a program induction problem in which a model is trained to execute and predict the output of simple arithmetic expressions, which we denote the Calculator domain.

Original: We considered common synthetic data generation strategies including one from tensor2tensor (Vaswani et al., 2018), an open-source deep learning library.
Revised: We considered common synthetic data generation strategies including one from tensor2tensor (Vaswani et al., 2018), an open-source deep learning library.

Original: Upon analysis, we ﬁnd evidence of undesirable artifacts resulting from certain biases in the generation algorithm.
Revised: Upon analysis, we ﬁnd evidence of undesirable artifacts resulting from certain biases in the generation algorithm.

Original: Our results indicate that models trained with common methodologies for synthesizing datasets fail to learn the full semantics of the DSL, even when they perform well on a test set, and suggest the need of a more principled way to generate synthetic datasets.
Revised: Our results indicate that models trained with common methodologies for synthesizing datasets fail to learn the full semantics of the DSL, even when they perform well on a test set, and suggest the need of a more principled way to generate synthetic datasets.

Original: For some program and input distributions, the state-of-the-art neural synthesis models perform quite poorly, often achieving less than 5% generalization accuracy.
Revised: For some program and input distributions, the state-of-the-art neural synthesis models perform quite poorly, often achieving less than 5% generalization accuracy.

Original: In our paper, we develop a new methodology for creating training distributions over programs in the DSL to mitigate some of these issues.
Revised: In our paper, we develop a new methodology for creating training distributions over programs in the DSL to mitigate some of these issues.

Original: Moreover, unlike previous works that have ignored considering the distributions over input space, we show that input distributions also play a signiﬁcant role in determining the synthesizer performance.
Revised: Moreover, unlike previous works that have ignored considering the distributions over input space, we show that input distributions also play a signiﬁcant role in determining the synthesizer performance.

Original: Our methodology involves deﬁning the distribution over DSL programs and input space using a set of random variables that encodes much of the valuable features that describe the data, e.g. in the Karel domain, the amount of control ﬂow nesting in programs or the number of markers present in the inputs.
Revised: Our methodology involves deﬁning the distribution over DSL programs and input space using a set of random variables that encodes much of the valuable features that describe the data, e.g. in the Karel domain, the amount of control ﬂow nesting in programs or the number of markers present in the inputs.

Original: Our methodology allows us to identify several specialized distributions over the input space and Karel programs on which the current state of the art synthesis models (Bunel et al., 2018) perform poorly when trained on traditional program and input distributions, and tested on our new distributions.
Revised: Our methodology allows us to identify several specialized distributions over the input space and Karel programs on which the current state of the art synthesis models (Bunel et al., 2018) perform poorly when trained on traditional program and input distributions, and tested on our new distributions.

Original: From this, we design new training distributions by ensuring greater uniformity over the random variables in our methodology.
Revised: From this, we design new training distributions by ensuring greater uniformity over the random variables in our methodology.

Original: By retraining the same architecture on these new training data distributions, we observe a greater ability to generalize, with signiﬁcant improvements when evaluated on the aforementioned test sets.
Revised: By retraining the same architecture on these new training data distributions, we observe a greater ability to generalize, with signiﬁcant improvements when evaluated on the aforementioned test sets.

Original: We also observe similar improvements in the Calculator domain as well.
Revised: We also observe similar improvements in the Calculator domain as well.

Original: This paper makes the following key contributions:
Revised: This paper makes the following key contributions:

Original: • We propose a new methodology to generate different desirable distributions over the space of datasets for program induction and synthesis tasks.
Revised: • We propose a new methodology to generate different desirable distributions over the space of datasets for program induction and synthesis tasks.

Original: • We instantiate the methodology for the Karel and Calculator domains and show that model generalization is worse on datasets generated by our technique.
Revised: • We instantiate the methodology for the Karel and Calculator domains and show that model generalization is worse on datasets generated by our technique.

Original: • We then retrain models in both domains and demonstrate that models achieve greater overall generalization performance when trained on datasets generated with our methodology.
Revised: • We then retrain models in both domains and demonstrate that models achieve greater overall generalization performance when trained on datasets generated with our methodology.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Training models with synthetic data.
Revised: Training models with synthetic data.

Original: In certain domains like computer vision and robotics, collecting high-quality real-world training data incurs signiﬁcant cost, and so many researchers have investigated the use of large amounts of synthetic data.
Revised: In certain domains like computer vision and robotics, collecting high-quality real-world training data incurs signiﬁcant cost, and so many researchers have investigated the use of large amounts of synthetic data.

Original: For example, Christiano et al.
Revised: For example, Christiano et al.

Original: ; Peng et al.
Revised: ; Peng et al.

Original: ; Pinto et al.
Revised: ; Pinto et al.

Original: ; Bousmalis et al.
Revised: ; Bousmalis et al.

Original: (2017) aim to learn robotics policies that compensate for differences between the real world and the simulation.
Revised: (2017) aim to learn robotics policies that compensate for differences between the real world and the simulation.

Original: Within computer vision, Shrivastava et al.
Revised: Within computer vision, Shrivastava et al.

Original: (2016) demonstrate learning from entirely synthetic images for gaze and pose estimation.
Revised: (2016) demonstrate learning from entirely synthetic images for gaze and pose estimation.

Original: Neural program induction and synthesis.
Revised: Neural program induction and synthesis.

Original: Program induction methods learn differentiable modules such as stack (Joulin & Mikolov, 2015), RAM (Kurach et al., 2016), GPU (Kaiser & Sutskever, 2015), and read-write external memory (Graves et al., 2014) to represent algorithms.
Revised: Program induction methods learn differentiable modules such as stack (Joulin & Mikolov, 2015), RAM (Kurach et al., 2016), GPU (Kaiser & Sutskever, 2015), and read-write external memory (Graves et al., 2014) to represent algorithms.

Original: Other methods attempt to learn differentiable control ﬂow operations (Gaunt et al., 2016; Neelakantan et al., 2015).
Revised: Other methods attempt to learn differentiable control ﬂow operations (Gaunt et al., 2016; Neelakantan et al., 2015).

Original: These approaches reconstruct outputs given inputs, inferring the underlying algorithms (Guu et al., 2017).
Revised: These approaches reconstruct outputs given inputs, inferring the underlying algorithms (Guu et al., 2017).

Original: Select different algorithms learn directly from program traces rather than from I/O examples
Revised: Select different algorithms learn directly from program traces rather than from I/O examples

Original: (Reed & de Freitas, 2015; Xiao et al., 2018;
Revised: (Reed & de Freitas, 2015; Xiao et al., 2018;

Original: Cai et al., 2017).
Revised: Cai et al., 2017).

Original: Devlin et al.
Revised: Devlin et al.

Original: ; Parisotto et al.
Revised: ; Parisotto et al.

Original: (2017) use neural program synthesis techniques for learning string editing programs in RobustFill.
Revised: (2017) use neural program synthesis techniques for learning string editing programs in RobustFill.

Original: Similarly, Balog et al.
Revised: Similarly, Balog et al.

Original: (2016) learn array programs in DeepCoder, and Bhupatiraju et al.
Revised: (2016) learn array programs in DeepCoder, and Bhupatiraju et al.

Original: (2017) learn to compose API calls.
Revised: (2017) learn to compose API calls.

Original: Bunel et al.
Revised: Bunel et al.

Original: (2018) apply neural program synthesis to the Karel domain we consider in our paper; we use their architecture and dataset.
Revised: (2018) apply neural program synthesis to the Karel domain we consider in our paper; we use their architecture and dataset.

Original: Many of these approaches report high test performance, but good performance on synthetically-speciﬁced programs need not indicate the model’s ability to generalize to arbitrary user-desired programs.
Revised: Many of these approaches report high test performance, but good performance on synthetically-speciﬁced programs need not indicate the model’s ability to generalize to arbitrary user-desired programs.

Original: In fact, our results show that under certain distributions, these models perform quite poorly.
Revised: In fact, our results show that under certain distributions, these models perform quite poorly.

Original: To verify these models appropriately generalize to reasonably complex arbitrary programs, the test set should sufﬁciently represent the universe of these programs and their speciﬁcations.
Revised: To verify these models appropriately generalize to reasonably complex arbitrary programs, the test set should sufﬁciently represent the universe of these programs and their speciﬁcations.

Original: Likewise, to avoid biasing the result, the test set should also draw uniformly from these programs and speciﬁcations.
Revised: Likewise, to avoid biasing the result, the test set should also draw uniformly from these programs and speciﬁcations.

Original: For example, for RobustFill, the data generation methodology only sampled programs uniformly from the string DSL, but did not take into account the distribution over input strings such as their length, frequencies of occurrence of regular expressions and their nesting, common words and constants etc.[Figure]
Revised: For example, for RobustFill, the data generation methodology only sampled programs uniformly from the string DSL, but did not take into account the distribution over input strings such as their length, frequencies of occurrence of regular expressions and their nesting, common words and constants etc.

Original:  O UR D ATA G ENERATION M ETHODOLOGY
Revised:  O UR D ATA G ENERATION M ETHODOLOGY

Original: Currently, automated data generation focuses in large part on a constructive process, whose parameters can be tuned.
Revised: Currently, automated data generation focuses in large part on a constructive process, whose parameters can be tuned.

Original: We propose a complementary approach in which we perform post-selection on this data to ensure certain properties.
Revised: We propose a complementary approach in which we perform post-selection on this data to ensure certain properties.

Original: We deﬁne a salient random variable as one whose distribution in the ﬁnal dataset is of interest.
Revised: We deﬁne a salient random variable as one whose distribution in the ﬁnal dataset is of interest.

Original: In the case of program synthesis, there are two kinds of salient variables: X = { X 1 , . . . , X n } and Z = { Z 1 , . . , Z m } , where the random variable X i denotes a certain important feature of a program in the DSL D and the random variable Z j denotes a feature for the input space.
Revised: In the case of program synthesis, we consider two kinds of salient variables: variables denoting important features of a program in the given DSL, such as its length and degree of nesting; and variables denoting features for the input space.

Original: In many cases, we can modify our sampling procedure to ensure a desirable distribution of a particular salient variable.
Revised: In many cases, we can modify our sampling procedure to ensure a desirable distribution of a particular salient variable.

Original: However, for some salient variables, it is infeasible to tune the parameters of a given sampling procedure in order to obtain a desired distribution for that salient variable.
Revised: However, for some salient variables, it is infeasible to tune the parameters of a given sampling procedure in order to obtain a desired distribution for that salient variable.

Original: For example, if we sample programs directly from a context-free grammar, we can end up biasing various salient variables such as program length, degree of nesting, etc.
Revised: For example, if we sample programs directly from a context-free grammar, it is difﬁcult to control the distribution of various salient variables such as program length, degree of nesting, etc.

Original: This is a notable problem in both the Karel and Calculator domains.
Revised: This is a notable problem in both the Karel and Calculator domains.

Original: Furthermore, within the context of program synthesis speciﬁcally, there is often an additional challenge: not all inputs are valid for all programs.
Revised: Furthermore, within the context of program synthesis speciﬁcally, there is often an additional challenge: not all inputs are valid for all programs.

Original: For example, in the Karel domain, Karel is not allowed to  move into walls or pickMarker in a cell containing no markers.
Revised: For example, in the Karel domain, an input for a given program would be invalid if the program attempts to perform illegal actions for that input (such as move into walls or pickMarker in a cell containing no markers).

Original: The requirement that the program/input pairs suit each other itself acts as an unpredictable post-selector that makes it difﬁcult to ensure uniformity of salient variables by tuning generation parameters.
Revised: The requirement that the program/input pairs suit each other itself acts as an unpredictable post-selector that makes it difﬁcult to ensure uniformity of salient variables by tuning generation parameters.

Original: Our proposed solution is an algorithm for  samplingfrom a given distribution p while homogenizing (ensuring the uniformity of) a certain salient variable  X  .
Revised: As such, we propose a methodology for randomly sampling a dataset D (consisting of elements of S ) while ensuring that a given salient variable ν : S → X (where X is ﬁnite and discrete), denoted as a random variable X = ν ( s ) , has a uniform distribution throughout D .

Original: Our process is as such: we produce a sample  s ∼ p ( S ) and then reject with probability1 − g ( s ) , where
Revised: To do this, we ﬁrst sample an example s ∼ q ( · ) from an original distribution q . We then add s to D with probability g ( s ) , where

Original: [Equation]
Revised: [Equation]

Original: with  the probabilities  calculated empirically based on  past samples.
Revised: with P q [ X ] , the probabilities induced over X via q , calculated empirically based on counts computed with past samples drawn from q . We repeat the above until D is of a desired size.

Original: If we were to set ε = 0 , this would ensure that X ( s ) would be uniform in the resulting distribution (proof in the Appendix, Section 8.2). Unfortunately, we would have no guarantees on runtime.
Revised: For full pseudocode see the Appendix, Section 8.2. We use ε ∈ R + as a hyperparameter to trade off the runtime of the above procedure with the level of X ’s uniformity in D . In Section 8.3 (in the Appendix), we provide a probabilistic bound on the uniformity of X in the resulting distribution, for the case where ε = 0 .

Original: If, however, ε is non-zero, we know that drawing a single sample is possible in O ( 1 (cid:15) ) calls to the original sampler (proof in the Appendix, Section 8.3, shown to be a reasonable bound in the Appendix, Section 8.4).
Revised: Also, for when ε > 0 , we can show that drawing a single sample is possible in O ( 1 ε ) calls to the original sampler (proof in the Appendix, Section 8.4, with empirical experiments in the Appendix, Section 8.5).

Original: Increasing ε increases the algorithm’s speed, at the cost of allowing P [ X ] to differ from uniform, reducing performance.
Revised: Increasing ε increases the algorithm’s speed, at the cost of allowing the distribution of X in D to further diverge from uniform.

Original:  A NALYSIS OF E XISTING M ODELS IN THE K AREL D OMAIN
Revised:  A NALYSIS OF E XISTING M ODELS IN THE K AREL D OMAIN

Original: Karel is an educational programming language (Pattis, 1981) where the programmer writes imperative programs with conditionals and loops to produce a sequence of actions for an agent (a robot named
Revised: Karel is an educational programming language (Pattis, 1981) where the programmer writes imperative programs with conditionals and loops to produce a sequence of actions for an agent (a robot named

Original: Karel) which lives in a rectangular m × n grid world.
Revised: Karel) which lives in a rectangular m × n grid world.

Original: For a detailed description of the particular instantiation of the Karel language and input grid speciﬁcation that we consider, see Figure 4 in the
Revised: For a detailed description of the particular instantiation of the Karel language and input grid speciﬁcation that we consider, see Figure 4 in the

Original: Appendix.
Revised: Appendix.

Original: The program synthesis task that we consider is as follows: given a set of pairs of input and output grids { ( i 1 , o 1 ) , · · · , ( i n , o n ) } , ﬁnd a Karel program π such that executing π on i 1 results in o 1 , i 2 results in o 2 , and so on.
Revised: The program synthesis task that we consider is as follows: given a set of pairs of input and output grids { ( i 1 , o 1 ) , · · · , ( i n , o n ) } , ﬁnd a Karel program π such that executing π on i 1 results in o 1 , i 2 results in o 2 , and so on.

Original: An example Karel synthesis task with the I/O examples and the corresponding Karel program to be synthesized is shown in Figure 1.
Revised: An example Karel synthesis task with the I/O examples and the corresponding Karel program to be synthesized is shown in Figure 1.

Original: In this section, we employ the Karel instantiation of our abstract data generation methodology in
Revised: In this section, we employ the Karel instantiation of our abstract data generation methodology in

Original: Section 4.1 to generate different test datasets.
Revised: Section 4.1 to generate different test datasets.

Original: By imposing a more uniform distribution over the salient random variables when generating the I/O speciﬁcations and target programs which make up the test set, we observe much lower accuracies of the previous Karel synthesis models (Bunel et al., 2018) compared to the original test set. 
Revised: By imposing a more uniform distribution over the salient random variables when generating the I/O speciﬁcations and target programs which make up the test set, we observe much lower accuracies of the previous Karel synthesis models (Bunel et al., 2018) compared to the original test set. [Figure]

Original: 4. I NSTANTIATION FOR THE K AREL DOMAIN
Revised: 4. I NSTANTIATION FOR THE K AREL DOMAIN

Original: We devised the following salient random variables to describe the input space in Karel:
Revised: We devised the following salient random variables to describe the input space in Karel (corresponding to variables Z i in Section 3):

Original: • Grid size : Dimensions of the grid in which Karel can act.
Revised: • Grid size : Dimensions of the grid in which Karel can act.

Original: • Marker ratio : Fraction of cells with at least one marker.
Revised: • Marker ratio : Fraction of cells with at least one marker.

Original: • Wall ratio : Fraction of cells which contain a wall.
Revised: • Wall ratio : Fraction of cells which contain a wall.

Original: • Marker count : Number of markers that are present in a cell containing markers.
Revised: • Marker count : Number of markers that are present in a cell containing markers.

Original: For the program space in the Karel DSL, we consider the following random variables:
Revised: For the program space in the Karel DSL, we consider the following random variables (corresponding to variables

Revised: X i in Section 3):

Original: • Program size : Size of the program in terms of number of tokens.
Revised: • Program size : Size of the program in terms of number of tokens.

Original: • Control ﬂow ratio : Number of control ﬂow structures appearing in the program.
Revised: • Control ﬂow ratio : Number of control ﬂow structures appearing in the program.

Original: • Nested control ﬂow : The amount of control ﬂow nesting in programs (e g. while inside if).
Revised: • Nested control ﬂow : The amount of control ﬂow nesting in programs (e.g. while inside if).

Original: 4. C HANGING I/O DISTRIBUTIONS
Revised: 4. C HANGING I/O DISTRIBUTIONS

Original: We reproduced the encoder-decoder model of Bunel et al.
Revised: We reproduced the encoder-decoder model of Bunel et al.

Original: and trained it using the provided synthetic training set with the teacher-forcing maximum likelihood objective.
Revised: and trained it using the provided synthetic training set with the teacher-forcing maximum likelihood objective.

Original: On the existing test set, our model achieves 73.52% generalization accuracy, slightly higher than the 71.91% accuracy reported in (Bunel et al., 2018).
Revised: On the existing test set, our model achieves 73.52% generalization accuracy, slightly higher than the 71.91% accuracy reported in (Bunel et al., 2018).

Original: Generalization accuracy denotes how often the model’s output is correct on the 5 I/O examples shown to the model and the remaining held-out 6 th I/O example.
Revised: Generalization accuracy denotes how often the model’s output is correct on the 5 I/O examples shown to the model and the remaining held-out 6 th I/O example.

Original: To test how the model may be sensitive to changes in the I/O examples used to specify the program, we created new test sets by sampling new input grids and running them on each of the programs in the existing test set to obtain new I/O pairs.
Revised: To test how the model may be sensitive to changes in the I/O examples used to specify the program, we created new test sets by sampling new input grids and running them on each of the programs in the existing test set to obtain new I/O pairs.

Original: By keeping the programs themselves the same, we avoid inadvertent changes in the inherent difﬁculty of the task.
Revised: By keeping the programs themselves the same, we avoid inadvertent changes in the inherent difﬁculty of the task.

Original: 4.2. S ALIENT RANDOM VARIABLES WITH UNIFORM DISTRIBUTION
Revised: 4.2. S ALIENT RANDOM VARIABLES WITH UNIFORM DISTRIBUTION

Original: We ﬁrst generated grids such that they would follow a distribution that is as uniform as possible in the salient features in Section 4.1.
Revised: We ﬁrst generated grids such that they would follow a distribution that is as uniform as possible in the salient features in Section 4.1.

Original: We used the following procedure to sample each grid: 1) sample the grid size (height and width) from x, y ∼ U{ 2 , . . .
Revised: We used the following procedure to sample each grid: 1) sample the grid size (height and width) from x, y ∼ U{ 2 , . . .

Original: 16 } ; 2) sample the marker ratio r marker ∼ U (0 , 1) and wall ratio r wall ∼ U (0 , 1) ; 3) for each cell ( i, j ) , 0 ≤ i < x, 0 ≤ j < y in the grid, sample m i,j ∼ Bernoulli ( r marker ) and w i,j ∼ Bernoulli ( r wall ) ; 4) if m i,j = 1 and w i,j = 0 ,
Revised: 16 } ; 2) sample the marker ratio r marker ∼ U (0 , 1) and wall ratio r wall ∼ U (0 , 1) ; 3) for each cell ( i, j ) , 0 ≤ i < x, 0 ≤ j < y in the grid, sample m i,j ∼ Bernoulli ( r marker ) and w i,j ∼ Bernoulli ( r wall ) ; 4) if m i,j = 1 and w i,j = 0 ,

Original: sample marker count mc i,j ∼ U{ 1 , . . .
Revised: sample marker count mc i,j ∼ U{ 1 , . . .

Original: 9 } , otherwise set mc i,j = 0 ; 5) place walls and markers in grid according to w i,j and mc i,j ; 6) place Karel at a random location (not containing a wall) and with a random orientation.
Revised: 9 } , otherwise set mc i,j = 0 ; 5) place walls and markers in grid according to w i,j and mc i,j ; 6) place Karel at a random location (not containing a wall) and with a random orientation.

Original: After generating 5 input grids for a given program, we ensure that the program does not crash on any of them and also check whether the 5 input grids exhibit complete branch coverage .
Revised: After generating 5 input grids for a given program, we ensure that the program does not crash on any of them and also check whether the 5 input grids exhibit complete branch coverage .

Original: If either of these conditions are not satisﬁed, we discard all 5 grids and start over. 
Revised: If either of these conditions are not satisﬁed, we discard all 5 grids and start over. [Table]

Original:  On this dataset, the model trained on existing data achieved generalization accuracy of only 27.9%, which was a drop of 44.6pp from the existing test set’s generalization accuracy of 73.52%.
Revised: [Table] On this dataset [Table] the model trained on existing data achieved generalization accuracy of only 27.9%, which was a drop of 44.6pp from the existing test set’s generalization accuracy of 73.52%.

Original: 4.2. S ALIENT RANDOM VARIABLES WITH NARROW DISTRIBUTIONS
Revised: 4.2. S ALIENT RANDOM VARIABLES WITH NARROW DISTRIBUTIONS

Original: We further investigated the performance drop noted above by synthesizing “narrower" datasets that captured different parts of the joint probability space over the salient input random variables.
Revised: We further investigated the performance drop noted above by synthesizing “narrower" datasets that captured different parts of the joint probability space over the salient input random variables.

Original: For each narrow dataset, we selected r wall and r marker (both between 0 and 1 ) as well as a distribution[Table] [Table]
Revised: For each narrow dataset, we selected r wall and r marker (both between 0 and 1 ) as well as a distribution

Original: D [Table] which would be the same for all I/O grids.
Revised: D marker count which would be the same for all I/O grids.

Original: Then, we follow the procedure below for each grid: 1) sample the grid size (height and width) x, y ∼ U{ 10 , . . .
Revised: Then, we follow the procedure below for each grid: 1) sample the grid size (height and width) x, y ∼ U{ 10 , . . .

Original: 16 } ; 2) randomly choose xy · r wall cells to contain walls, and xy · r marker cells for markers; 3) sample mc i,j ∼ D marker count for all cells ( i, j ) chosen to contain markers; 4) place Karel at a random location (not containing a wall) and with a random orientation.
Revised: 16 } ; 2) randomly choose xy · r wall cells to contain walls, and xy · r marker cells for markers; 3) sample mc i,j ∼ D marker count for all cells ( i, j ) chosen to contain markers; 4) place Karel at a random location (not containing a wall) and with a random orientation.

Original: In our experiments, we primarily used 3 different distributions for D marker count :
Revised: In our experiments, we primarily used 3 different distributions for D marker count :

Original: Geom (0 . 5) truncated at 9, U{ 1 , . . .
Revised: Geom (0 . 5) truncated at 9, U{ 1 , . . .

Original: 9 } , and 10 − Geom (0 . 5) which, when sampled, has a value equal to 10 minus a sample from Geom(0.5), truncated at 1.
Revised: 9 } , and 10 − Geom (0 . 5) which, when sampled, has a value equal to 10 minus a sample from Geom(0.5), truncated at 1.

Original: The results are shown in Table 1 (row 1, “Baseline (%)”).
Revised: The results are shown in Table 1 (row 1, “Baseline (%)”).

Original: We discovered that the most correlated factor with model performance was the distribution D marker count .
Revised: We discovered that the most correlated factor with model performance was the distribution D marker count .

Original: A more negative skew consistently lowered model performance, and this effect was more pronounced at higher values of r marker .
Revised: A more negative skew consistently lowered model performance, and this effect was more pronounced at higher values of r marker .

Original: 4. C HANGES IN THE DISTRIBUTION OF PROGRAMS
Revised: 4. C HANGES IN THE DISTRIBUTION OF PROGRAMS

Original: We will now examine how the existing model can surprisingly fail to perform well at synthesizing certain programs that are different from those in the existing validation and test sets.
Revised: We will now examine how the existing model can surprisingly fail to perform well at synthesizing certain programs that are different from those in the existing validation and test sets.

Original: Performance on complex DSL constructs.
Revised: Performance on complex DSL constructs.

Original: We examined whether or not the model could succeed in synthesizing programs which require nesting of conditional constructs.
Revised: We examined whether or not the model could succeed in synthesizing programs which require nesting of conditional constructs.

Original: This was of interest since these programs were relatively rare in the training dataset.
Revised: This was of interest since these programs were relatively rare in the training dataset.

Original: We obtained approximations for r wall , r marker and D marker count for the provided training set, and subsequently generated an evaluation dataset comprised solely of programs that contained while inside while statements, and another dataset in which all programs had while inside if statements.
Revised: We obtained approximations for r wall , r marker and D marker count for the provided training set, and subsequently generated an evaluation dataset comprised solely of programs that contained while inside while statements, and another dataset in which all programs had while inside if statements.

Original: We found that the model fared very poorly on these datasets, achieving only 0.64% and 2.23% accuracy respectively.
Revised: We found that the model fared very poorly on these datasets, achieving only 0. and 2. % accuracy respectively

Original: Programs only containing actions.
Revised: Programs only containing actions.

Original: Intuitively, much of the difﬁculty in the Karel program synthesis task should come from inferring the control ﬂow statements, i.e. if , ifElse , and while .
Revised: Intuitively, much of the difﬁculty in the Karel program synthesis task should come from inferring the control ﬂow statements, i.e. if , ifElse , and while .

Original: Synthesizing a Karel program that only contains actions is intrinsically a much more straightforward task, which a relatively simple search algorithm (such as A * ) can perform well. 
Revised: Synthesizing a Karel program that only contains actions is intrinsically a much more straightforward task, which a relatively simple search algorithm (such as A * ) can perform well. [Figure]

Original: We performed an experiment using test datasets generated by enumerating action-only programs of various lengths.
Revised: We performed an experiment using test datasets generated by enumerating action-only programs of various lengths.

Original: As there are ﬁve actions ( move , turnLeft , turnRight , putMarker , pickMarker ), there exist 5 L textually unique action-only programs for length L .
Revised: As there are ﬁve actions ( move , turnLeft , turnRight , putMarker , pickMarker ), there exist 5 L textually unique action-only programs for length L .

Original: We sampled up to 500 unique programs of lengths 1 , 2 , . . .
Revised: We sampled up to 500 unique programs of lengths 1 , 2 , . . .

Original: For each program, we generated 10 speciﬁcations, each containing 5 I/O pairs.
Revised: For each program, we generated 10 speciﬁcations, each containing 5 I/O pairs.

Original: We sampled each I/O pair from the set of all input grids in the existing training data (of which there are 6.7 million), as to match its distribution as closely as possible.
Revised: We sampled each I/O pair from the set of all input grids in the existing training data (of which there are 6.7 million), as to match its distribution as closely as possible.

Original: Table 2 shows the results.
Revised: Table 2 shows the results.

Original: Remarkably, even though the underlying programs have relatively low complexity, the model’s accuracy is lower on every one of these action-only test sets than the existing[Figure] provided test set.
Revised: Remarkably, even though the underlying programs have relatively low complexity, the model’s accuracy is lower on every one of these action-only test sets than the existing provided test set.

Original: The generalization accuracy grows as the program length becomes longer, even though those programs should be harder to synthesize.
Revised: The generalization accuracy grows as the program length becomes longer, even though those programs should be harder to synthesize.

Original: Among the existing action-only programs in the test set, the model’s generalization accuracy on that subset is 99.24%.
Revised: Among the existing action-only programs in the test set, the model’s generalization accuracy on that subset is 99.24%.

Original: Given the surprising nature of this result, we investigated the difference between the action-only programs we generated, and those in the existing test set.
Revised: Given the surprising nature of this result, we investigated the difference between the action-only programs we generated, and those in the existing test set.

Original: We found that in the existing training and test sets, all programs contain at least two actions, and also contain at least one move action somewhere in the program.
Revised: We found that in the existing training and test sets, all programs contain at least two actions, and also contain at least one move action somewhere in the program.

Original: These and any undiscovered differences in the distribution of programs seem to have caused the gap in performance.
Revised: These and any undiscovered differences in the distribution of programs seem to have caused the gap in performance.

Original:  T RAINING THE K AREL M ODEL ON N EW D ATASETS
Revised:  T RAINING THE K AREL M ODEL ON N EW D ATASETS

Original: In Section 4, we saw that the existing model performs much more poorly on certain test datasets that we constructed, compared to its performance on the existing test set as reported in Section 4.2.
Revised: In Section 4, we saw that the existing model performs much more poorly on certain test datasets that we constructed, compared to its performance on the existing test set as reported in Section 4.2.

Original: In light of the framework in Section 3, various imbalances of the salient random variables in the existing training data could have caused these gaps in performance.
Revised: In light of the framework in Section 3, various imbalances of the salient random variables in the existing training data could have caused these gaps in performance.

Original: Then a natural solution is to to train using datasets constructed to avoid undesirable skews in the salient random variables, which should hopefully perform better across a variety of distributions.
Revised: Then a natural solution is to to train using datasets constructed to avoid undesirable skews in the salient random variables, which should hopefully perform better across a variety of distributions.

Original: Training datasets with uniform I/O. We generated a training dataset by taking the programs of the existing training set and synthesizing I/O pairs using the procedure described in Section 4.2.1.
Revised: Training datasets with uniform I/O. We generated a training dataset by taking the programs of the existing training set and synthesizing I/O pairs using the procedure described in Section 4.2.1.

Original: We trained a model on this data and then evaluated it on the same set of narrow distribution evaluation datasets as mentioned in Section 4.2.2.
Revised: We trained a model on this data and then evaluated it on the same set of narrow distribution evaluation datasets as mentioned in Section 4.2.2.

Original: Figure 2 compares how this new model performs to the baseline model.
Revised: Figure 2 compares how this new model performs to the baseline model.

Original: The model trained on uniform I/O distributions maintains much higher inference accuracy on the test sets of Section 4.2.2 than the baseline model.
Revised: The model trained on uniform I/O distributions maintains much higher inference accuracy on the test sets of Section 4.2.2 than the baseline model.

Original: Note that the uniform I/
Revised: Note that the uniform I/

Original: O distribution is not simply a union of the tested distributions and is intended to cover all possible input speciﬁcations.
Revised: O distribution is not simply a union of the tested distributions and is intended to cover all possible input speciﬁcations.

Original: Real-world benchmarks.
Revised: Real-world benchmarks.

Original: We evaluated both the baseline model and the uniform model  on a set of 36 real-world Karel programming problems.
Revised: We evaluated both the baseline model, and the uniform model described in the previous paragraph, on a set of 36 real-world Karel programming problems.

Original: This dataset was compiled from the Hour of Code Initiative and Stanford University’s introductory computer science course, CS106A, with the problems being hand-designed as educational exercises for students.
Revised: This dataset was compiled from the Hour of Code Initiative and Stanford University’s introductory computer science course, CS106A, with the problems being hand-designed as educational exercises for students.

Original: We found that the baseline model got 4 correct (11.1%) while the uniform model got 7 correct (19.4%).
Revised: We found that the baseline model got 4 correct (11.1%) while the uniform model got 7 correct (19.4%).

Original: This further demonstrates the uniform model’s increased ability to generalize to out-of-distribution datasets, including those which are of interest to humans.
Revised: This further demonstrates the uniform model’s increased ability to generalize to out-of-distribution datasets, including those which are of interest to humans.

Revised: Both models’ accuracies are still low compared to the performance on the synthetically generated test set.

Revised: After further analysis, we believe the models face two challenges on the real-world test set: (1) many of the real-world problems require

Revised: [Figure]

Revised: long programs to solve compared to the synthetic test set, (2) the speciﬁcations in the real-world examples always contains fewer than 5 I/O pairs, and often only a single one.

Revised: However, the training methodology for the model assumes that it is provided with a diverse set of 5 I/O pairs.

Original: Adding action-only programs.
Revised: Adding action-only programs.

Original: We observed in Section 4.3 that the model fails to do well on either action-only programs or programs with many control-ﬂow statements.
Revised: We observed in Section 4.3 that the model fails to do well on either action-only programs or programs with many control-ﬂow statements.

Original: In the case of action-only programs, we found that the training data had been pruned to only include programs with at least two actions and at least one move , and in the case of programs with complex control ﬂow, we found a similar sparsity in the train set.
Revised: In the case of action-only programs, we found that the training data had been pruned to only include programs with at least two actions and at least one move , and in the case of programs with complex control ﬂow, we found a similar sparsity in the train set.

Original: As discussed in Section 3, the principled way to counteract this sparsity is to introduce uniformity into a set of salient variables.
Revised: As discussed in Section 3, the principled way to counteract this sparsity is to introduce uniformity into a set of salient variables.

Original: This methodology allows us to counteract both naturally sparse data[Figure] (such as complex control-ﬂow) and spurious data pre-processing (such as enforcing programs to have at least two actions).
Revised: This methodology allows us to counteract both naturally sparse data (such as complex control-ﬂow) and spurious data pre-processing (such as enforcing programs to have at least two actions).

Original: In our case, we introduce uniformity into the length of action-only programs by synthesizing 20, programs of each length 1 to 20, by uniformly selecting tokens from the ﬁve action choices and generating I/O with other salient random variables as close to the original training set as possible; we append these new programs to the original dataset to train a new model.
Revised: In our case, we introduce uniformity into the length of action-only programs by synthesizing 20, programs of each length 1 to 20, by uniformly selecting tokens from the ﬁve action choices and generating I/O with other salient random variables as close to the original training set as possible; we append these new programs to the original dataset to train a new model.

Original: Table 2 shows a clear improvement when homogenizing this salient random variable.
Revised: Table 2 shows a clear improvement when homogenizing this salient random variable.

Original: Narrowly speciﬁed training datasets.
Revised: Narrowly speciﬁed training datasets.

Original: As done in for the uniform training dataset, we generated “narrow” training datasets by keeping the same programs as in the existing training data and replacing the I/O pairs with the process from Section 4.2.2.
Revised: As done in for the uniform training dataset, we generated “narrow” training datasets by keeping the same programs as in the existing training data and replacing the I/O pairs with the process from Section 4.2.2.

Original: We trained a variety of models and evaluated them on 12 datasets of different I/O feature distributions.
Revised: We trained a variety of models and evaluated them on 12 datasets of different I/O feature distributions.

Original: Figure 5 summarizes the results of evaluating each model on each dataset by noting the performance of the model on the narrow dataset of the same type and the outcome on every other narrow dataset.
Revised: Figure 5 summarizes the results of evaluating each model on each dataset by noting the performance of the model on the narrow dataset of the same type and the outcome on every other narrow dataset.

Original: For the models trained on the low variance datasets, we observed that they all consistently achieved between 60 and 70 percent accuracy on their own training distribution; however, the uniform model was able to achieve similar performance (between 57 and 80 percent accuracy).
Revised: For the models trained on the low variance datasets, we observed that they all consistently achieved between 60 and 70 percent accuracy on their own training distribution; however, the uniform model was able to achieve similar performance (between 57 and 80 percent accuracy).

Original: Hence, we conclude that models can be trained to perform well on out-of-distribution data (e.g., the uniform model), while still achieving comparable performance on narrow sub-distributions, as measured against models explicitly trained on such distributions.
Revised: Hence, we conclude that models can be trained to perform well on out-of-distribution data (e.g., the uniform model), while still achieving comparable performance on narrow sub-distributions, as measured against models explicitly trained on such distributions.

Original:  C ALCULATOR
Revised:  C ALCULATOR

Original: The Calculator task is given as follows: given an expression such as "5+4*(2+3)" , compute the result mod 10; in this case, 5.
Revised: The Calculator task is given as follows: given an expression such as "5+4*(2+3)" , compute the result mod 10; in this case, 5.

Original: We are interested in the Calculator domain because, while Calculator is not a program synthesis task, creating data for the Calculator problem involves sampling from a context-free grammar.
Revised: Calculator is a program induction task rather than a program synthesis task like Karel; nevertheless, creating data for the Calculator problem involves sampling from a context-free grammar.

Original: Additionally, Calculator is not as intricate a domain as Karel and thus we can more completely control the environment of data generation with less fear of lurking variables.
Revised: Additionally, Calculator is not as intricate a domain as Karel and thus we can more completely control the environment of data generation with less fear of lurking variables.

Original: Calculator Model.
Revised: Calculator Model.

Original: Similar to the work by Zaremba & Sutskever (2014), we implement an LSTM that parses calculator expressions on a character level.
Revised: Similar to the work by Zaremba & Sutskever (2014), we implement an LSTM that parses calculator expressions on a character level.

Original: We perform a 10-class classiﬁcation problem using a dense network on the ﬁnal hidden state of the LSTM.
Revised: We perform a 10-class classiﬁcation problem using a dense network on the ﬁnal hidden state of the LSTM.

Original: The prediction is correct if it exactly matches the parsed expression value, mod 10. 
Revised: The prediction is correct if it exactly matches the parsed expression value, mod 10. [Table]

Original: Distributions of Calculator tasks.
Revised: Distributions of Calculator tasks.

Original: We propose 4 distributions for calculator tasks: direct CFG sampling (DCFG), tensor2tensor sampling (T2T), “runs” CFG sampling (RCFG), and balanced sampling (BAL).
Revised: We propose 4 distributions for calculator tasks: direct CFG sampling (DCFG), tensor2tensor sampling (T2T), “runs” CFG sampling (RCFG), and balanced sampling (BAL).

Original: Two of our distributions represent reasonable ways in which a researcher might choose to sample data.
Revised: Two of our distributions represent reasonable ways in which a researcher might choose to sample data.

Original: The ﬁrst is DCFG sampling, which involves returning a digit with some probability (1 − p ) , or else recursively sampling two productions and combining them with a − , ∗ , or + , each with probability p 3 .
Revised: The ﬁrst is DCFG sampling, which involves returning a digit with some probability (1 − p ) , or else recursively sampling two productions and combining them with a − , ∗ , or + , each with probability p 3 .

Original: This corresponds to a direct, weighted, sampling of the CFG for the calculator grammar.
Revised: This corresponds to a direct, weighted, sampling of the CFG for the calculator grammar.

Original: The second is T2T sampling 1 , which is used by the tensor2tensor library to sample arithmetic expressions in one[Table] of its examples.
Revised: The second is T2T sampling 1 , which is used by the tensor2tensor library to sample arithmetic expressions in one of its examples.

Original: It involves sampling a depth d , then ensuring that the resulting AST has depth d by forcing a random side of the operation production to be sampled to d − 1 and the other side to be sampled to a depth d (cid:48) ∼ U{ 0 , 1 , . . .
Revised: It involves sampling a depth d , then ensuring that the resulting AST has depth d by forcing a random side of the operation production to be sampled to d − 1 and the other side to be sampled to a depth d (cid:48) ∼ U{ 0 , 1 , . . .

Original: , d − 1 } .
Revised: , d − 1 } .

Original: The other two distributions represent potentially difﬁcult or nonstandard problems that might appear in practical environments.
Revised: The other two distributions represent potentially difﬁcult or nonstandard problems that might appear in practical environments.

Original: RCFG is similar to DCFG sampling but involves increasing the frequency of “runs” of the associative operations picking 2, 3, or 4 productions and then combining them with the given symbol.
Revised: RCFG is similar to DCFG sampling but involves increasing the frequency of “runs” of the associative operations picking 2, 3, or 4 productions and then combining them with the given symbol.

Original: Balanced sampling involves selecting a depth and then creating an AST that is a balanced binary tree at that depth.
Revised: Balanced sampling involves selecting a depth and then creating an AST that is a balanced binary tree at that depth.

Original: Importantly, regardless of sampling technique, redundant parentheses are removed.
Revised: Importantly, regardless of sampling technique, redundant parentheses are removed.

Original: This is to increase the difﬁculty somewhat as order of operations needs to be established.
Revised: This is to increase the difﬁculty somewhat as order of operations needs to be established.

Original: 6. S ALIENT V ARIABLES AND M ETHODOLOGY
Revised: 6. S ALIENT V ARIABLES AND M ETHODOLOGY

Original: We use the following salient variables: length (rounded to the nearest even number), number of operations, number of pairs of parentheses, mean parenthesized depth, and maximum parenthesized depth.
Revised: We use the following salient variables: length (rounded to the nearest even number), number of operations, number of pairs of parentheses, mean parenthesized depth, and maximum parenthesized depth.

Original: Parenthesized depth is deﬁned for each digit and refers to the number of nested parentheses it is in.
Revised: Parenthesized depth is deﬁned for each digit and refers to the number of nested parentheses it is in.

Original: For example in (1+2)*(3-4)+5 , the 1, 2, 3, and 4 are at depth 1 while the 5 is at depth 0.
Revised: For example in (1+2)*(3-4)+5 , the 1, 2, 3, and 4 are at depth 1 while the 5 is at depth 0.

Original: We constructed 2 × (1 + 5) distributions in total, corresponding to a total of 2 task distributions, T2T and DCFG, which represent the “natural” sampling techniques a researcher might employ, and 1 +homogenization strategies, one unhomogenized and ﬁve homogenized corresponding to each salient variable with ε = 0 .
Revised: We constructed 2 × (1 + 5) distributions in total, corresponding to a total of 2 task distributions, T2T and DCFG, which represent the “natural” sampling techniques a researcher might employ, and 1 +homogenization strategies, one unhomogenized and ﬁve homogenized corresponding to each salient variable with ε = 0 .

Original: We then evaluated each model on a fresh evaluation set sampled from a mixture of the four unhomogenized distributions (T2T, DCFG, RCFG, BAL).
Revised: We then evaluated each model on a fresh evaluation set sampled from a mixture of the four unhomogenized distributions (T2T, DCFG, RCFG, BAL).

Original: 6. R ESULTS
Revised: 6. R ESULTS

Original: The original performances and improvements created by homogenizing different random variables can be found in Table 3.
Revised: The original performances and improvements created by homogenizing different random variables can be found in Table 5.

Original: On average, homogenizing the DCFG and T2T distributions caused them to increase by 5.00pp and 2.84pp, respectively.
Revised: On average, homogenizing the DCFG and T2T distributions caused them to increase by 5.00pp and 2.84pp, respectively.

Original: We note that the Calculator domain is much simpler than Karel when considering both input complexity (grid worlds versus arithmetic expressions) and output complexity (a DSL program versus a single digit).
Revised: We note that the Calculator domain is much simpler than Karel when considering both input complexity (grid worlds versus arithmetic expressions) and output complexity (a DSL program versus a single digit).

Original: Furthermore, the difference in distributions between the naive sampling approaches and the versions with one homogenized random variable are not as different in Calculator as what we observed in Karel (see Table 1 for the dramatic effect of D marker ).
Revised: Furthermore, the difference in distributions between the naive sampling approaches and the versions with one homogenized random variable are not as different in Calculator as what we observed in Karel (see Table 1 for the dramatic effect of D marker ).

Original: We hypothesize that it is this difference in complexity that explains the smaller (but still consistent) effect of homogenizing salient random variables in Calculator as compared to in Karel.
Revised: We hypothesize that it is this difference in complexity that explains the smaller (but still consistent) effect of homogenizing salient random variables in Calculator as compared to in Karel.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We demonstrate that uniform sampling, a widely accepted method for randomly generating inputoutput examples, has unintended and overlooked distribution ﬂaws in both the Calculator and the
Revised: We demonstrate that uniform sampling, a widely accepted method for randomly generating inputoutput examples, has unintended and overlooked distribution ﬂaws in both the Calculator and the

Original: Karel domain.
Revised: Karel domain.

Original: These ﬂaws prevent predictive models from generalizing to many simple distributions.
Revised: These ﬂaws prevent predictive models from generalizing to many simple distributions.

Original: To resolve these problems, we propose a robust strategy for controlling and evaluating the bias of synthetic data distributions over programs and speciﬁcations by homogenizing salient random variables, such as the number of parentheses in a calculator expression.
Revised: To resolve these problems we propose a robust strategy for controlling and evaluating the bias of synthetic data distributions over programs and speciﬁcations by homogenizing salient random variables, such as the number of parentheses in a calculator expression.

Original: Equipped with our method, deep networks exhibit an increase in cross-distribution test accuracy, at the expense of a minor decrease in on-distribution test accuracy.
Revised: Equipped with our method, deep networks exhibit an increase in cross-distribution test accuracy, at the expense of a minor decrease in on-distribution test accuracy.

Original: We choose program synthesis as a convenient domain, but any domain with synthetically generated data would beneﬁt well from homogenization of salient variables.
Revised: We choose program synthesis as a convenient domain, but any domain with synthetically generated data would beneﬁt well from homogenization of salient variables.

Original:  A PPENDIX 8.T HE K AREL D OMAIN 
Revised:  A PPENDIX 8.

Revised: T HE K AREL D OMAIN

Original: Prog p :=
Revised: Prog p :=

Original: def main(): s
Revised: def main(): s

Original: Stmt s := while ( b ) : s | repeat ( r ) : s | s 1 ; s| a | if ( b ) : s | if ( b ) : s 1 else : s
Revised: Stmt s := while ( b ) : s | repeat ( r ) : s | s 1 ; s| a | if ( b ) : s | if ( b ) : s 1 else : s

Original: Cond b := markersPresent() | leftIsClear() | rightIsClear()
Revised: Cond b := markersPresent() | leftIsClear() | rightIsClear()

Original: | frontIsClear() | not ( b )
Revised: | frontIsClear() | not ( b )

Original: Action a :=
Revised: Action a :=

Original: move() | turnLeft()
Revised: move() | turnLeft()

Original: | turnRight() | pickMarker() | putMarker()
Revised: | turnRight() | pickMarker() | putMarker()

Original: Cste r := 

Revised: Cste r :=

Original: | 1 | · · · |
Revised: | 1 | · · · |

Original: GridWidth : m
Revised: GridWidth : m

Original: GridHeight : n
Revised: GridHeight : n

Original: Markers : { ( i, j, k ) l } l
Revised: Markers : { ( i, j, k ) l } l

Original: Walls : { ( i, j ) t } t
Revised: Walls : { ( i, j ) t } t

Original: KarelLoc : ( i, j )
Revised: KarelLoc : ( i, j )

Original: Orientation : d ∈ { N,S,E,W } 2 ≤ m, n ≤ 16; i ≤ m ; j ≤ n ; 1 ≤ k ≤(
Revised: Orientation : d ∈ { N,S,E,W } 2 ≤ m, n ≤ 16; i ≤ m ; j ≤ n ; 1 ≤ k ≤(

Original: [Figure]
Revised: [Figure]

Original: [Figure] A declarative speciﬁcation of the space of valid input worlds to Karel programs is shown in Figure 4(b).
Revised: ) (b) A declarative speciﬁcation of the space of valid input worlds to Karel programs is shown in Figure 4(b).

Original: As with Bunel et al.
Revised: As with Bunel et al.

Original: (2018), we assume a bound on the input grid size to be 2 ≤ m, n ≤ 16 .
Revised: (2018), we assume a bound on the input grid size to be 2 ≤ m, n ≤ 16 .

Original: Each cell ( i, j ) in a grid can either be empty, contain an obstacle (i.e. a wall, speciﬁed by the list Walls ), or contain k ≤ 9 markers (deﬁned using the list Markers ).
Revised: Each cell ( i, j ) in a grid can either be empty, contain an obstacle (i.e. a wall, speciﬁed by the list Walls ), or contain k ≤ 9 markers (deﬁned using the list Markers ).

Original: The agent starts at some cell denoted by
Revised: The agent starts at some cell denoted by

Original: KarelLoc in the grid (which may contain markers but no obstacle) with a particular orientation direction denoted by Orientation .
Revised: KarelLoc in the grid (which may contain markers but no obstacle) with a particular orientation direction denoted by Orientation .

Original: The grammar for the Karel DSL we consider in this work is shown in Figure 4(a).
Revised: The grammar for the Karel DSL we consider in this work is shown in Figure 4(a).

Original: The DSL allows the
Revised: The DSL allows the

Original: Karel agent to perform a move action to move one step in the grid in the direction of the orientation, actions turnLeft and turnRight to change its orientation direction, and actions pickMarker and putMarker to manipulate markers.
Revised: Karel agent to perform a move action to move one step in the grid in the direction of the orientation, actions turnLeft and turnRight to change its orientation direction, and actions pickMarker and putMarker to manipulate markers.

Original: The language contains if , ifElse , while constructs with conditionals {front,left,right}IsClear , markersPresent , and their negations.
Revised: The language contains if , ifElse , while constructs with conditionals {front,left,right}IsClear , markersPresent , and their negations.

Original: The repeat construct allows for a ﬁxed number of repetitions.
Revised: The repeat construct allows for a ﬁxed number of repetitions.

Original: Note that the language does not contain any variables or auxiliary functions.
Revised: Note that the language does not contain any variables or auxiliary functions.

Original: 8. S ALIENT V ARIABLE H OMOGENIZATION : P ROOF OF C ORRECTNESS
Revised: 8. S ALIENT V ARIABLE H OMOGENIZATION : A LGORITHM

Revised: Let S be some space that is sampled by some original distribution q : S → [0 , 1] .

Revised: Let X be the space of a salient variable which is calculated by ν : S → X ; it is a ﬁnite set.

Revised: Let ε be our tolerance.

Revised: procedure S AMPLED H OMOGENIZE ( q : S → [0 , 1] , ν : S → X , ε ∈ R , n ∈ N )

Revised: C ← { x → 0 : x ∈ X } (cid:46) Set up a dictionary of counts of each value of X t ←(cid:46)

Revised: The total number of samples seen

Revised: D ← {} (cid:46) D is originally an empty multiset while |D| < n do s ← a sample from q

Revised: C [ ν ( s )]

Original: We wish to show that if ε = 0 , the salient variable analysis technique will always result in a distribution where the homogenized salient variable X is uniform.
Revised: ← C [ ν ( s )] +t ← t +p min ← min x ∈ X C [ x ] t p curr ← C [ ν ( s )] t g ← p min + ε p curr + ε h ← (cid:26) 1 with probability g 0 with probability 1 − g if h = 1 then D ← D ∪ { s } end if end while return D end procedure

Revised: 8. S ALIENT V ARIABLE H OMOGENIZATION : P ROOF OF C ORRECTNESS

Original: Let the initial sampling distribution be q and the resulting distribution be r .
Revised: Let the initial sampling distribution be q and the resulting distribution be r .

Revised: We use the notation

Revised: P r [ X = x ] to refer to the probability that X = x given that S is sampled from the distribution r .

Revised: C [ x ] refers to the count for the salient variable value x ∈ X among past samples from q , as deﬁned in

Revised: S AMPLED H OMOGENIZE .

Revised: Theorem.

Original: Since we are performing rejection sampling, we have that the  probability of selecting any given sample s in our model is
Revised: The Salient Variable Homogenization algorithm produces samples from a distribution close to uniform. Formally, after 48log 2 | X | δ p m | X | 2 ξ 2 samples are drawn from distribution q , we have that the resulting homogenized distribution satisﬁes [Equation] with probability at least 1 − δ , where p m =

Revised: min x ∈ X P q

Revised: [ X = x ] > 0 and ε has been set to 0 .

Revised: Proof.

Revised: We can simplify the probability as

Revised: [Equation]

Revised: (for ε = 0 ; see Probability Simpliﬁcation Lemma below).

Revised: Let α = ξ | X | 4 .

Revised: We have that n > 3log 2 | X | δ α 2 p m by assumption and substituting in α .

Revised: By the Count

Revised: Bounding Lemma we have that (cid:12)(cid:12)(cid:12)

Revised: nP q

Revised: [ X = x ] C [ x ] − 1 (cid:12)(cid:12)(cid:12) ≤

Revised: α for all x with probability at least 1 − δ .

Revised: The following computations assume (cid:12)(cid:12)(cid:12) nP q [ X = x ] C [ x ] − 1 (cid:12)(cid:12)(cid:12) ≤ α for all x and thus are valid with probability 1 − δ .

Revised: We have nP q [ X = x ] C [ x ] ∈

Revised: [1 − α, 1 + α ] .

Revised: We also have (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)| X | (cid:88)

Revised: z ∈ X nP [ X = z ] /C

Revised: [ z ] − 1 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =

Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Revised: [Equation]

Revised: and thus 1 | X | (cid:80)

Original: where g ( s ) = 
Revised: z ∈ X nP [ X = z ] C [ z ] ∈ [Equation] − α,

Revised: [Equation]

Revised: + α ] .

Revised: Combining the previous two ranges via a division, we have

Revised: [Equation]

Revised: We have that 1 − α 1+ α = 1 − 2 α 1+ α ≥ 1 − 2 α ≥ 1 − 4 α and 1+ α 1 − α = 1 + 2 α 1 − α = 1 + 4 α 2 − 2 α ≤ 1 + 4 α since 2 − 2 α > 1 for small α .

Original: {+Thus, we have that (cid:104) 1 − α 
+}
Revised: {+Thus, we have that (cid:104) 1 − α 
+}

Revised: + α , 1+ α 1 − α (cid:105) ⊆

Revised: [1 − 4 α, 1 + 4 α ] and therefore we have | X | P r

Original: {+[ X = x ] ∈ [1 − 4 α, 1 + 
+}
Revised: {+[ X = x ] ∈ [1 − 4 α, 1 + 
+}

Revised: α

Revised: ] we thus have that (cid:12)(cid:12)(cid:12) P r [ X = x ] − 1 | X | (cid:12)(cid:12)(cid:12) ≤ 4 α | X | = ξ , completing our proof.

Revised: Lemma (Count Bounding) .

Revised: We have that if n > 3log 2 | X | δ α 2 p m samples have been drawn from q that

Revised: [Equation]

Original:  min x  P q [ X = x ]P q [ X = X
Revised: where p m = min x ∈ X P q [ X = x ]

Revised: Proof.

Revised: We can model C [ x ] as a sum of n independent Bernoulli trials with probability P q [ X = x ] .

Revised: Using Chernoff bound, we have that

Original: ( s )]
Revised: [Equation] and [Equation] Thus, we have by union bound that

Revised: [Equation]

Revised: For α ≤ (cid:112) 3 / 2 − 1 we have that

Revised: [Equation]

Revised: α √

Revised: [Equation]

Revised: [Equation]

Revised: α ≤ 1 + α and 1 1+ α = 1 − α 1+ α ≥ 1 − α .

Revised: Thus, we can restate the previous inequality as

Revised: [Equation]

Revised: or in other words

Revised: [Equation]

Revised: Thus we can bound the RHS as

Revised: [Equation]

Revised: X | where the ﬁrst inequality is since P q [ X = x ] ≥ p m and the second is since n > 3log 2 | X | δ α 2 p m .

Revised: We can again apply union bound to get that

Revised: [Equation]

Revised: Lemma (Probability Simpliﬁcation) .

Revised: [Equation]

Revised: Proof.

Revised: [Equation]

Revised: Since we can simplify (cid:88) s

Revised: [Equation]

Revised: S

Revised: [Equation]

Revised: s

Revised: [Equation]

Revised: x g ( s ) q

Revised: [Equation]

Revised: (cid:88) s

Original: ∝ P q [ X = X
Revised: [Equation] = [Equation] C

Revised: [Equation]

Original: ( s )]− 1 .
Revised: y ] C [ [Equation] s )]

Original: We thus have that
Revised: q ( s [Equation] y [Equation]

Original: P r [X = x ] =
Revised: C [ y ] C [ x ] (cid:88) s

Original: [Equation]
Revised: [Equation]

Original: And thus we have that P r
Revised: S [Equation] ( s [Equation]

Original: [ X = x ] = k
Revised: = [Equation] q ( s

Revised: [Equation]

Revised: = min y ∈ X C

Original: and is therefore uniform.
Revised: [ y ] C [ x ] P q [ X = x ] we have that [Equation]

Original:  8. S ALIENT V ARIABLE H OMOGENIZATION : E FFICIENCY A NALYSIS
Revised: P [Equation] 8. S ALIENT V ARIABLE H OMOGENIZATION : E FFICIENCY A NALYSIS

Revised: We show that the number of samples from the original distribution required to produce a sample from the homogenized distribution is O ( 1 ε ) in expectation.

Original: In the Salient Variable Homogenization algorithm, we have the probability of not rejecting a given sample as g ( s ) =
Revised: In the Salient Variable Homogenization algorithm, we have the probability of not rejecting a given sample as g ( s ) =

Original: min x P [ X = x ]+ ε P [ X = X
Revised: min x P [ X = x ]+ ε P [ X = X

Original: ( s )]
Revised: ( s )]

Original: + ε .
Revised: + ε .

Original: We know that
Revised: We know that

Original: [Equation]
Revised: [Equation]

Original:  and thus we have that the expected number of tries t = 1 g ( s ) ≤ 1+ 1 ε .
Revised: Since each sample is independent, we can model this as a geometric distribution, and thus we have that the expected number of tries t = 1 g ( s ) ≤ 1 + 1 ε .

Original: We thus have that in expectation, we need to sample O ( 1 (cid:15) ) samples from the original distribution to produce one homogenized sample.
Revised: We thus have that in expectation, we need to sample O ( 1 ε ) samples from the original distribution to produce one homogenized sample.

Original: [Figure] [Figure]
Revised: 8. E MPIRICAL E FFECT O F V ARYING ε

Original: The solid line is an upper bound ε 1+ ε derived in Section 8.3.
Revised: The solid line is an upper bound ε 1+ ε derived in Section 8.4.

Original: Seen in the measured samples for different values of ε , the upper bound appropriately reﬂects the maximum height of the samples, with  very little remaining space on the DCFG dataset and some but not much on the T2T dataset, and is thus a close bound.
Revised: Seen in the measured samples for different values of ε , the upper bound appropriately reﬂects the maximum height of the samples, with [Table] very little remaining space on the DCFG dataset and some but not much on the T2T dataset, and is thus a close bound.

Original: As the values of ε → ∞ the bound approaches the limit 1, indicating no samples are rejected by the algorithm.
Revised: As the values of ε → ∞ the bound approaches the limit 1, indicating no samples are rejected by the algorithm.

Original: Original ε = [Table]
Revised: Increasing the parameter ε has the theoretical affect of causing the homogenized distribution to deviate more from uniform in its salient random variables, as shown in 8.4.

Original: [Table] = [Table] [Table]
Revised: In practice, we discover that performance boosts tend to decrease with increasing ε , although the effect was not as pronounced on the T2T dataset, potentially because the unhomogenized T2T distribution is closer to uniform and thus homogenization has a limited effect for any larger ε values.

Original: =
Revised: 8. S ALIENT V ARIABLE H OMOGENIZATION : E MPIRICAL E VIDENCE FOR I NCREASE IN U NIFORMITY

Original: [Table] ε [Table]
Revised: Empirically, the Salient Variable Homogenization algorithm led to increases in uniformity in the variable being homogenized.

Original: T2T [Table] 33pp
Revised: We measure uniformity by KL divergence between the distribution being measured and the uniform distribution.

Original: [Table] 45pp DCFG [Table]
Revised: A table of relative improvements is given in Table 5.

Original: 42pp Increasing the parameter ε has the theoretical affect of causing the homogenized distribution to deviate more from uniform in its salient random variables, as shown in 8.3. In practice, we discover that performance boosts tend to decrease with increasing ε , although the effect was not as pronounced on the T2T dataset, potentially because the unhomogenized T2T distribution is closer to uniform and thus homogenization has a limited effect for any larger ε values.
Revised: [Table]