Original: L EARNING TO R EPRESENT P ROGRAMS WITH P ROPERTY S IGNATURES
Revised: L EARNING TO R EPRESENT P ROGRAMS WITH P ROPERTY S IGNATURES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce the notion of property signatures, a representation for programs and program speciﬁcations meant for consumption by machine learning algorithms.
Revised: We introduce the notion of property signatures, a representation for programs and program speciﬁcations meant for consumption by machine learning algorithms.

Original: Given a function with input type τ in and output type τ out , a property is a function of type: ( τ in , τ out ) → Bool that (informally) describes some simple property of the function under consideration.
Revised: Given a function with input type τ in and output type τ out , a property is a function of type: ( τ in , τ out ) → Bool that (informally) describes some simple property of the function under consideration.

Original: For instance, if τ in and τ out are both lists of the same type, one property might ask ‘is the input list the same length as the output list?’.
Revised: For instance, if τ in and τ out are both lists of the same type, one property might ask ‘is the input list the same length as the output list?’.

Original: If we have a list of such properties, we can evaluate them all for our function to get a list of outputs that we will call the property signature.
Revised: If we have a list of such properties, we can evaluate them all for our function to get a list of outputs that we will call the property signature.

Original: Crucially, we can ‘guess’ the property signature for a function given only a set of input/output pairs meant to specify that function.
Revised: Crucially, we can ‘guess’ the property signature for a function given only a set of input/output pairs meant to specify that function.

Original: We discuss several potential applications of property signatures and show experimentally that they can be used to improve over a baseline synthesizer so that it emits twice as many programs in less than one-tenth of the time.
Revised: We discuss several potential applications of property signatures and show experimentally that they can be used to improve over a baseline synthesizer so that it emits twice as many programs in less than one-tenth of the time.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Program synthesis is a longstanding goal of computer science research (Manna & Waldinger, 1971;
Revised: Program synthesis is a longstanding goal of computer science research (Manna & Waldinger, 1971;

Original: Waldinger et al., 1969; Summers, 1977; Shaw; Pnueli & Rosner, 1989; Manna & Waldinger, 1975), arguably dating to the 1940s and 50s (Copeland, 2012; Backus et al., 1957).
Revised: Waldinger et al., 1969; Summers, 1977; Shaw; Pnueli & Rosner, 1989; Manna & Waldinger, 1975), arguably dating to the 1940s and 50s (Copeland, 2012; Backus et al., 1957).

Original: Deep learning methods have shown promise at automatically generating programs from a small set of input-output examples (Balog et al., 2016;
Revised: Deep learning methods have shown promise at automatically generating programs from a small set of input-output examples (Balog et al., 2016;

Original: Devlin et al., 2017; Ellis et al., 2018b; 2019b).
Revised: Devlin et al., 2017; Ellis et al., 2018b; 2019b).

Original: In order to deliver on this promise, we believe it is important to represent programs and speciﬁcations in a way that supports learning.
Revised: In order to deliver on this promise, we believe it is important to represent programs and speciﬁcations in a way that supports learning.

Original: Just as computer vision methods beneﬁt from the inductive bias inherent to convolutional neural networks (LeCun et al., 1989), and likewise with LSTMs for natural language and other sequence data (Hochreiter & Schmidhuber, 1997), it stands to reason that ML techniques for computer programs will beneﬁt from architectures with a suitable inductive bias.
Revised: Just as computer vision methods beneﬁt from the inductive bias inherent to convolutional neural networks (LeCun et al., 1989), and likewise with LSTMs for natural language and other sequence data (Hochreiter & Schmidhuber, 1997), it stands to reason that ML techniques for computer programs will beneﬁt from architectures with a suitable inductive bias.

Original: We introduce a new representation for programs and their speciﬁcations, based on the principle that to represent a program, we can use a set of simpler programs .
Revised: We introduce a new representation for programs and their speciﬁcations, based on the principle that to represent a program, we can use a set of simpler programs .

Original: This leads us to introduce the concept of a property, which is a program that computes a boolean function of the input and output of another program.
Revised: This leads us to introduce the concept of a property, which is a program that computes a boolean function of the input and output of another program.

Original: For example, consider the problem of synthesizing a program from a small set of input-output examples.
Revised: For example, consider the problem of synthesizing a program from a small set of input-output examples.

Original: Perhaps the synthesizer is given a few pairs of lists of integers, and the user hopes that the synthesizer will produce a sorting function.
Revised: Perhaps the synthesizer is given a few pairs of lists of integers, and the user hopes that the synthesizer will produce a sorting function.

Original: Then useful properties might include functions that check if the input and output lists have the same length, if the input list is a subset of the output, if element 0 of the output list is less than element 42 , and so on.
Revised: Then useful properties might include functions that check if the input and output lists have the same length, if the input list is a subset of the output, if element 0 of the output list is less than element 42 , and so on.

Original: The outputs of a set of properties can be concatenated into a vector, yielding a representation that we call a property signature .
Revised: The outputs of a set of properties can be concatenated into a vector, yielding a representation that we call a property signature .

Original: Property signatures can then be used for consumption by machine learning algorithms, essentially serving as the ﬁrst layer of a neural network.
Revised: Property signatures can then be used for consumption by machine learning algorithms, essentially serving as the ﬁrst layer of a neural network.

Original: In this paper, we demonstrate the utility of property signatures for program synthesis, using them to perform a type of premise selection as in Balog et al.
Revised: In this paper, we demonstrate the utility of property signatures for program synthesis, using them to perform a type of premise selection as in Balog et al.

Original: More broadly, however, we envision that property signatures could be useful across a broad range of problems, including algorithm induction (Devlin et al., 2017), improving code readability (Allamanis et al., 2014), and program analysis (Heo et al., 2019).
Revised: More broadly, however, we envision that property signatures could be useful across a broad range of problems, including algorithm induction (Devlin et al., 2017), improving code readability (Allamanis et al., 2014), and program analysis (Heo et al., 2019).

Original: More speciﬁcally, our contributions are:
Revised: More speciﬁcally, our contributions are:

Original: • We introduce the notion of property signatures, which are a general purpose way of featurizing both programs and program speciﬁcations (Section 3).
Revised: • We introduce the notion of property signatures, which are a general purpose way of featurizing both programs and program speciﬁcations (Section 3).

Original: • We demonstrate how to use property signatures within a machine-learning based synthesizer for a general-purpose programming language.
Revised: • We demonstrate how to use property signatures within a machine-learning based synthesizer for a general-purpose programming language.

Original: This allows us to automatically learn a useful set of property signatures, rather than choosing them manually (Sections 3.2 and 4).
Revised: This allows us to automatically learn a useful set of property signatures, rather than choosing them manually (Sections 3.2 and 4).

Original: • We show that a machine learning model can predict the signatures of individual functions given the signature of their composition, and describe several ways this could be used to improve existing synthesizers (Section 5).
Revised: • We show that a machine learning model can predict the signatures of individual functions given the signature of their composition, and describe several ways this could be used to improve existing synthesizers (Section 5).

Original: • We perform experiments on a new test set of 185 functional programs of varying difﬁculty, designed to be the sort of algorithmic problems that one would ask on an undergraduate computer science examination.
Revised: • We perform experiments on a new test set of 185 functional programs of varying difﬁculty, designed to be the sort of algorithmic problems that one would ask on an undergraduate computer science examination.

Original: We ﬁnd that the use of property signatures leads to a dramatic improvement in the performance of the synthesizer, allowing it to synthesize over twice as many programs in less than one-tenth of the time (Section 4).
Revised: We ﬁnd that the use of property signatures leads to a dramatic improvement in the performance of the synthesizer, allowing it to synthesize over twice as many programs in less than one-tenth of the time (Section 4).

Original: An example of a complex program that was synthesized only by the property signatures method is shown in Listing 1.
Revised: An example of a complex program that was synthesized only by the property signatures method is shown in Listing 1.

Original: For our experiments we created a specialized programming language, called Searcho 1 (Section 2), based on strongly-typed functional languages such as Standard ML and Haskell.
Revised: For our experiments we created a specialized programming language, called Searcho 1 (Section 2), based on strongly-typed functional languages such as Standard ML and Haskell.

Original: Searcho is designed so that many similar programs can be executed rapidly, as is needed during a large-scale distributed search during synthesis.
Revised: Searcho is designed so that many similar programs can be executed rapidly, as is needed during a large-scale distributed search during synthesis.

Original: We release 2 the programming language, runtime environment, distributed search infrastructure, machine learning models and training data from our experiments so that they can be used for future research.
Revised: We release 2 the programming language, runtime environment, distributed search infrastructure, machine learning models, and training data from our experiments so that they can be used for future research.

Revised: 1 fun unique_justseen(xs :List<

Original: (cons_(list_elt, acc), list_elt, _0)}
Revised: Int>) -> List< Int> { let triple = list_foldl_<Int, (List< Int>, Int, Bool)>

Revised: ( xs, (nil<

Revised: Int>, 0, _1), \(list_elt, (acc, last_elt, first))

Revised: { cond_(or_(first, not_equal_(list_elt, last_elt)), \{(cons_(list_elt, acc), list_elt, _0)},

Original: Listing 1: A program synthesized by our system, reformatted and with variables renamed for readability.
Revised: Listing 1: A program synthesized by our system, reformatted and with variables renamed for readability.

Original: This program returns the sub-list of all of the elements in a list that are distinct from their previous value in the list.
Revised: This program returns the sub-list of all of the elements in a list that are distinct from their previous value in the list.

Original:  P ROGRAMMING B Y E XAMPLE AND T HE S EARCHO L ANGUAGE
Revised:  P ROGRAMMING B Y E XAMPLE AND T HE S EARCHO L ANGUAGE

Original: In Inductive Program Synthesis, we are given a speciﬁcation of a program and our goal is to synthesize a program meeting that speciﬁcation.
Revised: In Inductive Program Synthesis, we are given a speciﬁcation of a program and our goal is to synthesize a program meeting that speciﬁcation.

Original: Inductive Synthesis is generally divided into Programming by
Revised: Inductive Synthesis is generally divided into Programming by

Original: Example (PBE) and Programming by Demonstration (PBD).
Revised: Example (PBE) and Programming by Demonstration (PBD).

Original: This work is focused on PBE.
Revised: This work is focused on PBE.

Original: In PBE, we are given a set of input/output pairs such that for each pair, the target program takes the input to the corresponding output.
Revised: In PBE, we are given a set of input/output pairs such that for each pair, the target program takes the input to the corresponding output.

Original: Existing PBE systems include Winston (1970), Menon et al.
Revised: Existing PBE systems include Winston (1970), Menon et al.

Original: (2013), and
Revised: (2013), and

Original: Gulwani (2011).
Revised: Gulwani (2011).

Original: A PBE speciﬁcation might look like Listing 2:
Revised: A PBE speciﬁcation might look like Listing 2:

Original: Listing 2: An example PBE speciﬁcation.
Revised: Listing 2: An example PBE speciﬁcation.

Original: for which a satisfying solution would be the function squaring its input.
Revised: for which a satisfying solution would be the function squaring its input.

Original: Arbitrarily many functions satisfy this speciﬁcation.
Revised: Arbitrarily many functions satisfy this speciﬁcation.

Original: It is interesting but out of scope 3 to think about ways to ensure that the synthesis procedure recovers the ‘best’ or ‘simplest’ program satisfying the speciﬁcation.
Revised: It is interesting but out of scope 3 to think about ways to ensure that the synthesis procedure recovers the ‘best’ or ‘simplest’ program satisfying the speciﬁcation.

Original: Much (though not all) work on program synthesis is focused on domain speciﬁc languages that are less than maximally expressive (Gulwani, 2011; Balog et al., 2016; Wang et al., 2017; Alur et al., 2015).
Revised: Much (though not all) work on program synthesis is focused on domain speciﬁc languages that are less than maximally expressive (Gulwani, 2011; Balog et al., 2016; Wang et al., 2017; Alur et al., the search procedure used will tend to emit ‘shorter’ programs ﬁrst, and so there is an Occam’s-Razor-type argument (Spade & Panaccio, 2019) to be made that you should get this for free.

Original: We would like to focus on the synthesis of programs in a Turing complete language, but this presents technical challenges: First, general purpose languages such as C++ or Python are typically quite complicated and sometimes not fully speciﬁed; this makes it a challenge to search over partial programs in those languages.
Revised: We would like to focus on the synthesis of programs in a Turing complete language, but this presents technical challenges: First, general purpose languages such as C++ or Python are typically quite complicated and sometimes not fully speciﬁed; this makes it a challenge to search over partial programs in those languages.

Original: Second, sandboxing and executing code written in these languages is nontrivial.
Revised: Second, sandboxing and executing code written in these languages is nontrivial.

Original: Finally, searching over and executing many programs in these languages can be quite slow, since this is not what they were designed for.
Revised: Finally, searching over and executing many programs in these languages can be quite slow, since this is not what they were designed for.

Original: For these reasons, we have created a general-pupose, Turing complete programming language and runtime.
Revised: For these reasons, we have created a general-pupose, Turing complete programming language and runtime.

Original: The programming language is called Searcho and it and its runtime have been designed speciﬁcally with program synthesis in mind.
Revised: The programming language is called Searcho and it and its runtime have been designed speciﬁcally with program synthesis in mind.

Original: The language can roughly be thought of as a more complicated version of the simply typed lambda calculus or as a less complicated version of Standard
Revised: The language can roughly be thought of as a more complicated version of the simply typed lambda calculus or as a less complicated version of Standard

Original: ML or OCaml.
Revised: ML or OCaml.

Original: 4 Searcho code is compiled to bytecode and run on the Searcho Virtual Machine.
Revised: 4 Searcho code is compiled to bytecode and run on the Searcho Virtual Machine.

Original: Code is incrementally compiled, which means that the standard library and speciﬁcation can be compiled once and then many programs can be pushed on and popped off from the stack in order to check them against the speciﬁcation.
Revised: Code is incrementally compiled, which means that the standard library and speciﬁcation can be compiled once and then many programs can be pushed on and popped off from the stack in order to check them against the speciﬁcation.

Original: Searcho is strongly typed with algebraic datatypes (Pierce & Benjamin, 2002) 

Revised: Searcho is strongly typed with algebraic datatypes (Pierce & Benjamin, 2002)

Original: Searcho includes a library of 86 functions, all of which are supported by our synthesizer.
Revised: Searcho includes a library of 86 functions, all of which are supported by our synthesizer.

Original: This is a signiﬁcantly larger language and library than have been used in previous work on neural program synthesis.
Revised: This is a signiﬁcantly larger language and library than have been used in previous work on neural program synthesis.

Original: We have also implemented a baseline enumerative synthesizer.
Revised: We have also implemented a baseline enumerative synthesizer.

Original: The main experiments in this paper will involve plugging the outputs of a machine learning model into the conﬁguration for our baseline synthesizer to improve its performance on a set of human-constructed PBE tasks.
Revised: The main experiments in this paper will involve plugging the outputs of a machine learning model into the conﬁguration for our baseline synthesizer to improve its performance on a set of human-constructed PBE tasks.

Original:  P ROPERTY S IGNATURES
Revised:  P ROPERTY S IGNATURES

Original: Consider the PBE speciﬁcation in Listing 3:
Revised: Consider the PBE speciﬁcation in Listing 3:

Original: 1 io_pairs =
Revised: 1 io_pairs =

Original:  34567, 2345, 1]), ([True, False], [ [Table] [Table] [Table]
Revised: [ ([1, 2345, 34567], [1, 2345, 34567, 34567, 2345, 1]), ([True, False], [True, False, False, True]), (["Batman"], ["Batman", "Batman"]),

Original: , True]), (["

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table] [Table] [Table] Listing3: An example PBE Speciﬁcation.
Revised: Listing example PBE Speciﬁcation.

Original: We can see that the function concatenating the input list to its reverse will satisfy the speciﬁcation, but how can we teach this to a computer?
Revised: We can see that the function concatenating the input list to its reverse will satisfy the speciﬁcation, but how can we teach this to a computer?

Original: Following Balog et al.
Revised: Following Balog et al.

Original: we take the approach of training a machine learning model to do premise selection for a symbolic search procedure.
Revised: we take the approach of training a machine learning model to do premise selection for a symbolic search procedure.

Original: But how do we get a representation of the speciﬁcation to feed to the model?
Revised: But how do we get a representation of the speciﬁcation to feed to the model?

Original: In Balog et al.
Revised: In Balog et al.

Original: (2016), the model acts only on integers and lists of integers, constrains all integers to lie in [ − 256 , 256] , has special-case handling of lists, and does not deal with polymorphic functions.
Revised: (2016), the model acts only on integers and lists of integers, constrains all integers to lie in [ − 256 , 256] , has special-case handling of lists, and does not deal with polymorphic functions.

Original: It would be hard to apply this technique to the above speciﬁcation, since the ﬁrst example contains unbounded integers, the second example contains a different type than the ﬁrst 6 , and the third and fourth examples contain recursive data structures (lists of characters and lists of integers respectively).
Revised: It would be hard to apply this technique to the above speciﬁcation, since the ﬁrst example contains unbounded integers, the second example contains a different type than the ﬁrst 6 , and the third and fourth examples contain recursive data structures (lists of characters and lists of integers respectively).

Original: Thankfully, we can instead learn a representation that is composed of the outputs of multiple other programs running on each input/output pair.
Revised: Thankfully, we can instead learn a representation that is composed of the outputs of multiple other programs running on each input/output pair.

Original: We will call these other programs properties.
Revised: We will call these other programs properties.

Original: Consider the three properties in Listing 4.
Revised: Consider the three properties in Listing 4.

Original: = =
Revised: [Equation]

Original: Listing 4: Three function projections that can act on the speciﬁcation from Listing 3.similar
Revised: Listing 4: Three function projections that can act on the speciﬁcation from Listing 3.

Original: Types have been shown to substantially speed up synthesis

Original: See g. Figure

Original: Each of these three programs can be run on all 4 of the input output pairs to yield a Boolean .
Revised: Each of these three programs can be run on all 4 of the input output pairs to yield a Boolean .

Original: The ﬁrst always returns True for our spec, as does the second.
Revised: The ﬁrst always returns True for our spec, as does the second.

Original: The third always returns False on the given examples, although note that it would return True if the examples had contained the implicit base case of the empty list.
Revised: The third always returns False on the given examples, although note that it would return True if the examples had contained the implicit base case of the empty list.

Original: Thus, we can write that our spec has the ‘property signature’
Revised: Thus, we can write that our spec has the ‘property signature’

Original: [ True , True , False ] .
Revised: [ True , True , False ] .

Revised: [Equation]

Revised: In this paper, we will present illustrative programs in Haskell syntax to make them more broadly readable.

Revised: Searcho programs will be presented in Searcho syntax, which is similar.

Original:  How is this useful?
Revised: Types have been shown to substantially speed up synthesis. See e.g. Figure 6 of Feser et al. So any function satisfying the spec will be parametrically polymorphic. How is this useful?

Original: From the ﬁrst property we can infer that we should not throw away any elements of the input list.
Revised: From the ﬁrst property we can infer that we should not throw away any elements of the input list.

Original: From the third we might guess that we have to add or remove elements from the input list.
Revised: From the third we might guess that we have to add or remove elements from the input list.

Original: Finally, the second might imply that we need to create copies of the input elements somehow.
Revised: Finally, the second might imply that we need to create copies of the input elements somehow.

Original: This does not narrow our search down all the way, but it narrows it down quite a lot.
Revised: This does not narrow our search down all the way, but it narrows it down quite a lot.

Original: Since the properties are expressed in the same language as the programs we are synthesizing, we can emit them using the same synthesizer.
Revised: Since the properties are expressed in the same language as the programs we are synthesizing, we can emit them using the same synthesizer.

Original: Later on, we will describe how we enumerate many random properties and prune them to keep only the useful ones.
Revised: Later on, we will describe how we enumerate many random properties and prune them to keep only the useful ones.

Original: The property signatures that we consider in our experiments contain thousands of values.
Revised: The property signatures that we consider in our experiments contain thousands of values.

Original: Since the output of these properties is either always True, always False, or sometimes True and sometimes False, a neural network can learn embeddings for those three values and it can be fed a vector of such values, one for each applicable property, as the representation of a program speciﬁcation.
Revised: Since the output of these properties is either always True, always False, or sometimes True and sometimes False, a neural network can learn embeddings for those three values and it can be fed a vector of such values, one for each applicable property, as the representation of a program speciﬁcation.

Original: 3. A BSTRACTING P ROPERTIES INTO S IGNATURES
Revised: 3. A BSTRACTING P ROPERTIES INTO S IGNATURES

Original: Now we describe our representation for a program f :: τ in → τ out .
Revised: Now we describe our representation for a program f :: τ in → τ out .

Original: Each property is a program p :: ( τ in , τ out )
Revised: Each property is a program p :: ( τ in , τ out )

Original: → Bool that represents a single “feature” of the program’s inputs and outputs which might be useful for its representation.
Revised: → Bool that represents a single “feature” of the program’s inputs and outputs which might be useful for its representation.

Original: In this section, we assume that we have determined a sequence P =
Revised: In this section, we assume that we have determined a sequence P =

Original: [ p 1 . . .
Revised: [ p 1 . . .

Original: p n ] of properties that are useful for describing f , and we wish to combine them into a single representation of f .
Revised: p n ] of properties that are useful for describing f , and we wish to combine them into a single representation of f .

Original: Later, we will describe a learning principle for choosing relevant properties.
Revised: Later, we will describe a learning principle for choosing relevant properties.

Original: We want the property signature to summarize the output of all the properties in P over all valid inputs to f .
Revised: We want the property signature to summarize the output of all the properties in P over all valid inputs to f .

Original: To do this, we ﬁrst extend the notion of property to a set of inputs in the natural way.
Revised: To do this, we ﬁrst extend the notion of property to a set of inputs in the natural way.

Original: If S is a set of values of type τ in and p ∈ P , we deﬁne p ( S )
Revised: If S is a set of values of type τ in and p ∈ P , we deﬁne p ( S )

Original: =
Revised: =

Original: { p ( x, f ( x )) | x ∈ S } .
Revised: { p ( x, f ( x )) | x ∈ S } .

Original: Because p ( S ) is a set of booleans, it can have only three possible values, either p ( S ) =
Revised: Because p ( S ) is a set of booleans, it can have only three possible values, either p ( S ) =

Original: { True } , or p ( S ) =
Revised: { True } , or p ( S ) =

Original: { False } , or p ( S ) =
Revised: { False } , or p ( S ) =

Original: { True , False } , corresponding respectively to the cases where p is always true, always false, or neither.
Revised: { True , False } , corresponding respectively to the cases where p is always true, always false, or neither.

Original: To simplify notation slightly, we deﬁne the function Π as Π( { True } ) =
Revised: To simplify notation slightly, we deﬁne the function Π as Π( { True } ) =

Original: AllTrue , Π( { False } ) =
Revised: AllTrue , Π( { False } ) =

Original: AllFalse , and Π( { True , False } ) =
Revised: AllFalse , and Π( { True , False } ) =

Original: Mixed .
Revised: Mixed .

Original: Finally, we can deﬁne the property signature sig( P, f ) for a program f and a property sequence P as 
Revised: Finally, we can deﬁne the property signature sig( P, f ) for a program f and a property sequence P as sig( P, f )

Original: [Equation]
Revised: [ i ] =

Original:  where V ( τ in ) is the possibly inﬁnite set of all values of type τ in .
Revised: Π( p i ( V ( τ in ))) , where V ( τ in ) is the possibly inﬁnite set of all values of type τ in .

Original: Computing the property signature for f could be intractable or undecidable, as it might require proving difﬁcult facts about the program.
Revised: Computing the property signature for f could be intractable or undecidable, as it might require proving difﬁcult facts about the program.

Original: Instead, in practice, we will compute an estimated property signature for a small set of input-output pairs S io .
Revised: Instead, in practice, we will compute an estimated property signature for a small set of input-output pairs S io .

Original: The estimated property signature summarizes the actions of P on S io rather than on the full set of inputs V ( τ in ) .
Revised: The estimated property signature summarizes the actions of P on S io rather than on the full set of inputs V ( τ in ) .

Original: Formally, the estimated property signature is
Revised: Formally, the estimated property signature is

Original: [Equation]
Revised: [Equation]

Original: This estimate gives us an under-approximation of the true signature of f in the following sense: If we have (cid:99) sig( P, S ) =
Revised: This estimate gives us an under-approximation of the true signature of f in the following sense: If we have (cid:99) sig( P, S ) =

Original: Mixed , we must also have sig( P, f ) =
Revised: Mixed , we must also have sig( P, f ) =

Original: Mixed .
Revised: Mixed .

Original: If (cid:99) sig( P, S ) =
Revised: If (cid:99) sig( P, S ) =

Original: AllTrue , then either sig( P, f ) =
Revised: AllTrue , then either sig( P, f ) =

Original: AllTrue or sig( P, f ) =
Revised: AllTrue or sig( P, f ) =

Original: Mixed , and similarly with AllFalse .
Revised: Mixed , and similarly with AllFalse .

Original: Estimated property signatures are particularly useful for synthesis using PBE, because we can compute them from the input-output pairs that specify the synthesis task, without having the deﬁnition of f .
Revised: Estimated property signatures are particularly useful for synthesis using PBE, because we can compute them from the input-output pairs that specify the synthesis task, without having the deﬁnition of f .

Original: Thus we can use estimated property signatures to ‘featurize’ PBE speciﬁcations for use in synthesis.
Revised: Thus we can use estimated property signatures to ‘featurize’ PBE speciﬁcations for use in synthesis.

Original: 3. L EARNING U SEFUL P ROPERTIES
Revised: 3. L EARNING U SEFUL P ROPERTIES

Original: How do we choose a set of properties that will be useful for synthesis?
Revised: How do we choose a set of properties that will be useful for synthesis?

Original: Given a training set of random programs with random input/output examples, we generate many random properties.
Revised: Given a training set of random programs with random input/output examples, we generate many random properties.

Original: We then prune the random properties based on whether they distinguish between any of the programs.
Revised: We then prune the random properties based on whether they distinguish between any of the programs.

Original: Then, given a test suite of programs, we do an additional pruning step: among all properties that give the same value for every element of the test suite, we keep the shortest property, because of Occam’s razor considerations.
Revised: Then, given a test suite of programs, we do an additional pruning step: among all properties that give the same value for every element of the test suite, we keep the shortest property, because of Occam’s razor considerations.

Original: Given these ‘useful’ properties, we can train a premise selector (Balog et al., 2016) to predict library function usage given properties.
Revised: Given these ‘useful’ properties, we can train a premise selector (Balog et al., 2016) to predict library function usage given properties.

Original: Speciﬁcally, from the remaining properties, we compute estimated property signatures for each function in the training set, based on its input output examples.
Revised: Speciﬁcally, from the remaining properties, we compute estimated property signatures for each function in the training set, based on its input output examples.

Original: Then we use the property signature as the input to a feedforward network that predicts the number of times each library function appears in the program.
Revised: Then we use the property signature as the input to a feedforward network that predicts the number of times each library function appears in the program.

Original: In Section 4, we will give more details about the architecture of this premise selector, and evaluate it for synthesis.
Revised: In Section 4, we will give more details about the architecture of this premise selector, and evaluate it for synthesis.

Original: For now, we point out that this premise selector could itself be used to ﬁnd useful properties, by examining which properties are most useful for the model’s predictions.
Revised: For now, we point out that this premise selector could itself be used to ﬁnd useful properties, by examining which properties are most useful for the model’s predictions.

Original: 3. W HY ARE P ROPERTY S IGNATURES U SEFUL ?
Revised: 3. W HY ARE P ROPERTY S IGNATURES U SEFUL ?

Original: Experiments in the next section will establish that property signatures let our baseline synthesizer emit programs it previously could not, but we think that they can have broader utility:
Revised: Experiments in the next section will establish that property signatures let our baseline synthesizer emit programs it previously could not, but we think that they can have broader utility:

Original: • They allow us to represent more types of functions.
Revised: • They allow us to represent more types of functions.

Original: Property signatures can automatically deal with unbounded data types, recursive data types, and polymorphic functions.
Revised: Property signatures can automatically deal with unbounded data types, recursive data types, and polymorphic functions.

Original: • They reduce dependency on the distribution from which examples are drawn.
Revised: • They reduce dependency on the distribution from which examples are drawn.

Original: If the user of a synthesizer gives example inputs distributed differently than the training data, the ‘estimated’ properties might not change much.• They can be used wherever we want to search for functions by semantics.
Revised: If the user of a synthesizer gives example inputs distributed differently than the training data, the ‘estimated’ properties might not change much.• They can be used wherever we want to search for functions by semantics.

Original: Imagine a search engine where users give a speciﬁcation, the system guesses a property signature, and this signature guess is used to ﬁnd all the pre-computed functions with similar semantics.
Revised: Imagine a search engine where users give a speciﬁcation, the system guesses a property signature, and this signature guess is used to ﬁnd all the pre-computed functions with similar semantics.

Original: • Synthesized programs can themselves become new properties.
Revised: • Synthesized programs can themselves become new properties.

Original: For example, once I learn a program for primality checking, I can use primality checking in my library of properties.
Revised: For example, once I learn a program for primality checking, I can use primality checking in my library of properties.

Original:  P ROGRAM S YNTHESIS WITH PROPERTY SIGNATURES
Revised:  P ROGRAM S YNTHESIS WITH PROPERTY SIGNATURES

Original: We design an experiment to answer the following question: Can property signatures help us synthesize programs that we otherwise could not have synthesized?
Revised: We design an experiment to answer the following question: Can property signatures help us synthesize programs that we otherwise could not have synthesized?

Original: As we will show, the answer is yes!
Revised: As we will show, the answer is yes!

Original: 4. E XPERIMENTAL S ETUP
Revised: 4. E XPERIMENTAL S ETUP

Original: How Does the Baseline Synthesizer Work?
Revised: How Does the Baseline Synthesizer Work?

Original: Our baseline synthesizer is very similar to that in
Revised: Our baseline synthesizer is very similar to that in

Original: Feser et al.
Revised: Feser et al.

Original: and works by ﬁlling in typed holes 9 .
Revised: and works by ﬁlling in typed holes 9 .

Original: That is, we infer a program type τ in → τ out from the speciﬁcation and the synthesizer starts with a empty ‘hole’ of type τ in → τ out and then ﬁlls it in all possible ways allowed by the type system.
Revised: That is, we infer a program type τ in → τ out from the speciﬁcation and the synthesizer starts with a empty ‘hole’ of type τ in → τ out and then ﬁlls it in all possible ways allowed by the type system.

Original: Many of these ways of ﬁlling-in will yield new holes, which can in turn be ﬁlled by the same technique.
Revised: Many of these ways of ﬁlling-in will yield new holes, which can in turn be ﬁlled by the same technique.

Original: When a program has no holes, we check if it satisﬁes the spec.
Revised: When a program has no holes, we check if it satisﬁes the spec.

Original: We order the programs to expand by their cost, where the cost is essentially a sum of the costs of the individual operations used in the program.
Revised: We order the programs to expand by their cost, where the cost is essentially a sum of the costs of the individual operations used in the program.

Original: At the beginning of the procedure, the synthesizer is given a conﬁguration, which is essentially a weighted set of pool elements that it is allowed to use to ﬁll in the holes.
Revised: At the beginning of the procedure, the synthesizer is given a conﬁguration, which is essentially a weighted set of pool elements that it is allowed to use to ﬁll in the holes.

Original: A pool element is a rewrite rule that replaces a hole with a type-correct Searcho program, which may itself contain its own, new holes.
Revised: A pool element is a rewrite rule that replaces a hole with a type-correct Searcho program, which may itself contain its own, new holes.

Original: In our synthesizer, there is one possible pool element for each of the 86 library functions in
Revised: In our synthesizer, there is one possible pool element for each of the 86 library functions in

Original: Searcho, which calls the library function, with correctly-typed holes for each of its arguments.
Revised: Searcho, which calls the library function, with correctly-typed holes for each of its arguments.

Original: The conﬁguration will specify a small subset of these pool elements to use during search.
Revised: The conﬁguration will specify a small subset of these pool elements to use during search.

Original: It is through the conﬁguration that we will use machine learning to inform the search procedure, as we describe later.
Revised: It is through the conﬁguration that we will use machine learning to inform the search procedure, as we describe later.

Original: See Appendix A.1 for further details on this baseline system.programs.
Revised: See Appendix A.1 for further details on this baseline system.

Original: How is the Training Data Generated?
Revised: How is the Training Data Generated?

Original: Our test corpus contains programs with 14 different types.
Revised: Our test corpus contains programs with 14 different types.

Original: For each of those 14 types, we randomly sample conﬁgurations and then randomly generate training programs for each conﬁguration, pruning for observational equivalence.
Revised: For each of those 14 types, we randomly sample conﬁgurations and then randomly generate training programs for each conﬁguration, pruning for observational equivalence.

Original: We generate up 10, semantically distinct programs for each type, though of course some function types admit less distinct programs than this ( e.g. Bool → Bool ).
Revised: We generate up 10, semantically distinct programs for each type, though of course some function types admit less distinct programs than this ( e.g. Bool → Bool ).

Original: We also generate and prune random properties as described in Section 3.2.
Revised: We also generate and prune random properties as described in Section 3.2.

Original: See Listing 5 for examples of useful properties that were generated.
Revised: See Listing 5 for examples of useful properties that were generated.

Original: 1 \:(List<Int>, List<Int>)->Bool (input, output) { list_for_all_<Int> (input, \x {in_list_<Int> (x, output)})} 3 \:(List<Int>, List<Int>)->Bool (input, output) { not_ (is_even_ (list_len_<Int> output))} 5 \:(List<Int>, List<Int>)->Bool (input, output) { not_equal_<Int> ((ints_sum_ input), (ints_sum_ output))} 7 \:(List<Int>, List<Int>)->Bool (input, output) { gt_ ((list_len_<Int> input), (list_len_<Int> output))}
Revised: 1 \:(List<Int>, List<Int>)->Bool (input, output) { list_for_all_<Int> (input, \x {in_list_<Int> (x, output)})} 3 \:(List<Int>, List<Int>)->Bool (input, output) { not_ (is_even_ (list_len_<Int> output))} 5 \:(List<Int>, List<Int>)->Bool (input, output) { not_equal_<Int> ((ints_sum_ input), (ints_sum_ output))} 7 \:(List<Int>, List<Int>)->Bool (input, output) { gt_ ((list_len_<Int> input), (list_len_<Int> output))}

Original: Listing 5: 4 of the Properties with the highest discriminative power on functions of type
Revised: Listing 5: 4 of the Properties with the highest discriminative power on functions of type

Original: List<
Revised: List<

Original: Int> → List<
Revised: Int> → List<

Original: Int> .
Revised: Int> .

Original: The ﬁrst checks if every element of the input list is in the output list.
Revised: The ﬁrst checks if every element of the input list is in the output list.

Original: The second checks if the length of the output list is even.
Revised: The second checks if the length of the output list is even.

Original: The third checks if sum of the input and the output list is the same, and the fourth checks if the input list is longer than the output list.
Revised: The third checks if sum of the input and the output list is the same, and the fourth checks if the input list is longer than the output list.

Original: How was the Test Set Constructed?
Revised: How was the Test Set Constructed?

Original: We’ve constructed a test set of 185 human generated programs ranging in complexity from one single line to many nested function calls with recursion.
Revised: We’ve constructed a test set of 185 human generated programs ranging in complexity from one single line to many nested function calls with recursion.

Original: Programs in the test set include computing the GCD of two integers, computing the n -th ﬁbonacci number, computing the intersection of two sets, and computing the sum of all pairs in two lists.
Revised: Programs in the test set include computing the GCD of two integers, computing the n -th ﬁbonacci number, computing the intersection of two sets, and computing the sum of all pairs in two lists.

Original: We ensure that none of the test functions appear in the training set.
Revised: We ensure that none of the test functions appear in the training set.

Original: See the open source code for more details on this.
Revised: See the open source code for more details on this.

Original: What is the Architecture of the Model?
Revised: What is the Architecture of the Model?

Original: As mentioned above, we train a neural network to predict the number of times each pool element will appear in the output.
Revised: As mentioned above, we train a neural network to predict the number of times each pool element will appear in the output.

Original: This neural network is fully connected, with learned embeddings for each of the values AllTrue , AllFalse and Mixed .
Revised: This neural network is fully connected, with learned embeddings for each of the values AllTrue , AllFalse and Mixed .

Original: How does the Model Output Inform the Search Procedure?
Revised: How does the Model Output Inform the Search Procedure?

Original: Since we have a large number of pool elements (86), we can’t run the synthesizer with all pool elements if we want to ﬁnd programs of reasonable length.
Revised: Since we have a large number of pool elements (86), we can’t run the synthesizer with all pool elements if we want to ﬁnd programs of reasonable length.

Original: This is both because we will run out of memory and because it will take too long.
Revised: This is both because we will run out of memory and because it will take too long.

Original: Thus, we randomly sample conﬁgurations with less pool elements.
Revised: Thus, we randomly sample conﬁgurations with less pool elements.

Original: We then send multiple such conﬁgurations to a distributed synthesis server that tries them in parallel.
Revised: We then send multiple such conﬁgurations to a distributed synthesis server that tries them in parallel.

Original: When we use the model predictions, we sample pool elements in proportion to the model’s predicted number of times that pool element appears.
Revised: When we use the model predictions, we sample pool elements in proportion to the model’s predicted number of times that pool element appears.

Original: The baseline samples pool elements in proportion to their rate of appearance in the training set.
Revised: The baseline samples pool elements in proportion to their rate of appearance in the training set.

Original: 4. U SING P ROPERTY S IGNATURES L ETS US S YNTHESIZE N EW F UNCTIONS
Revised: 4. U SING P ROPERTY S IGNATURES L ETS US S YNTHESIZE N EW F UNCTIONS

Original: We ran 3 different runs of our distributed synthesizer for 100,000 seconds with and without the aid of property signatures.
Revised: We ran 3 different runs of our distributed synthesizer for 100,000 seconds with and without the aid of property signatures.

Original: The baseline synthesizer solved 28 test programs on average.
Revised: The baseline synthesizer solved 28 test programs on average.

Original: With property signatures, the synthesizer solved an average of 73 test programs.
Revised: With property signatures, the synthesizer solved an average of 73 test programs.

Original: See Figure 1 for more discussion.
Revised: See Figure 1 for more discussion.

Original: Indeed, it can be seen from the ﬁgure that not only did the synthesizer solve many more test programs using property signatures, but it did so much faster, synthesizing over twice as many programs in one-tenth of the time as the baseline.
Revised: Indeed, it can be seen from the ﬁgure that not only did the synthesizer solve many more test programs using property signatures, but it did so much faster, synthesizing over twice as many programs in one-tenth of the time as the baseline.

Original: P REDICTING PROPERTY SIGNATURES OF F UNCTION C OMPOSITIONS
Revised: 4. C OMPARISON WITH D EEP C ODER

Revised: We have conducted an experiment to compare premise selection using Property Signatures to the premise selection algorithm from (Balog et al., 2016). This required considerable modiﬁcations to the experimental procedure. First, since the premise-selection part of DeepCoder can only handle Integers and lists of Integers, we restricted the types of our training and test functions.

Revised: In particular, we read through (Balog et al., 2016) and found four function types in use:

Revised: 1 f :: [Int] -> [Int] 2 g :: [Int] -> Int 3 h :: ([Int], [Int]) -> Int 4 k :: ([Int], Int) ->

Revised: Int

Revised: Listing

Revised: The types of f and g in 6 are taken directly from (Balog et al., 2016).

Revised: The types of h and k are inferred from examples given in the appendix of (Balog et al., 2016).

Revised: Their DSL does not technically have tuples, but we have wrapped the inputs of their ‘two-input-functions’ in tuples for convienence.

Revised: Second, since DeepCoder can only handle integers betwen − 255 and 255 , we ﬁrst re-generated all of our random inputs (used for ‘hashing’ of generated training data) to lie in that range.

Revised: We then generated random training functions of the above four types.

Revised: We then made a data set of training functions associated with 5 input-output pairs, throwing out pairs where any of the outputs were outside the aforementioned range, and throwing out functions where all outputs contained some number outside that range.

Revised: Third, of the examples in our test set with the right types, we modiﬁed their input output pairs in a similar way.

Revised: We ﬁltered out functions that could not be so modiﬁed.

Revised: After doing so, we were left with a remaining test suite of 32 functions.

Revised: Finally, we trained a model to predict functions-to-use from learned embeddings of the input-output pairs, as in DeepCoder.

Revised: We didn’t see a description of how functions with multiple inputs had their inputs embedded, so we elected to separate them with a special character, distinct from the null characters that are used to pad lists.

Revised: Compared with the Property Signatures method, this technique results in far fewer synthesized test set programs.

Revised: We did 3 random restarts for each of DeepCoder, Property Signatures, and the Random

Revised: Baseline (recall that the random baseline itself is already a relatively sophisticated synthesis algorithm - it’s just the conﬁgurations that are random).

Revised: The 3 DeepCoder runs synthesized an average of 3 .

Revised: 33 test programs, while the Property Signature runs (trained on the same modiﬁed training data and tested on the same modiﬁed test data) synthesized 16 .

Revised: The random baseline synthesizedprograms on average.

Revised: A priori, this seems like a surprisingly large gap, but it actually ﬁts with what we know from existing literature.

Revised: Shin et al.

Revised: (2018) observe something similar: which is that DeepCoder-esque techniques tend to generalize poorly to a a test set where the input-output pairs come from a different distribution than they do in training.

Revised: This is the case in our experiment, and it will be the case in any realistic setting, since the test set will be provided by users.

Revised: Property Signatures are (according to our experiments) much less sensitive to such shift.

Revised: This makes intuitive sense: whether an input list is half the length of an output list (for instance) is invariant to the particular distribution of members of the list.

Revised: Note that even if Property Signatures did not outperform DeepCoder on this subset of our test set, they would still constitute an improvement due to their allowing us to operate on arbitrary programs and inputs types.

Revised:  P REDICTING PROPERTY SIGNATURES OF F UNCTION C OMPOSITIONS

Original: Most programs involve composing functions with other functions.
Revised: Most programs involve composing functions with other functions.

Original: Suppose that we are trying to solve a synthesis problem from a set of input/output examples, and during the search we create a partial program of the form f ( g ( x )) for some unknown g .
Revised: Suppose that we are trying to solve a synthesis problem from a set of input/output examples, and during the search we create a partial program of the form f ( g ( x )) for some unknown g .

Original: Since we know f , we know its property signature.
Revised: Since we know f , we know its property signature.

Original: Since we have the program speciﬁcation, we also have the estimated property signature for f ◦ g :
Revised: Since we have the program speciﬁcation, we also have the estimated property signature for f ◦ g :

Original: =
Revised: =

Original: f
Revised: f

Original: ( g ( x )) .
Revised: ( g ( x )) .

Original: If we could somehow guess the signature for g , we could look it up in a cache of previously computed functions keyed by signature.
Revised: If we could somehow guess the signature for g , we could look it up in a cache of previously computed functions keyed by signature.

Original: If we found a function matching the desired signature, we would be done.
Revised: If we found a function matching the desired signature, we would be done.

Original: If no matching function exists in the cache, we could start a smaller search with only the signature of g as the target, then use that result in our original search.
Revised: If no matching function exists in the cache, we could start a smaller search with only the signature of g as the target, then use that result in our original search.

Original: We could attempt to encode the relationship between f and g into a set of formal constraints and pass that to a solver of some kind (De Moura & Bjørner, 2008), and while that is potentially an effective approach, it may be difﬁcult to scale to a language like Searcho.
Revised: We could attempt to encode the relationship between f and g into a set of formal constraints and pass that to a solver of some kind (De Moura & Bjørner, 2008), and while that is potentially an effective approach, it may be difﬁcult to scale to a language like Searcho.

Original: Instead, we can simply train a machine learning model to predict the signature of g from the signature of f and the signature of f ◦ g .
Revised: Instead, we can simply train a machine learning model to predict the signature of g from the signature of f and the signature of f ◦ g .

Original: Here we present an experiment to establish a proof of concept of this idea.
Revised: Here we present an experiment to establish a proof of concept of this idea.

Original: First, we generated a data set of 10,000 random functions taking lists of integers to lists of integers.
Revised: First, we generated a data set of 10,000 random functions taking lists of integers to lists of integers.

Original: Then we randomly chose 50,000 pairs of functions from this list, arbitrarily designating one as f and one as g .
Revised: Then we randomly chose 50,000 pairs of functions from this list, arbitrarily designating one as f and one as g .

Original: We then computed the signatures of f , g and f ◦ g for each pair, divided the data into a training set of 45, elements and a test set of 5,000 elements, and trained a small fully connected neural network to predict the signature of g from the other two signatures.
Revised: We then computed the signatures of f , g and f ◦ g for each pair, divided the data into a training set of 45, elements and a test set of 5,000 elements, and trained a small fully connected neural network to predict the signature of g from the other two signatures.

Original: On the test set, this model had 87.5% accuracy, which is substantially better than chance.
Revised: On the test set, this model had 87.5% accuracy, which is substantially better than chance.

Original: We inspected the predictions made on the test set and found interesting examples like the one in Listing 6, where the model has learned to do something you might (cautiously) refer to as logical deduction on properties.
Revised: We inspected the predictions made on the test set and found interesting examples like the one in Listing 7, where the model has learned to do something you might (cautiously) refer to as logical deduction on properties.

Original: This result is suggestive of the expressive power of property signatures.
Revised: This result is suggestive of the expressive power of property signatures.

Original: It also points toward exciting future directions for research into neurally guided program synthesis.
Revised: It also points toward exciting future directions for research into neurally guided program synthesis.

Original: \:List<Int>->List<
Revised: [Equation] [Equation]

Original:  Int> inputs { consume_ (inputs, (list_foldl_<Int, Int> (inputs, int_min, mod_)))
Revised: [Equation] [Equation] Int> (outputs

Original: } 3 g: \:List<Int>->List<
Revised: _list_< Int>

Original: Int> inputs { list_map_<Int, Int> (inputs, neg_)} 5 prop: \:(List<Int>, List<Int>)->Bool (inputs, outputs) { list_for_all_<Int> (outputs, \x {in_list_<Int> (x, inputs)})}
Revised: (x, inputs) Listing 7: Example of successful prediction made by our composition predictor model. The property in question checks whether all the elements of the output list are members of the input list. For f , the value is AllTrue , and for f ◦ g the value is Mixed .

Revised: The model doesn’t know g or its signature, but correctly predicts that the value of this property for g must be Mixed .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: There is substantial prior work on program synthesis in general.
Revised: There is substantial prior work on program synthesis in general.

Original: We can hardly do it justice here, but see some of Gottschlich et al.
Revised: We can hardly do it justice here, but see some of Gottschlich et al.

Original: Solar-Lezama (2018); Gulwani et al.
Revised: Solar-Lezama (2018); Gulwani et al.

Original: ; Allamanis et al.
Revised: ; Allamanis et al.

Original: (2018) for more detailed surveys.
Revised: (2018) for more detailed surveys.

Original: Property Based Testing: Function properties are similar to the properties from Property Based
Revised: Property Based Testing: Function properties are similar to the properties from Property Based

Original: Testing, a software testing methodology popularized by the QuickCheck library (Claessen & Hughes, 2011) that has now spread to many contexts (Gallant, 2018; Holser, 2018; Hypothesis, 2018; Luu,2015; Elhage, 2017; MacIver, 2017).
Revised: Testing, a software testing methodology popularized by the QuickCheck library (Claessen & Hughes, 2011) that has now spread to many contexts (Gallant, 2018; Holser, 2018; Hypothesis, 2018; Luu,

Original: Quickcheck properties are human-speciﬁed and operate on functions, while our properties operate on input/output pairs.

Original: Automated Theorem Proving: Synthesizing programs using machine learning is related to the idea of proving theorems using machine learning (Irving et al., 2016).
Revised: Automated Theorem Proving: Synthesizing programs using machine learning is related to the idea of proving theorems using machine learning (Irving et al., 2016).

Original: Synthesis and theorem proving are formally related as well (Howard, 1980).
Revised: Synthesis and theorem proving are formally related as well (Howard, 1980).

Original: Program Synthesis from a Programming Languages Perspective: Most existing work on synthesis approaches is from the perspective of programming language design.
Revised: Program Synthesis from a Programming Languages Perspective: Most existing work on synthesis approaches is from the perspective of programming language design.

Original: Our baseline synthesizer borrows many ideas from Feser et al.
Revised: Our baseline synthesizer borrows many ideas from Feser et al.

Original: Polikarpova et al.
Revised: Polikarpova et al.

Original: (2016) use reﬁnement types (Freeman, 1994) (roughly, a decidable version of dependent types - see Pierce & Benjamin (2002)) to give program speciﬁcations, allowing the type-checker to discard many candidate programs.
Revised: (2016) use reﬁnement types (Freeman, 1994) (roughly, a decidable version of dependent types - see Pierce & Benjamin (2002)) to give program speciﬁcations, allowing the type-checker to discard many candidate programs.

Original: Property signatures can be thought of as a compromise between reﬁnement types and dependent types: we can write down speciﬁcations with them that would be impossible to express in reﬁnement types, but we can only check those speciﬁcations empirically.
Revised: Property signatures can be thought of as a compromise between reﬁnement types and dependent types: we can write down speciﬁcations with them that would be impossible to express in reﬁnement types, but we can only check those speciﬁcations empirically.

Original: ML-Guided Program Synthesis: More recently, researchers have used machine learning to synthesize and understand programs.
Revised: ML-Guided Program Synthesis: More recently, researchers have used machine learning to synthesize and understand programs.

Original: We have mentioned Balog et al.
Revised: We have mentioned Balog et al.

Original: , but see all of: Nye et al.
Revised: , but see all of: Nye et al.

Original: ; Ellis et al.
Revised: ; Ellis et al.

Original: (2018a);
Revised: (2018a);

Original: Zohar & Wolf (2018); Kalyan et al.
Revised: Zohar & Wolf (2018); Kalyan et al.

Original: ; Ellis et al.
Revised: ; Ellis et al.

Original: (2019a)
Revised: (2019a)

Original: ; Liang et al.
Revised: ; Liang et al.

Original: ; Alon et al.
Revised: ; Alon et al.

Original: (2019) as well.
Revised: (2019) as well.

Original: Menon et al.
Revised: Menon et al.

Original: (2013) introduces the idea of features: a predecessor to the idea of properties.
Revised: (2013) introduces the idea of features: a predecessor to the idea of properties.

Original: Features differ from properties in that they are hand-crafted rather than learned, and that they were applied only on a limited string processing domain.
Revised: Features differ from properties in that they are hand-crafted rather than learned, and that they were applied only on a limited string processing domain.

Original: Deepcoder: The relationship between this work and Balog et al.
Revised: Deepcoder: The relationship between this work and Balog et al.

Original: (2016) merits special discussion.
Revised: (2016) merits special discussion.

Original: Aside from the inclusion of property signatures, they differ in the following ways:
Revised: Aside from the inclusion of property signatures, they differ in the following ways:

Original: • We use a more expressive DSL.
Revised: • We use a more expressive DSL.

Original: Their DSL only allows linear control ﬂow with a small set of functions, whereas our language is Turing complete (it has looping, recursion, etc).
Revised: Their DSL only allows linear control ﬂow with a small set of functions, whereas our language is Turing complete (it has looping, recursion, etc).

Original: We also have a larger set of allowed component functions: 86 vs. 34.
Revised: We also have a larger set of allowed component functions: 86 vs. 34.

Original: • Their machine learning method does not work straightforwardly for arbitrary programs.
Revised: • Their machine learning method does not work straightforwardly for arbitrary programs.

Original: Their training and test programs only deal with integers and lists of integers, while we have 14 different function types.
Revised: Their training and test programs only deal with integers and lists of integers, while we have 14 different function types.

Original: It would thus not be feasible to compare the techniques on anything but a tiny subset of our existing test set.
Revised: It would thus not be feasible to compare the techniques on anything but a tiny subset of our existing test set.

Original: • The test cases in Balog et al.
Revised: • The test cases in Balog et al.

Original: (2016) are generated from their enumerative synthesizer.
Revised: (2016) are generated from their enumerative synthesizer.

Original: It is therefore guaranteed that the synthesizer will be able to emit them in a reasonable amount of time during testing, so their demonstrated improvements are ‘merely’ speed-ups.
Revised: It is therefore guaranteed that the synthesizer will be able to emit them in a reasonable amount of time during testing, so their demonstrated improvements are ‘merely’ speed-ups.

Original: Our test cases are human generated, and over half of the programs synthesized using property signatures were not synthesized at all 10 given over a day of time.
Revised: Our test cases are human generated, and over half of the programs synthesized using property signatures were not synthesized at all 10 given over a day of time.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this work, we have introduced the idea of properties and property signatures.
Revised: In this work, we have introduced the idea of properties and property signatures.

Original: We have shown that property signatures allow us to synthesize programs that a baseline otherwise was not able to synthesize, and have sketched out other potential applications as well.
Revised: We have shown that property signatures allow us to synthesize programs that a baseline otherwise was not able to synthesize, and have sketched out other potential applications as well.

Original: Finally, we have open sourced all of our code, which we hope will accelerate future research into ML-guided program synthesis.
Revised: Finally, we have open sourced all of our code, which we hope will accelerate future research into ML-guided program synthesis.

Revised: A CKNOWLEDGMENTS

Revised: We would like to thank Kensen Shi, David Bieber, and the rest of the Program Synthesis Team for helpful discussions.

Revised: We would like to thank Colin Raffel for reading a draft of the paper.

Revised: Most of all, we owe a substantial debt to Niklas Een, on whose Evo programming language ( https://github.com/tensorflow/deepmath/tree/master/ deepmath/zz/CodeBreeder )

Revised: the Searcho language is heavily based.

Original: A A PPENDIX A. F URTHER D ETAILS ON THE B ASELINE S YNTHESIZER
Revised: A A PPENDIX A. F URTHER D ETAILS ON THE B ASELINE S YNTHESIZER

Original: This section contains details on the baseline synthesizer that did not ﬁt into the main text.
Revised: This section contains details on the baseline synthesizer that did not ﬁt into the main text.

Original: Figure 2 gives a more formal description of the basic synthesis algorithm.
Revised: Figure 2 gives a more formal description of the basic synthesis algorithm.

Original: Listing 7 shows an example trajectory of partial program expansions.
Revised: Listing 8 shows an example trajectory of partial program expansions.

Original: 1 $1 \:(Int, Int)->(Int, Int) (a2, a3) {?} 2 $2 \:(Int, Int)->(Int, Int) (a2, a3) {(?, ?)} 3 $2 \:(Int, Int)->(Int, Int) (a2, a3) {(a3, ?)} 4 $2 \:(Int, Int)->(Int, Int) (a2, a3) {(a3, a2)}
Revised: 1 $1 \:(Int, Int)->(Int, Int) (a2, a3) {?} 2 $2 \:(Int, Int)->(Int, Int) (a2, a3) {(?, ?)} 3 $2 \:(Int, Int)->(Int, Int) (a2, a3) {(a3, ?)} 4 $2 \:(Int, Int)->(Int, Int) (a2, a3) {(a3, a2)}

Original: Listing 7: The trajectory the synthesizer took to generate the swap function, which just swaps the two elements of a tuple.
Revised: Listing 8: The trajectory the synthesizer took to generate the swap function, which just swaps the two elements of a tuple.

Original: Since it knows it needs to take a tuple of ints as an argument and return a tuple of ints, it starts with a hole of type ( Int , Int ) in line 1.
Revised: Since it knows it needs to take a tuple of ints as an argument and return a tuple of ints, it starts with a hole of type ( Int , Int ) in line 1.

Original: It then converts that hole into a tuple of holes, both of type Int in line 2, ﬁlls one of the holes with a reference to one of the arguments in line 3, and ﬁlls in the ﬁnal hole with a reference to the other argument in line 4.
Revised: It then converts that hole into a tuple of holes, both of type Int in line 2, ﬁlls one of the holes with a reference to one of the arguments in line 3, and ﬁlls in the ﬁnal hole with a reference to the other argument in line 4.

Original: Note that this listing doesn’t show all programs attempted, it just shows the sequence of partial programs that led to the ﬁnal solution.
Revised: Note that this listing doesn’t show all programs attempted, it just shows the sequence of partial programs that led to the ﬁnal solution.

Original: A. C OMPARISON WITH D EEP C ODER

Original: We have conducted an experiment to compare premise selection using Property Signatures to the premise selection algorithm from (Balog et al., 2016).

Original: This required considerable modiﬁcations to the experimental procedure.

Original: First, since the premise-selection part of DeepCoder can only handle Integers and lists of Integers, we restricted the types of our training and test functions.

Original: In particular, we read through (Balog et al., 2016) and found four function types in use:

Original: 1 f :: [Int] -> [Int] 2 g :: [Int] -> Int 3 h :: ([Int], [Int]) -> Int 4 k :: ([Int], Int) ->

Original: Int

Original: Listing 8: The four function types used in DeepCoder.

Original: The types of f and g in 8 are taken directly from (Balog et al., 2016).

Original: The types of h and k are inferred from examples given in the appendix of (Balog et al., 2016).

Original: Their DSL does not technically have tuples, but we have wrapped the inputs of their ‘two-input-functions’ in tuples for convienence.

Original: Second, since DeepCoder can only handle integers betwen − 255 and 255 , we ﬁrst re-generated all of our random inputs (used for ‘hashing’ of generated training data) to lie in that range.

Original: We then generated random training functions of the above four types.

Original: We then made a data set of training functions associated with 5 input-output pairs, throwing out pairs where any of the outputs were outside the aforementioned range, and throwing out functions where all outputs contained some number outside that range.

Original: Third, of the examples in our test set with the right types, we modiﬁed their input output pairs in a similar way.

Original: We ﬁltered out functions that could not be so modiﬁed.

Original: After doing so, we were left with a remaining test suite of 32 functions.

Original: Finally, we trained a model to predict functions-to-use from learned embeddings of the input-output pairs, as in DeepCoder.

Original: We didn’t see a description of how functions with multiple inputs had their inputs embedded, so we elected to separate them with a special character, distinct from the null characters that are used to pad lists.

Original: Compared with the Property Signatures method, this technique results in far fewer synthesized test set programs.

Original: We did 3 random restarts for each of DeepCoder, Property Signatures, and the Random

Original: Baseline (recall that the random baseline itself is already a relatively sophisticated synthesis algorithm - it’s just the conﬁgurations that are random).

Original: The 3 DeepCoder runs synthesized an average of 3 .

Original: 33 test programs, while the Property Signature runs (trained on the same modiﬁed training data and tested on the same modiﬁed test data) synthesized 16 .

Original: The random baseline synthesizedprograms on average.

Original: A priori, this seems like a surprisingly large gap, but it actually ﬁts with what we know from existing literature.

Original: Shin et al.

Original: (2018) observe something similar: which is that DeepCoder-esque techniques tend to generalize poorly to a a test set where the input-output pairs come from a different distribution than they do in training.

Original: This is the case in our experiment, and it will be the case in any realistic setting, since the test set will be provided by users.

Original: Property Signatures are (according to our experiments) much less sensitive to such shift.

Original: This makes intuitive sense: whether an input list is half the length of an output list (for instance) is invariant to the particular distribution of members of the list.

Original: Note that even if Property Signatures did not outperform DeepCoder on this subset of our test set, they would still constitute an improvement due to their allowing us to operate on arbitrary programs and inputs types.