O PTIMAL N EURAL P ROGRAM S YNTHESIS FROM M ULTIMODAL S PECIFICATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Multimodal program synthesis, which leverages different types of user input to synthesize a desired program, is an attractive way to scale program synthesis to challenging settings; however, it requires integrating noisy signals from the user (like natural language) with hard constraints on the program’s behavior.
This paper proposes an optimal neural synthesis approach where the goal is to ﬁnd a program that satisﬁes user-provided constraints while also maximizing the program’s score with respect to a neural model.
Speciﬁcally, we focus on multimodal synthesis tasks in which the user intent is expressed using combination of natural language (NL) and input-output examples.
At the core of our method is a top-down recurrent neural model that places distributions over abstract syntax trees conditioned on the NL input.
This model not only allows for efﬁcient search over the space of syntactically valid programs, but it allows us to leverage automated program analysis techniques for pruning the search space based on infeasibility of partial programs with respect to the user’s constraints.
The experimental results on a multimodal synthesis dataset (S TRUCTURED R EGEX ) show that our method substantially outperforms prior state-of-the-art techniques in terms of accuracy and explores fewer states during search.
I NTRODUCTION
In recent years, there has been a revolution in machine learning-based program synthesis techniques for automatically generating programs from high-level expressions of user intent, such as input-output examples (Balog et al., 2017; Chen et al., 2019a;
Devlin et al., 2017; Ellis et al., 2019; Kalyan et al., 2018; Shin et al., 2018) and natural language (Yaghmazadeh et al., 2017; Dong & Lapata, 2016;
Rabinovich et al., 2017; Yin & Neubig, 2017;
Desai et al., 2016; Wang et al., 2018).
Many of these techniques use deep neural networks to consume speciﬁcations and then perform model-guided search to ﬁnd a program that satisﬁes the user.
However, because the user’s speciﬁcation can be inherently ambiguous (Devlin et al., 2017; Yin et al., 2018), a recent thread of work on multimodal synthesis attempts to combine different types of cues, such as natural language and examples, to allow program synthesis to effectively scale to more complex problems.
Critically, this setting introduces a new challenge: how do we efﬁciently synthesize programs with a combination of hard and soft constraints from distinct sources?
In this paper, we formulate multimodal synthesis as an optimal synthesis task and propose an optimal synthesis algorithm to solve it.
The goal of optimal synthesis is to generate a program that satisﬁes any hard constraints provided by the user while also maximizing the score under a learned neural network model that captures noisy information, like that from natural language.
In practice, there are many programs that satisfy the hard constraints, so this maximization is crucial to ﬁnding the program that actually meets the user’s expectations: if our neural model is well-calibrated, a program that maximizes the score under the neural model is more likely to be the user’s intended program.
Our optimal neural synthesis algorithm takes as input multimodal user guidance.
In our setting, we train a neural model to take natural language input that can be used to guide the search for a program consistent with some user-provided examples.
Because our search procedure enumerates programs according to their score, the ﬁrst enumerated program satisfying the examples is guaranteed to be optimal according to the model.
A central feature of our approach is the use of a tree-structured neural model, namely the abstract syntax network (ASN) (Rabinovich et al., 2017), for constructing
[Figure] [Figure]
[Figure]
syntactically valid programs in a top-down manner.
The structure of the ASN model restricts search to programs that are syntactically correct, thereby avoiding the need to deal with program syntax errors (Kulal et al., 2019), and it allows us to search over programs in a ﬂexible way, without constraining a left-to-right generation order like seq2seq models do.
More importantly, the use of top-down search allows us to more effectively leverage automated program analysis techniques for proving infeasibility of partial ASTs.
As a result, our synthesizer can prune the search space more aggressively than prior work and signiﬁcantly speed up search.
While our network structure and pruning techique are adapted from prior work, we combine them and generalize them to this optimal neural synthesis setting in a new way, and we show that our general approach leads to substantial improvements over previous synthesis approaches.
We implement our method in a synthesizer called O P S YNTH and evaluate it on the challenging
S TRUCTURED R EGEX dataset (Ye et al., 2020a) for synthesizing regular expressions from linguistically diverse natural language descriptions and positive/negative examples.
We compare our approach against a range of approaches from prior work and ablations of our own method.
O P S YNTH achieves substantial gain over past work by solving 59.8% (resp.
46.9%) of the programs of Test (resp.
Test-E) set in S TRUCTURED R EGEX by exploring on average 560 (resp. 810) states, which surpasses previous state-of-the-art by 11.7% (resp.
10.9%) with 23 × (resp.
18 × ) fewer states.
P ROBLEM F ORMULATION
Context-free grammar.
In this work, we assume that the syntax of the target programming language L is speciﬁed as a context-free grammar G =
( V, Σ , R, S 0 ) where V is a set of non-terminals, Σ is the set of terminal symbols, R is a set of productions, and S 0 is the start symbol.
We use the notation s to denote any symbol in V ∪ Σ .
The grammar in Figure 1 has two nonterminals ( S 0 and V 1 ) and three terminals ( cat , <0> , and <1> ).
To simplify presentation in the rest of the paper, we assume that each grammar production is of the form v → f ( s 0 , . . .
, s n ) where f is a language construct (e.g., a constant like 0 or a built-in function/operator like cat , + , etc.).
We represent programs in terms of their abstract syntax trees (AST).
We assume that every node n in an abstract syntax tree is labeled with a grammar symbol s (denoted S ( n ) ), and that every node is labeled with a production r ∈ R (denoted R ( n ) ) that indicates which CFG production was used to assign a terminal symbol for that node (if applicable).
Figure 2 shows an AST representation of the program cat ( cat ( <0> , <1> ) , <0> ) generated using the simple grammar shown in Figure 1.
Partial programs.
For the purposes of this paper, a partial program is an AST in which some of the nodes are labeled with non-terminal symbols in the grammar (see Figure 3).
For a complete program , all node labels are terminal symbols.
We use the notation E XPAND ( P, l, r ) to denote replacing leaf l with production r , which adds n nodes s 1 , . . .
, s n to the tree corresponding to the yield of r .
Consistency with examples.
In this paper, we focus on multimodal synthesis problems where the user provides a logical speciﬁcation φ in addition to a natural language description.
More concretely, we focus on logical speciﬁcations that are in the form of positive and negative examples on the program behavior.
Each example is a pair ( x, y ) such that, for a positive example, we have P ( x ) =
y for the target program P , and for a negative example, we have P ( x ) (cid:54) =
y .
Given a set of examples
E = E + ∪ E − and program P , we write P | =
E , if we have P ( x ) =
y for every positive example in
E +
and we have P ( x ) (cid:54) =
y for every negative example in E − .
If P is a partial program, P (cid:54)| = φ indicates that there is no completion of P that satisﬁes the speciﬁcation φ .
Optimal multimodal synthesis problem.
A second input to our multimodal synthesis problem is a natural language description of the task.
We deﬁne a model M θ ( P | N ) that yields the probability of a given program conditioned on the natural language description (described in Section 3).
Given a programming language L speciﬁed by its context-free grammar, a logical speciﬁcation φ (e.g., a set of positive and negative examples), natural language description N and a model M θ , our goal is to ﬁnd the most likely program in the language satisfying the constraints:
[Equation]
O PTIMAL N EURAL S YNTHESIS A LGORITHM
We consider a class of models M θ that admit efﬁcient optimal synthesis.
Any model with the properties described in this section can be plugged into our synthesis algorithm (Section 3.2).
Deﬁnition 3.1.
(AST Path)
Given a node n in a partial program P , we deﬁne the AST path π
( P, n ) =
(( n 1 , i 1 ) , . .
, ( n k , i k )) to be a sequence of pairs ( n j , i j ) where (1) AST node n j +1 is the i j ’th child of AST node n j and (2) the i k ’th child of n k is n .
For instance, for the partial program in
Figure 3, we have π ( P, n 4 ) =
(( n 1 , 1) , ( n 2 , 1)) .
Deﬁnition 3.2.
(Concrete/Inconcrete nodes)
Given a partial program P , we deﬁne the concrete nodes of P as C ( P ) to be the nodes which have production rules assigned to them.
The inconcrete nodes I ( P ) are the non-terminal leaf nodes whose production rules haven’t been determined and need to be ﬁll in in order to form a complete program.
Given a partial program P , we deﬁne the probability of generating P as the product of the probabilities of applying the productions labeling each node in the AST.
There are a number of possible ways we could factor and parameterize this distribution, including PCFGs, where the distribution depends only on the parent, or as sequence models over a pre-order traversal of the tree (Dong & Lapata, 2016;
Yin & Neubig, 2017; Polosukhin & Skidanov, 2018).
We choose the following factorization, similar to that used in Abstract Syntax Networks (ASN) (Rabinovich et al., 2017), where a production rule depends on the derivation path leading to that nonterminal:
[Equation]
The chief advantage of this factorization is that the score of a partial program is invariant to the derivation order of that program : two derivations of the same tree P that differ only in the order that branches were generated are still assigned the same probability, allowing for ﬂexibility in the search process.
Second, for a partial program P , the distribution over rules of every unexpanded non-terminal leaf node does not depend on the others’, which allows the estimation of the upper bound (maximum possible probability) of the complete programs that can be derived from P .
Speciﬁcally, we deﬁne the upper bound of the complete programs that can possibly be derived from a partial program P as:
[Equation]
This bound incorporates the known probabilities of concrete nodes as well as the minimum cost of ﬁlling inconcrete non-terminals, and thus more accurately estimates the cost of the optimal complete program given this partial program.
A sequence model traversing the tree with a ﬁxed order cannot estimate such an upper bound as the probabilities of inconcrete nodes are not known.
3. N EURAL M ODEL
We instantiate the neural model deﬁned above using a simpliﬁed version of ASN (Rabinovich et al., 2017), which respects the p θ ( R ( n ) | π
( P, n ) , N ) factorization for the production of each node in the tree.
Figure 4 illustrates how ASN recursively computes the probability of labeling a node n as R ( n ) .
[Figure]
D
[Figure]
and(startwith(<let>,S),S)
cat(S,S) and(contain(S,S),S)
Infeasible
Feasible cat(rep(S,D),S)
[Figure]
(not(S),S) 0.
[Figure]
[Figure]
W o r k li s t
[Figure]
[Table]
over rules …
[Figure]
Consider the partial program cat ( S ( n 1 ), S ( n 2 )); we need to deﬁne the probability distribution over legal productions on the ﬁrst node n 1 : p θ ( R ( n 1 )
| π ( P, n ) , N ) =
p θ ( R ( n 1 ) | { ( cat,1 ) } , N ) .
We encode the AST path using an LSTM (Hochreiter & Schmidhuber, 1997).
Deﬁne
LSTM( h 0 , ( r j , i j ) ) to be an LSTM with initial state h 0 and which, at each timestep, consumes a tuple consisting of a node n j and a parent-child index
i j (i.e., an element in π ( P, n ) ).
1 We embed each tuple ( n j , i j ) by W R ( n j ) ,i j , where W is specialized to the rule and position.
Then: h root =
LSTM ( N ) and h n = LSTM ( h root , π ( P, n )) where LSTM ( N ) denotes an encoding of the natural language.
The hidden state h n encodes both the user’s NL speciﬁcation as well as where we are in the parse tree, allowing us to model which grammar symbol should be likely at this position.
Given this hidden state h n , the probability for each production rule at node n is computed using a feedforward neural network (FFNN) module and attention over the NL input:
[Equation]
During search, each Expand operation instantiates a node n with each possible rule according to the probabilities above, then computes the hidden states for any new nonterminals using the LSTM.
3. S YNTHESIS
In this section, we describe a search algorithm to solve the optimal neural synthesis problem deﬁned in Equation 1.
The key idea is to maintain a priority list Q of partial programs, ranked according to the upper bound ( u θ ( P ) ) probability of the complete programs that can be derived from this partial program.
Then, in each iteration of the search procedure, we pick the highest upper bound partial
[Figure]
program P in Q , check its feasibility using program analysis, and if it is feasible, expand one of the non-terminals in P using the applicable CFG productions.
Since complete programs are dequeued from Q in decreasing order of their probability according to M θ , the ﬁrst complete program that satisﬁes φ is guaranteed to be optimal under M θ (Proof in the in Appendix); thus, our algorithm is guaranteed to return an optimal program if a solution exists.
Infeasibility pruning Our top-down search allows us to exploit program analysis techniques to prune the search space, by determining whether P is infeasible with respect to the user’s hard constraints.
A common way of doing this is to use well-known abstract interpretation techniques from the programming languages literature to approximate program semantics (Cousot & Cousot, 1977; Nielson et al., 2015).
In particular, given a partial program P , the idea behind the feasibility checking procedure is to generate a pair of logical formulas ( ψ + , ψ − ) over-
and under-approximating
P ’s semantics respectively.
If there is any positive example e + ∈ E
+ that is inconsistent with ψ + , then the partial program is infeasible.
Similarly, if there is any negative example e − ∈ E − that satisﬁes ψ − , we can again conclude that P must be infeasible.
Figure 5 describes our feasibility checking procedure in terms of inference rules, where rules (a) and (b) generate a pair of over- and under-approximations of the program, and rule (c) checks feasibility of these approximations with respect to the provided examples.
Here, free variables x in the formula represent program inputs, and free variable y represents the program output.
The existentially quantiﬁed variables z corresponds to values of sub-expressions.
The ﬁrst rule states that “holes" (i.e., non-terminals) in the partial program are over-approximated using y = (cid:62) meaning the sub-program can return anything, and they are under-approximated using y = ⊥ , meaning that the sub-program returns nothing.
The second rule is used to (recursively) construct an approximation of a sub-AST rooted at node n .
This rule utilizes a pair of mappings Φ + , Φ − where Φ + (resp.
Φ − ) gives an over-approximating (resp.
under-approximating) semantics for each language construct.
In rule (b), each child formula ψ + i , ψ − i must be satisﬁed as well as the parent formula, and these are uniﬁed by a shared set of new existentially-quantiﬁed variables.
The ﬁnal rule uses the generated over- and under-approximations of the partial program to check feasibility.
In particular, we conclude that the partial program is infeasible if there is any positive example e + ∈ E
+ that is inconsistent with ψ + or any negative example e − ∈ E − that satisﬁes ψ − .
E XPERIMENTAL S ETUP
We evaluate our synthesizer on the S TRUCTURED R EGEX dataset for multimodal synthesis of regular expressions.
This dataset contains 3520 labeled examples, including an NL description, positive/negative examples, and the target regex.
We choose this dataset for our evaluation because (1) it is only the dataset containing both examples and NL where the NL description is written by humans, and (2) this dataset is quite challenging, with existing techniques achieving under 50% accuracy.
Implementation Details As stated in Section 3.1, our model is an Abstract Syntax Network tailored to ﬁt the regex DSL used in S TRUCTURED R EGEX .
We train our neural model to maximize the log likelihood of generating ground truth regexes given the NL using the Adam optimizer (Kingma & Ba, 2015), stopping when the performance on dev set converges.
More details are in the Appendix.
We implement the infeasibility checking procedure for our regex DSL by encoding the semantics of each operator in the theory of strings (Liang et al., 2014).
Since all existentially quantiﬁed variables in the resulting formula can be eliminated through substitution, the resulting constraints are of the form s ∈ R (or s (cid:54)∈ R ) where s is a string constant and R is a regular expression.
Thus, we can check the satisﬁability of these formulas using the Bricks library (Møller, 2017).
The supplementary material describes both the semantics of the DSL constructs as well as an example showing how to generate the encoding for a given partial program.
Because of our infeasibility check, the order of expanding non-terminals can impact the efﬁciency of our search.
We experimented with several methods of selecting a leaf node to expand, including pre-order traversal, choosing high-level nodes ﬁrst, and choosing lowest-entropy nodes ﬁrst.
Pre-order traversal seemed to work best; more details about the expansion order can be found in Appendix.
Baselines We compare our method against three programming-by-example ( PBE-only ) baselines,
A LPHA R EGEX (Lee et al., 2016), D EEP C ODER (Balog et al., 2017), and R OBUST F ILL (Devlin et al., 2017).
A LPHA R EGEX is an enumerative regex synthesizer that uses breadth-ﬁrst search to ﬁnd regexes that are consistent with the examples.
Both D EEP C ODER and R OBUST F ILL are neural program synthesis approaches.
D EEP C ODER places distribution over constructs and terminals based on examples, and uses this distribution to carry out DFS search, whereas R OBUST F ILL uses beam search to autoregressively build programs.
We further compare our method against prior multimodal program synthesis techniques,
S KETCH (Ye et al., 2020b) and T REE S EARCH (Polosukhin & Skidanov, 2018) with appropriate tuning of the hyperparameters and the S KETCH synthesizer for this setting.
We do not compare against
S KETCH A DAPT (Nye et al., 2019) because it relies on the assumption that every program consistent with examples is the gold program, which does not hold in our setting.
We also consider two NL-to-code models, Seq2Seq and TranX (Yin & Neubig, 2017), which we modify to ﬁlter out complete programs that are inconsistent with the examples (Chen et al., 2020;
Polosukhin & Skidanov, 2018).
A more sophisticated baseline (Ye et al., 2020a) uses example-guided pruning by ﬁltering the beam at every timestep during search.
We adopt these more sophisticated baselines proposed in Ye et al.
(2020a) to allow a fair comparison.
Implementation details of all our baselines are in the Appendix.
We refer to our Optimal Synthesis approach as O P S YNTH .
We also show ablations: ASN + P (ASN with our pruning during beam search), and O P S YNTH - P to further demonstrate the beneﬁts of our approach over models like Polosukhin & Skidanov (2018) that do not use such pruning.
Finally, we also consider an extension denoted as O P S YNTH + R , which extends O P S YNTH with the A TTENTION
A M ODEL from R OBUST F ILL (Devlin et al., 2017), which encodes the examples φ using another set of LSTM layers.
To combine these signals, we deﬁne the probability of applying rule r on n as:
[Equation]
Attn
[Equation] [Equation]
φ
[Equation]
R ESULTS AND A NALYSIS
In the following experiments, we evaluate our approach based on two criteria: (1) accuracy, deﬁned as the fraction of solved synthesis tasks, and (2) efﬁciency, deﬁned in terms of the number of partial programs searched.
Note that we deﬁne efﬁciency in this way because the bottleneck is applying the
E XPAND function on partial programs and symbolic evaluation of these partial programs rather than the neural net computation.
Main Results Our main results are shown in Table 1.
We report results on two test sets from
S TRUCTURED R EGEX ; Test-E is annotated by a distinct set of annotators from the training set.
As shown in the top part of Table 1, pure PBE approaches (top) do poorly on this dataset due to not utilizing NL.
These approaches either fail to ﬁnd a regex consistent with the examples within a time limit of 90 seconds or the synthesized regex is semantically different from the target one.
Thus, the comparison against PBE-only approaches demonstrates the importance of using a model that places distributions over programs conditioned on the NL description.
[Table]
The second and third part of Table 1 shows results from prior multimodal neural synthesis approaches and NL-to-code models augmented with example-based pruning (Ye et al., 2020a).
S KETCH slightly outperforms T REE S EARCH , solving 45% and 30% of the Test and Test-E set respectively.
Seq2Seq + P and TranX + P , which perform beam search guided by the Seq2Seq and TranX models but also check feasibility of partial programs before adding them to the beam, outperform these other techniques: TranX + P outperforms Seq2Seq + P and solves 53% of the benchmarks on Test and 38% for Test-E.
The last part of Table 1 provides results about O P S YNTH and its ablations.
O P S YNTH achieves a substantial improvement over TranX + P and is able to solve approximately 60% of benchmarks in
Test and 47% in Test-E.
In addition to solving more benchmarks, O P S YNTH also explores only a fraction of the states explored by TranX + P , which translates into improving synthesis time by roughly an order of magnitude.
We now compare O P S YNTH against three of its ablations.
O P S YNTH - P does not use program analysis to prune infeasible partial programs (hence, we do not report explored states as a measure of runtime), and ASN + P is similar to O P S YNTH except that it uses beam search (with beam size 20) combined with example-based pruning.
Both the program analysis component and optimal search are important: without these, we observe both a reduction in accuracy and an increase in the number of states explored.
The last row in Table 1 shows an extension of O P S YNTH described in Section 4 where we incorporate the R OBUST F ILL model.
We ﬁnd that R OBUST F ILL is ineffective on its own, and incorporating it into our base synthesizer actually decreases performance.
While such neural-guided
PBE approaches (D EEP C ODER (Balog et al., 2017) and R OBUST F ILL (Devlin et al., 2017)) have been successful in prior work, they do not appear to be effective on this challenging task, or not necessary in the presence of strong natural language hints.
Additionally, these models both rely on millions of synthetic examples in the original reported settings.
Optimality and efﬁciency.
We now explore the beneﬁts of optimal neural synthesis in more detail.
Speciﬁcally, Table 2 compares O P S YNTH with its ablations that perform beam search with varying beam sizes for Test-E.
For the purposes of this experiment, we terminate O P S YNTH ’s search after it has explored a maximum of 2500 states.
For the beam-search-based ablations, we terminate search when the beam is ﬁlled up with complete programs or the size of partial programs in the beam exceeds a speciﬁed threshold.
In Table 2, the column labeled “% Opt" shows the percentage of optimal programs found by the search algorithm.
We also show the gap (difference of log probability) between the best programs found by each approach and the optimal programs; this is reported in the column labeled “Gap”.
Finally, the last three columns show the fraction of solved problems (accuracy), the fraction of programs consistent with the examples, and the number of explored states respectively.
As seen in Table 2, our optimal synthesizer can ﬁnd the optimal program in 75.5% of cases and solves 46.9% problems after exploring 810 states on average.
Beam search with a beam size of can only ﬁnd 66.2% optimal programs, and it solves fewer problems (
8%) despite exploring 1 . 8 × more states.
Although larger beams explore more states than O P S YNTH and ﬁnd more I/O-consistent programs, they solve fewer problems overall.
We further evaluate the beneﬁt of ﬁnding model-optimal programs in Figure 6.
Here, we focus only on those programs that are consistent with the input-output examples.
The x-axis shows the distance from the optimal program, and the y-axis shows the % of programs that are functionally equivalent to the desired regex.
As shown in Figure 6, 62% of optimal programs are equivalent to the target regex, whereas only around 30% of the non-optimal programs match the ground truth functionally.
[Table]
Finally, Figure 7 plots the fraction of solved problems with respect to the number of states explored.
O P S YNTH consistently solves more problems than the other methods given the same budget.
Additionally, it also does not require specifying a beam size.
R ELATED W ORK
Natural Language to Logical Forms Semantic parsing (translating NL to executable logical forms) has been a long-standing research problem in the NLP community (Zelle & Mooney, 1996; Price, 1990).
Traditional grammar-based semantic parsers can construct database queries (Zelle & Mooney, 1996; Price, 1990), lambda calculus expressions (Zettlemoyer & Collins, 2005) and programs in other DSLs (Kushman & Barzilay, 2013; Wang et al., 2015).
Recent advances in deep learning have explored seq2seq (Jia & Liang, 2016) or seq2tree models (Dong & Lapata, 2016) that directly translate the NL into a logical form, and syntax-based models (Yin & Neubig, 2017) can also inject syntactic constraints.
Our approach relies on similar neural modeling to predict the distribution of target programs from NL.
However, search is much more complex in our example-guided synthesis setting, whereas prior neural semantic parsers approximate the best solution using beam search (Dong & Lapata, 2016; Yin & Neubig, 2017).
Optimal Synthesis with Examples Prior work on PBE considers various notions of optimality using cost functions (Bornholt et al., 2016; Feser et al., 2015; Schkufza et al., 2013) and machine learning (Menon et al., 2013).
The ﬁrst line of work allows users to specify the desired properties of the synthesized program; for instance, smaller program size, lower execution time, or more efﬁcient memory usage.
Menon et al.
(2013) deﬁne optimality as the most likely constructs given a set of examples under a probabilistic context free grammar.
In this work, we focus on a new setting where we guarantee the optimality with respect to a neural modal, which can encode speciﬁcations such as natural language that are hard to formulate as simple cost functions.
Multimodal Program Synthesis There has been recent interest in synthesizing programs using a combination of natural language and examples (Polosukhin & Skidanov, 2018;
Chen et al., 2019b;
Nye et al., 2019; Andreas et al., 2018; Raza et al., 2015).
Speciﬁcally, Chen et al.
(2020) and Ye et al.
(2020b) parse the natural language into an intermediate representation and then use it to guide enumeration, but they do not have any optimality guarantees with respect to the neural model.
Kulal et al.
(2019) synthesize programs by performing line-by-line translation of pseudocode to code and verify consistency with test cases at the end.
However, unlike our approach, their technique enumerates syntactically ill-formed programs, which they address using compiler error localization.
C ONCLUSION
In this paper we presented a technique for optimal synthesis from multimodal speciﬁcations.
On a benchmark of complex regex synthesis problems, we showed that this approach is substantially more accurate than past models and our synthesis algorithm ﬁnds the model-optimal program more frequently compared to beam search.
While we have evaluated this method in the context of regular expressions, our technique is also applicable for other synthesis tasks.
A G UARANTEE OF O PTIMALITY
Theorem 1 (Guarantee of Optimality) .
Suppose given a CFG G = ( V, Σ , R, S 0 ) , speciﬁcation φ , natural language N and model M θ , O P S YNTH returns a program P ∗ . Then, for any program P | = φ , M θ ( P ) ≤ M θ ( P ∗ ) .
Proof.
Assume P ∗ is the returned program of O P S YNTH ( G , φ, N, M θ ) and there exits a program P such that P | = φ and M θ ( P ) >
M θ ( P ∗ ) .
Since M θ ( P ) >
M θ ( P ∗ ) , P must have been present in the worklist and considered as a concrete program before the model visited P ∗ .
But then, given that
P | = φ , then O P S YNTH will return P rather than P ∗ , which contradicts the assumption.
B CFG FOR R EGULAR E XPRESSIONS
We present the CFG for the regex domain language taken from S TRUCTURED R EGEX (Ye et al., 2020a).
Its correspondence to the constructions in the standard regular expression is shown in the
Appendix A of Ye et al.
(2020a).
[Figure]
startwith
[Figure]
) | endwith ( V 1 ) | contain ( V 1 ) not ( V 1 ) | and ( V 1 , V 1 ) | or ( V 1 , V 1 )
[Figure]
C E NCODING FOR THE I NFEASIBLE P ROCEDURE FOR R EGEX
[Equation]
[Figure]
[Equation]
[Figure] InLang , y, x , z ) = ( y ∧ ( x ∈ z 0 ))
f ∈ { startwith , endwith , contain , not , optional , star } Φ { + , −} ( f, y, z ) = ( y = f ( z 1 ) f ∈ { cat , or , and , or , repeat , repatleast } Φ { + , −} ( f, y, z ) =
( y = f ( z 1 ,
[Equation]
We describe our detailed instantiation of the I NFEASIBLE procedure described in Section 3.2 in the regex domain.
Recall that I NFEASIBLE prunes a given partial program P by checking consistency between the approximated semantics and the given examples.
In the regex domain, we encode the semantics of a regex in terms of the set of strings it can match.
To enable checking consistency between a given example and the regex, given a string s , we use the program InLang ( s, P ) (denoted as P (cid:48) ) to represent whether s is in the set of strings that can be matched by P .
To encode a program
P (cid:48) for consistency checking, we use the set of encoding rules presented in Figure 9 to generate its over- and under- approximated semantics.
In the regex domain, for most of the constructs we can model the precise semantics except for the non-terminal symbols in the partial program.
Example.
Consider the following partial program P : cat ( or ( <0> , V 1 ) , <1> ) .
To utilize the positive and negative examples for pruning [Equation] we ﬁrst encode the semantics of the program P (cid:48) :
InLang ( x, P ) as follows:
[Equation]
We can simplify formulas ψ + , ψ − by eliminating the existentially quantiﬁed variables:
[Equation]
Let the positive example be i = "a1" , o = True and let the negative example be i = "01" , o =
True .
According rule (c) of Figure 5, we check if the following formula is unsat:
[Equation]
Since the under-approximated semantics of P contains the string "01" , this formula is indeed unsat and we are able to prune this partial program.
D N EURAL M ODEL D ETAILS
As described in Section 3.1, our neural model resembles an Abstract Syntax Network (Rabinovich et al., 2017) tailored to ﬁt the regex DSL used in S TRUCTURED R EGEX .
We show the grammar in
Figure 1.
As there is no production rule having optional or sequential cardinality, we do not include the “constructor ﬁeld module” from the ASN in our implementation.
We encode the NL using a single layer Bi-LSTM encoder with a hidden state size of 100.
In the decoding phase, we set the size of the hidden state in the decoder LSTM as well as the the size of the embedding of R ( n j , i j ) to be 100.
To obtain the contexts, we use the Luong general attention scheme (Luong et al., 2015).
To prevent overﬁtting, we apply a dropout of 0.3 to the all the embedding, outputs of recurrent modules, and context vectors.
Our model is trained using Adam (Kingma & Ba, 2015) with a learning rate of 0.003 and a batch size of 25.
E S ELECT L EAF F UNCTION D ETAILS
The S ELECT L EAF function selects one non-terminal leaf node in the partial program to expand.
We ﬁnd that when programmatic constraints are integrated into the search process, the order of choose which non-terminal to expand can impact the cost needed to synthesize the target program.
We give a concrete example of how the way we select non-terminal leaf nodes to expand can affect the cost of synthesis.
Consider a timestep where we obtain the feasible partial program cat( V 1 , V 2 ) from the queue, where both V 1 and V 2 can be expanded to <0> or <1> with a probabilities 0 .
9 and 0 .
1 respectively.
Suppose cat(<0>, V 2 ) is feasible, cat( V 1 ,<0>) is infeasible, and the only feasible complete program is cat(<1>,<1>) .
If we choose to expand
V 1 ﬁrst, then the search procedure goes as follows: {( cat(<0>, V 2 ) , (cid:51) )
→ ( cat(<0>,<0>) , (cid:55) ) → ( cat(<0>,<1>) , (cid:55) )
→ ( cat(<1>, V 2 ) , (cid:55) )
→ ( cat(<1>,<0>) , (cid:55) ) → ( cat(<1>,<1>) , (cid:51) )}, which takes 6 steps.
Now, if we expand V 2 ﬁrst, the search procedure is: {( cat( V 1 ,<0>) , (cid:55) )
→ ( cat( V 1 ,<1>) , (cid:51) ), → ( cat(<0>,<1>) , (cid:55) ), → ( cat(<1>,<1>) , (cid:51) )}, which only takes 4 steps.
We want to ﬁnd an order expand the nodes that leads to most effective pruning.
We tested the following ways of selecting leaf nodes: (1) pre-order traversal, (2) choosing the highest-level leaf node, (3) choosing the lowest-entropy leaf node.
We found that pre-order traversal worked better than the other strategies in most cases.
Given the same budget, using per-order traversal solves more programs while exploring less states compared to the other ways.
The superiority of pre-order traversal on the regex synthesis task can be attributed to that our I NFEASIBLE function needs concrete terminal leaf nodes to prune effectively, and using pre-order traversal prioritizes deepest nodes and usually yields terminal leaf nodes more quickly than other strategies.
F I MPLEMENTATION D ETAILS OF THE B ASELINES A LPHA R EGEX
We implemented the top-down enumerative synthesizer presented in Lee et al.
Although Lee et al.
(2016) only uses <0> and <1> as terminals, here we extended the synthesizer to support most of the ASCII characters.
D EEP C ODER
We implemented D EEP C ODER with a few modiﬁcations from its original implementation (Balog et al., 2017).
First, we assign each token in the examples with a class , and embed the token by both its value and its class.
For instance, consider the positive example (ax4,+) of the regex concat(repeat(<low>,2),repatleast(<num>,1) (2 lower letters followed byor more digits.
We assign “ a ” and “ b ” with the “ <low> ” class, and assign “ 4 ” with the “ <num> ” class.
The ﬁnal embedding of the token “ a ” is the concatenation of the embedding of the value
Emb ( a ) and the class Emb ( <low> ) .
We use such combined embeddings for better generalizability.
Then, we encode the examples with a Bi-LSTM encoder.
Each example is encoded into a hidden vector, which is later max-pooled.
Finally, we apply a linear layer on the pooled representation for the whole program, and predict the the set of probabilities for each of the constructs in the DSL.
We extended A LPHA R EGEX to synthesize programs using the probability of constructs obtained from the neural model.
In the S TRUCTURED R EGEX grammar, we associate each construct with the score returned from the neural network and calculate the score of a partial program by summing up the score of all the constructs that are used in the partial program.
We specify the synthesizer to prioritize exploring the partial programs with the highest score so far.
Recall that in Section 5 that D EEP C ODER doesn’t achieve high performance in the S TRUCTURE R EGEX dataset.
Since most of the constructs are recursive in the regex language and
D EEP C ODER search is essentially doing a depth-ﬁrst search, the synthesizer ﬁrst needs to exhaustively check all possible programs associated with the highest probability constructs before it can move on to explore those programs with any other constructs.
For example, suppose the concat has the highest probability and the synthesizer explores programs up to maximum depth 5 , the synthesizer will prioritize exploring programs like concat(concat(concat(concat(<low>)))) and searching in this way does not help the synthesizer to ﬁnd the ground truth regex.
R OBUST F ILL
We implemented the A TTENTION A model from R OBUST F ILL (Devlin et al., 2017), which predicts programs given I/O examples.
We encode the I/O with the the same I/
O embedding and I/
O encoder used in our implementation of D EEP C ODER .
We replaced the LSTM decoder in the original implementation with our ASN decoder.
During decoding, we extract a context vector from each of the examples provided in the example set, and pool them with max-pooling as the ﬁnal context vector.
The probability distribution over rules for node n is then given as:
[Equation]
We set the size of value embedding and class embedding to be 50, and the size of hidden state in encoder Bi-LSTM and LSTM in ASN to be 100.
T REE S EARCH
As the code of T REE S EARCH (Polosukhin & Skidanov, 2018) is not publicly available code, we implemented our own version of T REE S EARCH on top of T RAN X which is reported to be more powerful than the originally used S EQ 2T REE on various datasets (Yin & Neubig, 2018).
During search, we set the threshold to be 10 − 5 , and the max queue size to be 100.
O P S YNTH + R
We naturally combine O P S YNTH and R OBUST F ILL by concatenating the context vectors from NL and examples, as in Section 4.
The hyper-parameters for the NL encoder are the same as those for the base synthesizer, and the hyper-parameters for the I/O encoder are the same as
R OBUST F ILL .