G UIDING P ROGRAM S YNTHESIS BY L EARNING TO G ENERATE E XAMPLES
A BSTRACT
A key challenge of existing program synthesizers is ensuring that the synthesized program generalizes well.
This can be difﬁcult to achieve as the speciﬁcation provided by the end user is often limited, containing as few as one or two inputoutput examples.
In this paper we address this challenge via an iterative approach that ﬁnds ambiguities in the provided speciﬁcation and learns to resolve these by generating additional input-output examples.
The main insight is to reduce the problem of selecting which program generalizes well to the simpler task of deciding which output is correct.
As a result, to train our probabilistic models, we can take advantage of the large amounts of data in the form of program outputs, which are often much easier to obtain than the corresponding ground-truth programs
I NTRODUCTION
Over the years, program synthesis has been applied to a wide variety of different tasks including string, number or date transformations (Gulwani, 2011; Singh & Gulwani, 2012; 2016; Ellis et al., 2019; Menon et al., 2013; Ellis & Gulwani, 2017), layout and graphic program generation (Bielik et al., 2018; Hempel & Chugh, 2016; Ellis et al., 2019; 2018), data extraction (Barowy et al., 2014; Le & Gulwani, 2014; Iyer et al., 2019), superoptimization (Phothilimthana et al., 2016; Schkufza et al., 2016), code repair (Singh et al., 2013; Nguyen et al., 2013; D’Antoni et al., 2016), language modelling (Bielik et al., 2017), synthesis of data processing programs (Polosukhin & Skidanov, 2018;
Nye et al., 2019) or semantic parsing Shin et al.
(2019a). To capture user intent in an easy and intuitive way, many program synthesizers let its users provide a set of input-output examples I which the synthesized program should satisfy.
Generalization challenge A natural expectation of the end user in this setting is that the synthesized program works well even when I is severely limited (e.g., to one or two examples).
Because of this small number of examples and the big search space of possible programs, there are often millions of programs consistent with I .
However, only a small number of them generalizes well to unseen examples which makes the synthesis problem difﬁcult.
Existing methods
Several approaches have provided ways to address the above challenge, including using an external model that learns to rank candidate programs returned by the synthesizer, modifying the search procedure by learning to guide the synthesizer such that it returns more likely programs directly, or neural program induction methods that replace the synthesizer with a neural network to generate outputs directly using a latent program representation.
However, regardless of what other features these approaches use, such as conditioning on program traces (Shin et al., 2018;
Ellis & Gulwani, 2017; Chen et al., 2019) or pre-training on the input data (Singh, 2016), they are limited by the fact that their models are conditioned on the initial, limited user speciﬁcation.
This work
We present a new approach for program synthesis from examples which addresses the above challenge.
The key idea is to resolve ambiguity by iteratively strengthening the initial speciﬁcation I with new examples.
To achieve this, we start by using an existing synthesizer to ﬁnd a candidate program p 1 that satisﬁes all examples in I .
Instead of returning p 1 , we use it to ﬁnd a distinguishing input x ∗ that leads to ambiguities, i.e., other programs p i that satisfy I but produce different outputs p 1 ( x ∗ ) (cid:54) =
p i ( x ∗ ) .
To resolve this ambiguity, we ﬁrst generate a set of candidate outputs for x ∗ , then use a neural model (which we train beforehand) that acts as an oracle and selects
381 dp (Nexus4)
Input Speciﬁcation
I
Symbolic Search ∃ p ∈L .
∀ ( x i ,y i ) ∈I p ( x i ) =
y i or/and
Statistical Search argmax p ∈L q ( p | I )
Existing Android Layout Synthesizer InferUI ppp n 381 dp (Nexus4)
dp (P4Pro) p i ( x 1 ) p i ( x ∗ )
Synthesized Program Outputs
Neural Oracle argmax f ( y ∗ | x ∗ )
argmax f
( (cid:51)
| x ∗ ,y ∗ ) argmax f
( (cid:51)
| x ∗ ,y ∗ , I )
y ∗ ∈
y
Candidate Outputs y
=
{ p i ( x ∗ ) } ni =
( D U )
( D S )
( D + S ) 341 dp (P4Pro)
Selected Output Extend the Input Speciﬁcation
I ← I ∪ { ( 341 dp (P4 Pro) , ) } the most likely output, and ﬁnally, add x ∗ and its predicted output to the input speciﬁcation I . The whole process is then repeated. These steps are similar to those used in Oracle Guided Inductive Synthesis (Jha et al., 2010) with two main differences: (i) we automate the entire process by learning the oracle from data instead of using a human oracle, and (ii) as we do not use a human oracle to produce a correct output, we need to ensure that the set of candidate outputs contains the correct one.
Augmenting an existing Android layout synthesizer
In this work we apply our approach to a state-of-the-art synthesizer, called InferUI (Bielik et al., 2018), that creates an Android layout program which represents the implementation of a user interface.
Given an application design consisting of a set of views (e.g., buttons, images, text ﬁelds, etc.)
and their location on the device screen, InferUI synthesizes a layout program that when rendered, places the views at that same location.
Concretely, each input-output example ( x, y ) consists of a device screen x ∈ R 4 and a set of n views y ∈ R n × 4 , all of which are represented using their coordinates in a two dimensional euclidean space.
As an example, the input speciﬁcation I shown in Figure 1 contains a single example with absolute view positions for a Nexus 4 device and the InferUI synthesizer easily ﬁnds multiple programs that satisfy it (dashed box).
To apply our method and resolve the ambiguity, we ﬁnd a distinguishing input x ∗ , in this case a narrower P4 Pro device, on which some of the candidate programs produce different outputs.
Then, instead of asking the user to manually produce the correct output, we generate additional candidate outputs (to ensure that the correct one is included) and our learned neural oracle automatically selects one of these outputs (the one it believes is correct) and adds it to the input speciﬁcation I .
In this case, the oracle selects output p 2 ( x ∗ ) as both buttons are correctly resized and the distance between them was reduced to match the smaller device width.
In contrast, p 1 ( x ∗ ) contains overlapping buttons while in p n ( x ∗ ) , only the left button was resized.
Automatically obtaining real-world datasets An important advantage of our approach is that we reduce the problem of selecting which program generalizes well to the simpler task of deciding which output is correct.
This is especially useful for domains, such as the Android layout synthesis, for which the output correctness depends mostly on properties of the output and not on the program used to generate it.
As a result, obtaining a suitable training dataset can be easier as we do not require the hard to obtain ground-truth programs, for which currently no large real-world datasets exist (Shin et al., 2019b).
In fact, it is possible to train the oracle using unsupervised learning only, with a dataset consisting of correct input-output examples D U .
For example, in layout synthesis an autoencoder can be trained over a large number of views and their positions extracted from running real-world applications.
However, instead of training such an unsupervised model, in our work we use D U to automatically construct a supervised dataset D S by labelling the samples in D U as positive and generating a set of negative samples by adding suitable noise to the samples in D U .
Finally, we also obtain the dataset D S + that additionally includes the input speciﬁcation I .
In the domain of
Android layouts, although it is more difﬁcult, such a dataset can also be collected automatically by running the same application on devices with different screen sizes.
Our contributions We present a new approach to address the ambiguity in the existing Android layout program synthesizer InferUI by iteratively extending the user provided speciﬁcation with new input-output examples.
The key component of our method is a learned neural oracle used to generate new examples trained with datasets that do not require human annotations or ground-truth programs.
To improve generalization, InferUI already contains a probabilistic model that scores programs q ( p |I ) as well as handcrafted robustness properties, achieving 35% generalization accuracy on a dataset of Google Play Store applications.
In contrast, our method signiﬁcantly improves the accuracy to 71% while using a dataset containing only correct and incorrect program outputs.
We make our implementation and datasets available online at: https://github.com/eth-sri/guiding-synthesizers
R ELATED W ORK
In this section we discuss the work most closely related to ours.
Guiding program synthesis To improve scalability and generalization of program synthesizers several techniques have been proposed that guide the synthesizer towards good programs.
The most widely used approach is to implement a statistical search procedure which explores candidate programs based on some type of learned probabilistic model – log-linear models (Menon et al., 2013;
Long & Rinard, 2016), hierarchical Bayesian prior (Liang et al., 2010), probabilistic higher order grammar (Lee et al., 2018) or neural network (Balog et al., 2017; Sun et al., 2018).
Kalyan et al.
(2018) also takes advantage of probabilistic models but instead of implementing a custom search procedure, they use the learned model to guide an existing symbolic search engine.
In addition to approaches that search for a good program directly (conditioned on the input speciﬁcation), a number of works guide the search by ﬁrst selecting a high-level sketch of the program and then ﬁlling in the holes using symbolic (Ellis et al., 2018; Murali et al., 2017; Nye et al., 2019), enumerative or neural search (Bosnjak et al., 2017; Gaunt et al., 2016).
A similar idea is also used by (Shin et al., 2018), but instead of generating a program sketch the authors ﬁrst infer execution traces (or condition on partial traces obtained as the program is being generated (Chen et al., 2019)), which are then used to guide the synthesis of the actual program.
In comparison to prior work, a key aspect of our approach is to guide the synthesis by generating additional input-output examples that resolve the ambiguities in the input speciﬁcation.
Guiding the synthesizer in this way has several advantages – (i) it is interpretable and the user can inspect the generated examples, (ii) it can be used to extend any existing synthesizer by introducing a reﬁnement loop around it, (iii) the learned model is independent of the actual synthesizer (and its domain speciﬁc language) and instead is focused only on learning the relation between likely and unlikely input-output examples, and (iv) often it is easier to obtain a dataset containing program outputs instead of a dataset consisting of the actual programs.
Further, our approach is complementary to prior works as it treats the synthesizer as a black-box that can generate candidate programs.
We also note that several prior works explore the design of sophisticated neural architectures that encode input-output examples (Sun et al., 2018; Devlin et al., 2017; Parisotto et al., 2017) and incorporating some of their ideas might lead to further improvements to our models presented in Section 4.
Learning to rank To choose among all programs that satisfy the input speciﬁcation, existing program synthesizers select the syntactically shortest program (Liang et al., 2010; Polozov & Gulwani, 2015; Raychev et al., 2016), the semantically closest program to a reference program (D’Antoni et al., 2016) or a program based on a learned scoring function (Liang et al., 2010; Mandelin et al., 2005; Singh & Gulwani, 2015; Ellis & Gulwani, 2017; Singh, 2016).
Although the scoring function usually extracts features only from the synthesized program, some approaches also take advantage of additional information – Ellis & Gulwani (2017) trains a log-linear model using a set of handcrafted features deﬁned over program traces and program outputs while Singh (2016) leverages unlabelled data by learning common substring expressions shared across the input data.
Similar to prior work, our work explores various representations over which the model is learned.
Because we applied our work to a domain where outputs can be represented as images (rather than strings or numbers), to achieve good performance we explore different types of models (i.e., convolutional neural networks).
Further, we do not assume that the synthesizer can efﬁciently enumerate all programs that satisfy the input speciﬁcation as in Ellis & Gulwani (2017); Singh (2016).
For such synthesizers, applying a ranking of the returned candidates will often fail since the correct program is simply not included in the set of synthesized programs.
Therefore, the neural oracle is deﬁned over program outputs instead of actual programs.
This reduces the search space for the synthesizer as well as the complexity of the machine learning models.
Neural program induction Devlin et al.
(2017) and Parisotto et al.
(2017), as well as related line of work on neural machines (Graves et al., 2016;
Reed & de Freitas, 2016; Bošnjak et al., 2017;
Chen et al., 2018), explore the design of end-to-end neural approaches that generate the program output for a new input without the need for an explicit search.
In this case the goal of the neural network is not to ﬁnd the correct program explicitly, but rather to generate the most likely output for a given input based on the input speciﬁcation.
These approaches can be integrated in our work as one technique for generating a set of candidate outputs for a given distinguishing input instead of obtaining them using a symbolic synthesizer.
However, the model requirements in our work are much weaker – it is enough if the correct output is among the top n most likely candidates rather than requiring 100% precision for all possible inputs as in program induction.
L EARNING TO GENERATE NEW INPUT - OUTPUT EXAMPLES
Let I = { ( x i , y i ) } Ni =1 denote the input speciﬁcation consisting of user provided input-output examples.
Further, assume we are given an existing synthesizer which can ﬁnd a program p satisfying all examples in I , i.e., ∃ p ∈ L , ∀ ( x i , y i
) ∈
I .
p ( x i )=
y i , where p ( x i ) is the output obtained by running program p on input x i and L is a hypothesis space of valid programs.
To reduce clutter, we use the notation p |
= I to denote that p satisﬁes all examples in I .
We extend the synthesizer such that a program p not only satisﬁes all examples in I but also generalizes to unseen examples as follows:
Generate a candidate program p 1 |
= I that satisﬁes the input speciﬁcation I .
Find a distinguishing input x ∗ , a set of programs p 2 , . .
, p n that satisfy the input speciﬁcation I but produce different outputs when evaluated on x ∗ , and deﬁne candidate outputs as y
= { p 1 ( x ∗ ) , p 2 ( x ∗ ) , · · · , p n ( x ∗ ) } .
If no distinguishing input x ∗ exists, return program p 1 . 3.
Query an oracle to determine the correct output y ∗ ∈
y for the input x ∗ .
Extend the input speciﬁcation with the distinguishing input and its corresponding output I ← I ∪ { ( x ∗ , y ∗ ) } and continue with the ﬁrst step.
Finding a distinguishing input To ﬁnd the distinguishing input x ∗ we take advantage of existing symbolic synthesizers by asking the synthesizer to solve ∃ x ∗ ∈X , p 2 ∈L .
p 2 | = I∧ p 2 ( x ∗ ) (cid:54) =
p 1 ( x ∗ ) , where X denotes a set of valid inputs.
The result is both a distinguishing input x ∗ as well as a program p 2 that produces a different output than p 1 .
Programs p 1 and p 2 form the initial sequence of candidate outputs y =[
p 1 ( x ∗ ); p 2 ( x ∗ )] which is extended until the oracle is conﬁdent enough that y contains the correct output (described later in this section).
To make our approach applicable to any existing synthesizer, including those that can not solve the above satisﬁability query directly (e.g., statistical synthesizers), we note that the following sampling approach can also be applied: ﬁrst, use the synthesizer to generate the top n most likely programs, then randomly sample a valid input x ∗ not in the input speciﬁcation, and ﬁnally check if that input leads to ambiguities by computing the output of all candidate programs.
Finding candidate outputs To extend y with additional candidate outputs once the distinguishing input x ∗ is found, three techniques can be applied: (i) querying the synthesizer for another program with a different output: ∃ p ∈ L .
p
| = I ∧∀ y i ∈
y p ( x ∗ ) (cid:54) =
y i , (ii) sampling a program induction model
P ( y | x ∗ , I ) and using the synthesizer to check whether each sampled output is valid, or (iii) simply sampling more candidate programs, running them on x ∗ and keeping the unique outputs.
It is possible to use the second approach as we are only interested in the set of different outputs, rather than the actual programs.
The advantage of (i) is that it is simple to implement for symbolic synthesizers and is guaranteed to ﬁnd different outputs if they exist.
In contrast, (ii) has the potential to be faster as it avoids calling the synthesizer and works for both statistical and symbolic synthesizers.
Finally, (iii) is least effective, but it does not require pretraining and can be applied to any synthesizer.
Neural oracle
The key component of our approach is a neural oracle which selects the correct program output from a set of candidate outputs y .
Formally, the neural oracle is deﬁned as arg max y ∗ ∈
y f θ ( (cid:51) | x ∗ , y ∗ , I ) , where f is a function with learnable parameters θ (in our case a neural network) that returns the probability of the output y ∗ being correct given the input x ∗ and the input speciﬁcation I .
We train the parameters θ using a supervised dataset
D S + = { ( (cid:51) , x i , y i , I i ) }
Ni =1 ∪ { ( (cid:55) , x j , y j , I j ) } Mj =1 which, for a given distinguishing input x ∗ and input speciﬁcation I , contains both the correct ( (cid:51) ) as well as the incorrect ( (cid:55) ) outputs.
Because it might difﬁcult to obtain such a dataset in practice, we also deﬁne a simpler model f
θ ( (cid:51) | x ∗ , y ∗ ) that is trained using a supervised dataset D S
= { ( (cid:51) , x i , y i ) }
Ni =1 ∪ { ( (cid:55) , x j , y j ) } Mj =1 which does not include the input speciﬁcation.
In the extreme case, where the dataset contains only the correct inputoutput examples D U =
( x i , y i ) } Ni =1 , we deﬁne the oracle as f θ ( y ∗ | x ∗ ) .
Even though the dataset does not contain any labels, we can still train f in an unsupervised manner.
This can be achieved for example by splitting the output into smaller parts
y i
=
y 1 i · · · y ti (such as splitting a word into characters) and training f as an unsupervised language model.
Alternatively, we could also train an autoencoder that ﬁrst compresses the output into a lower dimensional representation, with the loss corresponding to how well it can be reconstructed.
To achieve good performance, the architecture used to represent f is tailored to the synthesis domain at hand, as discussed in the next section.
Dynamically controlling the number of generated candidate outputs Since generating a large number of candidate outputs is time consuming, we allow our models to dynamically control the number of sampled candidates for each distinguishing input x ∗ .
That is, instead of generating all candidate outputs y ﬁrst and only then querying the neural oracle to select the most likely one, we query the oracle after each generated candidate output and let the model decide whether more candidates should be generated.
Concretely, we deﬁne a threshold hyperparameter t ∈ R [0 , 1] which is used to return the ﬁrst candidate output y ∗ for which the probability of the output being correct is above this threshold.
Then, only if there are no candidate outputs with probability higher that t , we return arg max of all the candidates.
Note for t = 1 this formulation is equivalent to returning the arg max of all the candidates while for t = 0 , it corresponds to always returning the ﬁrst candidate, regardless of its probability of being correct.
We show the effect of different threshold values as well as the number of generated candidate outputs in Section 5.
I NSTANTIATION OF OUR APPROACH TO A NDROID LAYOUT SYNTHESIS
In this section we describe how to apply our approach to the existing Android layout program synthesizer InferUI
(Bielik et al., 2018).
Here, the input x ∈ R 4 deﬁnes the absolute positions of the top left and bottom right coordinates of a given device screen while the output y ∈ R n × 4 consists of n views and their absolute positions.
Finding a distinguishing input and candidate outputs Because InferUI uses symbolic search, ﬁnding a distinguishing input and candidate outputs is encoded as a logical query solved by the synthesizer as described in Section 3.
However, instead of synthesizing the layout program containing correct outputs of all the views at once (as done in InferUI ), we run the synthesizer n times, each time predicting the correct output for only a single view (starting from the largest view) which is then added as an additional input-output example (we provide a concrete example in Appendix C.4).
This is necessary since there are exponentially many combinations of the view positions when considering all the views at once and the InferUI synthesizer is not powerful enough to include the correct one in the set of candidate outputs (e.g., for samples with more than 10 views in less than 4%).
The advantage of allowing the position of only a single view to change, while ﬁxing the position of all prior views, is that the correct candidate output is much easier to include.
The disadvantage is that the neural oracle only has access to partial information (consisting of the prior view positions) and therefore performs a sequence of greedy predictions rather than optimizing all view positions jointly.
Neural oracle Because the input x has the same dimensions as each view, we encode it as an additional view in all our network architectures.
In Figure 2 we show three different neural architectures that implement the oracle function f , each of which uses a different way to encode the input-output example into a hidden representation followed by a fully-connected ReLU layer and a softmax that computes the probability that the output is correct.
In the following, we describe the architecture of all the models.
The formal feature deﬁnitions are included in Appendix A.
( x, y ) ( CNN ) Rendered Wireframe CNN ( x, y ) ( MLP ) Handcrafted Features aligned centered · · · intersect    · · ·  
ReLU ( x, y )
Encoder
LSTM
LSTM
LSTM vvv n ( RNN ) Per View Encoding where y
= [ v 1 ; . .
; v n ]
In ( CNN ) , the output is converted to an image (each is drawn as a rectangle with a 1px black border on a white background) and used as an to a convolutional neural network (CNN) with 3 convolutional layers with 5 × 5 ﬁlters of size 64, 32 and 16 and max pooling with kernel size 2 and stride 2.
To support outputs computed for different screen sizes, the image dimensions are slightly larger than the largest device size.
We the network by with a random such that they are still fully inside the of placing them in the
This is possible as the image used as input to the CNN is larger than the device size.
We provide visualization of this regularization in Appendix C.2.
In ( MLP ) , the output is transformed to a normalized feature vector and then fed into a feedforward neural network with 3 hidden layers of size 512 with ReLU activations.
To encode the properties of a candidate output y , we use high-level handcrafted features adapted from InferUI such as the number of view intersections or whether the views are aligned or centered.
By instantiating the features for both horizontal and vertical orientation we obtain a vector of size 30, denoted as ϕ MLP ( x ∗ , y ∗ ) , which is used as input to the neural network.
For the model f ( (cid:51) | x ∗ , y ∗ , I ) the network input is a concatenation of output features (as before)
ϕ
MLP ( x ∗ , y ∗ ) , features of each sample in the input speciﬁcation [ ϕ MLP ( x, y )]
( x,y ) ∈I , and their difference [ ϕ MLP ( x ∗ , y ∗ )
− ϕ MLP ( x, y )] ( x,y ) ∈I .
This difference captures how the views have been resized or moved between the devices with different screen dimensions.
This allows the model to distinguish between outputs that are all likely when considered in isolation but not when also compared to examples in I (as illustrated in Appendix C.3).
In ( RNN ) , we use the fact that each output consists of a set of views y ∗ =
[ v 1 ; . . .
; v n ] by using an encoder to ﬁrst compute a hidden representation of each view.
These are then combined with a LSTM to compute the representation of the whole output.
To encode a view v i , we extract pairwise feature vectors with all other views (including the input)
ϕ RNN ( v i , x ∗ , y ∗ ) =
[ φ ( v i , v j )] v j ∈{ x ∗ }∪ y ∗ \ v i and combine them using a LSTM.
Here, φ : R 4 × R 4 → R n is a function that extracts n real valued features from a pair of views.
For each pair of views, we apply 11 simple transformations of the view coordinates, capturing their distance (4 vertical, 4 horizontal), the size difference (in width and height) and the ratio of the aspect ratios.
For the model f ( (cid:51) | x ∗ , y ∗ ) we additionally use 17 highlevel features computed for each view that are adapted from InferUI .
When using f ( (cid:51) | x ∗ , y ∗ , I ) , these additional high-level features are not required and instead we only use the 11 simple transformations combined in the same way as for the MLP model.
That is, by concatenating ϕ RNN ( v i , x ∗ , y ∗ ) , [ ϕ RNN ( v i , x, y )] ( x,y ) ∈I and their difference [ ϕ RNN ( v i , x ∗ , y ∗ ) − ϕ RNN ( v i , x, y )]
( x,y ) ∈I .
Datasets To train our models we obtained three datasets D U , D S and D + S , each containing an increasing amount of information at the expense of being harder to collect.
The unsupervised D U = { ( x i , y i ) } Ni =1 is the simplest dataset and contains only positive input-output samples obtained by sampling
≈ 22 , 000 unique screenshots (including the associated metadata of all the absolute view positions) of Google Play Store applications taken from the Rico dataset (Deka et al., 2017).
Since the screenshots always consist of multiple layout programs combined together, [Table] we approximate the individual programs by sorting the views in decreasing order of their size and taking a preﬁx of random length (of up to 30 views).
For all of the datasets, we deduplicate the views that have the same coordinates and ﬁlter out views with a negative width or height.
The supervised D S =
{ ( (cid:51) , x i , y i ) }
( x i ,y i ) ∈D U ∪ (cid:8) (cid:83)
≤ 15 j =1 { ( (cid:55) , x i ,
y i + (cid:15) ij ) } (cid:9) ( x i ,y i ) ∈D
U contains both correct and incorrect input-output examples.
In our work this dataset is produced synthetically from
D U by extending it with incorrect samples.
Concretely, the positive samples correspond to those in the dataset D U and for each positive sample we generate up to 15 negative samples by applying a transformation (cid:15) ij to the correct output.
The transformations considered in our work are sampled from the common mistakes the synthesizer can make – resizing a view, shifting a view horizontally, shifting a view vertically or any combination of the above.
The supervised dataset is D S + = { ( (cid:51) , x i , y i , I i ) }
Ni =1 ∪{ ( (cid:55) , x j , y j , I j ) } Mj =1 , where each I i contains the same application rendered on multiple devices.
We downloaded the same applications as used in the Rico dataset from the Google Play Store and executed them on three Android emulators with different device sizes.
The number of valid samples is ≈ 600 since not all applications could be downloaded, executed or produced the same set of views (or screen content) when executed on three different devices.
The negative examples are generated by running the synthesizer with the input speciﬁcation I containing a single sample and selecting up to 16 outputs that are inconsistent with the ground-truth output for the other devices.
E VALUATION
We evaluate our approach by applying it to an existing Android layout synthesizer called
InferUI (Bielik et al., 2018) as described in Section 4.
InferUI is a symbolic synthesizer which encodes the synthesis problem as a set of logical constraints that are solved using the state-of-the-art
SMT solver Z3 (De Moura & Bjørner, 2008).
To improve generalization, InferUI already implements two techniques – a probabilistic model that selects the most likely program among those that satisfy the input speciﬁcation, and a set of handcrafted robustness constraints φ ( p ) that prevent synthesizing layouts which violate good design practices.
We show that even if we disable these two optimizations and instead guide the synthesizer purely by extending the input speciﬁcation with additional input-output examples, we can still achieve an accuracy increase from 35% to 71% .
In all our experiments, we evaluate our models and InferUI on a test subset of the D S + dataset which contains 85 Google Play Store applications, each of which contains the ground truth of the absolute view positions on three different screen dimensions.
We use one screen dimension as the input speciﬁcation I , the second as the distinguishing input and the third one only to compute the generalization accuracy.
The generalization accuracy of a synthesized program p |
= I is deﬁned as the percentage of views which the program p renders at the correct position.
InferUI Baseline To establish a baseline, we run InferUI in three modes as shown in Table 1.
The baseline mode returns the ﬁrst program that satisﬁes the input speciﬁcation, denoted as p | =
I , and achieves only 15.5% generalization accuracy.
In the second mode the synthesizer returns the most likely program according to a probabilistic model q ( p | I ) which leads to an improved accuracy of 24.7%.
The third mode additionally deﬁnes a set of robustness properties φ ( p ) that the synthesized program needs to satisfy, which together with the probabilistic model achieve 35.2% accuracy.
The generalization accuracy of all InferUI models is relatively low as we are using a challenging dataset where each sample contains on average 12 views.
Note however, that this is expected since increasing the number of views leads to an exponentially larger hypothesis space. [Table] [Table] Further
[Table] to establish an upper bound on how effective a candidate ranking approach can be [Table] we query the synthesizer for up to 100 different candidate programs (each producing a unique output) and check how often the correct program is included.
While for small samples (with up to 4 views) the correct program is almost always included, for samples with 6 views it is among the synthesized candidates in only 30% of the cases and for samples with more than 10 views in less than 4% [Table]
Sampling more outputs will help only slightly as increasing the number of views would require generating exponentially more candidates.
Our Work We apply our approach to the InferUI synthesizer by iteratively generating additional input-output examples that strengthen the input speciﬁcation.
The speciﬁcation initially contains absolute positions of all the views for one device and we extend the speciﬁcation by adding one view at a time (rendered on a different device) as described in Section 4.
In the experiments presented here we focus on evaluating the overall improvement of the InferUI synthesizer extended with our approach.
We provide additional experiments that evaluate the effectiveness of the neural oracle as well as an ablation study in Appendix B.
Generalization Accuracy
The results of our approach instantiated with various neural oracle models are shown in Table 2.
The best model trained on the dataset D S has almost the same accuracy as
InferUI with all its optimizations enabled.
This means that it is possible to replace existing optimizations and handcrafted robustness constraints by training on an easy to obtain dataset consisting of correct outputs and their perturbations.
More importantly, when training on the harder to obtain dataset D S + , the generalization accuracy more than doubles to 71% since the model can also condition on the input speciﬁcation I .
However, the results also show that the design of the model using the neural oracle is important for achieving good results.
In particular, both MLP models achieve poor accuracy since the high level handcrafted features adapted from the InferUI synthesizer are not expressive enough to distinguish between correct and incorrect outputs.
The CNN models achieve better accuracy but are limited for the opposite reason, they try to learn all the relevant features from the raw pixels which is challenging as many features require pixel level accuracy across large distances (e.g., whether two views in the opposite parts of the screen are aligned or centered).
The
RNN model performs the best, especially when also having access to the input speciﬁcation.
Even though it also processes low level information, such as distance or size difference between the views, it uses a more structured representation that ﬁrst computes individual view representations that are combined to capture the whole output.
Number of Candidate Outputs We show the effect of different threshold values t used by the neural oracle to dynamically control whether to search for more candidate outputs as well as the maximum number of candidate outputs in Table 3.
We can see that using the threshold both slightly improves the accuracy ( +0 . 3% ) but more importantly, signiﬁcantly reduces the average number of generated candidate outputs from 14 .
7 to 6 .
using the same number of maximum generated outputs | y | = 16 .
Incorporating User Feedback Even though our approach signiﬁcantly improves over the InferUI synthesizer, it does not achieve perfect generalization accuracy.
This is because for many synthesizers the perfect generalization is usually not achievable – the correct program and its outputs depends on a user preference, which is only expressed as severely underspeciﬁed set of input-output examples.
For example, for a given input speciﬁcation there are often multiple good layout programs that do not violate any design guidelines and which one is chosen depends on a particular user.
To achieve 100% in practice, we perform an experiment where the user can inspect the input-output examples generated by our approach and correct them if needed.
Then, we simply count how many corrections were required.
The applications in our dataset have on average 12 views and for our best model, no user corrections are required in 30% of the cases and in 27% , 15% , 12% , 5% of the cases the user needs to provide 1, 2, 3 or 4 corrections, respectively.
In contrast, InferUI with all optimizations enabled requires on average twice as many user corrections and achieves perfect generalization (i.e., zero user corrections) in only 3 .
5% of the cases.
C ONCLUSION
In this work we present a new approach to improve the generalization accuracy of existing program synthesizers.
The main components of our method are: (i) an existing program synthesizer, (ii) a reﬁnement loop around that synthesizer, which uses a neural oracle to iteratively extend the input speciﬁcation with new input-output examples, and (iii) a neural oracle trained using an easy to obtain dataset consisting of program outputs.
To show the practical usefulness of our approach we apply it to an existing Android layout synthesizer called InferUI (Bielik et al., 2018) and improve its generalization accuracy by 2 × , from 35% to 71% , when evaluated on a challenging dataset of real-world Google Play Store applications.
A PPENDIX
We provide three appendices.
Appendix A contains an in-depth description of the feature transformations for the MLP and RNN models.
Appendix B contains an additional ablation experiment as well as results from the oracle evaluation.
Appendix C provides visualizations of positive and negative candidate outputs, the CNN regularization as well as outputs from the end-to-end synthesis.
A F EATURE DEFINITIONS
In this section we describe in detail the feature transformations used in Section 4.
As mentioned in
Section 4 each view consists of the 4 coordinates:
x l ( x left ), y t ( y top ), x r ( x right ), y b ( y bottom ).
We use v.w for the width, v.h for the height and v.r for the aspect ratio ( v.w/v.h ) of view v .
A. MLP
In Table 4 we deﬁne the 7 feature types that lead to a vector of size 30 used in the MLP model.
All the features are normalized (divided) by the factor in the normalization column.
[Table]
Instantiated for the ratios 1, [3/4, 4/3] and [9/16, 9/16] | V |
[Table]
Pairwise comparison if two views intersect (without counting fully contained views.)
| V |
[Table]
The same dimension is deﬁned as: v 1 .w
= v 2 .w
∧ v 1 .h = v 2 .h
| V |
[Table]
Number
[Table]
view alignments
Pairwise comparison if two views align (e.g. for vertical alignment: v 1 .x l = v 2 .x
l ∨ v 1 .x l = v 2 .x
r ∨ v 1 .x
r
= v 2 .x
l ∨ v 1 .x r = v 2 .x r ) | V |
9 vertical + 9 horizontal Number of views with a speciﬁc margin to another view Instantiated for the margins 0, 16, 28, 32, 40, 48, 60, 64 and 96.
| V |
1 vertical + 1 horizontal Number of centered views Pairwise comparison if one view is centered within another view.
| V |
1 vertical + 1 horizontal Number of centered views between two different views Comparison if one view is centered between 2 other views.
| V |
A. RNN
We formally deﬁne the 11 transformations used as the pairwise view feature vector φ : R 4 × R 4 → R n in the RNN model.
These features capture properties like the view’s distance to the other view or the size difference.
The feature vectors extracted for each pair of views are combined to a ﬁxed length representation by passing them through the LSTM, in the decreasing order of view size.
In Figureall the properties are listed and visualized.
View View
View 1 View
Ratio Distance
Size di ﬀ erence
View 1 View
Name
Description
Formula
Distance
Horizontal distance from one view to another view d ll ( v 1 , v 2 )
d lr ( v 1 , v 2 )
d rl ( v 1 , v 2 )
d rr ( v 1 , v 2 ) =
=
= = v 1 .x
l − v 2 .x
l v 1 .x
l − v 2 .x
r v 1 .x r − v 2 .x
l v 1 .x
r − v 2 .x
r
Vertical distance from one view to another view d tt ( v 1 , v 2 ) d tb ( v 1 , v 2 )
d bt ( v 1 , v 2 ) d bb ( v 1 , v 2 )
=
=
= = v 1 .y
t − v 2 .y
t v 1 .y
t − v 2 .y
b v 1 .y
b − v 2 .y
t v 1 .y
b − v 2 .y
b
Size difference
Size difference of two views s w ( v 1 , v 2 )
s h ( v 1 , v 2 )
=
= v 1 .w − v 2 .w
v 1 .h − v 2 .h
Ratio
Relation of the aspect ratio r ( v 1 , v 2 )
= v 1 .r/v 2 .r
To guide the model for f ( (cid:51) | x ∗ , y ∗ ) , we deﬁned 17 more abstract features, adapted from InferUI , which are deﬁned using the simple transformations shown in Figure 3.
Concretely, we deﬁne the following 17 features:
• For alignments (8 features: 4 horizontal, 4 vertical): Compare if view v 1 is aligned with v 2 , such that one of the 4 distance functions is 0, e.g. d ll ( v 1 , v 2 ) =
• For centering (2 features: 1 horizontal, 1 vertical):
Compare if view v 1 is centered in v 2 , such that d ll ( v 1 , v 2 ) = − d rr ( v 1 , v 2 ) .
• For overlaps (4 features: 2 horizontal, 2 vertical): Check if the v 1 and v 2 can possibly intersect, i.e. have overlapping x-coordinates in the horizontal case: v 1 .x l ≥ v 2 .x
l and v 1 .x
l ≤ v 2 .x r .
• For the same size (2 features): Compare if the height or width difference of v 1 and v 2 is equal to 0, such that s w ( v 1 , v 2 )
=
and s h ( v 1 , v 2 )
= 0 .
• For the same ratio (1 feature): Compare if the ratio v 1 and v 2 ’s aspect ratios is 1, such that r ( v 1 , v 2 )
= 1 .
[Table]
B E XPERIMENTS
In this appendix
[Table]
we evaluate the oracle performance and include an ablation study of our models.
B. O RACLE E VALUATION
We evaluate the oracle’s performance (shown in Table 5) using two metrics: The pairwise accuracy and one vs. many accuracy.
The pairwise accuracy is the binary ranking problem – for each pair consisting of a positive and negative candidate, it is considered to be correct, if the score of the positive example is higher than of the negative example (the pair was ranked correctly).
To capture if the network can’t distinguish between the samples, we also report a second value in square brackets in which we count as correct if the score of the positive example is higher or equal to the score of the negative example.
In addition, we measure the one vs. many accuracy, in which the oracle is correct if the score of the correct candidate was higher than the scores of all incorrect candidates (since selecting the correct candidate out of many candidates represents what we are interested in during the end-to-end synthesis experiment).
Usually, models with a higher pairwise accuracy also have a higher one vs. many accuracy since more pairs are ranked correctly.
The one vs. many accuracy might be lower than expected if the mistakes of the pairwise accuracy spread over a wide number of different samples (instead of having many mistakes in a small set of samples).
In general, the results of the one vs. many accuracy correlate with the results of the end-to-end synthesis.
For example, RNN+CNN and RNN trained on D S + achieve the highest scores in both of the experiments.
One exception is the MLP model trained on D S + dataset which performs worse on the end-to-end synthesis experiment even though the oracle performs better.
A reason for this is that the MLP often makes mistakes when predicting the early views which then affect the positioning of the subsequent views.
For the same reason, the accuracies of the oracle experiment are in general higher than the accuracy we observe in the synthesis experiment.
Another interesting insight is that the MLP often scores the positive and the negative examples the same since there is a large difference in the pairwise accuracy depending on if we count the same score as correct or not.
This shows that the MLP’s feature functions are not expressive enough.
B. A BLATION S TUDY
To investigate which high-level properties are learned by our models, we generate an ablation dataset from the test dataset of D S + .
The results of the study are collected in Table 6 and Table 7.
Recall from Section
that ϕ
MLP ( x, y ) computes a vector of size 30 with the handcrafted functions which are described in detail in Appendix A.1.
For each pair of positive ( x ∗ , y pos ) and negative examples ( x ∗ , y neg ) and their corresponding sample in the input speciﬁcation ( x, y ) we compute the difference of their handcrafted feature functions d
= [ ϕ MLP ( x ∗ , y pos )
− ϕ MLP ( x, y )] ( x,y ) ∈I −
[Table]
[ ϕ MLP ( x ∗ , y neg ) − ϕ MLP ( x, y )] ( x,y ) ∈I .
We add a pair to the ablation dataset for the violated property if d (cid:54) = 0 , that is, if the property on the positive sample is different from the negative sample.
For example, if the number of view intersections of the positive sample and input speciﬁcation are the same, but different for the negative sample, we add this pair of positive and negative samples to the intersections-ablation dataset.
The MLP models perform very well on all the ablation properties.
The reason is that MLP’s features are the handcrafted features which are designed to capture the properties in the ablation dataset.
The
MLP’s performance on the whole test dataset is worse, which indicates that some properties in the dataset are not captured by the handcrafted features.
The CNN fails at preserving the aspect ratios due to the pooling and the downsampling operations.
Alignments and intersection properties can be learned with local ﬁlters detecting overlapping views.
Off-screen views can be detected by checking the downsampled representation after the convolutional layers.
On these three properties, the CNN performs best.
The RNN performs better on the ablation dataset than the CNN since it processes the exact numeric values of the view coordinates.
The overall performance on the ablation dataset is better than the one on the whole test dataset for all models, indicating that there are some features not expressed in the handcrafted functions.
Furthermore, the results of the ablation study should be considered carefully: the ablation study shows tendencies, but features could be correlated or have common hidden features. [Table]
C V ISUALIZATION
In this section [Table] we provide visualizations of two concepts introduced in Section 4 – the rendered output candidates and how the CNN input is shifted for regularization.
Further, we visualize the steps of the end-to-end synthesis and render outputs of synthesized programs.
C. P OSITIVE AND NEGATIVE OUTPUT CANDIDATES
Figure 4 shows four different rendered output candidates.
The correct candidate is on the left and the three incorrect ones are on the right.
The candidates differ in their 6th view which is moved around and overlaps in all of the three incorrect candidates.
The three negative examples are generated by the synthesizer as described in Section 4 (for dataset D S + ) or applying perturbations sampled from the synthesizer mistakes (for dataset D S ).
[Figure]
C. CNN REGULARIZATION
Figure 5 visualizes how the robustness of the CNN is increased by placing the input randomly within the input image as described in Section 4.
C. A DVANTAGE OF HAVING THE ADDITIONAL INPUT SPECIFICATION IN THE D S + DATASET
Figure 6 visualizes the absolute view positions in the input speciﬁcation on the left and the four candidates (differing in the red view) on the distinguishing device x ∗ on the right.
Looking at the four candidates, the ﬁrst candidate is incorrect, since the red view is not centered.
The third candidate is not the correct one either, since the red view intersects with the view below.
It is hard to decide between the second and fourth candidate since the red view is centered in both cases.
When taking the input speciﬁcation into consideration, the second candidate is the correct one, since there is no space between the red view and the view below in candidate 4 (unlike in second candidate and the input speciﬁcation).
[Figure]
C. E ND - TO - END SYNTHESIS PROCEDURE
In Figure 7 we present an example of the end-to-end synthesis described in Section 3 and Section 4.
The input of the synthesizer is on the top left of the Figure 7 and consists of an input speciﬁcation I with a single input-output pair ( x, y ) .
The synthesis loop starts by the synthesizer generating a candidate program p 1 |
= I (not shown) and checking whether ambiguities exists.
To do this, it ﬁnds a distinguishing input x ∗ , in our example a smaller device x ∗ =
[0 , 0 , 1400 , 2520] and a set of programs p 2 , . .
, p n that satisfy the input speciﬁcation I but produce different outputs when evaluated on x ∗ .
Three possible candidate outputs p 1 ( x ∗ ) , p 2 ( x ∗ ) , p 3 ( x ∗ ) are visualized in Figure 7 (column
Iteration 1).
Note that as described in Section 4, the synthesis proceeds by synthesizing the views iteratively therefore, the ﬁrst iteration contains only the ﬁrst view v 1 .
The oracle (in this example the
CNN+RNN model trained on D S + ) selects the second candidate (the oracle prediction is indicated by the dashed rectangle) since its score is the largest with 0.88.
The position of v 1 on the device x ∗ is added to the speciﬁcation and the iterative synthesis proceeds by selecting the correct output for the second view.
In the second iteration we perform the same steps, except now we synthesize different candidate outputs for the second view v 2 on the device x ∗ , given that the position of the ﬁrst view v 1 is ﬁxed from the previous iteration.
The same process is repeated until the absolute positions of all the views are predicted for the distinguishing input x ∗ .
The new input speciﬁcation, shown at the bottom in Figure 7, contains the absolute view positions for both the original input x as well as the distinguishing input x ∗ .
After that, the whole process repeats by querying for another distinguishing device and extending the input speciﬁcation even further.
Note that in our evaluation we restrict the distinguishing input to be among those device sizes for which we trained a model.
Since our datasets consist of three different screen sizes the choice is small and all the experiments in Section 5 are evaluated with a single distinguishing input x ∗ .
However, we could generate more than three dimensions for the training dataset and use them to train additional models.
This was however not needed as three different devices are typically enough to resolve majority of ambiguities (as long as the oracle does not make mistakes).
C.4. L IMITATIONS OF THE MLP FEATURES
Figure 8 visualizes eight candidate outputs in the ﬁrst synthesis iteration for the same application as in Figure 7.
The MLP oracle assigns a score of 0 to all the output candidates in which view v 1 is off-screen.
However, the handcrafted features are not expressive enough to distinguish between the candidate outputs in the bottom row which all have a score of 0.76.
In general, the MLP performs bad in selecting the ﬁrst views (earlier iterations) since there are only few views from which properties e.g. margins can be extracted.
On the bottom right, the synthesis result is compared to the correct output.
Since the ﬁrst view was not selected correctly in the synthesis, all the subsequent inner views are also slightly off in comparison to the correct output.
C. E XAMPLES OF S YNTHESIS O UTPUTS
Figure 9 visualizes the rendered output of the synthesizer for the distinguishing device x ∗ .
The correct output is on the left while the synthesis output of the different oracles is on the right.
At ﬁrst glance, the outputs of the MLP look visually appealing since the handcrafted features contain properties like intersections or off-screen and the model learns to avoid them.
However, it is often not expressive enough, since the view sizes or positions are often different to the correct output.
This happens in particular if there are only few views which is why the second view is often misplaced which leads to consecutive errors.
This also explains the bad performance in the synthesis experiment in Table 1 in comparison to the oracle experiment in Table 5.
The CNN outputs look much worse and often contain views which are misaligned or not centered.
The RNN performs quite well one the ﬁrst and third application but fails on the second application.
The RNN+CNN predicts the correct output for the ﬁrst application but fails on ﬁve views
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
I
[Table]
= ∪ (x*, [
[Table]
,1924]) … 2 = 1 ∪ (x*,[
[Table] [Table]
=
I
I 4 ∪ (x*,[
[Table]
[Table]
I Final =
(x,y) ∪ (x*, y*) [
[Table]
[Table] [Table]
[Table]
,1296,1544] [144,1608,1296,1800] [144,664,1296,776] [0,0,1440,2560] [0,500,1400,1924] [80,580,1320,1844] [144,836,1256,1524] [144,1588,1256,1780] [144,644,1256,756] [0,0,1400,2520] x = y
=
x
* = y* =
I
I
I v 1 :[-40,500,1400,1924] vv 1 :[-40,480,1400,1904]
Incorrect
Score: 0.
v
Incorrect
Score: 0.
v 1 :[-40,520,1400,1944]
Incorrect
Score: 0.
v 1 :[0,500,1440,1924]
Incorrect
Score: 0.
vvvv 1 :[0,480,1440,1904]
Incorrect
Score: 0.
v 1 :[0,480,1400,1904] v 1 :[0,500,1400,1924]
Incorrect
Score: 0.
Correct
Score: 0.
v 1 :[0,520,1400,1904]
Incorrect
Score: 0.
Synthesized with
MLP
Correct Output vvv
[Figure]