S YNTAX -D
IRECTED
V ARIATIONAL
A UTOENCODER
FOR S TRUCTURED D ATA
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep generative models have been enjoying success in modeling continuous data.
However it remains challenging to capture the representations for discrete structures with formal grammars and semantics, e.g. , computer programs and molecular structures.
How to generate both syntactically and semantically correct data still remains largely an open problem.
Inspired by the theory of compiler where syntax and semantics check is done via syntax-directed translation (SDT), we propose a novel syntax-directed variational autoencoder (SD-VAE) by introducing stochastic lazy attributes .
This approach converts the ofﬂine SDT check into onthe-ﬂy generated guidance for constraining the decoder.
Comparing to the stateof-the-art methods, our approach enforces constraints on the output space so that the output will be not only syntactically valid, but also semantically reasonable.
We evaluate the proposed model with applications in programming language and molecules, including reconstruction and program/molecule optimization.
The results demonstrate the effectiveness in incorporating syntactic and semantic constraints in discrete generative models, which is signiﬁcantly better than current state-of-the-art approaches.
I NTRODUCTION
Recent advances in deep representation learning have resulted in powerful probabilistic generative models which have demonstrated their ability on modeling continuous data, e.g. , time series signals (Oord et al., 2016; Dai et al., 2017) and images (Radford et al., 2015).
Despite the success in these domains, it is still challenging to correctly generate discrete structured data, such as graphs, molecules and computer programs.
Since many of the structures have syntax and semantic formalisms, the generative models without explicit constraints often produces invalid ones.
Conceptually an approach in generative model for structured data can be devided in two parts, one being the formalization of the structure generation and the other one being a (usually deep) generative model producing parameters for stochastic process in that formalization.
Often the hope is that with the help of training samples and capacity of deep models, the loss function will prefer the valid patterns and encourage the mass of the distribution of the generative model towards the desired region automatically.
Arguably the simplest structured data are sequences, whose generation with deep model has been well studied under the seq2seq (Sutskever et al., 2014) framework that models the generation of sequence as a series of token choices parameterized by recurrent neural networks (RNNs).
Its widespread success has encourage several pioneer works that consider the serialization of more complex structure data into sequences and apply sequence models to the represented sequences.
G´omez-Bombarelli et al.
(2016) (CVAE) is a representative work of such paradigm for the chemical molecule generation, using the SMILES line notation (Weininger, 1988) for representing molecules.
However, because of the lack of formalization of syntax and semantics serving as the restriction of the particular structured data, underﬁtted general-purpose string generative models will often lead to invalid outputs.
Therefore, to obtain a reasonable model via such training procedure, we need to prepare large amount of valid combinations of the structures, which in general is not practical.
To tackle such a challenge, one approach is to incorporate the structure restrictions explicitly into the generative model.
For the considerations of computational cost and model generality, contextfree grammars (CFG) have been taken into account in the decoder parametrization.
For instance, in
[Figure]
molecule generation tasks, Kusner et al.
(2017) proposes a grammar variational autoencoder (GVAE) in which the CFG of SMILES notation is embedded into the decoder.
The model generates the parse trees directly in a top-down direction, by repeatedly expanding any nonterminal with its production rules.
Although the CFG provides a mechanism for generating syntactic valid objects, it is still incapable to regularize the model for generating semantic valid objects (Kusner et al., 2017).
For example, in molecule generation, the semantic of the SMILES languages requires that the rings generated must be closed; in program generation, the referenced variable should be deﬁned in advance and each variable can only be deﬁned exactly once in each local context (illustrated in Fig 1b).
All the examples require cross-serial like dependencies which are not enforceable by CFG, implying that more constraints beyond CFG are needed to achieve semantic valid production in VAE.
In the theory of compiler, attribute grammars, or syntax-directed deﬁnition has been proposed for attaching semantics to a tree yield of context-free grammar.
That is, semantics are attached to an already materialized sequence and its CFG generating tree, thus one straightforward but not practical application of attribute grammars is, after generating a syntactic valid molecule candidate, to conduct ofﬂine semantic checking.
This process needs to be repeated until a semantically valid one is discovered, which is at best computationally inefﬁcient and at worst infeasible, due to extremely low rate of passing checking.
As a remedy, we propose the syntax-direct variational autoencoder (SDVAE), in which a semantic restriction component is advanced to the stage of syntax tree generator.
This allows the generator with both syntactic and semantic validation.
The proposed syntax-direct generative mechanism in the decoder further constraints the output space to ensure the semantic correctness in the tree generation process.
The relationships between our proposed model and previous models can be characterized in Figure 1a.
Our method brings theory of formal language into stochastic generative model.
The contribution of our paper can be summarized as follows:
• Syntax and semantics enforcement : We propose a new formalization of semantics that systematically converts the ofﬂine semantic check into online guidance for stochastic generation using the proposed stochastic lazy attribute .
This allows us effectively address both syntax and semantic constraints.
• Efﬁcient learning and inference: Our approach has computational cost O ( n ) where n is the length of structured data.
This is the same as existing methods like CVAE and GVAE which do not enforce semantics in generation.
• Strong empirical performance : We demonstrate the effectiveness of the SD-VAE through applications in two domains, namely (1) the subset of Python programs and (2) molecules.
Our approach consistently and signiﬁcantly improves the results in evaluations including generation, reconstruction and optimization.
B ACKGROUND
Before introducing our model and the learning algorithm, we ﬁrst provide some background knowledge which is important for understanding the proposed method.
2. V ARIATIONAL A UTOENCODER
The variational autoencoder (Kingma & Welling, 2013; Rezende et al., 2014) provides a framework for learning the probabilistic generative model as well as its posterior, respectively known as decoder and encoder.
We denote the observation as x , which is the structured data in our case, and the latent variable as z .
The decoder is modeling the probabilistic generative processes of x given the continuous representation z through the likelihood p θ ( x | z ) and the prior over the latent variables p ( z ) , where θ denotes the parameters.
The encoder approximates the posterior p θ ( z | x ) ∝ p θ ( x | z ) p ( z ) with a model q ψ ( z | x ) parametrized by ψ .
The decoder and encoder are learned simultaneously by maximizing the evidence lower bound (ELBO) of the marginal likelihood, i.e. ,
[Equation]
where X denotes the training datasets containing the observations.
2. C ONTEXT F REE G RAMMAR AND A TTRIBUTE G RAMMAR Context free grammar
A context free grammar (CFG) is deﬁned as G =
(cid:104)V , Σ , R , s (cid:105) , where symbols are divided into V , the set of non-terminal symbols, Σ , the set of terminal symbols and s ∈ V , the start symbol.
Here R is the set of production rules.
Each production rule r ∈ R is denoted as r = α → β , where α ∈ V is a nonterminal, and β = u 1 u 2 . . .
u | β | ∈ ( V (cid:83) Σ) ∗ is a sequence of terminals and/or nonterminals.
Attribute grammar To enrich the CFG with “semantic meaning”, Knuth (1968) formalizes attribute grammar that introduces attributes and rules to CFG.
The attribute is an attachment to the corresponding nonterminal symbol in CFG, written in the format <
v>.a where v ∈ V .
There can be two types of attributes assigned to non-terminals in G : the inherited attributes and the synthesized attributes.
An inherited attribute depends on the attributes from its parent and siblings, while a synthesized attribute is computed based on the attributes of its children.
Formally, for a production u 0 → u 1 u 2 . . .
u | β | , we denote I ( u i ) and S ( u i ) be the (disjoint) sets of inherited and synthesized attributes of u i , i ∈ { 0 , . . .
, | β |} .
2.2. A MOTIVATIONAL EXAMPLE
We here exemplify how the above deﬁned attribute grammar enriches CFG with non-context-free semantics.
We use the following toy grammar, a subset of SMILES that generates either a chain or a cycle with three carbons.
Production
[Table]
where we show the production rules in CFG with → on the left, and the calculation of attributes in attribute grammar with ← on the left.
Here we leverage the attribute grammar to check (with attribute matched ) whether the ringbonds come in pairs: a ringbond generated at (cid:104) atom (cid:105) 1 should match the bond type and bond index that generated at (cid:104) atom (cid:105) 2 , also the semantic constraint expressed by (cid:104) s (cid:105) .ok
requires that there is no difference between the set attribute of (cid:104) atom (cid:105) 1 and (cid:104) atom (cid:105) 2 .
Actually such constraint in SMILES is known as cross-serial dependencies (CSD) (Bresnan et al., 1982) which is non-context-free (Shieber, 1985).
Another example of CSD is a sequence of multiple different types of parentheses where each separately balanced disregarding the others.
Figure 2a further illustrates the example. Here all the attributes are synthetic , i.e. , calculated in a bottom-up direction. In the semantic correctness checking procedure, one need to perform (possibly multiple) bottom-up and top-down procedures for calculating the attributes after the parse tree is generated, however, in
<atom><bond> -
Start symbol
Nonterminals
Terminals <atom> 2 C <digit> C <bond> <digit> CBottom-up Attribute Grammar
Synthesized Attribute
Synthesize dependency
Semantics: ringbonds come in pairs (cross serial dependency) val=‘-’ val=‘1’ set={‘-1’} val=‘-’ val=‘1’ set={‘-1’} matched={‘-1’} ∩ {‘-1’} ok=True
[Figure]
<atom><bond> -
Start symbol
Nonterminals
Terminals <atom> 2 C <digit> C <bond> <digit> CTop-down Stochastic Attribute val=‘-’ val=‘1’ set={‘-1’} matched={ ‘-1’ }
Stochastic Lazy Attribute Lazy evaluation of
Synthesized attribute Inherit dependency sa=[1] (b) Top-down tree generation.
the structure generating process, the parse tree is not ready for semantic checking, since the synthesized attributes coming from children are not generated yet.
Due to such dilemma, it is nontrivial to use the attribute grammar to guide the top-down generation of the tree-structured data.
One straightforward way is using acceptance-rejection sampling scheme, i.e. , using the CFG decoder in grammar VAE Kusner et al.
(2017) as a proposal and the semantic checking as the threshold.
It is obvious that since the decoder does not include semantic guidance, the proposal distribution may raise semantically invalid candidate frequently, therefore, wasting the computational cost in vain.
S YNTAX -D IRECTED V ARIATIONAL A UTOENCODER
As described in Section 2.2.1, directly using attribute grammar to address both syntax and semantics constraints is not efﬁcient.
In this section we describe how to bring forward the attribute grammar online and incorporate it into variational autoencoders such that our VAE generates both syntactic and semantic valid outputs by deﬁnition.
We name our proposed method Syntax-Directed Variational Autoencoder (SD-VAE).
3. S TOCHASTIC S YNTAX -D IRECTED D ECODER
By scrutinizing the tree generation, the major difﬁculty in incorporating the attributes grammar into the processes is the appearance of the synthesized attributes.
For instance, when expanding the start symbol (cid:104) s (cid:105) , the corresponding synthesized attribute (cid:104) s (cid:105) .
matched is not ready yet.
Since none of its children is generated, their synthesized attributes are also absent at this time, making the (cid:104) s (cid:105) .
matched unable to be computed.
To enable the on-the-ﬂy computation of the synthesized attributes for semantic validation during tree generation, besides the two types of attributes, we introduce the stochastic lazy attributes to enlarge the existing attribute grammar, so that the synthesized attributes will be transformed to inherited constraints in generating procedure and instantiated once all the dependent attributes are ready (also named as lazy linking in the following content).
We demonstrate how the decoder with stochastic lazy attributes will generate semantic valid output through a pedagogical example with the subset of SMILES grammar in ﬁgure 2(b).
Following the terminology in compiler theory, we named it as stochastic syntax-directed decoder.
The tree generation procedure is indeed sampling from the decoder p θ ( x | z ) , which can be decomposed into several steps that elaborated below:
i) stochastic predetermination: in ﬁgure 2(b), we start from the node (cid:104) s (cid:105) with the synthesized attributes (cid:104) s (cid:105) .
matched determining the index and bond type of the ringbond that will be matched at node (cid:104) s (cid:105) . Since we know nothing about the children nodes right now, the only thing we can do is to ‘guess’ a value. That is to say, we associate a stochastic attribute (cid:104) s (cid:105) .
sa ∈ { 0 , 1 } C a ∼ (cid:81) C
a i =1 B ( sa i | z ; p ) as a predetermination for the sake of the absence of synthesized attribute (cid:104) s (cid:105) .
matched . B ( · ) is the bernoulli distribution. Here C a is the maximum cardinality 1: Global variables: CFG: G =
( V , Σ , R , s ) , decoder network parameters θ 2: procedure G EN T REE ( node , T ) 3:
Sample stochastic lazy attribute node.
sa ∼ B θ ( sa | node, T ) (cid:46) when introduced on node 4:
Sample production rule r = ( α → β ) ∈
R ∼ p θ ( r | ctx, node, T ) .
(cid:46)
The conditioned variables encodes the semantic constraints in tree generation.
5: ctx ← RNN ( ctx, r ) (cid:46) update context vector 6: for i = 1 , . .
. , | β | do 7: v i ← Node ( u i , node, { v j } i − 1 j =1 ) (cid:46) node creation wtih parent and siblings’ attributes 8:
GenTree( v i , T ) (cid:46) recursive generation of children nodes 9:
Update synthetic and stochastic attributes of node with v i (cid:46)
Lazy linking 10: end for 11: end procedure possible 1 for the corresponding attribute a .
In above example, the 0 indicates no ringbond andindicates one ringbond at both (cid:104) atom (cid:105) 1 and (cid:104) atom (cid:105) 2 , respectively.
ii) constraints as inherited attributes: we pass the (cid:104) s (cid:105) .
sa as inherited constraints to the children of node (cid:104) s (cid:105) , i.e. , (cid:104) atom (cid:105) 1 and (cid:104) atom (cid:105) 2 to ensure the semantic validation in the tree generation.
iii) sampling under constraints: assume in the valid order, (cid:104) atom (cid:105) 1 is selected before (cid:104) atom (cid:105) 2 , we then sample the rules from p θ ( r |(cid:104) atom (cid:105) 1 , (cid:104) s (cid:105) , z ) for expanding (cid:104) atom (cid:105) 1 , and so on and so forth to generate the subtree recursively.
Since we carefully designed sampling distribution that is conditioning on the stochastic property, the inherited constraints will be eventually satisﬁed.
In the example, due to the (cid:104) s (cid:105) .
sa = ‘1’ , when expanding (cid:104) atom (cid:105) 1 , the sampling distribution p θ ( r |(cid:104) atom (cid:105) 1 , (cid:104) s (cid:105) , z ) only has positive mass on rule (cid:104) atom (cid:105) → ‘C’ (cid:104) bond (cid:105) (cid:104) digit (cid:105) .
iv) lazy linking: once we complete the generation of the subtree rooted at (cid:104) atom (cid:105) 1 , the synthesized attribute (cid:104) atom (cid:105) 1 .
set is now available.
According to the semantic rule for (cid:104) s (cid:105) .
matched , we can instantiate (cid:104) s (cid:105) .
matched =
(cid:104) atom (cid:105) 1 .
set = { ‘-1’ } .
When expanding (cid:104) atom (cid:105) 2 , the (cid:104) s (cid:105) .
matched will be passed down as inherited attribute to regulate the generation of (cid:104) atom (cid:105) 2 .
In summary, the general syntax tree T ∈ L ( G ) can be constructed step by step, within the languages
L ( G ) covered by grammar G .
In the beginning, T (0) = root ,
where root.
symbol =
s which contains only the start symbol s .
At step t , we will choose an nonterminal node in the frontier 2 of partially generated tree T ( t ) to expand.
The generative process in each step t = 0 , 1 , . . .
can be described as:
Pick node v ( t ) ∈
Fr ( T ( t ) ) where its attributes needed are either satisﬁed, or are stochastic attributes that should be sampled ﬁrst according to bernoulli distribution B ( ·| v ( t ) , T ( t ) ) ; 2.
Sample rule r ( t ) =
α ( t ) → β ( t ) ∈ R according to distribution p θ ( r ( t ) | v ( t ) , T ( t ) ) , where v ( t ) .
symbol =
α ( t ) , and β ( t ) =
u ( t ) 1 u ( t ) 2 . . .
u ( t ) | β ( t ) | , i.e. , expand the nonterminal with production rules deﬁned in CFG.
T ( t +1)
=
T ( t ) (cid:83) { ( v ( t ) , u ( t ) i ) } | β ( t ) | i =1 , i.e. , grow the tree by attaching β ( t ) to v ( t ) .
Now the node v ( t ) will have children represented by symbols in β ( t ) .
The above process continues until all the nodes in the frontier of T ( T ) are all terminals after T steps.
Then, we obtain the algorithm 1 for sampling both syntactic and semantic valid structures.
In fact, in the model training phase, we need to compute the likelihood p θ ( x | z ) given x and z .
The probability computation procedure is similar to the sampling procedure in the sense that both of them requires tree generation.
The only difference is that in the likelihood computation procedure, the tree structure, i.e. , the computing path, is ﬁxed since x is given, while in sampling procedure, it is sampled following the learned model.
Speciﬁcally, the generative likelihood can be written as:
[Equation]
where ctx (0) =
z and ctx ( t ) =
RNN ( r t , ctx ( t − 1) ) .
Here RNN can be commonly used LSTM, etc. .
3. S TRUCTURE -B ASED E NCODER
As we introduced in section 2, the encoder, q ψ ( z | x ) approximates the posterior of the latent variable through the model with some parametrized function with parameters ψ .
Since the structure in the observation x plays an important role, the encoder parametrization should take care of such information.
The recently developed deep learning models (Duvenaud et al., 2015; Dai et al., 2016; Lei et al., 2017) provide powerful candidates as encoder.
However, to demonstrate the beneﬁts of the proposed syntax-directed decoder in incorporating the attribute grammar for semantic restrictions, we will exploit the same encoder in Kusner et al.
(2017) for a fair comparison later.
We provide a brief introduction to the particular encoder model used in Kusner et al.
(2017) for a self-contained purpose.
Given a program or a SMILES sequence, we obtain the corresponding parse tree using CFG and decompose it into a sequence of productions through a pre-order traversal on the tree.
Then, we convert these productions into one-hot indicator vectors, in which each dimension corresponds to one production in the grammar.
We will use a deep convolution neural networks which maps this sequence of one-hot vectors to a continuous vector as the encoder.
3. M ODEL L EARNING
Our learning goal is to maximize the evidence lower bound in Eq 1.
During training, each instance x is ﬁrst parsed into syntax tree with CFG parser.
Given the encoder, we can then map the structure input into latent space z .
The variational posterior q ( z | x ) is parameterized with Gaussian distribution, where the mean an variance are the output of corresponding neural networks.
The prior of latent variable p ( z ) =
N (0 , I ) .
Since both the prior and posterior are Gaussian, we use the closed form of KL-divergence that proposed in Kingma & Welling (2013).
In the decoding stage, our goal is to maximize p θ ( x | z ) .
Using the Algorithm 1, we can compute the corresponding conditional likelihood.
For efﬁcient learning, during the training time we divide the calculation in our stochastic decoder into two phases: the ﬁrst phase generates tree and the second phase only consists of a sequence of updates to the context vector.
This decoupling is possible since in training time we know the all decisions in the decoder since the tree’s calculation is deterministic.
In doing so, the ﬁrst phase can be accelerated using multiple CPU cores in parallel and the second one can effectively computed using any mini-batch batch optimization on GPU.
In practice, we observe no signiﬁcant time penalty measured in wall clock time compared to previous works.
R ELATED WORK
Generative models with discrete structured data have raised increasing interests among researchers in different domains.
The classical sequence to sequence model (Sutskever et al., 2014) and its variations have also been applied to molecules (G´omez-Bombarelli et al., 2016).
Since the model is quite ﬂexible, it is hard to generate valid structures with limited data.
Techniques including data augmentation (Bjerrum, 2017), active learning (Janz et al., 2017) and reinforcement learning (Guimaraes et al., 2017) have also been proposed to tackle this issue.
However, according to the empirical evaluations from Benhenda (2017), the validity is still not satisfactory.
Even when the validity is enforced, the models tend to overﬁt to simple structures while neglect the diversity.
Since the structured data often comes with formal grammars, it is very helpful to generate its parse tree derived from CFG, instead of generating sequence of tokens directly.
The Grammar
VAE(Kusner et al., 2017) introduced the CFG constrained decoder for simple math expression and SMILES string generation.
The rules are used to mask out invalid syntax such that the generated sequence is always from the language deﬁned by its CFG.
Parisotto et al.
(2016) uses a
RecursiveReverse-Recursive Neural Network (R3NN) to capture global context information while expanding with CFG production rules.
Although these works follows the syntax via CFG, the context sensitive information can only be captured using variants of sequence/tree RNNs (Alvarez-Melis & Jaakkola, 2016;
Dong & Lapata, 2016; Zhang et al., 2015), which may not be time and sample
In our work, we capture the semantics with proposed stochastic lazy attributes when generating structured outputs.
By addressing the most common semantics to harness the deep networks, it can greatly reshape the output domain of decoder (Hu et al., 2016).
As a result, we can also get a better generative model for discrete structures.
E XPERIMENTS
We show the soundness and utility of our proposed SD-VAE using two sets of applications: programs and molecules.
We compare our method with GVAE (Kusner et al., 2017) and CVAE (G´omezBombarelli et al., 2016).
CVAE operates on character level and GVAE on the context-free grammar level.
To make a fair comparison, we largely follow the experimental protocols that were set up in
Kusner et al.
The training details are discussed in Appendix B.
Our method proves signiﬁcantly superior results than previous works, with better reconstruction accuracy and prior validity by large margins, while also having better diversity as a generative model.
More importantly, our model produces a smooth latent space which enables the optimization for ﬁnding best programs and molecules.
5. S ETTINGS
Here we ﬁrst describe our datasets in detail.
The ﬁrst is for modeling of programs that are represented as a list of statements.
Each statement is an atomic arithmetic operation on variables (labeled as v0 , v1 , · · · , v9 ) and/or immediate numbers ( 1 , 2 , . . .
Some examples are list in the following: v
[Equation]
v
[Equation]
v
[Equation]
v
[Equation]
v
[Equation]
v0*v9;return:v v2=
[Equation]
v
[Equation]
v7=v
[Equation]
v
[Equation]
v7);v [Equation] (v9);return:v Here v0 is always the input, and the variable speciﬁed by return (respectively v5 and v8 in the examples) is the output, therefore it actually represent univariate functions f : R → R .
Note that a correct program should, besides the context-free grammar speciﬁed in Appendix A.1, also respect the semantic constraints such as that a reference to variable should not consult the unknown variables.
We randomly generate 130 , 000 program each consisting of 1 to 5 valid statements.
We hold 2000 programs out for testing and the reset for training VAE.
The second dataset is for molecules.
It contains 250 , 000 SMILES string, prepared by Kusner et al.
(2017) using randomly extraction from the ZINC database (G ´ omez-Bombarelli et al., 2016).
We use 5000 SMILES strings as the holdout set for testing and the reset for training.
For syntax, our formalization of SMILES follows the grammar speciﬁed in Appendix A.2.
For our SD-VAE, we address some of the most common semantics:
Program semantics We address the following: a ) variables should be deﬁned before use, b ) program must return a variable, c ) number of statements should be less than 10.
Molecule semantics The SMILES semantics we addressed includes: a ) ringbonds should satisfy cross-serial dependencies, b ) explicit valence of atoms should not go beyond permitted. .
For more details about the semantics, please refer to Appendix A.3.
5. R ECONSTRUCTION A CCURACY AND P RIOR V ALIDITY
[Table] One metric of the soundness of VAE is to measure the ability of the model to encode data (in our case program/molecule) into a representation in the latent space and reconstruct the input by decoding from that point.
Another metric is how often the model can decode into a valid data when the prior is randomly sampled.
Since both encoding and decoding are stochastic, we follow the estimation by
Monte Carlo method similar to that proposed in Kusner et al.
For reconstruction, we select 5000 programs in the hold-out test set true data, and for each of them we encode it 10 times and decoded (for each encoded latent space representation) 25 times, and report the portion of decoded molecules that are the same as the input one;
For validity of prior, we sample 1000 latent representation z ∼ N ( O, I ) , for each of them decode 100 times, and calculate the portion of 100,000 decoded results that corresponds to valid Program or SMILES sequences.
Program
We show in the left part of Table 1 that our model has near perfect reconstruction rate, and most importantly, a perfect valid decoding program from prior.
This huge improvement is due to our model that utilizes the semantics that previous work ignores, thus in theory guarantees perfect valid prior and in practice enables high reconstruction success rate.
For a fair comparison, we run and tune the baselines in 10% of training data and report the best result.
In the same place we also report the reconstruction successful rate grouped by number of statements.
It is shown that our model keeps high rate even with the size of program growing.
SMILES
The CVAE and GVAE results are included directly from Kusner et al.
We show in the right part of Table 1 that our model produces a much higher rate of successful reconstruction, and a huge increase in ratio of valid prior.
Figure 7 in Appendix C.2 also demonstrates some decoded molecules from our method.
Note that the invalid SMILES decoded by our algorithm is mainly due to the stack overﬂow error in Python.
5. B AYESIAN O PTIMIZATION
The variational autoencoder realizes the conversion from data (program/molecule) space to a continuous latent space via the encoder and vice versa via the decoder.
This naturally leads to following two important applications: First, we can now train an extra model that predicts the data’s property from the representation in latent space, as suggested in G ´ omez-Bombarelli et al.
Second and more importantly, the continuous nature of latent space makes possible the optimization of ﬁnding new data with better properties.
Following the protocol used in Kusner et al.
(2017), we use
Bayesian Optimization (BO) to search the programs and molecules with desired properties in latent space.
Details about BO settings and parameters can be found in Appendix C.1.
Finding program
In this application the model is tasked with ﬁnding in the latent space a point corresponding a program close to a ground truth program which the model does not know.
Here the closeness is measured by log(1 + MSE ) between two programs’ returned value estimated by sampling 1000 value for the input v0 ∈ [ − 5 , 5] in evenly paced sapce.
In Figure 3 we show that our method ﬁnds the best program to the ground truth one compared to CVAE and GVAE.
Interestingly, the GVAE fail shot of ﬁnding non-trivial program.
We hypothesize that this is due to the deep generative model in GVAE being highly relying on the space of production sequences from CFG syntax, which may not be optimization-friendly without constraints such as ones in our SD-VAE. [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] Molecules Here we optimize the drug properties of molecules.
In this problem, we ask the model to optimize for octanol-water partition coefﬁcients (a.k.a log P ), an important measurement of druglikeness of a given molecule.
As G´omez-Bombarelli et al.
(2016) suggests, for drug-likeness assessment log P is penalized by other properties including synthetic accessibility score (Ertl & Schuffenhauer, 2009).
In Figure 4 we show the the top-3 best molecules found by each method, where our method found molecules with much better scores than previous works.
[Figure]
5. P REDICTIVE PERFORMANCE OF LATENT REPRESENTATION
[Table]
We seek to to know how well our latent space predicts the properties of programs and molecules.
We train the same sparse Gaussian Process as in Sec 5.3, with the same target value (namely the error for programs and the drug-likeness for molecules) for regression.
We test the performance in the hold-out test dataset.
In Table 2, we report the result in Log Likelihood (LL) and Regression
Mean Square Error (RMSE), which show that our SD-VAE always produces latent space that are more discriminative than both CVAE and GVAE baselines.
This also shows that, with a properly designed decoder, the quality of encoder will also be improved via end2end training.
5. D IVERSITY OF GENERATED MOLECULES
[Table] Inspired by Benhenda (2017), here we seek to measure the diversity of generated molecules as an assessment of our methods.
The intuition is that a good generative model should be able to generate diverse data and avoid model collapse in the learned space.
In detail, we conduct this experiment in SMILES dataset, where we sample 100 points from the prior distribution, and for each point, we associate it as a molecule, which is the most frequent occurring valid SMILES decoded.
(we use 50 decoding attempts since the decoding is stochastic and the baseline may not always produce valid decoded SMILES string).
We then, with one of several molecular similarity, compute the pairwise similarity and report the mean and standard deviation in Table 3.
We see both methods do not have the model collapse problem, while our model always produces higher diversity.
It indicates that although our method has more restricted decoding space than baselines, the diversity is instead boosted.
This is because we never rule-out the valid molecules.
And a more compact decoding space leads to much higher probability in obtaining valid molecules.
5. V ISUALIZING THE L ATENT S PACE
We seek to visualize the latent space as an assessment of how well our generative model is able to produces a coherent and smooth space of program and molecules. [Figure] [Table] [Figure]
[Table]
[Figure]
[Table] [Figure] [Table] Program
Following Bowman et al.
, we visualize the latent space of program by interpolation between two programs.
More speciﬁcally, given two programs which are encoded to p a and p b respectively in the latent space, we pick 9 evenly spaced points between them and for each point as prior generate a program using the decoder.
In Table 4 we compare our results with previous works.
Our SD-VAE can pass though points in the latent space that can be decoded into valid programs without error and with visually more smooth interpolation than previous works.
Meanwhile, CVAE makes both syntactic and semantic errors, and GVAE produces only semantic errors (reference of undeﬁned variables), but still in a considerable amount.
OH O N O
NH OH O N O NH F
OH O N O N F
OH O N O N F
NOH O O NH
NOH O O NH
NH + O N O
NH + O N O
NH + O ONH F
NH + ONH F NH + NH ONH F NH + NH ONH F NH + NH ONH F
OH O O NH F
OH OH O O NH F OH O O NH F
OH O N O
NH F
OH O N O
NH F
NOH O O NH
NH + O O O NH
NH + O ONHF
NH + O ONHF NH
+ NH ONHF NH + NH ONH F NH + NH ONH F NH ONH F
OH O O NH F
OH O O NH F
OH O N O
NH F
OH O N O
NH F OH O N O NH F OH O N O
NH F OH O N O NH F NH
+ O ONHF
NH +
NH ONHF NH + NH ONHF NH O NH F NH O NH F NH O NH F
OH O O NH F
OH O O NH F
OH O N O
NH F F
OH O N O
NH F OH O N O NH F OH O N O
NH F OH O N O
NH F O O NH F NH O NH F NH O NH F NH O NH F NH O NH F NH O NH F
OH O O NH F OH O O
NH F OH O N O
NH F OH O N O NH F OH O N O
NH F O O
NH F O O
NH F O O NH F NH O NH F NH O NH F NH O NH F NH O NH F NH O NH F
OH O O NH F F
OH O O NH F OH O O NH F O N ONH F
O O NH
F O O
NH F O O
NH F O O NH F NH O NH F NH O NH F NH O NH F NH O NH F NH O NH F
OH O O NH F OH O O NH F O N ONH F O N ONH F
O O NH
F O O
NH F O O
NH F O O NH F NH O NH F NH O NH F NH O NH F NH O NH F NH O NH F
O O NH F O N ONH F O NN ONH F O NN ONH F O O
NH F O O NH F ON O
NH F O O NH F NH O NH F NH O NH F NH O NH F NH O NH F NH O NH F
O N ONH F O N ONH F O N ONH F O N ONH F ON O NH F ON O NH F ON O NH F NH O NH F NH O NH F NH O NH F NH O NH F OH NH
ONHF
OH O O ONHF
O N ONH F O ONH F O ONH F O N ONH F ON O NH F ON O NH F NHN O NH F ON O
NH F O O O
NH F N
OH O O ONHF N
O O NH
O O NH
O O O
NH O
ONH F N
O ONH F N O ONH F N
ON O NH F N
ONO O
NH F N
OO NH OO NH
O NH O NH
O O NH
O NH O NH O NH O NH
O O NH
O O NH O NH
O NH O
NH O NH O NH
O NH
O NH O NH
O NH
O NH O NH O NH O NH O NH O NH
O NH O NH O NH O NH O NH N O NH
O NH O NH
O NH O NH
O NH
O NH O NH O NH
O NH O NH
O NH O NH O NH O NH
O NH
O NH
NH NH
NH
NH ONH N O NH ONH N O N O
NH
NO N
N O NH
NO NH
N ONH N O
N
N ONH N O
N N O NH S S
NH
N O NH S
NH O N
N NH O S
NH O N
NH NH
NH
NH ONH N O N O
NH
NO N
N O NH
NO NH
N O NH
NO N
N O NH
NO N
N O NH
NO N
N O NH
NO N
N O NH S S NH N N O NH S
NH O N
N N O S
NH O
NH
NH ONH OH NH ONHO N ONH N O O
H
N
O
NH
NO N
N O NH
NO N
N O NH
NO N
N O NH
NO N
N O NH
NO N
N O NH
NO N
N O NH S S NH N N O NH S S NH N
N NH O S
NH O NH
NH ONH O - NH ONH OH N ONH O O
N
ONH
N
O O
H
N
O
NH
NO N
N O NH
NO N
N O NH
NO N
N O NH
NO N
N O NH
NO N
N O NH
NO N
N O NH
NO N
N O NH S S NH N
N NOHNH S S NH N
NH ONH ON
O NH O N N
N O
NH O N N N O NH N O N
NH H
N O NH
NO N
N O NH
NO N
N O NH
NO N
N O NH
NO N
N O NH
NO N
N O NH
NO N
N O NH
NO N
N O
NH O
N N OH
NH
O N
N O
NH O O H
N O
NH O N H N O NH S N H N O NH O N
N O NH NO N N O NH
NO N N O NH NO N N O NH
NO N
N O NH NO N
N O NH
NO N
N O NH
NO N
N O
NH O
NN O NH O N
N O
NH O N N O NH N O N H
N O
NH O N H N O NH O H
N O NH
NO N
N O NH
NO N
N O NH
NO N
N O NH
NO N
N O NH
NO N
N O NH
NO N
NN O NH
NO NH
NN O
NH O
NN O NH O N
N O NH OOHNH H
N O NH SH
N O NH H 2 N O N H
N O NH O H
N O NH
NO N
N O NH
NO N
N O NH
NO N
N O NH
NO N
N O NH
NO N
NN O NH
NO N
NN O NH
NO
NN O NH S S
NH NH
NN O NH O N
N O N O + OH H
N ONH NHSNN H
N O NH H 2 N O N H
N O NH O H
N O NH
NO N
N O NH
NO N
N O NH
NO N
NN O NH
NO N
NN O NH
NO N
NN O NH N O N H
NN O NH
NO
NN O NH NH S
NH NH
NN O NH O N
N O NH NSHO N ONH NHSNN H
N O NH H 2 N S
NN H
N O
NH N O N H N O NH
NO N
N O NH
NO N
NN O NH
NO N
NN O NH
NO N
NN O NH
NO N
NN O NH N O N H
NN O NH
NO
NN O NH NH S O NH
NN O NH O N
N O N S + O H
N O NH NH 2 S NNH 2 H N O NH NH 2 S NNH 2 N O NH N S NNH
NN O NH N O N N H NN O NH N O N N H
NN O NH
NO N
NN O NH N O NN O NH N O
NN O NH N O NN
O NH N O NH
NN O NH
NH O O NH
NN O NH O N
N O N S + O
NH 2 N ONH
NH 2 SNNH 2 N O NH NH 2 S NNH 2 NN O NH NNH 2 S NNH
NN ONH N ONN
NN O NH N O N N H NN O NH N O
NH NN
O
NH N O
NN O NH N O NN O NH N O N
NN O NH N O NH
NN O O
NH OO NN
O O O
NN O N S + O SHH
NN O N NH S NNH 2 H
NN ONH NH 2 S NNH 2 NN ONH NNH 2 S NNH
NN ONH N SNN
NN O NH N O NHNH 
 NN O NH N O
NH NN
O NH N O
NH NN
O
NH N O
NN O NH N O N NN ONH N ONH NN
O O
NH OO NN O O S
NH O
Space visualization.
We start from the center molecule and decode the neighborhood latent vectors (neighborhood in projected 2D space).
SMILES For molecules, we visualize the latent space in 2 dimensions.
We ﬁrst embed a random molecule into latent space.
Then we randomly generate 2 orthogonal unit vectors A .
To get the latent representation of neighborhood, we interpolate the 2-D grid and project back to latent space with pseudo inverse of A .
Finally we show decoded molecules.
In Figure 5, we present two of such grid visualizations.
Subjectively compared with ﬁgures in Kusner et al.
(2017), our visualization is characterized by having smoother differences between neighboring molecules, and more complicated decoded structures.
Appendix A G RAMMAR A. G RAMMAR FOR P ROGRAM S YNTAX
The syntax grammar for program is a generative contest free grammar start with (cid:104) program (cid:105)
[Table]
(cid:104) program (cid:105) → (cid:104) stat list (cid:105) (cid:104) stat list (cid:105) → (cid:104) stat (cid:105) ‘ ; ’ (cid:104)
[Table]
| (cid:104) stat (cid:105) (cid:104) stat (cid:105) → (cid:104)
[Table]
|
[Table]
(cid:104) assign
[Table]
→
[Table]
lhs
[Table]
‘ = ’
[Table]
return (cid:105) → ‘ return: ’ (cid:104) lhs
[Table]
(cid:104) lhs
[Table]
→ (cid:104) var
[Table]
(cid:104) var (cid:105)
[Table]
var
[Table]
digit
[Table] [Table]
’ |
[Table]
[Table] [Table]
’ |
[Table]
’ |
[Table]
’ | ‘
[Table] [Table]
A. G RAMMAR FOR M ODECULE S YNTAX
Our syntax grammar is based on OpenSMILES standard, a generative context free grammar starting with (cid:104) s (cid:105) .
Speciﬁcly
[Table]
we use the Kekul ´ e form for representing aromaticity
[Table]
[Figure] [Figure]
[Table]
A. E XAMPLES OF SMILES SEMANTICS
Here we provide more explanations of the semantics constraints that contained in SMILES language for molecules.
Speciﬁcally, the semantics we addressed here are:
Ringbond matching: the ringbonds should come in pairs.
Each pair of ringbonds has an index and bond-type associated.
So the SMILES semantics requires something exactly the same as well-known cross-serial dependencies (CSD).
CSD is also appeared in some natural languages, such as Dutch and Swiss-German.
See Figure 6 for an illustration.
Explicit valence control: Intuitively, the semantics says each atom cannot have too many bonds associated with it.
For example, a normal Carbon atom has maximum valence of 4.
This means associating a Carbon atom with two triple-bonds will violate the semantics.
A. D EPENDENCY GRAPH INTRODUCED BY ATTRIBUTE GRAMMAR
Suppose there is a production r = u 0 → u 1 u 2 . . .
u | β | ∈ R and an attribute u i .a
we denote the dependency set D r ( u i .a )
= { u j .b |
u j .b is required for calculating u i .a } .
The union of all dependency sets D ( att ) T =
(cid:83) r ∈T ,u i ∈
r D r ( u i .a ) induces a dependency graph, where nodes are the attributes and directed edges represents the dependency relationships between those attributes computation.
Here T is an (partial or full) instantiation of the generated syntax tree of grammar G .
Let
D r ( u i ) =
{ u j |∃ a, b : u
j .b ∈ D r ( u i .a ) } and D T
= (cid:83) r ∈T ,u i ∈
r D r ( u i ) , i.e. , D T is constructed from
D ( att ) T by merging nodes with the same symbol but different attributes, we call D ( att ) T is noncircular if the corresponding D T is noncircular.
In our paper, we assume the noncircular property of the dependency graph.
Such property will be exploited for top-down generation in our decoder.
B T RAINING D ETAILS
Since our proposed SD-VAE differentiate itself from previous works (CVAE, GVAE) on the formalization of syntax and semantics, we therefore use the same deep neural network model architecture for a fair comparison.
In encoder, we use 3-layer 1 dimension convolution neural networks (CNNs) following by a full connected layer, whose output would be feed to two separate afﬁne layers for producing µ and σ respectively in reparameterization trick; and in decoder we use 3 layer RNNs following by a afﬁne layer activated by softmax that gives probability for each production rule.
In detail, we use 56 dimensions the latent space and the dimension of layers as the same number as in
Kusner et al.
As for implementation, we use Kusner et al.
(2017)’s open sourced code for baselines, and implement our model with PyTorch framework 3 .
In a 10% validation set we turn the following hyper parameters and report the test result from setting with best valid loss.
For a fair comparison, all tunings are also conducted in the baselines.
We explore the use of widely adopted Gated Recurrent Units (GRUs)
Cho et al.
and
recently proposed Simple Recurrent Unit (SRU)
Lei & Zhang (2017) that are much effective for training in the choice of RNN cells.
The observations shows that for our tasks, SRU is consistently faster in training measured in wall clock time but converges to a slightly worse result, therefore we stick to
GRU in all experiments.
We use ReconstructLoss + α KLDivergence as the loss function for training.
A natural setting is α = 1 , but Kusner et al.
(2017) suggested in their open-sourced implementation 4 that using α = 1 / LatentDimension would leads to better results.
We explore both settings.
Speciﬁcally for the ZINC molecule dataset, we explore two popular ways to represent aromaticity:
The Kekule form using the single and double bound alternatively, and the use of lowercase latter for aromatic atoms.
C M ORE EXPERIMENT DETAILS
NH 2+
O NH O NH
O Target
NH 2+
O NH O NH
O Decoded:
NH 2+
O NH O NH
O Decoded:
NH 2+
O NH O NH
O Decoded:
NH 2+
O NH O NH
O Decoded:
NH +
OH
S Br
Target
NH +
OH
S Br
Decoded:
NH +
OH
S Br
Decoded:
NH +
OH
S Br
Decoded:
NH +
OH
S Br
Decoded:
NH
O NH N N N N
Target
NH
O NH N N N
Decoded:
NH
O NH N N N N
Decoded:
NH
O NH N N N N
Decoded:
NH
O NH N N N N
Decoded:
Cl O O NH N
Target
Cl O O NH N
Decoded:
Cl O O NH N
Decoded:
Cl O O NH N
Decoded:
Cl O O NH N
Decoded:
O O
O
O O
N
O
Target
O
O O O O
N
O
Decoded:
O
O
O O O
N
Br O O N
Decoded:
O O
O
O O
N
O
Decoded:
O
O O O O
N
O
Decoded:
N
N NH
Target
OH
N NH
Decoded:
NH 2 N
N NH
Decoded:
OH
N NH
Decoded:
Br N N NH
Decoded:
O O NH F
Target
NH O NH F
Decoded:
NH O NH F
Decoded:
NH O NH F
Decoded:
O O NH F
Decoded:
N NH N O NH N N
Target
N N N O NH NH NHNH
Decoded:
N N N O NH O NHN
Cl
Decoded:
N N N
O NH N NH NH
Decoded:
N N N O
NH NHNHN
Cl
Decoded:
N N NH O N O
NH O
Target
N N NH O N O
NH O
Decoded:
N N NH O N O
NH O
Decoded:
N N NH O N O
NH O
Decoded:
N N NH O N O
NH O
Decoded:
O OH
O N
H 2 N N OH Target
O OH
O N HO
NH 2 N OH
Decoded:
O OH
O N
H 2 N N O Decoded:
O OH
O N
H 2 N N O Decoded:
O OH
O N HO
NH 2 N O
Decoded:
NN O NH O S N S
O NH
Target
NN O NH O S N S
O NH
Decoded:
NN O NH O S N S
O NH
Decoded:
NN O NH O S N S
O NH
Decoded:
NN O NH O S N S
O NH
Decoded:
N
NH
O O NH N N NH
+
Target
N
NH
O O NH N N NH
+
Decoded:
N
NH
O O NH N N NH
+
Decoded:
N
NH
O O NH N N NH
+
Decoded:
N
NH
O O NH N N NH
+
Decoded:
C. B AYESIAN OPTIMIZATION
The Bayesian optimization is used for searching latent vectors with desired target property.
For example, in symbolic program regression, we are interested in ﬁnding programs that can ﬁt the grammarVAE/issues/  given input-output pairs; in drug discovery, we are aiming at ﬁnding molecules with maximum drug likeness.
To get a fair comparison with baseline algorithms, we follow the settings used in Kusner et al.
Speciﬁcally, we ﬁrst train the variational autoencoder in an unsupervised way.
After obtaining the generative model, we encode all the structures into latent space.
Then these vectors and corresponding property values ( i.e. , estimated errors for program, or drug likeness for molecule) are used to train a sparse Gaussian process with 500 inducing points.
This is used later for predicting properties in latent space.
Next, 5 iterations of batch Bayesian optimization with the expected improvement (EI) heuristic is used for proposing new latent vectors.
In each iteration, 50 latent vectors are proposed.
After the proposal, the newly found programs/molecules are then added to the batch for next round of iteration.
During the proposal of latent vectors in each iteration, we perform 100 rounds of decoding and pick the most frequent decoded structures.
This helps regulates the decoding due to randomness, as well as increasing the chance for baselines algorithms to propose valid ones.
C. R ECONTRUCTION
We visualize some reconstruction results of SMILES in Figure 7.
One can see that, in most cases the decoder successfully recover the exact origin input.
Due to the stochasticity of decoder, it may have some small variations.