L EARNING A GENERATIVE MODEL FOR VALIDITY IN COMPLEX DISCRETE STRUCTURES
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep generative models have been successfully used to learn representations for high-dimensional discrete spaces by representing discrete objects as sequences, for which powerful sequence-based deep models can be employed.
Unfortunately, these techniques are signiﬁcantly hindered by the fact that these generative models often produce invalid sequences: sequences which do not represent any underlying discrete structure.
As a step towards solving this problem, we propose to learn a deep recurrent validator model, which can estimate whether a partial sequence can function as the beginning of a full, valid sequence.
This model not only discriminates between valid and invalid sequences, but also provides insight as to how individual sequence elements inﬂuence the validity of the overall sequence, and the existence of a corresponding discrete object.
To learn this model we propose a reinforcement learning approach, where an oracle which can evaluate validity of complete sequences provides a sparse reward signal.
We believe this is a key step toward learning generative models that faithfully produce valid sequences which represent discrete objects.
We demonstrate its effectiveness in evaluating the validity of Python 3 source code for mathematical expressions, and improving the ability of a variational autoencoder trained on SMILES strings to decode valid molecular structures.
I NTRODUCTION
Deep generative modeling has seen many successful recent developments, such as producing realistic images from noise (Radford et al., 2015) and creating artwork (Gatys et al., 2016).
We ﬁnd particularly promising the opportunity to leverage deep generative models for search in high-dimensional discrete spaces (G ´ omez-Bombarelli et al., 2016b; Kusner et al., 2017).
Discrete search is at the heart of problems in drug discovery (G ´ omez-Bombarelli et al., 2016a), natural language processing (Bowman et al., 2016; Guimaraes et al., 2017), and symbolic regression (Kusner et al., 2017).
The application of deep modelling to search involves ‘lifting’ the search from the discrete space to a continuous space, via an autoencoder (Rumelhart et al., 1985).
An autoencoder learns two mappings: 1) a mapping from discrete space to continuous space called an encoder ; and 2) a reverse mapping from continuous space back to discrete space called a decoder .
The discrete space is presented to the autoencoder as a sequence in some formal language — for example, in G´omez-Bombarelli et al.
(2016b) molecules are encoded as SMILES strings — and powerful sequential models (e.g.,
LSTMs (Hochreiter & Schmidhuber, 1997) GRUs (Cho et al., 2014), DCNNs (Kalchbrenner et al., 2014)) are applied to the string representation.
When employing these models as encoders and decoders, generation of invalid sequences is however possible, and using current techniques this happens frequently.
Kusner et al.
(2017) aimed to ﬁx this by basing the sequential models on parse tree representations of the discrete structures, where externally speciﬁed grammatical rules assist the model in the decoding process.
This work boosted the ability of the model to produce valid sequences during decoding, but its performance achieved by this method leaves scope for improvement, and the method requires hand-crafted grammatical rules for each application domain.
In this paper, we propose a generative approach to modeling validity that can learn the validity constraints of a given discrete space.
We show how concepts from reinforcement learning may be used to deﬁne a suitable generative model and how this model can be approximated using sequencebased deep learning techniques.
To assist in training this generative model we propose two data
C
C 0.
B C
N O
S P
F I c n ...+ = # / \ ( ) [ ] 0.
embedding
LSTMcell
LSTMcell augmentation techniques.
Where no labelled data-set of valid and invalid sequences to learn from is available, we propose a novel approach to active learning for sequential tasks inspired by classic mutual-information-based approaches (Houlsby et al., 2011; Hern´andez-Lobato et al., 2014).
In the context of molecules, where data sets containing valid molecule examples do exist, we propose an effective data augmentation process based on applying minimal perturbations to known-valid sequences.
These two techniques allow us to rapidly learn sequence validity models that can be used as a) generative models, which we demonstrate in the context of Python 3 mathematical expressions and b) a grammar model for character-based sequences, that can drastically improve the ability of deep models to decode valid discrete structures from continuous representations.
We demonstrate the latter in the context of molecules represented as SMILES strings.
A MODEL FOR SEQUENCE VALIDITY
To formalise the problem we denote the set of discrete sequences of length T by X = { ( x 1 , . .
, x T ) : x t ∈ C} using an alphabet C = { 1 , . . .
, C } of size C .
Individual sequences in X are denoted x 1: T .
We assume the availability of a validator v : X → { 0 , 1 } , an oracle which can tell us whether a given sequence is valid.
It is important to note that such a validator gives very sparse feedback: it can only be evaluated on a complete sequence.
Examples of such validators are compilers for programming languages (which can identify syntax and type errors) and chemo-informatics software for parsing
SMILES strings (which identify violations of valence constraints).
Running the standard validity checker v ( x 1: T ) on a partial sequence or subsequence (e.g., the ﬁrst t < T characters of a computer program) does not in general provide any indication as to whether the complete sequence of length T is valid.
We aim to obtain a generative model for the sequence set X + = { x 1: T ∈ X : v ( x 1: T ) = 1 } , the subset of valid sequences in X .
To achieve this, we would ideally like to be able to query a more-informative function ˜ v ( x 1: t ) which operates on preﬁxes x 1: t of a hypothetical longer sequence x 1: T and outputs
[Equation]
where c [ x 1: t , x t +1: T ] concatenates a preﬁx and a sufﬁx to form a complete sequence.
The function ˜ v ( x 1: t ) can be used to determine whether a given preﬁx can ever successfully yield a valid outcome.
Note that we are indifferent to how many sufﬁxes yield valid sequences.
With access to ˜ v ( x 1: t ) , we could create a generative model for X + which constructs sequences from left to right, a single character at a time, using ˜ v ( x 1: t ) to provide early feedback as to which of the next character choices will surely not lead to a “dead end” from which no valid sequence can be produced.
We frame the problem of modelling X + as a Markov decision process (Sutton & Barto, 1998) for which we train a reinforcement learning agent to select characters sequentially in a manner that avoids producing invalid sequences.
At time t = 1 , . . .
, T , an agent is in state x <t = x 1: t − 1 and can take actions x t ∈ C .
At the end of an episode, following action x T , the agent receives a reward of v ( x 1: T ) .
Since in practice we are only able to evaluate v ( x 1: T ) in a meaningful way on complete sequences, the agent does not receive any reward at any of the intermediate steps t < T .
The optimal Q -function
Q (cid:63) ( s, a ) (Watkins, 1989), a function of a state s and an action a , represents the expected reward of an agent following an optimal policy which takes action a at state s .
This optimal Q -function assigns value 1 to actions a = x t in state s = x <t for which there exists a sufﬁx x t +1: T such that c [ x 1: t , x t
+1: T ] ∈ X + , and value 0 to all other state/action pairs.
This behaviour exactly matches the desired preﬁx validator in (1), that is, Q (cid:63) ( x <t , x t )
=
˜ v ( x 1: t ) , and so for the reinforcement learning environment as speciﬁed, learning ˜ v ( x 1: t ) corresponds to Q-learning.
Having access to Q (cid:63) would allow us to obtain a generative model for X + .
In particular, an agent following any optimal policy π (cid:63) ( x <t ) =
argmax x t ∈C Q (cid:63) ( x <t , x t ) will always generate valid sequences.
If we sample uniformly at random across all optimal actions at each time t = 1 , . .
, T , we obtain the joint distribution given by
[Equation]
where Z ( x <t ) = (cid:80) x t Q (cid:63) ( x <t , x t ) are the per-timestep normalisation constants.
This distribution allows us to sample sequences x 1: T in a straightforward manner by sequentially selecting characters x t ∈ C given the previously selected ones in x <t .
In this work we focus on learning an approximation to (2).
For this, we use recurrent neural networks, which have recently shown remarkable empirical success in the modelling of sequencial data, e.g., in natural language processing applications (Sutskever et al., 2014).
We approximate the optimal
Q -function with a long-short term memory (LSTM) model (Hochreiter & Schmidhuber, 1997) that includes one output unit per character in C , with each output unit using a logistic activation function (see ﬁgure 1), such that the output is in the closed interval [0 , 1] .
We denote by y( x t | x <t , w ) the value at time t of the LSTM output unit corresponding to character x t when the network weights are w the input is the sequence x <t .
We interpret the neural network output y( x t | x <t , w ) as p ( Q (cid:63) ( x <t , x t ) = 1) , that is, as the probability that action x t can yield a valid sequence given that the current state is x <t .
Within our framing a sequence x 1: T will be valid according to our model if every action during the sequence generation process is permissible, that is, if Q (cid:63) ( x <t , x t )
= 1 for t = 1 , . . .
, T .
Similarly, we consider that the sequence x 1: T will be invalid if at least one action during the sequence generation process is not valid 1 that is, if Q (cid:63) ( x <t , x t ) =
at least once for t = 1 , . . .
, T .
This speciﬁes the following log-likelihood function given a training set D
= { ( x n 1: T , y n ) } Nn =1 of sequences x n 1: T ∈ X and corresponding labels y n = v ( x 1: T ) :
[Equation]
where, following from the above characterisation of valid and invalid sequences, we deﬁne
[Equation]
−
[Equation]
according to our model’s predictions.
The log-likelihood (3) can be optimised using backpropagation and stochastic gradient descent and, given sufﬁcient model capacity, results in a maximiser ˆ w such that y( x t | x <t , ˆ w ) ≈ Q (cid:63) ( x <t , x t ) .
Instead of directly maximising (3), we can follow a Bayesian approach to obtain estimates of uncertainty in the predictions of our LSTM model.
For this, we can introduce dropout layers which stochastically zero-out units in the input and hidden layers of the LSTM model according to a
Bernoulli distribution (Gal & Ghahramani, 2016).
Under the assumption of a Gaussian prior p ( w ) over weights, the resulting stochastic process yields an implicit approximation q ( w ) to the posterior distribution p ( w |D )
∝ exp( L ( w |D ))
p ( w ) .
We do this to obtain uncertainty estimates, allowing us to perform efﬁcient active learning, as described in section 3.1.
O NLINE GENERATION OF SYNTHETIC TRAINING DATA
One critical aspect of learning w as described above is how to generate the training set D in a sensible manner.
A na¨ıve approach could be to draw elements from X uniformly at random.
However, in many cases, X contains only a tiny fraction of valid sequences and the uniform sampling approach produces extremely unbalanced sets which are useless in practice.
While rejection sampling can be used to increase the number of positive samples, the resulting additional cost makes such an alternative infeasible in most practical cases.
The problem gets typically worse as the sequence length
T increases since |X| always grow as |C| T , while |X + | will typically grow at a lower rate.
We employ two approaches for artiﬁcially constructing balanced sets that permit learning these models in far fewer samples than |C| T .
In settings where we do not have a corpus of known valid sequences,
Bayesian active learning can automatically construct the training set D .
These methods work by iteratively selecting sequences in X that are maximally informative about the model parameters w given the data collected so far (MacKay, 1992).
When we do have a set of known valid sequences, we can use these to seed a process for generating balanced sets by applying random perturbations to valid sequences.
3. A CTIVE LEARNING
Let x 1: T denote an arbitrary sequence and let y be the unknown binary label indicating whether x 1: T is valid or not.
Our model’s predictive distribution for y , that is, p ( y | x 1: T , w ) is given by (4).
The amount of information on w that we expect to gain by labeling and adding x 1: T to D can be measured in terms of the expected reduction in the entropy of the posterior distribution p ( w |D ) .
That is,
[Equation]
(5) where H ( · ) computes the entropy of a distribution.
This formulation of the entropy-based active learning criterion is, however, difﬁcult to approximate, because it requires us to work with the entropy of p ( w |D ) , an intractable quantity.
To obtain a simpler expression we follow Houlsby et al.
and note that α ( x 1: T ) is equal to the mutual information between y and w given x 1: T and D
[Equation]
which is easier to work with as the required entropy is now that of Bernoulli predictive distributions, an analytic quantity.
Let a B ( p ) denote a Bernoulli distribution with probability p , with probability mass p z (1 − p ) 1 − z for values z ∈ { 0 , 1 } .
The entropy of B ( p ) can be easily obtained as
[Equation]
The expectation with respect to p ( w |D ) can be easily approximated by Monte Carlo.
We could attempt to sequentially construct D by optimising (6).
However, this optimisation process would still be difﬁcult, as it would require evaluating α ( x 1: T ) exhaustively on all the elements of X .
To avoid this, we follow a greedy approach and construct our informative sequence in a sequential manner.
In particular, at each time step t = 1 , . . .
, T , we select x t by optimising the mutual information between w and Q (cid:63) ( x <t , x t ) , where x <t denotes here the preﬁx already selected at previous steps of the optimisation process.
This mutual information quantity is denoted by α ( x t | x <t ) and its expression is given by
[Equation]
The generation of an informative sequence can then be performed efﬁciently by sequentially optimising (8), an operation that requires only |C| × T evaluations of α ( x t | x <t ) .
To obtain an approximation to (8), we ﬁrst approximate the posterior distribution p ( w |D ) with q ( w ) and then estimate the expectations in (8) by Monte Carlo using K samples drawn from q ( w ) .
The resulting estimator is given by
[Equation]
where w 1 , . . . , w K ∼ q ( w ) and g ( · ) is deﬁned in (7).
The nonlinearity of g ( · ) means that our Monte
Carlo approximation is biased, but still consistent.
In practice, reasonable estimates can be obtained even for small K .
In our experiments we use K = 16 .
The iterative procedure just described is designed to produce a single informative sequence.
In practice, we would like to generate a batch of informative and diverse sequences.
The reason for this is that, when training neural networks, processing a batch of data is computationally more efﬁcient than processing individually multiple data points.
To construct a batch with L informative sequences, we propose to repeat the previous iterative procedure L times.
To introduce diversity in the batch-generation process, we “soften” the greedy maximisation operation at each step by injecting a small amount of noise in the evaluation of the objective function (Finkel et al., 2006).
Besides introducing diversity, this can also lead to better overall solutions than those produced by the noiseless greedy approach (Cho, 2016).
We introduce noise into the greedy selection process by sampling from
[Equation]
for each t = 1 , . . .
, T , which is a Boltzmann distribution with sampling temperature θ .
By adjusting this temperature parameter, we can trade off the diversity of samples in the batch vs. their similarity.
3. D ATA AUGMENTATION
In some settings, such as the molecule domain we will consider later, we have databases of knownvalid examples (e.g. collections of known drug-like molecules).
Training directly on sequences which are valid has a danger of overﬁtting to only the subset of the space covered by the data, rather than the entire domain of X ; furthermore, if the model views primarily valid sequences during training then the batches of data will be quite non-uniform.
We address this by augmenting a database of valid sequences x 1: T with additional, minimallyperturbed invalid sequences x (cid:48) 1: T .
This is constructed by setting each x (cid:48) t to be a symbol selected independently from C with probability γ , while remaining the original x t with probability 1 − γ .
In expectation this changes γT entries in the sequence.
We choose γ = 0 .
05 , which results in synthetic data that despite its high per-character overlap with original SMILES strings is less than 1% valid.
E XPERIMENTS
We test the proposed validity checker in two environments.
First, we look at ﬁxed length Pythonmathematical expressions, where we derive lower bounds for the support of our model and compare the performance of active learning with that achieved by a simple passive approach.
Secondly, we look at molecular structures encoded into string representation, where we utilise existing molecule data sets together with our proposed data augmentation method to learn the rules governing molecule string validity.
We test the efﬁcacy of our validity checker on the downstream task of decoding valid molecules from a continuous latent representation given by a variational autoencoder.
4. M ATHEMATICAL EXPRESSIONS
We illustrate the utility of the proposed validity model and sequential Bayesian active learning in the context of Python 3 mathematical expressions.
Here, X consists of length 25 sequences built from the alphabet of numbers and symbols shown in table 1.
The validity of any given expression is then determined using the Python 3 eval function to evaluate their validity: this either returns the result of the calculation if the sequence is valid or raises an exception.
[Table]
[Equation]
Within this problem domain we do not assume the existence of a data set of positive examples.
Instead, we rely solely on sampling sequences from X .
We compare two methods of selecting these training examples: passive , where training sequences are sampled from a uniform distribution over X , and active , where we use the procedure described in section 3.1 to select training sequences.
We train
[Figure]
AU
C
[Figure]
passive 0 .0 .0 .
.validity fraction of samples     e n t r opy active
passive
[Table]
an otherwise identical model using each of the two methods, and present metrics of their respective performance in ﬁgure 2.
With no validation data set to measure performance on, we compare the models in terms of their capability provide high entropy distributions over valid sequences – a validity-entropy area under curve, the details of which are described below.
We ﬁnd that the active method is able to learn a model of sequence validity much more rapidly than sampling uniformly from X
[Table]
and the corresponding converged model is capable of generating many more distinct valid sequences than that trained using the passive method.
This is unsurprising, as sampling from X uniformly yields a valid sequence on the order of one in every ten thousand samples.
In table 2 we present lower bounds on the support of the two respective models.
The details of how this lower bound is computed can be found in appendix B.
In order to measure the performance of the model in terms of its ability to generate diverse valid sequences without reference to any speciﬁc data set, we deﬁne a generative procedure to sample from the model and measure the validity and entropy of the samples.
To sample stochastically, we use a
Boltzmann policy, i.e. a policy which samples next actions according to
[Equation]
where τ is a temperature constant that governs the trade-off between exploration and exploitation.
Note that this is not the same as the Boltzmann distribution used as a proposal generation scheme during active learning, which was deﬁned not on Q -function values but rather on the estimated mutual information.
We obtain samples { x (1) , . . .
, x ( N ) } τ i for a range of temperatures τ
i and compute the validity fraction and entropy of each set of samples.
These points now plot a curve of the tradeoff between validity and entropy that a given model provides.
Without a preferred level of sequence validity, the area under this validity-entropy curve (V-H AUC) can be utilised as a metric of model quality.
'CC1CN(C(=O)C2=F' example invalid sequences: 'CC1CN(C(=OC' 'CC1CN(C()' ']' reason:
O has too many bonds
F cannot have double bond cannot have empty branch () cannot have ] before [ 'CC1CN(C(=O)C2=CC(BrC' 'CC1CN(C(=O)C2=CC(BrF' 'CC1CN(C(=)'
Br has too many bonds
Br, F have too many bonds branch must have atoms
SMILES: 'CC1CN(C(=O)C2=CC(Br)=CN2C)CCC1[NH3+]' has heatmap molecule is in Figure 4 in
To provide some context for the entropy values we estimate an information theoretic lower bound for the fraction of the set X + that our model is able to generate.
This translates to upper bounding the false negative rate for our model.
We also see Bayesian active learning converge faster per sample, with little additional computational cost; per 10,000 data points, training run time is 31 seconds with the passive approach, and 38 seconds with active.
4. SMILES MOLECULES
SMILES strings (Weininger, 1970) are one of the most common representations for molecules, consisting an ordering of atoms and bonds.
It is attractive for many applications because it maps the graphical representation of a molecule to a sequential representation, capturing not just its chemical composition but also structure.
This structural information is captured by intricate dependencies in
SMILES strings based on chemical properties of individual atoms and permissable atom connectivities.
For instance, the atom Bromine can only bond with a single other atom, meaning that it may only occur at the beginning or end of a SMILES string, or within a so-called ‘branch’, denoted by a bracketed expression (Br) .
We illustrate some of these rules, including a Bromine branch, in ﬁgure 3, a graphical representation of a molecule alongside its corresponding SMILES string.
There, we also show examples of how a string may fail to form a valid SMILES molecule representation.
The full SMILES alphabet is presented in Table 3.
[Table]
B
[Equation] [Equation]
The intricacy of SMILES strings makes them a suitable testing ground for our method.
There are two technical distinctions to make between this experimental setup and the previously considered Python 3 mathematical expressions.
As there exist databases of SMILES strings, we leverage those by using the data augmentation technique described in section 3.2.
The main data source considered is the
ZINC data set Irwin & Shoichet (2005), as used in Kusner et al.
We also use the USPTO 15k reaction products data (Lowe, 2014) and a set of molecule solubility information (Huuskonen, 2000) are used as withheld test data.
Secondly, whereas we used ﬁxed length Python 3 expressions in order to obtain coverage bounds, molecules are inherently variable length.
We deal with this by padding all molecules to ﬁxed length.
As a ﬁrst test of the suitability of our proposed validity model, we train it on augmented ZINC data and examine the accuracy of its predictions on a withheld test partition of that same data set as well as the two unseen molecule data sets.
Accuracy is the ability of the model to accurately recognise which perturbations make a certain SMILES string invalid, and which leave it valid – effectively how well the model has captured the grammar of SMILES strings in the vicinity of the data manifold.
Recalling that a sequence is invalid if ˜ v ( x 1: t )
=
at any t ≤ T , we consider the model prediction for molecule x 1: T to be (cid:81)
Tt
=
I [y( x t | x <t , w ) ≥ 0 .
5] , and compare this to its true label as given by rdkit, a chemical informatics software.
The results are encouraging, with the model achieving 0.
accuracy on perturbed ZINC (test) and 1.000 accuracy on both perturbed USPTO and perturbed
Solubility withheld data.
Perturbation rate was selected such that approximately half of the perturbed strings are valid.
To demonstrate the models capability of improving preexisting generative models for discrete structures, we show how it can be used to improve the results of previous work, a character variational autoencoder (CVAE) applied to SMILES strings (G´omez-Bombarelli et al., 2016b; Kingma & Welling, 2013).
Therein, an encoder maps points in X + to a continuous latent representation Z and a paired decoder maps points in Z back to X + .
A reconstruction based loss is minimised such that training points mapped to the latent space decode back into the same SMILES strings.
The fraction of test points that do is termed reconstruction accuracy.
The loss also features a term that encourages the posterior over Z to be close to some prior, typically a normal distribution.
The decoder functions by outputting a set of weights f ( x t | z ) for each character x t in the reconstructed sequence conditioned on a latent point z ∈ Z ; the sequence is recovered by sampling from a multinomial according to these weights.
To integrate our validity model into this framework, we take the decoder output for each step t and mask out choices that we predict cannot give valid sequence continuations.
We thus sample characters with weights given by f ( x t | z ) · I [y( x t | x <t , w ) ≥ 0 . 5] .
A key metric for the performance of variational autoencoder models for discrete structures is the fraction of points sampled from the prior over Z that decode into valid molecules.
If many points do not correspond to valid molecules, any sort of predictive modelling on that space will likely also mostly output invalid SMILES strings.
The validity model provides an order of magnitude improvement on this metric when applied to the character VAE.
Furthermore, the binary nature of the learnt model means that it does not affect the reconstruction accuracy.
In fact, some modest gains are present.
The inclusion of our validity model addition is completely post-hoc, and thus can be applied to any exiting character-based VAE model where elements of the latent space correspond to a structured discrete sequence.
Table 4 contains a comparison of our work to a plain CVAE and to the
Grammar VAE approach.
We use a Kekul ´ e format of the ZINC data in our experiments, a speciﬁc representation of aromatic bonds that our model handled particularly well.
Note that the results we quote for Grammar VAE are taken directly from Kusner et al.
(2017) and on non-Kekul´e format data.
The CVAE model is trained for 100 epochs only, as per previous work – further training improves reconstruction accuracy.
[Table]
D ISCUSSION
In this work we proposed a modelling technique for learning the validity constraints of discrete spaces.
The proposed likelihood makes the model easy to train, is unaffected by the introduction of padding for variable length sequences and due to its optimum being largely independent of the training data distribution, allows for active learning techniques to be utilised.
Through experiments we found that the proposed technique is only as strong as the data it is given.
It is vital to show the model informative examples of validity constraints being validated.
Where no informative data sets exist, we proposed a mutual-information-based active learning scheme that uses model uncertainty to select training sequences.
We used principled approximations to make that learning scheme computationally feasible.
Where data sets of positive examples is available, we proposed a simple method of perturbations to create informative examples of validity constraints being broken.
The model showed promise on the Python mathematical expressions problem, especially when combined with active learning.
In the context of SMILES molecules, the model as able to learn near-perfectly the validity of independently perturbed molecules.
When applied to the variational autoencoder benchmark on SMILE strings, the proposed method beat the previous results by a large margin on prior sample validity, the relevant metric for the downstream utility of the latent space.
The method is can be simply overlaid onto existing character-based model and can be trained easily using data produced through our augmentation method.
The perturbations used do not, however, capture every way in which a molecule may be mis-constructed.
Correlated changes such as the insertion of matching brackets into expressions are likely what is missing from our scheme.
An improved data augmentation method, perhaps one combining the advantages of uncertainty-based exploration with the ability to exploit existing data sets effectively, could lead to further improvements to the here presented results.
A A PPENDIX : MODEL PARAMETERS AND IMPLEMENTATION DETAILS
Model hyperparameters.
There are a handful of free parameters in both the LSTM model and the training procedure.
We document the values used in all experiments in table 5.
For both models we used a distributed representation of the characters with a size of 20 as input to the LSTM (i.e., an embedding layer of size 20).
[Table]
B C OVERAGE ESTIMATION
Ideally, we would like to check that the learned model y( x t | x <t , w ) assigns positive probability to exactly those points which may lead to valid sequences, but for large discrete spaces this is impossible to compute or even accurately estimate.
A simple check for accuracy could by to evaluate whether the model correctly identiﬁes points as valid in a known, held-out validation or test set of real data, relative to randomly sampled sequences (which are nearly always invalid).
However, if the validation set is too “similar” to the training data, even showing 100% accuracy in classifying these as valid may simply indicate having overﬁt to the training data: a discriminator which identiﬁes data as similar to the training data needs to be accurate over a much smaller space than a discriminator which estimates validity over all of X .
Instead, we propose to evaluate the trade-off between accuracy on a validation set, and an approximation to the size of the effect support of (cid:81) t y( x t | x <t , w ) over X .
Let X + denote the valid subset of X .
Suppose we estimate the fraction valid f + = |X + | / |X| by simple Monte Carlo, sampling uniformly from X .
We can then estimate N + = |X
+ | by f + |X| , where |X| =
C T , a known quantity.
A uniform distribution over N + sequences would have an entropy of log N + .
We denote the entropy of output from the model H .
If our model were perfectly uniform over the sequences it can generate, it would then be capable of generating N model =
e H .
As our model at its optimum is extremely not uniform over elements x ∈ X , this is very much a lower bound a coverage.
[Figure]
H I
N O
P S 1357# ( ) + - = [ ]
C a n d i d a t e s y m b o l
[Figure]