Drawing out of Distribution with Neuro-Symbolic Generative Models
Anonymous Author(s)
AffiliationAddress
Abstract
Learning general-purpose representations from perceptual inputs is a hallmark of human intelligence.
For example, people can write out numbers or characters, or even draw doodles, bycharacterizing these tasks as different instantiations of the same generic underlying process—compositional arrangements of different forms of pen strokes.
Crucially, learning to do one task,say writing, implies reasonable competence at another, say drawing, on account of this sharedprocess.
We present D rawing o ut o f D istribution (DooD), a neuro-symbolic generative model ofstroke-based drawing that can learn such general-purpose representations.
In contrast to priorwork, DooD operates directly on images, requires no supervision or expensive test-time inference,and performs unsupervised amortised inference with a symbolic stroke model that better enablesboth interpretability and generalization.
We evaluate DooD on its ability to generalise acrossboth data and tasks.
We first perform zero-shot transfer from one dataset (e.g. MNIST) toanother
(e.g. Quickdraw), across five different datasets, and show that DooD clearly outperformsdifferent baselines.
An analysis of the learnt representations further highlights the benefits ofadopting a symbolic stroke model.
We then adopt a subset of the Omniglot challenge tasks,and evaluate its ability to generate new exemplars (both unconditionally and conditionally), andperform one-shot classification, showing that DooD matches the state of the art.
Taken together,we demonstrate that DooD does indeed capture general-purpose representations across both dataand task, and takes a further step towards building general and robust concept-learning systems.
Introduction
[Figure]
Humans can learn representations of data that are generalpurpose and meaningful.
Being general-purpose permits effective reuse when characterizing novel observations, and beingmeaningful facilitates tasks like generating or classifying observations.
Key to this is a generic process for characterizingobservations—inferring what features are relevant and how theycompose to generate the observations [9, 13, 33].
For example,when observing handwritten numbers, we learn to characterisethem as sequential compositions ( how ) of different pen strokes( what ).
This is general-purpose as it allows characterizing novelobservations, say doodles instead of numbers, simply as novelcompositions of previously learnt pen strokes.
It is also meaningful since pen-strokes themselves are symbolic and interpretable.
Current computational approaches capture important aspects of generalizability, but none of theseare simultaneously efficient, reliable, interpretable, and unsupervised [25].
At one end, symbolicapproaches like Lake et al.
[24] attribute generalization to an explicit hierarchical composition processinvolving sub-strokes, strokes, and characters, and build concomitant models that demonstrate humanlike generalization abilities across different tasks.
At the other end, neural approaches like deepgenerative models [9, 18, 29] and deep meta-learning [11, 32, 34] favour scalable learning from raw
STN ˜ x < t l t s t ⊙ δ R o t x t ⊗ x < t x ≤ t ˜ s
t
h lt h st
Generative Model
STN ˜ ∆ x t l t s t ∆ x t o t ⊖ x
< t x h lt h st
Recognition Model s t −h st −• h st x < t l t −h lt −• h lt x < t stop_gradient
STN
SpatialTransformers[20]
Both models are autoregressive via two shared with hidden h and lt . perceptual data, unfettered by explicit representational biases such as strokes and their compositions.
Each comes with its own shortcomings—symbolic approaches typically need additional supervisionor data processing along with expensive special-purpose inference, and neural approaches fail togeneralise well and don’t capture interpretable representations.
Neuro-symbolic approaches [10] seekto make the best of both worlds by judiciously combining neural processing of raw perceptual inputswith symbolic processing of extracted features, but typically involve a different set of trade-offs.
We present D rawing o ut o f D istribution (DooD), a neuro-symbolic generative model of stroke-baseddrawing that can learn general-purpose representations (Fig. 1).
Our model operates directly onimages, requires no supervision, pre-processing, or expensive test-time inference, and performsefficient amortised inference with a symbolic stroke model that helps with both interpretability andgeneralization, setting us apart from the current state-of-the-art in neuro-symbolic approaches [10, 17].
We evaluate on two axes (a) generalization across data, which measures how well the learnt representations can be reused to characterise out-of-distribution data, and (b) generalization across task, wherewe measure how useful the learnt representations are for auxiliary tasks drawn from the Omniglotchallenge set [24].
We show that DooD significantly outperforms baselines on generalization acrossdatasets, highlighting the quality of the learnt representations as a factor, and on generalization acrosstasks, show that it outperforms neural models, while being competitive against SOTA neuro-symbolicmodels without requiring additional support such as supervision or data augmentation.
Method
The framework for DooD involves a generative model over sequences of strokes and their layouts,a recognition model that conditions on a given observation to predict where to place what strokes,and an amortised variational-inference learning setup that uses these models to estimate an evidencelower bound (ELBO) as the objective.
2.1 Generative Model
Conceptually, the model can be seen as drawing a figure over a sequence of steps, building up to thefinal image (as seen in Fig. 1).
At each step t the model identifies a region of the image canvas todraw in, puts down Bézier curve control points within that region, renders the curve in a differentiablemanner, and then composites this rendered stroke over the previously rendered canvas.
We refer tothese as the layout , stroke , rendering , and compositing modules respectively (elaborated below).
The model sits on a substrate of recurrent neural networks (RNNs), one each for the layout and strokemodules, with hidden states h lt and h st respectively.
The complete setup is depicted in Fig.
2 alongwith example values (images) for the different variables involved.
See Appendix B for details.
Formally, the generative model defines a joint distribution over an image x composed over T steps,with latent variables l t and s t that characterise a stroke’s location and form, and a binary latentvariable o t that determines stopping, as
[Equation]
where terminal step T is the last t where o t
= 1 , x t the rendered stroke given ( s t , l t ) , x <t thecanvas-so-far, and all variables at step t = 0 initialised to 0 .
Note that the likelihood is evaluatedonly at the terminal step T .
Layout Module: At step t , given the canvas-so-far x <t and corresponding layout-RNN hiddenstate h lt , we define the layout as a distribution over affine transforms.
This allows transforming thecanvas into a “glimpse” ˜ x <t using a Spatial Transformer Network (STN) [20], which allows focussingon a particular canvas region.
The affine transform is constructed from appropriately constrainedscale ( l sc t ), translation ( l tr t ), and rotation ( l rt ) random variables, by employing a Gaussian Mixture
Model (GMM) with M components over the collection as
[Equation]
Stroke Module:
Given the sampled affine transform l t , selected “glimpse” ˜ x <t , and correspondingstroke-RNN hidden state h st , this module defines a distribution over strokes parametrised as D th order
Bézier splines, constructing a GMM with K components for each spline control point as
[Equation]
Rendering Module: Note that the Bézier spline control points sampled from the stroke module aretaken to be in a canonical centered coordinate frame.
While this can help simplify learning by reducingthe variation required to be captured by the stroke module, the points can’t be rendered onto thecanvas as is.
We situate them properly within the context of the previously determined “glimpse” bysimply applying the affine transform l t to the control points as ˜ s t =
l t ⊙ s t .
The transformed controlpoints now describe a stroke to be drawn over the whole canvas for step t , through a differentiablerenderer δ R, to produce the rendered stroke as x t = δ R (˜ s t ) (see Appendix B.1 for details).
Using theintermediate canvas to guide model unrolling is termed execution guidance (EG).
Compositing Module: At step t this defines a distribution over whether the model should continuedrawing strokes given the rendered canvas-so-far x <t as
[Equation]
Once the model has decided to stop drawing, it stops permanently.
At step t , the rendered stroke x t iscomposited with the canvas-so-far x <t to generate x ≤ t = x <t ⊗ x t .
When the model stops, i.e., atthe last t when o t = 1 (denoted T ), the likelihood, with global learnable parameter σ , is given asp [Equation] [Equation] |
[Equation]
≤
[Equation]
) =
Laplace ( x | x ≤
[Equation]
, diag ( σ )) .
[Equation]
2.2 Recognition Model
As with prior approaches, we construct an approximate posterior to facilitate learning with amortisedvariational inference.
Using the same notation from the generative model section, we defineq
[Equation]
A couple of things stand out.
First, where the generative model made heavy use of the canvas-sofar x <t , the recognition model primarily uses the residual ∆ x t = x − x <t .
Second, being given the target observation x itself, the information available to the layout and stroke modules is quitedifferent.
While they needed to speculate where, and what stroke to draw, in the generative model,here their task is simply to isolate a part of the drawing (“glimpse”) and fit a spline to that.
As a consequence of these different characteristics, the distributions over layout and strokes in therecognition model do not need to be as flexible as the generative model—locating a curve in theresidual and fitting it with a spline does not typically involve much ambiguity.
To factor this in, andhave the variational objective be reasonable, we define corresponding distributions in the recognitionmodel using just a single component of the corresponding GMMs in the generative model as
[Equation]
2.3 Learning
Having defined the generative and recognition models, we now bring them together in order toconstruct the variational objective that will enable learning both models simultaneously from data.
[Equation]
Note that except for the stopping criterion o t which is a Bernoulli random variable, all other distributions employed are reparametrizable.
In order to construct an effective variational objective with thisdiscrete variable, we employ a control variate method, NVIL [28], that helps reduce the variance ofthe standard REINFORCE estimator, as is also done in related work [9] (see Appendix C.2).
Furthermore, in order to ensure that the ELBO objective is appropriately balanced, we employadditional weighting β for the KL-divergence over stopping criterion o t within the objective [2, 19](see Appendix C.1 for details).
This weight plays a crucial role as a mismatch could result in themodel either stopping too early or too late, resulting in incomplete or incorrect figures respectively.
Experiments
We wish to understand how well DooD generalises across both datasets (§ 3.1) and tasks (§ 3.2).
Foracross-dataset generalization
, we train DooD and Attend-Infer-Repeat (AIR) [9] 1 , an unsupervisedpart-based model, on each of five stroke-based image datasets (i) MNIST (handwritten digits) [26],(ii) EMNIST (handwritten digits and letters) [7], (iii) KMNIST (cursive Japanese characters) [6],(iv)
Quickdraw (doodles) [15], and (v) Omniglot (handwritten characters from multiple alphabets)[24], and evaluate how well the model generalises to unseen exemplars both within the same datasetand across other datasets.
We find that DooD significantly outperforms AIR, which from ablationstudies, is attributed to explicit stroke modelling and execution guidance.
Note that we only compareagainst a fully-unsupervised approach since most datasets do not provide additional data in theform of stroke labels (as required elsewhere [10]).
For across-task generalization, we primarilyfocus on Omniglot and evaluate on three out of the five challenge tasks for this dataset [25], whichinclude contextual generation and classification.
We find that our model outperforms unsupervisedbaselines where appropriate, and is competitive against SOTA neuro-symbolic models withoutrequiring additional support in the form of supervision or data augmentation.
We include exact detailsabout datasets (Appendix A), our model and the baselines (Appendix B), the training procedure(Appendix C), and the evaluation procedure (Appendix D) in the supplementary material.
3.1 Across-Dataset Generalization
MNIST-trained transfer.
To understand how our model and AIR generalise to new datasets, welook at sequential reconstructions (Fig. 3).
We train on MNIST and show sample reconstructions fromall five datasets without fine tuning.
Each model renders one step at a time by rendering latent parsesof increasing length, allowing us to evaluate and compare the performance of part decomposition andinference.
Note that we limit the maximum number of strokes to 6 throughout all experiments.
[Figure]
MNIST
[Figure]
EMNIST
KMNIST
QuickDraw
Omniglot
[Figure]
A
[Figure]
oo
[Figure]
D oo
[Figure]
G
[Figure]
e
[Figure]
c e n [Figure]
Ablation studies.
To better understand why our model generalises well, we evaluate two furthervariants of DooD that ablate a key component each: an explicit spline decoder (DooD- δ R) andexecution-guided inference (DooD-EG) (Fig. 3a).
In the model without an explicit spline decoder, wereplace the differentiable spline renderer by a neural network decoder similar to AIR.
This model stilldiffers from the AIR in terms of the learnable sequential prior and the fact that we enforce explicitconstrains over the latent variable ranges—e.g. enforcing the mean of the control-point Gaussianto not stray too far away from the image frame.
In the variant without execution guidance, we donot perform intermediate rendering, removing the direct dependence of the generative model and therecognition model on the canvas-so-far x <t and the residual ∆ x t .
Both the explicit spline decoder and the execution guidance prove to be important.
Without theexplicit spline decoder (DooD- δ R), the reconstruction quality suffers—the strokes are blurry (e.g. firstthree QuickDraw images), strokes go missing (e.g. the last EMNIST image), or the reconstructionsare inaccurate (e.g. the last Omniglot character is interpreted as a “9” due to overfitting).
However,even without the explicit spline decoder, the model learns to be parsimonious, using fewer strokes toreconstruct simpler images (Fig.
3b-d).
On the other hand, without execution guidance (DooD-EG),the model is unable to be selective with the number of strokes, always using the maximum allowednumber.
And while the reconstructions are better than AIR and DooD- δ R, it still shows instances ofmissing strokes (e.g. some Omniglot characters).
Note that although we use the canvas-so-far in amanner that disallows gradients ( stop_gradient ), just providing it as a conditioning variable forthe different components (layout, stroke, RNN hidden states) has a tangible effect.
Quantifying zero-shot transfer.
We look at how DooD and AIR trained on each of the five datasetstransfers to each other dataset to further understand how our model generalises.
Models are trainedon each “source” dataset and tested on each “target” dataset, resulting in a 5 × 5 table for each model(Fig.
Each cell shows the log marginal likelihood of the target dataset using the model trained onthe source dataset, estimated using the importance weighted autoencoder (IWAE) objective [3] with200 samples (mean and standard deviation over five runs).
We also show reconstructions obtained byrunning the model trained on the source dataset on a few examples from the target dataset.
Our model generalises significantly better than AIR across datasets (off diagonal cells), while alsoperforming better within dataset (diagonal cells).
For both models, the values on the diagonal are thehighest in any given column, suggesting that not training on directly on the target dataset results ina worse performance, as expected.
For both models, the row values for MNIST and Omniglot arelower than in other rows, indicating that transfer learning performance is the worst when the sourcedataset is MNIST or Omniglot—potentially due to a larger distribution shift since MNIST has lowdiversity and Omniglot has little to no variation in stroke thickness, in contrast to the other datasets.
However, we note that our reconstructions are high quality despite transferring out of distribution,unlike reconstructions from AIR which are qualitatively worse.
For example, when transferring fromsimple datasets (MNIST), AIR makes incomplete, incorrect and blurry reconstructions, as we haveseen before, while AIR trained on complex datasets like Omniglot results in blurry reconstructionsfor both in-distribution and out-of-distribution datasets.
Furthermore, AIR fails when trained on
Omniglot using a Laplace likelihood (used as standard across all other model-dataset combination).
We thus employ a Gaussian likelihood just for Omniglot-trained AIR, and highlight it as an outlier.
Understanding learned representations.
To better understand DooD’s generalization ability, weinvestigate its learnt representations by clustering the inferred strokes using k -means clustering( k = 8 ), and study the clusters both qualitatively and quantitatively.
For AIR, we cluster thecorresponding part-representation latents.
We then visualise things, using a t-SNE plot (Fig. 5) of theclusters, with exemplar strokes overlaid.
We find that DooD has better-clustered representations, withclusters denoting largely distinct types of strokes—e.g., clusters for a “/”, “c”, and its horizontallyflipped version.
In contrast, the clusters from AIR are less sensible with some clusters even capturingfull characters (“0”), comprising multiple strokes.
There are also clusters which contain visuallydifferent strokes, and many visually similar strokes are assigned to different clusters.
Quantitatively, following Aksan et al.
[1], we found DooD’s better cluster consistency is reflected in a higher
Silhouette Coefficient [30] than AIR (0.21 for DooD, 0.11 for AIR).
3.2 Across-Task Generalization
Here, we focus on a subset of the Omniglot challenge tasks [24], to evaluate our model’s utility ona range of auxiliary tasks that it was not trained to do.
Despite much progress in deep generativemodelling, relevant models are still not fully task general and often result in unrealistic (e.g. blurry)samples [10, 25].
DooD combines handling raw perceptual inputs with the compositional structureof strokes, which we evaluate on three out of the five Omniglot challenge tasks: unconditionalgeneration, conditional generation, and one-shot classification.
We compare against AIR and astate-of-the-art neuro-symbolic model (GNS [10]), where relevant.
Note that GNS requires strokeand character-class supervision and practically, at least for now, only applies to Omniglot.
Unconditional generation.
DooD generates realistic unconditional samples of all datasets (Fig.
6),indicating that the model has learned the high-level characteristics of each dataset.
The strokesare sharp, and the stroke structure composes into realistic images from each dataset.
For example,there are clear digits in the MNIST samples, there are recognizable objects (cars, bicycles, glasses,and smileys) in the QuickDraw samples, and the samples for EMNIST, KMNIST and Omniglotcan be easily recognised as possible instances coming from those datasets.
It generates samples ofcomparable fidelity to GNS without requiring any supervision, and as evaluated using the Fréchetinception distance (FID) [16] (smaller is better), outperforms GNS (0.051 versus 0.133).
The key to being able to generate realistic prior samples is the learnable sequential prior and thesymbolic latent representation.
AIR doesn’t have a sequential prior, so although it is possible to getgood reconstructions, it is impossible for it to generate realistic unconditional samples.
Character-conditioned generation.
In order to generate new exemplars of the same Omniglotcharacter, we follow Feinman and Lake [10], Lake et al.
[24] and extend our model to a hierarchicalgenerative model of an abstract character “type” or “template” that generates a concrete instanceof a character “token”, which is rendered out to an image.
We consider the previously used latentvariables as the type latent variable and introduce a token model which conditions on the type latentvariable. The token model introduces (i) a drawing noise represented by adding a Gaussian noisewith fixed standard deviations to spline control points and (ii) and an affine transformation on thenoise perturbed points, whose parameters are also sampled from a Gaussian distribution (describedin Appendix B.4). To sample a new exemplar of a character, we first sample the type variable from
[Figure] [Figure]
[Figure]
our recognition model, and produce different exemplars by sampling and rendering different tokenvariables given this type variable.
Distinctly from [10, 24], the parameters of the token model are notlearned through supervision but simply set to a sensible value by examining the noise level in theoutput – any learned statistics can be straightforwardly plugged in.
DooD generates realistic new exemplars of complex QuickDraw drawings and Omniglot characters(Fig.
thanks to the accurate inference and the ability to add noise to explicitly parametrisedstrokes.
While we can add an equivalent token model for AIR by (i) adding a Gaussian noise to theuninterpretable feature vector representing each part and (ii) applying a Gaussian affine transformationto the rendered image, the new exemplars are not as realistic both because of worse inference andthe hard-to-control variations of the part vectors.
GNS generates realistic conditional samples, butnotably still makes unnatural samples in multiple instances (e.g., in column 1, 3 the detachments ofstrokes) despite having a hierarchical model learned with multi-levels of supervision.
Partial completion.
As with inferring an entire figure in the previous case, we can interpretconditional generation in a slightly different way as well—where the condition is an initialization ofa number, character, or figure, and the model tries to extend/complete it as best it can (Fig.
To dothis, we first employ the recognition model over the partial figure to compute the hidden states of theshared recurrent networks.
Next, starting with these computed states, we set the canvas-so-far x <tto be the partial figure itself and then unroll the generative model from that point onwards.
Ascan be seen in the figure, DooD can generate a varied range of completions for each initial stroke,demonstrating its versatility and the utility of its learnt representations.
[Table]
One-shot classification.
Finally, we can apply the type-tokenhierarchical generative model used for generating new exemplars to perform within-alphabet, 20-way one-shot classification.
The key quantity needed for performing this task is the posterior predictive score of the query image x
( T )
given a set of support images { x ( c ) } c , ˆ c = arg max c p ( x ( T ) | x ( c ) ) , which requiresmarginalizing over the token variables corresponding to x ( T ) andx ( c ) , and the type variable of x ( c ) .
Following [10, 24], we approximate this score by sampling from the recognition modelgiven x ( c ) , and perform gradient-based optimization to marginalize out the token variable of x ( T ) (details in Appendix D.2.3).
We find that DooD outperforms the neural baseline (AIR), whileattaining a competitive accuracy in comparison to other baselines (Table 1) without requiring additional forms of support such as data augmentation, supervision forstrokes, or more complex ways of computing the accuracy such as two-way scoring (predictingˆ c = arg max c p ( x ( c ) | x ( T ) )
p ( x ( c ) ) p ( x ( T )
| x ( c ) ) ).
Related Work
Our work takes inspiration from Lake et al.
[24]’s symbolic generative modelling approach whichhypothesises that the human ability to generalise comes from our causal and compositional understanding that characters are generated by composing substrokes into strokes, strokes into charactersand rendering characters to images.
As a result, Lake et al.
[24] demonstrate human-like generalization on a wide range of tasks.
However, it is trained using stroke sequences, and inference isperformed using expensive MCMC sampling.
We combine features of neuro-symbolic generative models and deep generative models to be ableto generalise well across tasks while using amortised inference and being unsupervised.
Fromneuro-symbolic models
, we share key features of Feinman and Lake [10]’s model like (i) using thecanvas-so-far in the generative model and adopt a similar feature in the recognition model like Elliset al.
[8], (ii) parametrizing parts as splines and using a differentiable spline renderer, (iii) extendingthe model to have a type-token hierarchy for generating new exemplars and performing one-shotclassification.
Our model can be seen as an extension of [10] that learns directly from images anduses a recognition model for amortised inference.
Like Hewitt et al.
[17], we learn how to infer astroke sequence directly from images using a differentiable renderer but infer strokes directly insteadof learning a stroke bank and use a more flexible parametrization of strokes based on a differentiablespline renderer, leading to a more accurate model.
Similar to deep generative modelling approaches like [9, 14, 23, 29], we use attention to focus on partsof the canvas we want to generate to or recognise, and more generally exploit the compositionality ofobjects from which allows neural networks to learn simpler and hence more generalizable mappings.
To be able to train our model from unsupervised images, we adopt the NVIL control variate [28]used by Eslami et al.
[9] to be able to train a model with a discrete stop-drawing latent variable.
Thisfamily of models, along with deep meta-learning approaches [11, 32, 34], is easier to learn due tothe lack of symbolic variables and results in a fast amortised recognition model.
However, the lackof strong inductive biases leads to poor and unreliable generalization [25].
We also share idea withother works combining deep learning and explicit stroke modelling [1, 12, 15, 27], but we focus onlearning a principled generative model which allows tackling tasks like one-shot classification andgenerating new exemplars, in addition to conditional and unconditional sampling.
Conclusion
We demonstrated that DooD generalises across datasets and across tasks thanks to an explicit symbolicparametrization of strokes and execution guidance.
This allows us to train on one dataset such as
MNIST and generalise to a more complex, out-of-distribution dataset such as Omniglot.
Given acompositional representation and an associated learned sequential prior, DooD can be applied toadditional tasks in the Omniglot challenge like generating new exemplars and one-shot classificationby extending it to have a type-token hierarchy.
Our model produces realistic new exemplars withoutblur and artefacts unlike deep generative models.
More broadly, DooD is an example of a system that successfully combines symbolic generativemodels to achieve generalization and deep learning models to handle raw perceptual data and performfast amortised inference while being learned from unsupervised data.
We believe these principles canbe useful for building fast, reliable and robust learning systems going beyond stroke-based data.