Diversity vs. Recognizability: Human-like generalization in one-shot generative models
Anonymous Author(s)
AffiliationAddress email
Abstract
Robust generalization to new concepts has long remained a distinctive feature ofhuman intelligence.
However, recent progress in deep generative models has nowled to neural architectures capable of synthesizing novel instances of unknownvisual concepts from a single training example.
Yet, a more precise comparisonbetween these models and humans is not possible because existing performancemetrics for generative models (i.e., FID, IS, likelihood) are not appropriate forthe one-shot generation scenario.
Here, we propose a new framework to evaluateone-shot generative models along two axes: sample recognizability vs. diversity(i.e.
, intra-class variability).
Using this framework, we perform a systematic evaluation of representative one-shot generative models on the Omniglot handwrittendataset.
We first show that GAN-like and VAE-like models fall on opposite endsof the diversity-recognizability space.
Extensive analyses of the effect of keymodel parameters further revealed that spatial attention and context integrationhave a linear contribution to the diversity-recognizability trade-off.
In contrast,disentanglement transports the model along a parabolic curve that could be used tomaximize recognizability.
Using the diversity-recognizability framework, we wereable to identify models and parameters that closely approximate human data.
Introduction
Our ability to learn and generalize from a limited number of samples is a hallmark of human cognition.
In language, scientists have long highlighted how little training data children need in comparison tothe richness and complexity of the language they learn so efficiently to master [12, 40].
Similarly,children and adults alike are able to learn novel object categories from as little as a single trainingexample [17, 8].
From a computational point of view, such feats are remarkable because they suggestthat learners must be relying on inductive biases to overcome such challenges [33] – from an abilityto detect suspicious coincidences or ’non-accidental’ features [43, 51] to exploiting the principle ofcompositionality [32, 33].
While a common criticism of modern AI approaches is their reliance on large training datasets,progress in one-shot categorization has been significant.
One-shot categorization involves predictingan image category based on a unique training sample per class.
Multiple algorithms have beenproposed including meta-learning algorithms [18, 46, 14, 37] or metric-learning algorithms [47, 31,48] that are now starting to approach human accuracy.
Perhaps a less studied problem is the one-shotgeneration problem – aimed at creating new variations of a prototypical shape seen only once.
Sincethe seminal work of Lake et al.
[32] who introduced the Bayesian Program Learning algorithm, onlya handful of promising one-shot generative algorithms have been proposed [42, 15, 1] (see section 2.2for a more exhaustive description of prior work).
Why have so few
algorithms for one-shot image generation vs. image categorization
been proposed?
We argue that one of the main reasons for this lack of progress is the absence of an adequate evaluationmetric.
As of today, one-shot generative models are evaluated using methods initially developed formodels producing samples that belong to the training categories and trained on large datasets.
Thosemetrics include the likelihood, the FID (Frechet Inception Distance), or the IS (Inception Score).
In the one-shot image generation scenario in which training images are scarce and the generatedsamples represent new visual concepts, the likelihood, the FID, and the IS are biased [3, 13, 38](see 2.1 for more details).
These limitations urge us to look for new metrics tailored for one-shotimage generation.
Recent psychophysics work [51] has characterizedhumans’ ability for one-shot generation along twomain axes: samples diversity (i.e., intra-class vari
ability) and samples recognizability (i.e., how easy orhard they are to classify).
According to this frame
work, ideal generalization corresponds to a combination of high recognizability and high diversity.
Asillustrated in Fig.
1, an ideal model should be ableto generate samples that span the entire space withinthe decision boundary of a classifier (Box 1 ).
In comparison, the model of Box 2 has learned to makeidentical copies of the prototype (i.e., low diversitybut high accuracy).
Such a model has failed to generalize the visual concept exemplified by the prototype.
Similarly, if the model’s samples are so diverse thatthey cannot be recognized accurately as shown in the
Box 3 of Fig.
1, then the generated samples won’tlook like the prototype.
Here, we borrow from this work and adapt it to createthe first framework to evaluate and compare humansand one-shot generative models.
Using this framework, we systematically evaluate an array of representative one-shot generative models on the Omniglotdataset [32].
We show that GAN-like and VAE-likeone-shot generative models fall on opposite ends ofthe diversity-recognizability space: GAN-like modelsfall on the high recognizability — low diversity endof the space while VAE-like models fall on the lowrecognizability — high diversity end of the space.
We further study some key model parameters thatmodulate spatial attention, context integration, and disentanglement.
Our results suggest that spatialattention and context have an (almost) linear effect on the diversity vs. recognizability trade-off.
Incontrast, varying the disentanglement moves the models on a parabolic curve that could be used tomaximize the recognizability.
Last but not least, we have leveraged the diversity vs. recognizabilityspace to identify models and parameters that best approximate the human data on the Omniglothandwritten dataset.
Related work
2.1 Metrics to evaluate generative models and their limitations for one-shot generation tasks
Different types of generative models are typically evaluated using different metrics.
On the onehand, likelihood-based algorithms (e.g., VAE [29], PixelCNN [39], GLOW [30], etc.) are evaluatedusing their own objective function applied on a testing set.
Likelihood provides a direct estimate ofthe KL divergence between the data points and the model’s samples.
On the other hand, implicitgenerative models such as Generative Adversarial Networks (GANs) [20] for which the loss functioncannot be used, are typically evaluated using other scores such as the Inception Score (IS) [45] or
Frechet Inception Distance (FID) [22].
IS and FID are heuristic measures used to aggregate both thesample quality and diversity in one single score.
The IS scores a sample quality according to the confidence with which an Inception v3 Net [49] assigns the correct class label to it.
The FID score isthe Wasserstein-2 distance between two Gaussian distributions: one fitted on the features of the datadistribution and the other on the features of the model distribution (the features are also extractedfrom an Inception v3 Network).
All these metrics are problematic for the one-shot generation scenario for 2 main reasons that areintrinsically related to the task: the low number of samples per class, and the dissimilarity betweentraining and testing visual concepts.
IS and FID rely on statistical distances (either KL divergencefor IS or Wasserstein-2 distance for FID) that require a high number of data points N to produce anunbiased estimator of the distance.
Even when used in the traditional settings (i.e., N = 50000 ), ithas been demonstrated that both scores are biased [13].
This is to be compared with the N = 20samples typically available in popular few-shot learning datasets such as Omniglot [32].
Anotherproblem caused by the limited number of samples per class in the training set is the overfitting of the
Inception Net used to extract the features to compute the IS and FID [7].
The second limitation of these metrics appears because the training and the testing samples are toodissimilar.
Likelihood scores are known to yield higher scores for out-of-domain data compared toin-domain data [38].
Therefore, the evaluation of the novel visual concepts generated by one-shotgenerative models will be biased toward higher scores.
In addition, both FID and IS rely on distancebetween features extracted by an Inception Net which comes with no guarantee that it will producemeaningful features for novel categories.
For example, class misalignment has been reported when the
Inception Net was trained on ImageNet and tested on CIFAR10 [3].
Because of all the aforementionedlimitations, it is pretty clear that new procedures are needed to evaluate the performance of few-shotgenerative algorithms.
2.2 One-shot generative models
One can distinguish between two broad classes of one-shot generative models: structured modelsand statistical models [16].
Structured models have strong inductive biases and rigid parametricassumptions based on a priori knowledge such as for example a given hierarchy of features, a knowngrammar or program [44].
A prominent example of a structured model includes the very first algorithmfor one-shot image generation, the Bayesian Program Learning (BPL) model [32].
Statistical modelslearn visual concepts by learning statistical regularities between observed patterns [42, 15, 19].
Here, we focus on representative architectures of one-shot generative statistical models, which wesummarize below.
• VAE with Spatial Transformer Network ( VAE-STN ) [42].
The VAE-STN is a sequential and conditional Variational Auto-Encoder (VAE) constructing images iteratively.
The VAE-STNalgorithm uses a recurrent neural network (i
e., an LSTM) to encode the sequence of localpatches extracted by an attentional module.
A key ingredient of the VAE-STN is an attention module composed of a Spatial Transformer Network (STN) [26] to learn to shift attention todifferent locations of the input image.
The STN is a trainable module to learn all possibleaffine transformations (i e., translation, scaling, rotation, shearing) of an input image (see S8 for samples and details of the VAE-STN ).
• Neural statistician ( VAE-NS ) [15]: The Neural Statistician is an extension of the conditionalVAE model including contextual information.
Therefore, in addition to learning an approximateinference network over latent variables for every image in the set (as done in a VAE), the
approximate inference is also implemented over another latent variable, called the contextvariable, that is specific to the considered visual concept.
The context inference network is fed
with a small set of images representing variations of a given visual concept.
The VAE-NS has
been extended to include attention and hierarchical factorization of the generative process [19]
(see S9 for samples and details of the VAE-NS ).
• Data-Augmentation GAN (DA-GAN) [1]: Data -Augmentation GAN is a generative adversarialnetwork conditioned on a prototype image.
The DA-GAN generator is fed with a concatenationof a vector drawn from a normal distribution and a compressed representation of the prototype.
The discriminator is trained to differentiate images produced by the generator from imagesof the dataset, while the generator has to fool the discriminator.
We have trained 2 different
DA-GAN, one is based on the U-Net architecture ( DA-GAN-UN ) and the other one on the
ResNet architecture ( DA-GAN-RN ) (see S10 for samples and details of the DA-GAN-UN and S11 for samples and details of the DA-GAN-RN ).
All these models are generative models conditioned by an image prototype extracted from the trainingor the test set.
The way we have selected the prototypes is detailed in Eq.
To the best of ourknowledge, these models offer a representative set of one-shot generative models.
We have reproducedall these models (sometimes with our own implementation when it was not available online).
Our codecould be found at https://anonymous.4open.science/r/diversity_recognizability/ .
The diversity vs. accuracy framework
Let { x ji } be a dataset composed of K concepts (i.e., classes) with N samples each ( i ∈ [[1 , N ]] andj ∈
[[1 , K ]] ).
The framework we propose aims at evaluating the performance of a generative modelp θ , parameterized by θ , that produces new images v ji based on a single sample (or prototype) of aconcept given to the generator ˜ x
j (i.e., v ji ∼ p θ ( ·| ˜ x j ) ).
For each concept j , we define a prototype asthe sample closest to the center of mass for the concept j :
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
i
[Equation] [Equation]
In Eq.
1, f denotes a function that projects the input image from the pixel space to a feature space.
We will detail the feature extractor f shortly.
Note that this definition of a prototype is not unique(one could also select the prototype randomly within individual classes j ), nevertheless this selectionmechanism is a guarantee that the selected sample will be representative of the concept.
Dataset.
In this article, we use the Omniglot dataset [32] with a weak generalization split [42].
Omniglot is composed of binary images representing 1 , 623 classes of handwritten letters and symbols(extracted from 50 different alphabets) with just 20 samples per class.
We have downsampled theoriginal dataset to be 50 × 50 pixels.
The weak generalization split consists of a training set composedof all available symbols minus 3 symbols per alphabet which are left aside for the test set.
It is said tobe weak because all the alphabets were shown during the training process (albeit not all symbols inthose alphabets).
As the Omniglot dataset is hand-written by humans, we consider that these samplesreflect a human generative process and we refer to this later as the human model.
Diversity.
In the proposed framework, diversity refers to the intra-class variability of the samplesproduced by a generative model p θ ( ·| ˜ x j ) .
For a given prototype ˜ x j , we compute the diversity as thestandard deviation of the generated samples in the feature space f :
[Equation]
We use the Bessel-corrected standard deviation to keep a good estimate of the data dispersion despitethe relatively small number of samples (e.g., N = 20 for the Omniglot dataset used here).
To verifythat this diversity measure is robust to the specific choice of the feature extractor f , we explored twodifferent settings: features learned with class supervision by a Prototypical Net [47] and featureslearned with self-supervision by a SimCLR network [10].
In both cases, we extracted the featuresfrom the first fully-connected layer following the last convolutional layer.
The Prototypical Netwas optimized so that images that belong to the same category share similar latent representationsas measured by the ℓ 2 -norm.
Similarly, SimCLR leverages a contrastive loss to define a latentrepresentation such that a sample is more similar to its augmented version than to other imagesamples.
In SimCLR, this similarity is computed with cosine similarity.
These two approachesrepresent two ends of a continuum of methods to learn suitable representational spaces without theneed to explicitly learn to classify images and are thus more suitable for few-shot learning tasks [35].
For the sake of comparison, we have used the exact same network architecture for both featureextractors (see sections S1 and S2 for more details on Prototypical Net and SimCLR, respectively).
We computed the samples diversity for all 150 categories of the Omniglot test set (i.e., v ji = x ji in thisexperiment) using both the supervised and unsupervised settings.
We found a high linear correlation( ρ = 0 .
86 , p-value < 10 − 5 ) and a high rank-order Spearman correlation ( ρ = 0 .
85 , p-value <
10 − 5 )between the two settings (see section S3.1).
Hence, the two feature extraction methods producecomparable diversity measures and henceforth, we will report results using the unsupervised setting.
As an additional control, we have also verified that the SimCLR metric is robust to changes to theaugmentation method used (see section S3.3) and to the specific choice of the dispersion metric (seesection S3.2 for more details on this comparison).
We have compared the feature space of the Prototypical Net and SimCLR using a t-SNE analysis (see section S3.4).
We observed a strong clusteringof samples belonging to the same category for both networks.
It suggests that the augmentationmethods used by the SimCLR contrastive loss are sufficient to disentangle the class information.
Fig.
2 shows the 10 concepts from the Omniglot test set with the lowest and highest samplesdiversity, respectively (for more diversity-ranked concepts with unsupervised or supervised setting,see sections S4 and S5, respectively).
One can see that the proposed diversity metric is qualitativelysimilar to human judgment.
Concepts with low diversity are composed of very few relatively basicstrokes (e.g., lines, dots, etc) with little room for any kind of “creativity” in the generation processwhile more diverse concepts are composed of more numerous and more complex stroke combinationswith many more opportunities for creativity.
[Figure]
Recognizability.
We evaluate the recognizability of the samples produced by the one-shot generativemodels by leveraging one-shot classification models.
In order to make sure our classification accuracymeasure is robust to the choice of the model, we test different models which belong to the twomain approaches used in machine learning for one-shot classification: metric learning and metalearning [35].
We selected the Prototypical Net [47] as a representative metric-learning approach andthe Model-Agnostic Meta-Learning (MAML) [18] model as a representative meta-learning approach.
Both models were trained and tested in a 1-shot 20-ways setting (see section S6 for more detailson the MAML architecture and training details).
We report a high Pearson (negative) correlationbetween the logits produced by Prototypical Net and MAML ( ρ = − 0 .
60 , p-value < 10 − 5 ) as wellas a strong Spearman rank-order correlation between the classification accuracy of both networks( ρ = 0 .
62 , p-value < 10 − 5 ).
See section S7 for more details about this control experiment.
Hence,our recognizability metric is robust to the choice of the one-shot classification model (even whenthose models are leveraging different approaches) and henceforth, we will report results using the Prototypical Net model.
Results4.1 GAN-like vs. VAE-like models
For all algorithms listed in section 2.2 we have explored different hyper-parameters (see section 4.2for more details), leading to various models represented in the diversity vs. recognizability plotin Fig. 3a.
In this figure, we have reduced each model to a single point by averaging the diversityand recognizability over all classes of the Omniglot testing set.
The black star corresponds tothe human model, and colored data points are computed based on the samples generated by the VAE-NS , VAE-STN , DA-GAN-UN and the DA-GAN-RN . The base architectures for all algorithms (highlighted with bigger points in Fig. 3a) have a comparable number of parameters ( ≈ 6-7 M, see S8,
S9 and S10 for more details on the base architectures).
We observe that the GAN-like models (i.e., DA-GAN-UN and DA-GAN-RN ) tend to be located at theupper left side of the graph while VAE-like models (i.e., VAE-NS and VAE-STN ) spread on the rightside of the graph.
Therefore, the GAN-like models produce very recognizable samples that are highlysimilar to each other (high recognizability and low diversity).
In contrast, VAE-like models generatemore diverse but less recognizable samples.
The samples in Fig.
3b illustrate this observation.
Thedifference between GAN and VAE-like samples could be explained by their loss functions [36].
The
GANs’ adversarial loss tends to drop some of the modes of the training distribution.
In general, thedistribution learned by GANs put excessive mass on the more likely modes but discards secondarymodes [2].
This phenomenon leads to sharp and recognizable generations at the cost of reducedsamples diversity.
On the other hand, VAEs (and likelihood-based models in general) are sufferingfrom over-generalization: they cover all the modes of the training distribution and put mass inspurious regions [4].
We refer the reader to Fig.
4 of Lucas et al.
[36] for an illustration of modedropping in GANs and over-generalization in VAEs.
Our diversity vs. recognizability plot in Fig.
3ashows that this phenomenon is holding even when the testing distribution is different from the trainingdistribution as in the case of the one-shot generation scenario.
[Figure]
a comparable number of parameters ( ≈ 6 − 7 M).
The human data-point is computed based on the testing samples of the Omniglot dataset.
Samples produced by the different models in their base architectures (corresponding to the bigger circles in Fig. 3a).
Prototypes for individual concepts are shown a next
4.2 VAE-NS vs. VAE-STN
In this section we compare some key hyper-parameters of the VAE-NS and VAE-STN .
The coreidea of the VAE-NS is to integrate context information to the sample generation process.
Duringtraining, the context is composed of several samples that all represent variations of the same visualconcept.
During the testing phase, the VAE-NS infers the context statistics using a single image(i.e.
, the prototype).
We evaluate the effect of the context on the position of the VAE-NS models onthe diversity-recognizability space by varying the number of samples used to compute the contextstatistics during the training phase (from 2 to 20 samples).
Importantly, varying the number ofcontext samples do not change the number of parameters of the network.
For all tested runs, weobserve a monotonic decrease of the samples diversity (see Fig. S11a) and a monotonic increase ofthe samples recognizability (see Fig.
S11b) when the number of context samples is increased.
Inthe diversity-recognizability space, the resulting curve is monotonically transporting models from the lower-right side to the upper-left side of the plot (see Fig.
4a, dark blue curve).
The effect of thenumber of context samples is large: the diversity is almost divided by 2 (from 2 .
4 to 1 . 2 )
and theclassification accuracy is increased by 80% (from 53% to 96% ).
[Figure]
In contrast to the VAE-NS , the VAE-STN uses spatial attention to sequentially attend to sub-partsof the image to decompose it into simpler elements.
These sub-parts are then easier to encode andsynthesize.
In the VAE-STN , one can vary the number of attentional steps (i.e., the number ofattended locations) to modulate spatial attention.
Importantly, varying the number of attentional stepsdoes not change the number of parameters.
We have varied the number of attentional steps from 20 to90 .
The relationship between the number of attentional steps, the samples diversity, and the samplesrecognizability is non-monotonic.
We have used a parametric curve fitting method (i.e., the leastcurve fitting method from [21]) to parameterize the curve while maintaining the order of the datapoint (see S13 for more details on the fitting procedure).
We report a convex parabolic relationshipbetween the number of attentional steps and the samples diversity (see Fig. S12a).
This curve isminimal at 60 steps.
We observe a concave parabolic relationship between the number of attentionalsteps and the recognizability of samples.
This curve is maximal at 60 attentional steps (see Fig. S12b).
In Fig.
4b we have plotted the parametric fit illustrating the position of the VAE-STN models in thediversity-recognizability space when one increases the number of attentional steps (the light bluecurve).
This curve follows a quasi-linear trend with a sharp turn-around (at 60 attentional steps).
Theeffect of the number of attentional steps on the diversity-recognizability is limited compared to theeffect of the number of context samples.
Both VAE-NS and VAE-STN are trained to maximize the Evidence Lower Bound (ELBO), it isthen possible to tune the weight of the prior in the loss function.
One can operate such a modulationby changing the β coefficient in the ELBO loss function [24].
We refer the reader to S15 for moremathematical details about the ELBO.
A high β value enforces the latent variable to be closer toa normal distribution and increases the information bottleneck in the latent space.
Increasing β isknown to force the disentanglement of the generative factors [9].
We observe a monotonic decreasingrelationship between the value of β and the samples diversity for both the VAE-STN and the VAENS (see Fig.
S13a and Fig.
S14a, respectively).
We report a concave parabolic relationship betweenβ and the samples recognizability.
We use the least curve fitting method to find the optimal paraboliccurves [21].
This curve is maximal at β = 2 . 5 for the VAE-STN and at β = 3 for the VAE-NS (see
Fig.
S13b and Fig.
S14b, respectively).
The overall effect of β on the position of the VAE-likemodels on the diversity-recognizability space is relatively similar for both the VAE-STN and the
VAE-NS and follows a clear parabolic trend.
These curves demonstrate that one could modulate thevalue of β to maximize the recognizability.
We have also varied the size of the latent space of the DA-GAN-UN , DA-GAN-RN , VAE-NS , and
VAE-STN .
We did not find any common trend between the size of the latent variable, the diversity,and the recognizability (see S16 for more details).
We observe that the DA-GAN-UN tends to produceslightly more recognizable but less diverse samples than the DA-GAN-RN while both architectureshave the same number of parameters.
It suggests that the extra skip connections included in the U-Netarchitecture, in between the encoder and the decoder of the DA-GAN-UN , allow to trade diversityfor recognizability.
4.3 Comparison with humans
[Figure]
We now compare the tested models with the human data in the diversity-recognizability space.
Toperform such a comparison, we first normalize all the model’s diversity and recognizability (includinghumans) using the z-score such that both axes are scaled and centered similarly.
Then, for all models,we compute the ℓ 2 -distance between models and humans in the diversity-recognizability space.
Weremind that the human data point is computed using the samples of the Omniglot test set.
Distancesto humans as well as their distributions are reported for all models in Fig.
5a.
The median of VAENS models is closer to humans, followed by DA-GAN-UN , DA-GAN-RN and VAE-STN (mediansare indicated in Fig.
5a with horizontal bars).
The VAE-NS model showing the smallest distance isalmost at the human level (see dark blue square Fig.
5a).
It has a context size of 20 samples (thehighest possible context size), and a β = 2 . 5 .
The VAE-STN model that best approximates humanhas a β = 2 .
25 and 60 attentional steps (see light blue square in Fig.
5a).
So far, we have reduced all models to single points by averaging the diversity and recognizabilityvalues over all classes.
We now study distances to humans for individual classes and for the VAE- NS and the VAE-STN models showing the shortest distance to humans (indicated by blue squaresin Fig. 5a).
In Fig.
5b, we report distances to human for these 2 models and for 16 visual concepts.
The visual concepts 1 to 8 and 9 to 16 are selected so that they minimize the distance to humans withthe VAE-STN and the VAE-NS , respectively.
We observe that these visual concepts are differentfor the VAE-NS and the VAE-STN model.
Therefore, both models are well approximating humandata for some visual concepts but not for others.
Interestingly, we qualitatively observe that thevisual concepts 1 to 8 look simpler (i.e., made with fewer strokes) than the visual concepts 9 to16 .
It suggests that the spatial attention mechanism used by VAE-STN provides a better humanapproximation for simple visual concepts, while the context integration method leveraged by the
VAE-NS is more relevant to mimic human data on more complex visual concepts.
Discussion
In this article, we have described a novel framework for comparing computational models withhuman participants on the one-shot generation task.
The framework measures the diversity and therecognizability of the produced samples using metrics compatible with the one-shot scenario.
Tothe best of our knowledge, this is the first and only framework specifically tailored to evaluate andcompare one-shot image generation models.
Among all tested algorithms, the VAE-NS is the best human approximator on Omniglot (see Fig.
5a).
It suggests that the context integration mechanism of the VAE-NS is an important component toreach human-like generalization.
Interestingly, motor learning experiments have demonstrated thathuman generalization performances are also strongly related to contextual information [50].
Inaddition, we demonstrate that one can tune β so that the model becomes closer to human data(see Fig. 4b).
This is consistent with a prior computational neuroscience study that has shown thatdisentangled VAEs (with β > 1 ) provide a good model of face-tuned neurons in the inferotemporalcortex [23].
Our comparison between the VAE-NS and the VAE-STN suggests that a model whichuses a spatial attention mechanism better fits human data for simple visual concepts.
In contrast, thecontext integration mechanism of the VAE-NS appears to be a better human approximator for morecomplex visual concepts.
One could thus try to combine both mechanisms towards improving thesimilarity with human data independent of the complexity of the visual concept.
We have also foundthat GAN-like models ( DA-GAN-RN and DA-GAN-UN ) better account for human recognizabilitywhile VAE-like models ( VAE-NS and VAE-STN ) better account for human diversity.
An interestingapproach would be to leverage a hybrid architecture (such as the VAE-GAN [34]) to try to bettermatch human data.
Other candidate ingredients include the ability to harness compositionality [32] or the recurrentprocesses thought to be crucial for human generalization [53].
Compositionality could be introducedin one-shot generative algorithms by quantizing the latent space (as in the VQVAE [52]).
As a result,each coordinate of the latent variable represents an address in a codebook, and the role of the prioris then to combine simpler concepts to generate more complex samples.
One promising way toinclude recurrent processing into generative models is through the predictive coding framework [41].
Predictive Coding suggests that each processing step is part of an inference scheme that minimizesthe prediction error [6].
Previous work has demonstrated that such networks are more robust andexhibit improved generalization abilities [5, 11].
All these ingredients could be tested and comparedagainst human abilities using the diversity/recognizability framework we have proposed in this paper.
In the current version of the Omniglot dataset, the intra-class variability does not reflect the humanlevel of creativity.
It is mainly due to the experimental protocol in which one asks human participantsto copy a given visual concept.
The Omniglot dataset could be enriched with more diverse samples, byexplicitly asking human participants to be as creative as possible.
Other drawing databases with morecomplex symbols such as Quick Draw!
[27] could also be considered to strengthen the comparisonwith humans.
By decomposing the performance of the one-shot generation task along the recognizability vs.diversity axes we wanted to shed light on the relationship between generalization and creativity(quantified by the samples diversity in our framework).
We hope one can make use of our frameworkto validate key hypotheses about human generalization abilities so that we can better understandthe brain.
We argue that the best way to reach human-like generalization abilities is to unleash thealgorithms’ creativity.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Described in the conclusion.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We have briefly discussed societal impact in S17.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
No theoretical
results
(b) Did you include complete proofs of all theoretical results?
[N/A]
No theoretical results
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] All the codes
are available on github ( link given in section 2.2)
(b) Did you specify all the training details (e.g., data splits, hyper-parameters,
how they
were chosen)?
[Yes] All the training details, as well as the architectural details are given in the supplementary materials.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes] Error bars have been reported on 3 different runs
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
In the Supplementary Information at section S
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes] All the github linksare disclosed in the Supplementary information, and the
contributing authors are citedin the main article.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
The github link of our code is included in the article
(d) Did you discuss whether and how consent was obtained
from
people whose data you’re using/ curating?
[N/A] All assets we are using
are
publicly
available and do not require
any consent
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A
No personal information in Omniglot
If you
used crowdsourcing or conducted research with human subjects
(a) Did you include the full text of instructions given to participants and screenshots
, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board
(IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Supplementary Information
S1 More details on Prototypical Net Architecture
Table S1 describes the architecture of the Propotypical Net [47] we are using in this article.
We usethe Pytorch convention to describe the layers of the network.
[Table]
The overall number of parameters of the Prototypical Net we are using is around 292 K parameters.
The loss of the Prototypical Net is applied on the output of the last fully connected layers (of size128 ).
For the computation of the samples diversity, we extract the features on the first fully-connectedlayer after the last convolutional layer (i.e., of size 256 ).
Training details
The Prototypical Net is trained in a 1-shot 60-ways setting and tested on a 1 -shot 20 -ways setting.
Thesize of the query set is always 1 for both training and testing phase.
The model is trained during 80epochs, with a batch size of 128 .
For training, we are using an Adam optimizer [28] with a learningrate of 1 × 10 − 3 (all other parameters of the Adam optimizer are the default ones).
We are schedulingthe learning rate such that it is divided by 2 every 20 epochs.
At the end of the training, the training accuracy (evaluated on 1000 episodes) has reached 100% andthe testing accuracy reaches a plateau at 96 .
S2 More details on SimCLRS2.1 Architecture and Data Augmentation
The architecture we are using for SimCLR [10] is the exact same than the one used for Prototypical
Net (see Table S1).
In SimCLR, we also extract the features on the first fully-connected layer afterthe last convolutional layer (i.e., of size 256 ).
The augmentations we use are randomly chosen amongthe 3 following transformations
• Random resized crop: it crops random portion of the image and resizes it to a given size.
2 sets of parameters are used for this transformation: the scale and the ratio.
The scale
parameter specifies the lower and upper bounds for the random area of the crop.
The ratioparameter specifies the lower and upper bounds for the random aspect ratio of the crop.
Our scale range is ( 0 . 1 , 0 . 9 ) and our ratio range is ( 0 . 8 , 1 . 2 ).
• Random affine transformation: it applies a random affine transformation of the image
while keeping the center invariant.
The affine transformation is a combination of a rotation
(from − 15 ◦ to 15 ◦ ), a translation (from − 5 pixels to 5 pixels), a zoom (with a ratio from
75 to 1 . 25 )
and a shearing (from − 10 ◦ to 10 ◦ ).
• Random perspective transformation: apply a scale distortion with a certain probability
to simulate 3D transformations.
The scale distortion we have chosen is 0 . 5 , and it is applied to the image with a probability of 50%
Please see the site https://pytorch.org/vision/main/auto_examples/plot_transforms.html for illustration of the transformations.
Note that we have tried different settings for theaugmentations (varying the parameters of the augmentations), and we have observed a very limitedimpact of those settings on the computation of the samples diversity (see S3.3 for more details).
S2.2 Training details
Our SimCLR network is trained for 100 epochs with a batch size of 128 .
We used an RMSpropoptimizer [25], with a learning rate of 10 − 3 (all other parameters of the RMSprop are the defaultones).
S3 Control experiments for the samples diversity computationS3.1 Comparing the supervised and the unsupervised settings for the computation of thesamples diversity
To compare the unsupervised with the supervised setting, we have computed for all of the 150 classesof the Omniglot testing set the samples diversity.
We plot the samples diversity values for eachcategory and for both settings in Fig.
S1.
We report a linear correlation coefficient R 2 = 0 .
74 and a
Spearman rank order correlation ρ = 0 . 85 (see Table S2, first line).
It does mean that the samplesdiversity, as computed with one of the setting, is strongly correlated both in terms of rank order andexplained variance, with the samples diversity as computed with the other setting.
deviation (see Eq. 2) on SimCLR features (for x-axis) or on the features of Prototypical Net (for y-axis)
S3.2 More control experiments on the effect of the dispersion measure
To make our analysis more robust we have conducted additional control experiments with differentmeasures of dispersion.
In Eq.
2 we have presented a classical measure of dispersion that is thestandard deviation.
Another measure of data dispersion is the pair-wise cosine distance among the
[Table]
samples belonging to the same class:
[Equation]
In Eq.
3, C denotes the cosine similarity.
In Fig.
S2a, we plot the samples diversity for both featureextraction networks but with a dispersion measure based on the pairwise cosine distance as formulatedin Eq. 3.
We report a linear correlation of R 2 = 0 .
57 and a Spearman rank order correlation ofρ = 0 .
71 (see second line of Table S2).
This control experiment suggests that even by using adifferent dispersion metric (i.e., the pairwise cosine distance), the 2 feature extraction networksproduce samples diversity values that are heavily correlated.
This strengthen our observation madein S3.1: the representations produced by the SimCLR and Prototypical Net are similar.
Anotherinteresting control experiment is to compare the impact of the dispersion measure on the samplesdiversity metric.
To do so, we have compared the samples diversity computed with one featureextractor (either Prototypical Net in Fig.
S2b or SimCLR in Fig.
S2c)
but for 2 different dispersionmetrics (i.e., the standard deviation as formulated in Eq.
2 and the pairwise cosine distance as definedin Eq. 3).
In both cases, we have a non negligible linear correlation (i.e., R 2 > 0 . 44 ) and a strong
Spearman rank order correlation (i.e., ρ > 0 .
63 , see third and fourth lines of Table S2).
All thesecontrol experiments confirm that our computation of the samples diversity is robust to 1) the type ofapproach we used to extract the features and 2) the measure of dispersion we are using to computethe intraclass variability.
[Figure]
S3.3 Impact of the image augmentation on the diversity measure
To test the impact of the image augmentations on the SimCLR network we have trained 3 SimCLRnetworks with different augmentation levels.
• With moderate level of image augmentation.
All the augmentations here are those describedin section S2.• With a low level of image augmentation.
Here the scale of the random resized crop isvaried from 0 .
05 to 0 .
95 and the crop ratio is ranging from 0 .
9 to 1 . 1 .
The rotation of theaffine transformation is ranging from − 7 deg to 7 deg , the translation from − 3 pixels to 3pixels, the zoom from 0 .
9 to 1 . 1 and the shearing from − 5 deg to 5 deg .
The scale distortionapplied to the image is 0 .
25 (with a probability of 50 %).•
With a high level of image augmentation.
In this setting, the scale of the random resizedcrop is varied from 0 .
2 to 0 .
8 and the crop ratio is ranging from 0 .
6 to 1 . 4 .
The rotationof the affine transformation is ranging from − 30 deg to 30 deg , the translation from − 10pixels to 10 pixels, the zoom from 0 .
5 to 1 . 5 and the shearing from − 20 deg to 20 deg .
Thescale distortion applied to the image is 0 .
75 (with a probability of 50 %).
In Fig.
S3, we compare the samples diversity obtains for each category of the Omniglot testing setwhen we train the SimCLR network with moderate level of image augmentation and with a low levelof image augmentation (see Fig.
S3a), or with a high level of image augmentation (see Fig.
S3b).
Wealso report the Spearman correlation in Table S3.
[Figure]
[Table]
We observe a high linear correlation as well as a high Spearman rank order correlation between the
[Table] tested settings. It suggests that the samples diversity is relatively independent to the level of imageaugmentations used during the SimCLR training.
S3.4 T-SNE of the SimCLR and Prototypical Net latent space
In Fig.
S4a and Fig.
S4b, we show a t-SNE analysis of the feature space of Prototypical Net and
SimCLR respectively.
In Fig.
S4a, the t-SNE analysis of the Prototypical Net feature space reveals
[Figure]
a strong clustering of the samples belonging to the same class.
Note that this phenomenon is notsurprising as the loss of the Prototypical Net forces the samples belonging to the same class to beclose in the feature space.
More surprisingly, we also observe a clustering effect in the SimCLRt-SNE analysis (see Fig.
S4b).
Note that SimCLR is a fully unsupervised algorithm: there is no classinformation given to the algorithm.
Consequently, the strong clustering effect we observe suggeststhat forcing the proximity between a sample and its augmented version is enough to retrieve the classinformation.
This observation might explain why contrastive learning algorithms are in general soefficient in semi-supervised (or even unsupervised) classification tasks.
S6 MAML architecture and training details
The architecture we have used for the MAML classifier is exactly the same used for the Prototypical
Net (see Table S1).
The only difference is the last fully-connected layer that is : Linear(256, 20).
Indeed, as the MAML network is directly predicting the logits (and not a distance metric), the lastlayer needs to have the same dimension than the number of class of the experiment.
In a 1 -shot20 -way classification experiment, the number of classes is 20 .
We have used a 2 nd order meta-learning scheme [18].
The outer-loop optimizer is an Adam optimizerwith a learning rate of 10 − 3 , and the inner-loop optimizer is a simple Stochastic Gradient Decent
witha learning rate of 10 − 2 .
The number of inner loops is set to 5 during the training and to 10 during thetesting.
The number of tasks for each outer-loop is set to 4 .
S7 Control experiments: Comparing Prototypical Net and MAML
To rigorously compare MAML and Prototypical Net, we have conducted 2 types of control experiments.
First we have verified whether the classification accuracy obtained for each class were rankedin the same order for both MAML and Prototypical Net.
To do so, we have presented the same seriesof categorization tasks to both algorithms.
The high Spearman rank coefficient ( ρ = 0 . 60 ) indicatesthat both classifiers rank each category’ classification accuracy similarly (see section S4).
To confirm this result, we have computed the correlation between the logits generated by both models.
In the case of the MAML model, extracting the logits is straightforward.
For Prototypical Net, weuse the distance to prototypes as logits.
This explain why both model’s logits are anti-correlated:the MAML logits are the (un-normalized) probability of belonging to a given classes whereas the
Prototypical Net logits correspond to the distance to the category (so the lower the distance, thehigher the probability).
We report a strong negative correlation ( r = − 0 . 62 ) between the logits of the
MAML network and those of Prototypical Net (see section S4).
[Table]
S details of the
VAE
STN S8.1 Architecture of the VAE-STN
The VAE-STN is a sequential VAE that allows for the iterative construction of a complex image [42].
A pseudo-code of the algorithm is described in Algo 1.
At each iteration, the algorithm focuses itsattention on a specific part of the image ( x ), the prototype ( ˜ x ) and the residual image ( ˆ x ) using the
Reading Spatial Transformer Network (STN r ).
Then the extracted patch is passed to an encodingnetwork (EncBlock) to transform it into a latent variable.
This latent variable is concatenated to apatch extracted from the prototype and then passed to the RecBlock network.
The produced hiddenstate is first passed to DecBlock to recover the original patch, and then to the STN w to replace andrescale the patch into the original image.
The LocNet network is used to learn the parameter of theaffine transformation we used in the STN.
Note that the affine parameters used in STN w are simplythe inverse of those used in STN r .
Algorithm
Input: image: x , prototype: ˜ x c ←θ 1 ← [[1 , 0 , 0] , [0 , 1 , 0]] h 1 ←for i = 1 to N steps do ˆ x = x − sigmoid ( c ) r , ˆ r , ˜ r
=
STN r ( θ t , x ) , STN r ( θ t , ˆ x ) , STN r ( θ t , ˜ x ) r ← [ r , ˆ r , ˜ r , h t ] µ , σ = EncBlock ( r ) z
= µ + ϵσ with ϵ ∼ N (0 , 1)
z ← [ z , ˜ r ] p =
DecBlock ( h t ) c
← c + STN w ( θ − 1 t , p ) h t +1 ← RecBlock ( z , h t ) θ t + 1 ← LocNet ( h t +1 ) end for
The STN modules take 2 variables in input: an image (or a patch in the case to the STN w ) and amatrix (3 × 2) describing the parameters of the affine transformation to apply to the input image [26].
All other modules are made with MLPs networks, and are described in Table S5.
In the Table S5 weuse the following notations:
• s z : This the size of the latent space.
In the base architecture, we set s z = 80 .•
s LSTM :
This is the size of the output of the Long-Short Term Memory (LSTM) unit.
In thebase architecture, we set s LSTM =
400• s r : This is the resolution of the patches extracted by the Spatial Transformer Net (STN)during the reading operation.
In the base architecture we set s r = 15 .
• s loc :
This is the number of neurons used at the input of the localization network.
In the base architecture, we set s loc =
• s w : This is the resolution of the patch passed to the the STN network for the writing operation.
In the base architecture s w = 15 .
For the base architecture we used N steps = 60 .
The base architecture of the VAE-STN has 6 .
2millions parameters.
For more details on the loss function, please refer to [42].
S8.2 Training details of the VAE-STN
The VAE-STN is trained for 500 epochs, with batches of size 128 .
We use an Adam optimizer with alearning rate of 1 × 10 − 3 and β 1 = 0 .
All other parameters are the default Pytorch parameters.
Toavoid training instabilities we clip the norm of the gradient to 5 .
The learning rate was divided by 2when the evaluation loss has not decreased for 10 epochs (reduce on plateau strategy).
[Table]
EncBlock(s r , s LSTM , s z )
[Table]
DecBlock(s LSTM , s loc , s w )
[Table]
RecBlock(s z , s
[Table]
, s LSTM )
[Table] [Table]
[Table]
S8.3 VAE-STN samples
Table S6 describes the base architecture of the Neural Statistician model adopted from [19] which isa close approximation of [15].
We make minor changes in the network architecture to accommodatethe higher input image size of 50 × 50 of the Omniglot dataset.
The Neural Statistician model iscomposed of the following sub-networks:
• Shared encoder x (cid:55)→ h :
An instance encoder E that takes each individual datapoint x i to a
feature representation h i
=
E ( x i ) .•
Statistic network q ( c | D, ϕ ) : h 1 , ..., h k (cid:55)→ µ c , σ 2 c : A pooling layer that aggregates the matrix ( h 1 , ..., h k ) to a single
pre-statistic vector
v .
uses sample mean for theirexperiments
Which is
followed by a post-pooling network that takes v to a parametrization
of a Gaussian.
• Inference network q
( z | x, c, ϕ ) : h, c (cid:55)→ µ z , σ 2 z : Inference network gives an approximateposterior over latent variables.• Latent decoder network p ( z | c ; θ ) :
c (cid:55)→ µ z , σ 2 z• Observation decoder network p ( x | c, z ; θ ) : c, z (cid:55)→ µ x
The overall number of parameters of the base model (which has the same architecture as used in[15]) for the Neural Statistician we are using is around 7.48M parameters.
Training details
The Neural Statistician is trained for 300 epochs, with batch size of 32 and learning rate of 1 × 10 − 3 .
We adopt the same setting of the Neural Statistician as used in [15] for the omniglot dataset.
We constructed context sets by splitting each class into datasets of size 5 while training, anduse a single out-of-distribution exemplar while testing.
As discussed in the paper, we createnew classes by reflecting and rotating characters.
We based our implementation from https://github.com/georgosgeorgos/hierarchical-few-shot-generative-models and https://github.com/comRamona/Neural-Statistician .
[Table]
ConvBlock
[Table]
ConvBlock
[Table]
ConvBlock
[Table]
ConvBlock
[Table]
ConvBlock
[Table] ConvBlock
[Table]
[Table]
[Table]
[Table]
ConvBlock
[Table]
ConvBlock
[Table] [Table]
BatchNorm1d
[Table]
(cid:55)→
[Table]
[Table] [Table]
{3× FcBlock(
[Table]
Linear(
[Table]
32), BatchNorm1d(
[Table]
to µ z
[Table]
σ
[Table]
Linear(512, 256) (cid:55)→ c , ELU
[Table]
{3× FcBlock(
[Table]
Linear(256, 32), BatchNorm1d(1) to µ z , log σ 2 z
Observation decoder network
FcBlock(512, 256) (cid:55)→ z 3,324,
FcBlock(512, 256) (cid:55)→ c combine z and c, ELU
FcBlock(256, 256*4*4)
ConvBlock(256, 256, 1) ConvBlock(256, 256, 1)
DeConvBlock(256, 256)
ConvBlock(256, 128, 1)
ConvBlock(128, 128, 1)
DeConvBlock(128, 128)
ConvBlock(128, 64, 1)
Conv2d(64, 64, 4, 1, 0)
DeConvBlock(64, 64)
ConvBlock(64, 32, 1)
Conv2d(32, 32, 2, 1, 0)
DeConvBlock(32, 32)
Conv2d(32, 1, 1)
S10 Architecture and training details of the DA-GAN based on U-Net (DA-GAN-UN)
Architecture
Table S7 describes the base architecture of the DA-GAN-UN’s Generator model adopted from [1].
We have modified the architecture of the DA-GAN-UN model such that it can accommodate a higherinput image size 50 × 50 .
Also, we reduced the number of trainable parameters in the original
DA-GAN-UN architecture to have a fair comparison with other few-shot models.
Following are thenotations used in Table S7:
• s z : This is the size of the latent space.
In the base architecture, we set s z =
• Generator G ( x, z ) :
A generator network that takes data points and Gaussian noise as input,and generate new samples.
The base architecture of the DAGAN model we are using in our experiments has around 6.8 millionparameters.
Training details
[Table]
The DA-GAN-UN model was trained for 30 epochs, with batches of size 32.
We use an Adamoptimizer with a learning rate of 1 × 10 − 4 and β 1 = 0.9.
We update our generator after every 5updates of discriminator.
We based our implementation from https://github.com/amurthy1/dagan_torch
[Table]
In c , Out c , s l )
Conv2d(In c , Out c , 3, stride= s l , padding=1) Out
[Table]
[Table]
LeakyReLU(0.2), BatchNorm2d(Out c )
[Table]
Out c DeConvBlock(In c , Out c , s l )
ConvTranspose2d(In c , Out c , 3, stride=
s l , padding=1)
Out
[Table]
× (In
[Table]
3 × 3 + 1)
LeakyReLU(0.2), BatchNorm2d(Out c )
[Table]
Out c
EncoderBlock(In p , In c , Out c )
ConvBlock(In p , In p )
ConvBlock(In c
+ In p , Out c )
Conv2d(In c + Out c , Out c )
Conv2d(In c +
× Out c , Out c )
DecoderBlock(In p , In c , Out c )
DeConvBlock(In p , In p , 1)
ConvBlock(In c
+In p , Out c , 1)
DeConvBlock(In p , In p , 1)
ConvBlock(In c
+ In p + Out c , Out c , 1)
DeConvBlock(In c
+ 2 × Out c , Out c , 1)
Generator(s z )
ConvBlock(1, 64, 2) 6,813,
EncoderBlock(1, 64, 64) EncoderBlock(64, 64, 128)
EncoderBlock(128, 128, 128)
Linear(s z , 4 × 4 × 8)
DecoderBlock(0, 136, 64)
Linear(s z , 7 × 7 × 4)
DecoderBlock(128, 260, 64)
Linear(s z , 13 × 13 × 2)
DecoderBlock(128, 194, 64)
DecoderBlock(64, 128, 64) DecoderBlock(64, 65, 64) ConvBlock(64, 64, 1)
ConvBlock(64, 64, 1) Conv2d(64, 1, 3, stride=1, padding=1)
[Figure]
We use the same base architecture of DA-GAN-UN , except we remove the skip connections betweenthe contracting path (encoder) and the expansive path (decoder).
[1] used a combination of UNetand ResNet in their results, in DA-GAN-RN we consider only a ResNet type architecture.
Training details
Refer S10 for training details.
[Figure]
We observe a monotonic decrease of the diversity and a monotonic increase of the recognizabilitywhen the number of context samples increases.
We vary the number of context samples from 2 to20 .
This experiment has been conducted with 3 different seeds (i.e., different network initialization),represented with red, green and blue data points, respectively.
For each seed, we report 19 datapoints.
To highlight the trend in the diversity-recognizability space, we have smoothed the curves in
Fig.
S11a and Fig.
S11b, using a Savitzky-Golay filter (second order, window size of 7).
S13 Effect of the number of attentional steps on the diversity/recognizability framework
[Figure]
when one varies the number of attentional steps from 20 to
In this experiment, we have varied the number of attentional steps from 20 to 90 .
Note that wecould not go below 20 attentional steps to make sure the attentional process is fully covering theentire image.
We did not go over 90 attentional steps because we faced some training instabilitiesbeyond this point.
We observe a non-monotonic evolution of the diversity and the recognizabilitywith the increase of the number of attentional steps.
This experiment has been conducted with 3different seeds (i.e., different network initialization), represented with red, green and blue data points,respectively.
For each seed we report 8 data points.
In order to properly assess the type of parametriccurves that govern the evolution of the diversity-recognizability space when one varies the number ofattentional steps, we have used a least curve fitting method [21].
This method involves finding the best polynomial fit (second order in our case) for the 3 curves (Fig.
S12a, b and c) simultaneously.
This method is iteratively refining all the fits to minimize the sum of all least square error.
S14 Mathematical formulation of the ELBO
Let us consider a dataset X
= { x ( i ) }
Ni =1 composed of N i.i.d samples of a random variable x .
Weassume that x is generated by some random process involving an unobserved random variable z .
Thelatent variable z is sampled from a Gaussian distribution (see Eq. 5).
The mean of the likelihood isparametrized by µ θ (in which θ denotes the parameters) and its variance is considered constant.
[Equation]
The Variational Auto Encoder is optimized by maximizing the Evidence Lower Bound (ELBO), asformalized in its simplest form in Eq. 6:
[Equation] [Equation]
One could observe that the β coefficient is tuning the importance of the prior (through the KL).
Ifβ > 1 , then the latent space will be forced to be closer to the prior distribution but will attenuate theweight of the reconstruction loss.
Such a scenario tends to improve the disentanglement of the latentspace [24].
On the contrary, if β is low, then the reconstruction loss (i.e., E q ϕ ( z | x ) [log p θ ( x | z )] )will take over, and then the latent space will be less regularized.
Note that in the extreme case whereβ = 0 , the VAE becomes an auto-encoder.
The ELBO loss can be updated to include a latent variable encoding for the context c as in the
VAE-NS .
In this formulation, the context corresponds to a dataset D (see Eq. 7):
[Equation]
The ELBO could also be extended to include a sequential generative process as in the VAE-STN .
Inthis case, the latent variable z is time-indexed and is now a sequence of random variables denoted( z 1 , .., z T ) .
In Eq.
8, z <k indicates the collection of all latent variables from step t = 1 to t = k .
[Equation]
S15 Effect of the beta coefficient on the diversity/recognizability framework
In this experiment, we have varied the value of the β coefficient from 0 .
25 to 4 for the VAE-STN andfrom 0 .
25 to 5 for VAE-NS model.
This experiment has been conducted with 3 different seeds(i.e.
, different network initialization), represented with red, green and blue data points, respectively.
For the VAE-STN and for each seed, we have collected 16 data points (see Fig. S13), and 20 forthe VAE-NS (see Fig. S14).
We use a similar method than in S13 to find a polynomial fit (secondorder in our case) of the curves shown in Fig.
S13a, b, and c and Fig.
S14a, b, and c.
We reporta quasi-monotonic decline of the diversity when the beta value is increased (see Fig.
S13a and
Fig. S14a).
In contrast, the recognizability follows a parabolic relationship when the beta value isincreased.
For the VAE-STN , the maximum recognizability ( ≈ 80% ) is reached for a β value of 2 .
25(see Fig.
S13b).
For the VAE-NS , the maximum recognizability ( ≈ 91% ) is reached for a β value of3 (see Fig.
S14b).
Even if the change of amplitude in recognizability and in diversity is larger for the
VAE-STN than for VAE-NS , the shapes of the curves are very similar.
[Figure]
[Figure]
[Figure]
Effect
[Figure]
the size of the
[Figure]
space on the
[Figure]
recognizability
[Figure]
[Figure]
[Figure]
[-(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Note (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58) the (cid:58) 
-]
S17 Computational Resources
All the experiments of this paper have been performed using Tesla V100 with 16 Gb memory.
Thetraining time is dependent on the hyper-parameters, but varies between 4h to 24h per simulation.
S18 Broader Impact
This work does not present any foreseeable negative societal consequences.
We think the societal [Figure] impact of this work is positive.
It might help the neuroscience community to evaluate the different [Figure] mechanisms that allow human-level generalization, and then better understand the brain.
[Figure] [Figure]