Few-Shot Diffusion Models
Abstract
Denoising diffusion probabilistic models (DDPM) are powerful hierarchical latent variable models with remarkable sample generation quality and training stability.
These properties can be attributed to parameter sharing in the generative hierarchy, as well as a parameter-free diffusion-based inference procedure.
In this paper, we present Few-Shot Diffusion Models (FSDM), a framework for few-shot generation leveraging conditional DDPMs.
FSDMs are trained to adapt the generative process conditioned on a small set of images from a given class by aggregating image patch information using a set-based Vision Transformer (ViT).
At test time, the model is able to generate samples from previously unseen classes conditioned on as few as 5 samples from that class.
We empirically show that FSDM can perform few-shot generation and transfer to new datasets.
We benchmark variants of our method on complex vision datasets for few-shot learning and compare to unconditional and conditional DDPM baselines.
Additionally, we show how conditioning the model on patch-based input set information improves training convergence.
Introduction
[Figure]
Humans are exceptional few-shot learners able to grasp concepts and function of objects never encountered before [54, 95, 55].
This is because we build internal models of the world so we can combine our prior knowledge about object appearance and function to make well-educated inferences from very little data [96, 56, 98].
In contrast, traditional machine learning systems have to be trained tabula rasa and therefore need orders of magnitude more data.
A particularly challenging problem is few-shot adaptation in generative latent variable models [19, 79, 81, 7].
Few-shot generation has been limited to simple datasets and shallow tasks, using handcrafted aggregation and conditioning mechanisms.
Recently diffusion models [92, 35] have shown impressive generative performance for vision [64, 37, 36], language [39, 3], speech [52], biological data [40, 62, 107], and multimodal [65, 74] generation, providing an important step toward general and stable pure generative models.
Unconditional diffusion models are expressive likelihood-based density estimators with high sample quality [48].
This expressivity arises from the Monte Carlo (layer) sampling that we can perform during training thanks to the special structure of the forward process: a parameterfree diffusion process for which the posterior can be computed at each step in closed form [35].
[Figure]
However such effectiveness is at the cost of posterior flexibility and absence of latent space structure.
For this reason the few-shot capacities of this class of models is largely unexplored and conditional adaptation is challenging.
In this work we aim to study adaptation mechanisms and improve few-shot generation in latent variable models on realistic and complex visual data (Fig. 1).
The setting we consider is that of learning from a large quantity of homogeneous sets, where each set is an un-ordered collections of samples of one concept or class.
At test time, the model will be provided with sets of concepts never encountered during training.
We consider explicit conditioning in a hierarchical formulation, where global variables carry information about the set at hand .
The conditional hierarchical model can naturally represent a family of generative models, each specified by a different conditioning set-level variable.
We formulate FSDM using vision transformers [17] and diffusion models [35, 64].
We propose to process the input set in patches and condition the generative model with a learnable attention mechanism using a tokenized representation for the input set.
Our contributions are:
I) a new framework to perform few-shot generation for realistic sets of images in the DDPM framework.
II) Learnable Attentive Conditioning (LAC), a conditioning mechanism where the input set is processed as a collection of patches and used to condition a DDPM through attention between sample-level and set-level variables.
III)
Experimental evidence that our model speeds up training, increases sample quality and variety, and improves transfer for conditional and few-shot generation compared to relevant unconditional and conditional
DDPM-based baselines
Few-Shot Diffusion Models
For background on diffusion models see Appendix A. For related work see Appendix B
In this paper, our goal is to learn to quickly adapt to new generation tasks.
That is, we want to perform few-shot generation conditioned on a set X containing previously unseen samples from a new task.
We approach this problem using diffusion models: We learn a diffusion model p θ ( x | X ) conditioned on the set X .
We refer to our approach as Few-Shot Diffusion Models (FSDM).
Our model can be broken down into two main parts: 1) A neural network h ϕ that produces a context representation c
= h ϕ ( X ) of the set X , and 2) a conditional diffusion model that generates novel samples conditioned on the context c .
See Fig.
3 for an illustration.
[Figure]
[Figure]
Generative Model.
The generative model is a conditional diffusion model
[Equation]
conditioned on the set X through the context c produced by the context net h ϕ .
In practice, we use a
Vision Transformer (ViT [17]) as the context net h ϕ , but we also experiment with a UNet encoder.
We discuss the use of ViT in Sec.
C.1.
The generative path p θ ( x t − 1 | x t , c ) is parameterized by a
UNet, as is common practice for diffusion models.
However, since we have an additional context c , we need the UNet to fuse the information in x t and c to predict x t − 1 .
In this work we consider two main mechanisms for this, 1) a mechanism based on FiLM [70] and 2) we propose Learnable
Attentive Conditioning (LAC), inspired by [83].
We discuss these in greater detail in Sec.
C.2.
The prior could also have been conditioned on c , i.e. p θ ( x T | c ) , but for simplicity we use a standard unconditional Gaussian p θ ( x T ) = N ( 0 , I ) .
Inference Model.
Given the special structure of DDPM and FSDM, the inference model is parameter-free and we do not need to condition on c .
This is a great simplification during training.
In practice FSDM employs a diffusion parameter-free posterior that degrades the information in the data at each step adding noise as presented in Eq. 4.
Loss and Training.
The negative ELBO can be expressed as a conditional version of Eq.
L FSDM =
L c 0 + (cid:80) Tt =
L ct − 1 + L cT .
As for regular DDPMs, the loss can be decomposed into a sum of terms, one per layer, that can be computed independently.
Training thus enjoys the same benefits where we can get efficient stochastic estimates of the objective by Monte Carlo sampling terms.
The conditional per-layer loss L ct − 1 can then formulated as
[Equation]
L cT is unconditional and fixed in our model formulation, and L c 0 is a negated conditional discretized normal likelihood, L c
=
E q ( x 1 | x 0 ) [ − log p θ ( x 0 | x 1 , c )] .
Experiments
Setup.
We use as backbone the standard DDPM model proposed in [35, 64] with fixed T =and linear β schedule.
We reduce the number of channels in the model to 64 obtaining a 25M parameter model.
We train using L =
L ϵ + λL vlb with λ = 0 .
001 from [64] that we found gave us a good balance between sample quality and training stability.
We employ a Unet (10M) and a ViT (5M) as set encoders.
We use the standard unconditional DDPM and conditional DDPM variants as baselines.
In general our approach can be applied to condition any unconditional diffusion model [35, 49, 93].
For this reason we limit our attention to standard DDPM models with a discrete number of layers.
More details on datasets, baselines, and additional experiments in Appendix D.
Few-Shot Generation.
We compare the generative models in terms of denoising capacity, summing over T steps Eq. 2, FID [33] for sample quality, sFID [63] to capture spatial relationships, Precision and Recall [53] for measuring variety and mode coverage.
We consider two main scenarios: indistribution (In), testing on classes seen during training; and out-distribution (Out), testing on classes unknown during training (the few-shot scenario).
We perform qualitative experiments on Omniglot,
CIFAR100 and CelebA, and quantitative experiments on CIFAR100 and miniImageNet in Table 1 and
Table 3.
In Table 1 FSDM outperforms the unconditional and conditional baselines on both datasets and scenarios, providing evidence that a token-based representation jointly with cross-attention conditioning are effective mechanisms for few-shot generation.
FSDM is a better denoiser ( L ϵ ) and image generator (FID, sFID, P, R) than strong conditional baselines.
We notice that in-distribution,
DDPM and the conditional baselines (cDDPM and sDDPM) perform well as expected, but tend to under-perform out-distribution.
This is expected for unconditional DDPM.
For the conditional variants global aggregation is not expressive enough to represent complex novel realistic classes.
Processing the set using FSDM-s tends to work better than the baselines but under-performs FSDM.
Additionally, FSDM is an efficient learner, being able to extract more information from less data and converge faster than the baselines (Fig. 2). [Table]
Transfer.
The goal of FSDM is to perform few-shot generation on objects never seen during training.
However there are multiple challenges when dealing with new classes, in particular if these new classes are from new categories and datasets.
Imagine to train a model on cats and lions [Table] and then provide tiger at test time: even if the model has never seen a tiger , the encoder can extract information from a small set of tigers leveraging classes with similar animals.
In a way the model can "interpolate" between the set at hand and similar classes.
But if we train on apples and oranges and test on tiger , the model is challenged in a more fundamental way.
There is no way to interpolate with known classes and the model has to rely mostly on the conditioning set.
Between these two extremes there is a spectrum of challenges in few-shot generation and we want to explore how far our model can adapt to new information.
For this reason we test few-shot transfer on a different dataset.
We take models trained on CIFAR100 and test them (without gradient based adaptation) on
MinImageNet.
This is a difficult generalization task.
We report results in the bottom part of Table 1.
All the models struggle compared to new classes from the same dataset.
However FSDM is still able to extract more information than the baselines, providing additional evidence that our framework can be used in a variety of small and large adaptation tasks.
Sampling.
FSGM can sample known and unknown classes conditioning on few samples.
When sampling known classes we are simply sampling conditional iid from a set X summarized by a certain c .
When performing real few-shot sampling we condition on unknown classes using small sets of samples.
In Fig 4 we show samples from known-classes (left panel) and from unknown-classes (right panel).
The visual quality of the unknown classes is obviously worse than the known one.
However the model can extract content information from few-samples and complex realistic classes in an effective way.
We report additional visualizations on Omniglot and CelebA in Fig 10.
Conclusion
We presented Few-Shot Diffusion Models, a flexible framework to adapt quickly to different generative processes at test-time, leveraging advances in Vision Transformers and Diffusion Models.
We show how conditioning a diffusion model with rich, expressive information gives superior performance in a wide range of experiments in and out-distribution.
Few-shot generation is performed on realistic, complex sets of images, showcasing a promising direction for large-scale few-shot latent variable generative models
A Background
Few-Shot Generation.
Few-shot generation is the task of adapting quickly to new classes or objects at test time given a small amount of instances from a novel category.
In standard few-shot learning [54, 55, 41], given a support set X s = { x s } Ss =1 and a query sample x q , we condition a learner on X s and predict on x q with a model of the form p ( x q | f ϕ ( X s )) .
The conditioning can be explicit on the representations [23, 22, 102, 90, 68, 94, 88] or implicit on the parameters like in meta-learning [87, 38, 21, 27, 76] and optimization [85, 77, 1].
Few-shot generation [55] borrows a similar setting but for the more challenging task to generate objects given few samples of that object at test time.
In particular, given context information X , we want to learn a conditional generative model that adapts quickly to new objects.
There are two main ways to do this: learn a set-based generative model p ( X )
=
(cid:82) p ( X | c ) p ( c )
d c and perform few-shot generation as a downstream task leveraging the per-set posterior q ϕ ( c | X new ) as in [19, 34, 25].
Alternatively, we can learn a conditional model of the form p ( x | X ) =
(cid:82) p ( x | c ) p ( c | X )
d c similarly to [7, 81, 79] and perform few-shot generation using the model directly for p ( c | X new ) .
We consider the deterministic case of this, where the encoder p ( c | X ) =
δ ( c − h ϕ ( X )) is a deterministic set-based neural network c ← h ϕ ( X ) .
Diffusion Denoising Probabilistic Models (DDPM).
Let x 0 denote the observed data which is either continuous x 0 ∈ R D or discrete x 0 ∈ { 0 , ..., 255 } D .
Let x 1 , ..., x T denote T latent variables in R D .
We now introduce, the forward or diffusion process q , the reverse or generative process p θ and the objective L .
The forward or diffusion process q is defined as [35]:
[Equation]
The beta schedule β 1 , β 2 , ..., β T is chosen such that the final latent image x T is nearly Gaussian noise.
The generative or inverse process p θ is defined as:
[Equation]
where p ( x T ) =
N ( x T | 0 , I ) , and σ 2 t often is fixed (e.g. to σ 2 t = β t ).
The neural network µ θ ( x t , t ) is shared among all time steps and is conditioned on t .
The model is trained with a re-weighted version of the ELBO that relates to denoising score matching [92].
The negative ELBO L can be written as
[Equation]
where L 0 = E q ( x 1 | x 0 ) [ − log p ( x 0 | x 1 )] is the likelihood term (parameterized by a discretized Gaussian distribution) and, if β 1 , ...
β T are fixed, L T = KL [ q ( x T | x 0 ) , p ( x T )] is a constant.
The terms
L t − 1 for t = 2 , ..., T can be written as L t − 1 =
E q ( x [Equation]
| x [Equation] [ KL [ [Equation]
( x t − 1 | x t , x 0 ) | p ( x t − 1 | x t )]] .
By further applying the reparameterization trick [50], the terms L 1: T − 1 can be rewritten as a prediction of the noise ϵ added to x 0 in q ( x t | x 0 ) .
Parameterizing µ θ using the noise prediction ϵ
θ , we can write
[Equation]
where w t = [Equation] (1 − ¯ α t ) , which corresponds to the ELBO objective.
The weights w t can also be written in terms of signal-noise-ratio as proposed in [49].
Empirically [35] shows superior sample quality and stable training when using a re-weighted ELBO objective using w t = 1 with a predictable drop in likelihood performance.
We call this loss L ϵ in this paper.
B Related Work
Conditional Diffusion Models.
The standard DDPM [35] can be improved using likelihood-based training [18, 42], continuous time modeling [93, 99], learnable noise scales [49], efficient sampling mechanism [91, 4, 43, 86, 44, 103, 51], and exploiting powerful (variational) autoencoders for dimensionality reduction [83, 71, 69].
Methods to condition DDPM have been proposed, conditioning at sampling time [13], learning a class-conditional score [93], explicitly conditioning on class information [64, 66], physical properties [107, 105, 40], side information [6, 36], and temporal structure [37, 30].
We present a more general class of methods to condition diffusion models based on set-conditioning: We learn a parametric conditioning mechanism at the set-level and a conditional diffusion process at the sample-level.
A conditional DDPM has been proposed for point-cloud generation [62].
However they limit their attention to a specific application, where we consider the general idea to encode sets of generic data and use for few-shot generation and transfer.
Retrievalbased approaches [8, 2] use an external database and pre-trained contrastive embeddings [73].
We similar increase expressivity using a collection of sample similar to the input selecting a set from the same class without relying on a retrieval mechanism, pre-trained model and an external database.
In [83] DDPM leverages a large vector quantized [67] pretrained autoencoder to encode the data in latent space, and such encoder can be used to condition on generic data.
This approach is effective but expensive and rely on large pre-trained models on relevant datasets not always easily available.
VAE-DDPM models [99, 69] have been proposed to learn conditional models in latent space.
Text-toimage diffusion models [65, 74] have been recently proposed for guided generation.
Our approach can be easily adapted to work with text tokens instead of visual tokens, simply changing the patch encoder.
These results rely on massive computation and paired datasets, where we use small images, little data, relying on set information.
Vision Transformers.
Transformers [100] have shown remarkable performance on unstructured text based data.
Vision Transformers ([17], ViT) have recently emerged as a transformer variant to process image-like data using a patch-based approach.
Then these patches are encoded as tokens and fed to a standard transformer encoder.
ViT has been used for discriminative tasks with remarkable results [97, 109, 12, 60].
However less work has been done to use ViT in the context of generative latent variable models.
Recently masked autoencoders [32], based on the ViT formulation, have been proposed for self-supervised learning and pretraining [5].
ViT variants for small and little data [58, 59, 31] have been proposed and our patch aggregation relies on similar ideas.
Learning from Sets.
In recent years a large corpus of work studied the problem of learning from sets [108], and more generally learning in exchangeable deep models [14, 10, 9].
These models can be formulated in a variety of ways, but they all have in common a form of permutation invariant aggregation (or pooling mechanism) over the input set.
Deep Sets [108] formalized the framework of exchangeable models.
The Neural Statistician [19] was the first model proposing to learn from sets in the variational autoencoder framework and used a simple and effective mean pooling mechanism for aggregation.
The authors explored the representation capacities of such model for clustering and few-shot supervised learning.
Generative Query Networks performs neural rendering [20] where the problem of pooling views arises.
The Neural process family [23, 46], where a set of point is used to learn a context set and solve downstream tasks like image completion and few-shot learning.
Set
Transformers [57] leverages attention to solve problems involving sets.
PointNet [72] models point clouds as a set of points.
Graph Attention Networks [101] aggregate information from related nodes using attention.
Associate Compression Network [28] can be interpreted in this framework, where a prior for a VAE is learned using the top-knn retrieved in latent space.
In this work we build on ideas and intuitions in these works, with a focus on generative models for sets.
SetVAE [47] proposes a
VAE for point-clouds, showing that processing the input set at multiple resolutions is a promising direction for set-based latent variable models.
Few-Shot Latent Variable Models.
Historically the machine learning community has focused its attention on supervised few-shot learning [54, 55], solving a classification or regression task on new classes at test time given a small number of labeled examples.
The problem can be tackled using metric based approaches [102, 90, 68, 94, 88], gradient-based adaptation [21, 1, 38], optimization [77], and posterior inference [22, 26, 80, 27, 76].
More generally, the few-shot learning task can be recast as Bayesian inference in hierarchical modelling [27, 76].
In such models, typically parameters or representations are conditioned on the task, and conditional predictors are learned for such task.
In [106] an iterative attention mechanism is used to learn a query-dependent task representation for supervised few-shot learning.
Modern few-shot generation in machine learning was introduced in [54].
The Neural Statistician [19] is one of the first few-shot learning models in the context of
VAEs [50, 82].
The model has been improved further increasing expressivity for the conditional prior using powerful autoregressive models [34], a non-parametric formulation for the context [104], hierarchical learnable aggregation for the input set [24], and exploiting supervision [23].
[81] proposed a recurrent and attentive sequential generative model for one-shot generation based on [29].
Powerful autoregressive decoders and gradient-based adaptation are employed in [79] for one-shot generation.
The context c in this model is a deterministic variable.
In GMN [7] a variational recurrent model learns a per-sample context-aware latent variable.
However the context-aware representation scales quadratic with the input size, there is no separation between global and local information in latent space, and the input set is processed in an arbitrary autoregressive order, and not in a permutation invariant manner.
Finally, recent large-scale autoregressive language models [11] exhibit non-trivial few-shot capacities.
C Conditioning the model C.1 ViT as Set Encoder
Transformers [100] are the de-facto standard for natural language processing tasks and text generation [15, 11].
Recently Vision transformers (ViT, [17]) unlocked the general power of attention for vision tasks.
However the use of transformers in latent variable models for generation is still limited.
ViT gives us a flexible way to process images at the patch level.
We adapt ViT to handle sets of images (sViT) similarly to [58]: in particular we want to handle small sets (1-10) of images.
The fundamental idea is that we want to extract global information from the set and each patch should contain global information for a specific region in the image (Fig. 5).
In general we can condition the ViT encoder on the layer embedding t using ViT ( X , t ; ϕ ) : doing so we obtain a cheap way to learn a per layer-dependent context, coarse for large T , and more refined for small t .
The use of tokens as input opens the door for a general domain-agnostic few-shot latent variable generator: our approach can be effortlessly employed for few-shot and conditional generation with any modality (text, speech, vision) simply tokenizing the input set and fine-tuning the patch embedding layer, without the need of any modification to the set-encoder, conditioning mechanism or generative process.
C.2 Conditioning the Generative Process
After processing the patches and obtaining c , we need to find a way to condition the DDPM generative path.
We work with c in two different forms:
A vector c ∈ R d or a collection of N tokens , c ∈ R N × d as summarized in Table 2.
In this work we consider two main conditioning mechanisms: FiLM [70] and Learnable Attentive Conditioning (LAC), inspired by [83].
[Table]
Vector (V).
One approach can be to condition the intermediate feature maps u in the DDPM UNet on c , for example using a FiLM like mechanism [70], u
= m ( c ) u + b ( c ) ,
where m and b are learnable and context-dependent.
Given the special structure of the generative model, where all the layers share parameters θ and differ only through an embedding of the step t , the conditioning mechanism can be generically written as u = f ( u , c , t ) =
f ( u t , c t ) .
Merging together c with the step embedding we can condition each layer, defining a generic per-step conditioning mechanism.
In practice we found u ( c , t ) = m ( c ) u ( t )
+ b ( c ) being the best performing and flexible approach.
Tokens (T).
Alternatively, c can be a collection of variables c
= { c sp } N s ,N p s =1 ,p =1 , where N p is the number of patches per sample, and N s the number of samples in the set.
In this case, attention can be used to fuse information between the context c and the feature maps u .
In principle we could use the patches directly, i.e. u = att ( u , { c sp } N s ,N p s =1 ,p =1 ) .
However, this approach scales badly with the number of samples in the set N s .
Another option is to use a per-patch aggregation where we average of the set dimension c p =
1 N s (cid:80) N s s =1 c sp to obtain N p tokens { c p } N p p =1 that we feed to the ViT. We then use cross-attention [83] on the per-patch averaged tokens u = att ( u , { c p } N p p =1 ) to condition
DDPM.
Using per-patch aggregation, we can process any number of samples without increasing the number of tokens used to condition DDPM and, more importantly, aggregate information from different samples in the context c .
C.3 Variational FSDM
Alternatively to our formulation of FSDM, we could specify a latent variable model where the context c is a latent variable and the set X is generated conditioned on c .
We refer to this model as Variational
FSDM (VFSDM) and write it like
[Equation]
In this case, the inference model will be a combination of the parameter-free diffusion posterior and a parameterized encoder for c ,
[Equation]
Furthermore, the negative ELBO will contain an extra KL term between the encoder q ϕ ( c | X 0 ) and the prior p ( c ) ,
[Equation]
We originally worked with this model, but found the training to be more challenging, resulting in under-performance or poor conditioning properties.
D Experiments D.1 Experimental Setup
Baselines.
We compare FSDM with unconditional and conditional baselines.
For the conditional baselines, we adapt conditional diffusion models in the literature [16, 89, 99] to the few-shot class generation scenario.
We use a DDPM [35, 64] as unconditional baseline.
We then compare with two main conditional diffusion models: a cDDPM , where a Unet encoder [84] processes independently the images in X
= { x s } Ss =1 and aggregates using a mean operator, c = 1 S (cid:80)
Ss =1 f ϕ ( x s ) ;
the
Unet encoder has the same structure of the guiding classifier in [16] used to learn class-conditional models.
We then consider a sDDPM adapting ideas in [89] without contrastive learning, where a ViT encoder [17] splits X in patches and processes them jointly, as depicted in Fig.
5, and aggregates all the patches using a mean operator, c =
[Equation]
sp .
We also train a variational variant inspired by [99], vDDPM , that uses standard amortized variational inference [45, 50] on the per-set latent variable.
For cDDPM, sDDPM
and vDDPM, c is a vector and the conditioning mechanism is FiLM based [70].
We also compare explicit conditioning with test-time (or sampling-time) conditioning as proposed in [13].
We consider also two variants for FSDM, called FSDM-s and vFSDM : FSDM-s employees a different way to extract and aggregate set-information using ViT: we stack all the samples on the channel dimension and process them as one entity as proposed in [58].
vFSDM is a variational formulation where we learn a distribution over a set of tokens in the conditioning mechanism and we deal with amortized variational inference with quantized latents.
We adapt the relaxation proposed in [75, 65] and the vector quantization proposed in [67, 78] to learn this model.
Datasets.
We extensively test the baselines and our approach on 4 image datasets with different complexity and size: Omniglot (28) [54] using the binarization provided in [7].
FS-CIFAR100 (32) [68] using the original class split and mixing all the classes together ( CIFAR100mix ).
miniImageNet (32) [102, 77] dataset to test few-shot and transfer capacity.
CelebA (64) [61] for additional visualizations.
We refer to FS-CIFAR100 as CIFAR100 in the following.
D.2 Test-time Conditioning
[Table]
In this paper we argue the case that explicit adaptation during training is a powerful way to condition diffusion models.
However inference-time adaptation [13] has also been shown to be an effective mechanism to adapt diffusion models to new distributions and does not require retraining the model or a parametric encoder.
In
Table 3 we compare the ILVR method, a powerful conditioning mechanism at sampling time, with FSDM, that condition the model during training, in the few-shot generation scenario, i [Table] e. conditioning on samples from unknown classes during testing.
We use the same datasets and evaluation procedure proposed in Table 1.
We see that ILVR improves the result compared to a standard unconditional
DDPM in terms of few-shot generation.
However the task is challenging and the adaptation we require is on new classes and not only on new attributes.
FSDM outperforms ILVR in this setting, providing additional evidence that explicit conditioning during training is essential for few-shot generation of realistic and complex objects.
D.3 Additional Experiments
In this section we discuss additional experiments and visualizations for FSDM.
Fig.
6 shows conditioning sets with cardinality 5 (left) and 20 conditional samples (right) for a FSDM using a large number of indistribution classes from CIFAR100.
We can see that samples are high quality and have large variability.
In Fig.
7 we compare conditional samples on in-distribution classes using CIFAR100 (top) and
CIFAR100-mix (bottom).
In Fig.
8 we perform the same experiment on out-of-distribution classes.
When using CIFAR100 the out-of-distribution sets are not only from novel classes never seen during training but also from new categories, i.e. training on cats and testing on cars .
When using
CIFAR100-mix all the classes are mixed and the out-of-distribution sets are from new classes but not necessarily from novel categories, i.e. training on cats and testing on tigers .
Sample quality and variety decrease for out-of-distribution samples as expected.
When FSDM is presented with out-of-distribution sets from CIFAR100, the model cannot rely on similar classes and few-shot samples have reduced variability.
But when presented with out-of-distribution sets from
CIFAR100mix, the model can rely on similar classes and the few-shot generation task is easier, giving rise to better samples.
Smarter conditioning mechanisms and expressive set-level latent variables can help in improving generalization and we will explore such possibilities in future work.
In Fig.
9 we use L ϵ as a proxy signal to evaluate the capacity of DDPM and FSDM to distinguish between in-distribution and out-of-distribution samples.
FSDM performs better than DDPM for this task, consistently distinguishing better between in-distribution and out-of-distribution classes.
In Fig.
10 we show visualizations for Omniglot and CelebA using short training runs.
We train FSDM only for 100K iterations.
The goal is to explore how fast the conditioning mechanism can extract and aggregate context information.
As expected for a simple dataset as Omniglot the conditioning quality is high.
For CelebA the conditioning mechanism struggles using so few training iterations.
However, most of the context information is extracted successfully and samples are compatible with the content in the conditioning sets.
[Figure]
[Figure] [Figure]
steps
[Figure]
un t
FSDM_in
DDPM_in
FSDM_out
DDPM_out
FSDM_transfer
DDPM_transfer
E Experimental Details
[Table]