Self-Guided Diffusion Models
Abstract
Diffusion models have demonstrated remarkable progress in image generation quality, especially when guidance is used to control the generative process.
However, guidance requires a large amount of image-annotation pairs for training and is thus dependent on their availability, correctness and unbiasedness.
In this paper, we eliminate the need for such annotation by instead leveraging the ﬂexibility of self-supervision signals to design a framework for self-guided diffusion models.
By leveraging a feature extraction function and a self-annotation function, our method provides guidance signals at various image granularities: from the level of holistic images to object boxes and even segmentation masks.
Our experiments on single-label and multi-label image datasets demonstrate that self-labeled guidance always outperforms diffusion models without guidance and may even surpass guidance based on ground-truth labels, especially on unbalanced data.
When equipped with self-supervised box or mask proposals, our method further generates visually diverse yet semantically consistent images, without the need for any class, box, or segment label annotation.
Self-guided diffusion is simple, ﬂexible and expected to proﬁt from deployment at scale.
Introduction
The image ﬁdelity of diffusion models is spectacularly enhanced by conditioning on class labels [12].
Classiﬁer guidance goes a step further and offers control over the alignment with the class label, by using the classiﬁer gradient to guide the image generation [12].
Classiﬁer-free guidance [19] replaces the dedicated classiﬁer with a diffusion model that is trained by randomly setting the condition to the special non-label class.
This has proven a fruitful research line for several other condition modalities, such as text [38; 32], image layout [36], visual neighbors [1] and image features [14].
However, all these conditioning and guidance methods require ground-truth annotations.
This is an unrealistic and too costly assumption in many domains.
For example, medical images require domain experts to annotate very high-resolution data, which is infeasible to do exhaustively [29].
In this paper, we propose to remove the necessity of any ground-truth annotation for guidance diffusion models.
Inspired by progress in self-supervised learning [9; 7], in this paper, we propose self-guided diffusion , a framework for image generation using guided diffusion without the need for any annotated imagelabel pairs.
The framework encompasses a feature extraction function and a self-annotation function, that are compatible with recent self-supervised learning advances. Furthermore, we leverage the ﬂexibility of self-supervised learning to generalize the guidance signal from the holistic image level to (unsupervised) local bounding boxes and segmentation masks for more ﬁne-grained guidance. We demonstrate the potential of our proposal on single-label and multi-label image datasets, where self-labeled guidance always outperforms diffusion models without guidance and may even surpass guidance based on ground-truth labels.
When equipped with self-supervised box or mask proposals, our method further generates visually diverse yet semantically consistent images, without the need for any class, box, or segment label annotation.  Approach
Before detailing our self-guided diffusion framework, if readers are not familiar with diffusion models and the classiﬁer-free guidance technique, please refer to Appendix A.
2.1 Self-Guided Diffusion
The equations describing diffusion by classiﬁer-free guidance implicitly assume dataset D and its images each come with a single manually annotated class label.
We prefer to make the label requirement explicit.
We denote the human annotation process as the function ξ ( x ; D , C ) :
D → C , where C deﬁnes the annotation taxonomy, and plug this into Equation (7):
[Equation]
We propose to replace the supervised labeling process ξ with a self-supervised process that requires no human annotation:
[Equation]
where g is a self-supervised feature extraction function parameterized by φ that maps the input data to feature space H , g : x → g φ ( x ) , ∀ x ∈ D , and f is a self-annotation function parameterized by ψ to map the raw feature representation to the ultimate guidance signal k , f ψ : g φ ( · ; D )
→ k .
The guidance signal k can be any form of annotation , e.g., label, box, pixel, that can be paired with an image, which we derive by k
= f ψ ( g φ ( x ; D ); D ) .
The choice of the self-annotation function f can be non-parametric by heuristically searching over dataset D based on the extracted feature g φ ( · ; D ) , or parametric by ﬁne-tuning on the feature map g φ ( · ; D ) .
For the noise prediction function (cid:15) θ ( · ) , we adopt the traditional UNet network architecture [37] due to its superior image generation performance, following [20; 43; 33; 38].
Stemming from this general framework, we present three methods working at different spatial granularities, all without relying on any ground-truth labels.
Speciﬁcally, we cover image-level, box-level and pixel-level guidance by setting the feature extraction function g φ ( · ) , self-annotation function f ψ ( · ) , and guidance signal k to an approximate form.
Self-labeled guidance.
To achieve self-labeled guidance, we need a self-annotation function f that produces a representative guidance signal k ∈ R K .
Firstly, we need an embedding function g φ ( x ) , x ∈ D which provides semantically meaningful image-level guidance for the model.
We obtain g φ ( · ) in a self-supervised manner by mapping from image space, g φ ( · ) : R W × H × 3 → R C , where W and H are image width and height and C is the feature dimension.
We may use any type of feature for the feature embedding function g , which we will vary and validate in the experiments.
As the image-level feature g φ ( · ; D ) is not compact enough for guidance, we further conduct a non-parametric clustering algorithm, e.g., k -means, as our self-annotation function f .
For all features g φ ( · ) , we obtain the self-labeled guidance via self-annotation function f ψ ( · ) : R C → R K .
Motivated by Rolfe [35], we use a one-hot embedding k ∈ R K for each image to achieve a compact guidance.
We inject the guidance information into the noise prediction function (cid:15) θ by concatenating it with timestep embedding t and feed the concatenated information concat [ t, k ] into every block of the
UNet.
Thus, the noise prediction function (cid:15) θ is rewritten as:
[Equation]
where k = f ψ ( g φ ( x ; D ); D ) is the self-annotated image-level guidance signal.
For simplicity, we ignore the self-annotation function f ψ ( · ) here and in the later text.
Self-labeled guidance focuses on the image-level global guidance.
For more ﬁne-grained spatial guidance e.g., box, mask, please check
Appendix B.
Experiments
Evaluation metric & Baselines & implementation details. To evaluate both diversity and ﬁdelity of samples, we use the Fréchet Inception Distance (FID) [18] and Inception Score (IS) [39] as our [Figure]
main metrics. Throughout our experiments, we always use the diffusion model following [20]. We train with timestep T =1000 , guidance drop probability p =0 . 1 and the linear variance schedule.
For sampling, we set the guidance strength w =2 and deploy a clipping operation in every sampling timestep. As the baseline for a guided diffusion model with ground-truth labels we follow classiﬁerfree guidance [19]. We use DDIM [42] samplers with 250 steps, σ t =0 . All hyperparameter of our self-guided diffusion and the baselines are the same, allowing us to compare methods under a ﬁxed compute budget.
For details of the learning rate, optimizer, and hyperparameters, we refer to Appendix F. All code will be released.
3.1 Self-labeled Guidance
We use ImageNet32/64 [11] to validate the efﬁcacy of self-labeled guidance.
For better evaluation of sampling quality, we also adopt the Inception Score (IS) [39], following the common practice on this dataset [12; 22; 5]. IS measures how well a model ﬁts into the full ImageNet class distribution. [Table] [Table]
[Table]
DINO
[Table] [Table] Choice of feature extraction function g .
We ﬁrst measure the inﬂuence of the feature extraction function g used before clustering.
We consider two supervised feature backbones: ResNet50 [16] and ViT-B/16 [13], and four self-supervised backbones: SimCLR [9],
MAE [17], MSN [2] and DINO [7].
To assure a fair comparison we use 10k clusters for all architectures.
From the results in Table 1, we make the following observations.
First, features from the supervised ResNet50 , and ViT-B/ lead to a satisfactory FID performance, at the expense of relatively limited diversity (low IS).
However, they still require label annotation, which we strive to avoid in our work.
Second, among the self-supervised feature extraction functions, the MSN - and DINO -pretrained ViT backbones have the best trade-off in terms of both FID and IS.
They even improve over the label-supervised backbones.
This implies the label assignment for the guidance is not unique, pretext labels on top of self-supervised learning can still provide inﬂuential guidance signal in comparison with human annotated labels. Also, the diversity of label-supervised ViT-B/16 is much lower than self-supervised
ViT-B/16 with an IS of 7.81 vs. 10.41, suggesting that self-supervised guidance leads to more unbiased representation in comparison to supervised guidance.
From now on we pick the DINO ViT-B/16 architecture as our self-supervised feature extraction function g .
Effect of number of clusters.
Next, we ablate the inﬂuence of the number of clusters on the overall sampling quality.
We consider 1, 10, 100, 500, 1,000, 5,000 and 10,000 clusters on the extracted CLS token from the DINO ViT-B/16 feature.
For efﬁcient, yet uniform, comparison we only run 20 epochs on ImageNet32.
To put our sampling results in perspective, we also provide FID [Table] and IS results for DDPM and classiﬁer-free guidance.
From the result in Figure 1 we ﬁrst observe that when the cluster number is ranging from 1 to 5,000, our model’s performance monotonously increases and always surpasses the DDPM model.
Beyond 1,000 clusters, we are competitive with classiﬁer-free guidance using ground-truth labels.
For 5,000 clusters, there is a sweet spot where we outperform classiﬁer-free guidance with an FID of 16.4 vs. 17.9 and an IS of 9.94 vs. 10.35, see also the generated images in Figure 1.
The performance of FID starts to deteriorate from 5,000 to 10, clusters.
We conclude that self-labeled guidance outperforms DDPM without any guidance beyond a single cluster, is competitive with classiﬁer-free guidance beyond 1,000 clusters, and is even able to outperform guidance by ground-truth labels for 5,000 clusters.
From now on we use 5,000 clusters for self-labeled guidance on ImageNet.
Varying guidance strength w .
Next we consider the inﬂuence of the guidance strength w on our sampling results.
As the validation set of ImageNet32 is strictly balanced, we also consider an unbalanced setting which is more similar to real-world deployment. Under both settings we compare the FID between our self-labeled guidance and ground-truth guidance. We train both models for 100 epochs. For the standard ImageNet32 validation setting in Figure 2a, our method achieves a 17.8% improvement for respective optimal guidance strength of the two methods.
Self-labeled guidance is especially effective for lower values of w .
We observe similar trends for the unbalanced setting in Figure 2b, be it that the overall FID results are slightly higher for both methods.
The improvement increases to 18.7%.
We conjecture this is due to the unbalanced nature of the k -means algorithm [24], and clustering based on the statistics of the overall dataset can potentially lead to more robust performance in unbalanced setting.
Self-labeled comparisons on ImageNet32/64.
We compare our self-labeled guidance with groundtruth labels guidance, which utilizes the technique of classiﬁer-free guidance [19].
We train all experiments for 100 epochs which take about 6 days to converge on four RTX A5000 GPUs.
All hyperparameters are the same between the two methods to make the comparison as fair as possible.
Results on ImageNet32 and ImageNet64 are in Table 2.
Similar to Dhariwal & Nichol [12], we observe that any guidance setting improves considerably over the unconditional & no-guidance model.
Surprisingly, our self-labeled model even outperforms the ground-truth labels by a large gap in terms of FID of 2.9 and 4.7 points respectively.
We hypothesize that the ground-truth taxonomy might be suboptimal for learning generative models and the self-supervised clusters offer a better guidance signal due to better alignment with the visual similarity of the images.
This suggests that the label-conditioned guidance from [19] can be completely replaced by guidance from self-supervision, which would enable guided diffusion models to learn from even larger (unlabeled) datasets than feasible today.
Conclusion
We have explored the potential of self-supervision signals for diffusion models and propose a framework for self-guided diffusion models.
[-FID ↓ IS ↑ Ground-truth labels (32) (cid:55) 10 .
 None (33) (cid:51) 14 .10 .36 .10 .-]By leveraging a feature extraction function and a self-annotation function, our framework provides guidance signals at various image granularities: from the level of holistic images to object boxes and even segmentation masks.
Our experiments indicate that self-supervision signals are an adequate replacement for existing guidance methods that generate images by relying on annotated image-label pairs during training.
Furthermore, both self-boxed and self-segmented approaches demonstrate that we can acquire ﬁne-grained control over the image content, without any ground-truth bounding boxes or segmentation masks.
Ultimately, our goal is to enable the beneﬁts of self-guided diffusion for unlabeled and more diverse datasets at scale, wherein we believe this work is a promising ﬁrst step.
Acknowledgments
The work of DWZ is part of the research programme Perspectief EDL with project number P16project 3, which is ﬁnanced by the Dutch Research Council (NWO) domain Applied and Engineering Sciences (TTW) D Main framework
E
E.1 Correlation between NMI and FID in different feature backbones. . . . . . . . . . E.2 Precision and Recall in ImageNet32/64 dataset . . . . . . . . . . . . . . . . . . .
E.3 Cluster number ablation in self-boxed guidance . .
E.4 Trend visualization of training loss and validation FID . . . . . . . . . . . . . . .
F More experimental details
F.
UNet structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
Training Parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
Dataset preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
LOST, STEGO algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
G More qualitative results
A Background
Diffusion models.
Diffusion models [41; 20] gradually add noise to an image x 0 until the original signal is fully diminished.
By learning to reverse this process one can turn random noise x T into images.
This diffusion process is modeled as a Gaussian process with Markovian structure:
[Equation]
where β 1 , . .
. , β T is a ﬁxed variance schedule on which we deﬁne α t := 1 − β t and α t :=
(cid:81) ts =1 α s .
All latent variables have the same dimensionality as the image x 0 and differ by the proportion of the retained signal and added noise.
Learning the reverse process reduces to learning a denoiser x t ∼ q ( x t | x 0 ) that recovers the original image as ( x t − (1 − α t ) (cid:15) θ ( x t , t )) / √ α t ≈ x 0 .
[20] optimize the network parameters θ by minimizing the error of the noise prediction:
[Equation]
in which (cid:15) ∼ N ( 0 , I ) , x ∈ D is a sample from the training dataset D and the noise prediction function (cid:15) θ ( · ) are encouraged to be as close as possible to (cid:15) .
The standard sampling [20] requires many neural function evaluations to get good quality samples.
Instead, the faster Denoising Diffusion Implicit Models (DDIM) sampler [42] has a non-Markovian sampling process:
[Equation]
where (cid:15) ∼ N ( 0 , I ) is standard Gaussian noise independent of x t .
Classiﬁer-free guidance.
To trade off mode coverage and sample ﬁdelity in a conditional diffusion model, Dhariwal & Nichol [12] propose to guide the image generation process using the gradients of a classiﬁer, with the additional cost of having to train the classiﬁer on noisy images.
Motivated by this drawback, [19] introduce label-conditioned guidance that does not require a classiﬁer.
They obtain a combination of a conditional and unconditional network in a single model, by randomly dropping the guidance signal c during training.
After training, it empowers the model with progressive control over the degree of alignment between the guidance signal and the sample by varying the guidance strength w :
[Equation]
A larger w leads to greater alignment with the guidance signal, and vice versa.
Classiﬁer-free guidance [19] provides progressive control over the speciﬁc guidance direction at the expense of labor-consuming data annotation.
In this paper, we propose to remove the necessity of data annotation using a self-guided principle based on self-supervised learning.
B Extra Method
Self-boxed guidance.
Bounding boxes specify the location of an object in an image [34; 6] and complements the content information of class labels. Our self-boxed guidance approach aims to attain this signal via self-supervised models.
We represent the bounding box as a feature map ( W × H ) rather than coordinates ( [ X, Y, W, H ] ).
We propose the self-annotation function f that obtains a bounding box k s ∈ R W × H by mapping from feature space H to the bounding box space via f ψ ( · ; D ) : R W × H × C
→ R W × H , and inject the guidance signal by concatenating in the channel dimension: x t := concat [ x t , k s ] Usually in self-supervised learning, the derived bounding box is class-agnostic [44; 45].
To inject a self-supervised pseudo label to further enhance the guidance signal, we again resort to clustering to obtain k and concatenate it with the time embedding t := concat [ t, k ] .
To incorporate such guidance, we reformulate the noise prediction function (cid:15) θ as:
[Equation]
in which k s is the self-supervised box guidance obtained by self-annotation functions f ψ , k is the self-supervised image-level guidance from clustering.
The design of f ψ is ﬂexible as long as it obtains self-supervised bounding boxes by f ψ ( · ; D ) : R W × H × C
→ R W × H .
Self-boxed guidance guides the diffusion model by boxes, which speciﬁes the box area in which the object will be generated.
Sometimes, we may need an even ﬁner granularity, e.g., pixels, which we detail next. [Figure]
The difference between them is slightly more prominent for unbalanced data, we conjecture that is because our self-labeled guidance is obtained by clustering based on the statistics of the overall dataset, which can potentially lead to more robust performance in unbalanced setting.
Self-segmented guidance.
Compared to a bounding box, a segmentation mask is a more ﬁnegrained signal.
Additionally, a multichannel mask is more expressive than a binary foregroundbackground mask.
Therefore, we propose a self-annotation function f that acts as a plug-in built on feature g φ ( · ; D ) to extract the segmentation mask k s via function mapping f ψ ( · ; D ) : R W × H × C →
R W × H × K , where K is the number of segmentation clusters.
To inject the self-segmented guidance into the noise prediction function (cid:15) θ , we consider two pathways for injection of such guidance.
We ﬁrst concatenate the segmentation mask to x t in the channel dimension, x t := concat [ x t , k s ] , to retain the spatial inductive bias of the guidance signal.
Secondly, we also incorporate the image-level guidance to further amplify the guidance signal along the channel dimension.
As the segmentation mask from the self-annotation function f ψ already contains image-level information, we do not apply the image-level clustering as before in our selflabeled guidance.
Instead, we directly derive the image-level guidance from the self-annotation result f ψ ( · ) via spatial maximum pooling: R W × H × K → R K , and feed the image-level guidance ˆ k into the noise prediction function via concatenating it with the timestep embedding t := concat [ t, ˆ k ] .
The concatenated results will be sent to every block of the UNet.
In the end, the overall noise prediction function for self-segmented guidance is formulated as:
[Equation]
in which k s is the spatial mask guidance obtained from self-annotation function f , ˆ k is a multi-hot image-level guidance derived from the self-supervised learning mask k s .
We have described three variants of self-guidances by setting the feature extraction function g φ ( · ) , self-annotation function f ψ ( · ) , guidance signal k to an approximate form.
In the end, we arrive at three noise prediction functions (cid:15) θ , which we utilize for diffusion model training and sampling, following the standard guided [19] diffusion approach as detailed in Appendix A.
C Extra results. C.1 Self-boxed Guidance
We report on Pascal VOC and COCO_20K to validate the efﬁcacy of self-boxed guidance.
To obtain class-agnostic object bounding boxes, we use LOST [40] as our self-annotation function f .
We report train FID for Pascal VOC and train/validation FID for COCO_20K. For image-level clustering to attain the guidance signal, we empirically found k =100 works best on both datasets as those datasets are relatively small-scale in images and labels compared to ImageNet.
We train our diffusion model for 800 epochs with input image size 64×64.
See Appendix F for more details.
Self-boxed comparisons on Pascal VOC and COCO_20K. For the ground-truth labels guidance baseline
, we condition on a class embedding.
Since there are now multiple objects per image, we
[Table]
[Figure]
C.2 Self-segmented Guidance
[-Ground-truth labels (32) (cid:55) 
-]
[-None (33) (cid:51) 
-]
Finally, we validate the efﬁcacy of self-segmented guidance on Pascal VOC and COCO-Stuff.
For
COCO-Stuff we follow the split from [15; 21; 10; 46], with a train set of 49,629 images and a validation set of 2,175 images.
Classes are merged into 27 (15 stuff and 12 things) categories.
For self-segmented guidance we apply STEGO [15] as our self-annotation function f .
We set the cluster number to 27 for COCO-Stuff, and 21 for Pascal VOC, following STEGO .
We train all models on images of size 64×64, for 800 epochs on Pascal VOC, and for 400 epochs on COCO-Stuff.
We report the train FID for Pascal VOC and both train and validation FID for COCO-Stuff.
More details on the dataset and experimental setup are provided in Appendix F.
Self-segmented comparisons on Pascal VOC and COCO-Stuff.
We compare against both the ground-truth labels guidance baseline from the previous section and a model trained with ground-truth semantic masks guidance.
The results in Table 4 demonstrate that our self-segmented guidance still outperforms the ground-truth labels guidance baseline on both datasets.
The comparison between ground-truth labels and segmentation masks reveals an improvement in image quality when using the more ﬁne-grained segmentation mask as the condition signal.
But these segmentation masks are one of the most costly types of image annotations that require every pixel to be labeled.
Our self-segmented approach avoids the necessity for annotations while narrowing the performance gap, and more importantly offering ﬁne-grained control over the image layout.
We demonstrate this controllability with examples in Figure 4.
These examples further highlight a robustness against [Table] [Figure] noise in the segmentation masks, which our method acquires naturally due to training with noisy segmentations.
UnlabeledDataset
[Figure] Sampling [Figure]
𝑥 !
$’ 𝑥 ’
We illustrate the pipeline of our framework in Figure 5. [Equation]
[Table]
To assess the correlation between cluster quality and sample ﬁdelity
[Table] we consider the Normalized
Mutual Information (NMI), which is commonly adopted as a mutual information-derived metric to assess clustering quality based on provided ground truth labels. In Figure 6 we plot the connection between NMI and FID. For the label-supervised functions
[Table]
the NMI is unrelated to the FID
[Table] but for the self-supervised functions, NMI and FID are negatively correlated
[Table]
suggesting that NMI — a metric commonly applied to assess the quality of self-supervised methods — is also predictive of the model’s usefulness in our setting
[Table]
E.2 Precision and Recall in ImageNet32/64 dataset
We show the extra results of ImageNet on precision and recall in Table 5.
We follow the evaluation code of precision and recall from ICGAN [8], our self-labeled guidance also outperforms ground-truth labels in precision and remains competitive in recall.
E.3 Cluster number ablation in self-boxed guidance
In table 6, we empirically evaluate the performance when we alter the cluster number in our self-boxed guidance. We ﬁnd the performance will increase from k = 21 to k = 100 , and saturated at k = 100 .
E.4 Trend visualization of training loss and validation FID
We visualize the trend of training loss and validation FID in Figure 7.
[Figure]
F More experimental details
Training details.
For our best results, we train 100 epochs on 4 GPUs of A5000 (24G) in ImageNet.
We train 800/800/400 epochs on 1GPU of A6000 (48G) in Pascal VOC, COCO_20K, COCO-Stuff, respectively.
All qualitative results in this paper are trained in the same setting as mentioned above.
We train and evaluate the Pascal VOC, COCO_20K, COCO-Stuff in image size 64, and visualize them by bilinear up sampling to 256, following [26].
Sampling details.
We sample the guidance signal from the distribution of training set in our all experiments.
For each timestep, we need twice of Number of Forward Evaluation (NFE), we optimize them by concatenating the conditional and unconditional signal along the batch dimension so that we only need one time of NFE in every timestep.
Evaluation details.
We use the common package Clean-FID [30], torch-ﬁdelity [28] for FID, IS calculation, respectively.
For IS, we use the standard 10-split setting, we only report IS on ImageNet, as it might be not an appropriate metric for non object-centric datasets [3].
For checking point, we pick the checking point every 10 epochs by minimal FID between generated sample set and train set.
[Figure]
F.1 UNet structure
Guidance signal injection.
We describe the detail of guidance signal injection in Figure 8.
The injection of self-labeled guidance and self-boxed/segmented guidance is slightly different.
The common part is by concatenation between timestep embedding and noisy input, the concatenated feature will be sent to every block of the UNet.
For the self-boxed/segmented guidance, we not only conduct the information fusion as above, but also incorporate the spatial inductive-bias by concatenating it with input, the concatenated result will be fed into the UNet.
Timestep embedding.
We embed the raw timestep information by two-layer MLP: FC(512, 128) → SiLU → FC(128, 128).
Guidance embedding.
The guidance is in the form of one/multi-hot embedding R K , we feed it into two-layer MLP: FC(K, 256) → SiLU → FC(256, 256), then feed those guidance signal into the
UNet following in Figure 8.
Cross-attention.
In training for non object-centric dataset, we also tokenize the guidance signal to several tokens following Imagen Saharia et al.
[38], we concatenate those tokens with image tokens (can be transposed to a token from typical feature map by R W × H × C
→ R C × WH ), the cross-attention
[36; 4] is conducted by CA ( m , concat [ k , m ] ). Due to the quadratic complexity of transformer [23; 27], we only apply the cross-attention in lower-resolution feature maps. F.2 Training Parameter
[Table]
F.3 Dataset preparation
The preparation of unbalanced dataset.
There are 50,000 images in the validation set of ImageNet with 1,000 classes (50 instances for each).
We index the class from 0 to 999, for each class
c
i , the instance of the class c i is (cid:98) i × 50 / 1000 (cid:99)
=
(cid:98) i/ 200 (cid:99) .
Pascal VOC.
We use the standard split from [40].
It has 12,031 training images.
As there is no validation set for Pascal VOC dataset, therefore, we only evaluate FID on train set.
We sample 10, images and use 10,000 random-croped 64-sized train images as reference set for FID evaluation.
COCO_20K. We follow the split from [40; 44; 25].
COCO_20k is a subset of the COCO trainval dataset, consisting of 19,817 randomly chosen images, used in unsupervised object discovery [40; 44].
We sample 10,000 images and use 10,000 random-croped 64-sized train images as reference set for FID evaluation.
COCO-Stuff.
It has a train set of 49,629 images, validation set of 2,175 images, where the original classes are merged into 27 (15 stuff and 12 things) high-level categories.
We use the dataset split following [15; 21; 10; 46], We sample 10,000 images and use 10,000 train/validation images as reference set for FID evaluation.
F.4 LOST, STEGO algorithms
LOST algorithm details.
We conduct padding to make the original image can be patchiﬁed to be fed into the ViT architecture [13], and feed the original padded image into the LOST architecture using ofﬁcial source code 2 .
LOST can also be utilized in a two-stage approach to provide multi-object, due to its complexity, we opt for only single-object discovery in this paper.
STEGO algorithm details.
We follow the ofﬁcial source code 3 , and apply padding to make the original image can be fed into the ViT architecture to extract the self-segmented guidance signal.
For COCO-Stuff dataset, we directly use the ofﬁcial pretrained weight.
For Pascal VOC, we train STEGO ourselves using the ofﬁcial hyperparameters.
In STEGO ’s pre-processing for the k -NN, the number of neighbors for k -NN is 7.
The segmentation head of STEGO is composed of a two-layer MLP (with ReLU activation) and outputs a 70-dimension feature.
The learning rate is 5 e − 4 , batch size is 64.
G More qualitative results
[Figure]
Guidance signal from training set:
Guidance signal from validation set:
[Figure]
[Figure]
[Figure]
Sampled images are sampled by DDIM [42] with 250 steps.
Guidance strength w is 2.
Firstly, we construct a gallery which is composed of equivalent number of sampled and real images, then we ablate two experiments by querying using sampled image or real images in feature space and image space.
Conclusion: We can easily see, regardless of the feature space or image space, the k -NN query results are always highly semantic similar, and they show the diffusion model is not only to memorize the training data/real images, but also can generalize well to synthesize novel images.
[Figure]
[-P R None (33) 14 .10 .0 .
 [Table]-][Figure]
[Figure]