L ABEL -E FFICIENT S EMANTIC S EGMENTATION WITH D IFFUSION M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Denoising diffusion probabilistic models have recently received much research attention since they outperform alternative approaches, such as GANs, and currently provide state-of-the-art generative performance.
The superior performance of diffusion models has made them an appealing tool in several applications, including inpainting, super-resolution, and semantic editing.
In this paper, we demonstrate that diffusion models can also serve as an instrument for semantic segmentation, especially in the setup when labeled data is scarce.
In particular, for several pretrained diffusion models, we investigate the intermediate activations from the networks that perform the Markov step of the reverse diffusion process.
We show that these activations effectively capture the semantic information from an input image and appear to be excellent pixel-level representations for the segmentation problem.
Based on these observations, we describe a simple segmentation method, which can work even if only a few training images are provided.
Our approach signiﬁcantly outperforms the existing alternatives on several datasets for the same amount of human supervision.
The source code of the project is available online 1 .
I NTRODUCTION
Denoising diffusion probabilistic models (DDPM)
(Sohl-Dickstein et al., 2015; Ho et al., 2020) have recently outperformed alternative approaches to model the distribution of natural images both in the realism of individual samples and their diversity (Dhariwal & Nichol, 2021).
These advantages of
DDPM are successfully exploited in applications, such as colorization (Song et al., 2021), inpainting (Song et al., 2021), super-resolution (Saharia et al., 2021; Li et al., 2021b), and semantic editing (Meng et al., 2021), where DDPM often achieve more impressive results compared to GANs.
So far, however, DDPM were not exploited as a source of effective image representations for discriminative computer vision problems.
While the prior literature has demonstrated that various generative paradigms, such as GANs (Donahue & Simonyan, 2019) or autoregressive models (Chen et al., 2020a), can be used to extract the representations for common vision tasks, it is not clear if
DDPM can also serve as representation learners.
In this paper, we provide an afﬁrmative answer to this question in the context of semantic segmentation.
In particular, we investigate the intermediate activations from the U-Net network that approximates the Markov step of the reverse diffusion process in DDPM.
Intuitively, this network learns to denoise its input, and it is not clear why the intermediate activations should capture semantic information needed for high-level vision problems.
Nevertheless, we show that on the certain diffusion steps, these activations do capture such information, therefore, can potentially be used as image representations for downstream tasks.
Given these observations, we propose a simple semantic segmentation method, which exploits these representations and successfully works even if only a few labeled images are provided.
On several datasets, we show that our DDPM-based segmentation method outperforms the existing baselines for the same amount of supervision.
To sum up, the contributions of our paper are:
We investigate the representations learned by the state-of-the-art DDPM and show that they capture high-level semantic information valuable for downstream vision tasks.
anonymous URL
We design a simple segmentation approach that exploits these representations and outperforms the alternatives in the few-shot operating point.
We compare the DDPM-based representations with their GAN-based counterparts on the same datasets and demonstrate the advantages of the former in the context of semantic segmentation.
R ELATED WORK
In this section, we brieﬂy describe the existing research lines relevant to our work.
Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) are a class of generative models that approximate the distribution of real images by the endpoint of the Markov chain that originates from a simple parametric distribution, typically a standard Gaussian.
Each Markov step is modeled by a deep neural network that effectively learns to invert the diffusion process with a known Gaussian kernel.
(Ho et al., 2020) highlighted the equivalence of diffusion models and score matching (Song & Ermon, 2019; 2020), showing them to be two different perspectives on the gradual conversion of a simple known distribution into a target distribution via the iterative denoising process.
Very recent works (Nichol & Dhariwal, 2021; Dhariwal & Nichol, 2021) have developed more powerful model architectures as well as different advanced objectives for DDPM, which led to the “victory” of DDPM over GANs in terms of generative quality and diversity.
DDPM have been widely used in several applications, including image colorization (Song et al., 2021), super-resolution (Saharia et al., 2021; Li et al., 2021b), inpainting (Song et al., 2021), semantic editing (Meng et al., 2021).
In our work, we demonstrate that one can also successfully use them for semantic segmentation.
Image segmentation with generative models is an active research direction at the moment, however, existing methods are primarily based on GANs.
The ﬁrst line of works (Voynov & Babenko, 2020; Voynov et al., 2021; Melas-Kyriazi et al., 2021) is based on the evidence that the latent spaces of the state-of-the-art GANs have directions corresponding to effects that inﬂuence the foreground/background pixels differently, which allows producing synthetic data to train segmentation models.
However, the approaches from (Voynov & Babenko, 2020; Voynov et al., 2021; MelasKyriazi et al., 2021) are currently able to perform binary segmentation only, and it is not clear if they can be used in the general setup of semantic segmentation.
The second line of works (Zhang et al., 2021; Tritrong et al., 2021; Xu & Zheng, 2021) is more relevant to our study since they are based on the intermediate representations obtained in GANs.
In particular, the method proposed in (Zhang et al., 2021) trains a pixel class prediction model on these representations and conﬁrms their label efﬁciency.
In the experimental section, we compare the method from (Zhang et al., 2021) to our DDPM-based one and demonstrate several distinctive advantages of our solution.
Representations from generative models for discriminative tasks.
The usage of generative models as representation learners has been widely investigated for global prediction (Donahue & Simonyan, 2019; Chen et al., 2020a), and dense prediction problems (Zhang et al., 2021; Tritrong et al., 2021; Xu & Zheng, 2021;
Xu et al., 2021).
While previous works highlighted the practical advantages of these representations, such as out-of-distribution robustness (Li et al., 2021a), generative models as representation learners receive less attention compared to alternative unsupervised methods, e.g., based on contrastive learning (Chen et al., 2020b).
The main reason is probably the difﬁculty of training a high-quality generative model on a complex, diverse dataset.
However, given the recent success of DDPM on Imagenet (Deng et al., 2009), one can expect that this direction will attract more attention in the future.
R EPRESENTATIONS FROM DIFFUSION MODELS
In the following section, we investigate the image representations learned by diffusion models.
First, we provide a brief overview of the DDPM framework.
Then, we describe how to extract features from them and investigate what kind of semantic information these features might capture.
Background.
Diffusion models transform noise x T ∼ N (0 , I ) to the sample x 0 by gradually denoising x T to less noisy samples x t .
Formally, we are given a forward diffusion process:
[Equation]
[Figure]
for some ﬁxed variance schedule β 1 , . . .
, β t .
Importantly, a noisy sample x t can be obtained directly from the data x 0 :
[Equation]
(2) where (cid:15) ∼ N (0 , 1) and α t := 1 − β t , ¯ α t := (cid:81) ts =1 α s .
Pretrained DDPM approximates a reverse process:
[Equation]
In practice, rather than predicting the mean of the distribution in Equation (3), the noise predictor network (cid:15) θ ( x t , t ) predicts the noise component at the step t ; the mean is then a linear combination of this noise component and x t .
The covariance predictor Σ θ ( x t , t ) can be either a ﬁxed set of scalar covariances or learned as well (the latter was shown to improve the quality of models (Nichol &
Dhariwal, 2021)).
The denoising model (cid:15) θ ( x t , t ) is typically parameterized by different variants of the UNet architecture (Ronneberger et al., 2015), and in our experiments we investigate the state-of-the-art one proposed in (Dhariwal & Nichol, 2021).
Extracting representations.
For a given real image x 0 ∈ R H × W × 3 , one can compute T sets of activation tensors from the noise predictor network (cid:15) θ ( x t , t ) .
Formally for t = T . . .
1 we ﬁrst corrupt x 0 by adding a Gaussian noise: x t = √ ¯ α t x 0 + √ 1 − ¯ α t (cid:15) .
The noisy x t is used as an input of (cid:15) θ ( x t , t ) parameterized by the UNet model.
The UNet’s intermediate activations are then upsampled to H × W with bilinear interpolation.
This allows treating them as pixel-level representations of x 0 .
3. R EPRESENTATION ANALYSIS
We analyze the representations produced by the noise predictor (cid:15) θ ( x t , t ) for different t .
We consider the state-of-the-art DDPM checkpoints trained on the LSUN-Bedroom and FFHQ datasets 2 .
The intermediate activations from the noise predictor capture semantic information.
For this experiment, we take a few images from the LSUN-Bedroom and FFHQ datasets and manually assign each pixel to one of the 28 and 34 semantic classes, respectively.
Our goal is to understand whether the pixel-level representations produced by DDPM effectively capture the information about semantics.
To this end, we train a multi-layer perceptron (MLP) to predict the pixel semantic label from its features produced by one of the 18 UNet decoder blocks on a speciﬁc diffusion step t .
Note that we consider only the decoder activations because they also aggregate the encoder ones through the skip connections.
MLP is trained on 20 images and evaluated on 20 hold-out ones.
The predictive performance is measured in terms of mean IoU.
[Figure] [Figure]
The predictive performance for small-sized objects starts growing later in the reverse process.
The deeper blocks (e.g., 2 , 4 ) are more informative for larger objects and the shallow blocks (e.g., 10 , 12 ) are for smaller ones.
A similar evaluation for other datasets is provided in Appendix A.
The evolution of predictive performance across the different blocks and diffusion steps t is presented in Figure 2.
The blocks are numbered from the deep to shallow ones.
Figure 2 shows that the discriminability of the features produced by the noise predictor (cid:15) θ ( x t , t ) varies for different blocks and diffusion steps.
In particular, the features corresponding to the later steps of the reverse diffusion process typically capture semantic information more effectively.
In contrast, the ones corresponding to the early steps are generally uninformative.
Across different blocks, the features produced by the layers in the middle of the UNet decoder appear to be the most informative on all diffusion steps.
Additionally, we split the LSUN-Bedroom semantic classes into small-sized and big-sized based on the average area in the annotated dataset.
Then, we evaluate mean IoU for these classes independently across the different UNet blocks and diffusion steps.
The results are presented in Figure 3.
As expected, for big-sized objects, the predictive performance starts growing earlier in the reverse process.
The shallow blocks ( 10 , 12 ) are more informative for small-sized objects, while the deeper ones ( 2 , 4 ) are for big-sized ones.
Figure 2 implies that for certain UNet blocks and diffusion steps, similar DDPM-based representations correspond to the pixels of the same semantics.
Figure 1 shows the k-means clusters ( k =5 ) formed by the features extracted from the blocks { 6 , 8 , 10 , 12 } on the diffusion steps { 50 , 250 , 450 , 650 , 850 } , and conﬁrms that clusters can span coherent semantic objects and objectparts.
In the block B =6 , the features correspond to coarse semantic masks, e.g., they cannot discriminate between various face parts.
On the other extreme, the features from B =12 do not exhibit semantic meaningness, reafﬁrming the predictive performance behavior from Figure 2.
The block
B =8 appears to be a “sweet spot” demonstrating ﬁne-grained semantic fragmentation.
Across different diffusion steps, the most meaningful features correspond to the later ones.
We attribute this behavior to the fact that on the earlier diffusion steps, the global structure of a DDPM sample has not emerged yet, therefore, it is hardly possible to predict segmentation masks at this stage.
This
[Table]
[Figure]
intuition is qualitatively conﬁrmed by the masks in Figure 1.
For t =850 , the masks poorly reﬂect the content of actual images, while for smaller values of t , the masks and images are semantically coherent.
3. DDPM- BASED REPRESENTATIONS FOR FEW - SHOT SEMANTIC SEGMENTATION
The potential effectiveness of the intermediate DDPM activations observed above implies their usage as image representations for dense prediction tasks.
Figure 4 schematically presents a simple approach for image segmentation, which exploits the discriminability of these representations.
In more detail, we consider a few-shot semi-supervised setup, when a large number of unlabeled images { X 1 , . . .
, X N } ⊂ R H
× W × 3 from the particular domain are available, and only for n training images { X 1 , . .
X n } ⊂ R H
× W × 3 the groundtruth K -class semantic masks { Y 1 , . . .
, Y n }
⊂ R H × W ×{ 1 ,...,K } are provided.
As a ﬁrst step, we train a diffusion model on the whole { X 1 , . . .
, X N } in an unsupervised manner.
Then, this diffusion model is used to extract the pixel-level representations of the labeled images using the subset of the UNet blocks and diffusion steps t .
In this work, we use the representations from the middle blocks B = { 6 , 7 , 8 , 9 } of the UNet decoder and later steps t = { 50 , 150 , 250 , 350 } of the reverse diffusion process.
These blocks and time steps are motivated by the insights from
Section 3.1 but intentionally not tuned for each dataset.
While the feature extraction at the particular time step is stochastic, we ﬁx the noise for all timesteps t and ablate this in Section 4.1.
The extracted representations from all blocks B and steps t are upsampled to the image size and concatenated, forming the feature vectors for all pixels of the training images.
The overall dimension of the pixel-level representations is 8192 .
Then, following (Zhang et al., 2021), we train an ensemble of independent multi-layer perceptrons (MLPs) on these feature vectors, which aim to predict a semantic label of each pixel available for training images.
We adopt the ensemble conﬁguration and training settings from (Zhang et al., 2021) and exploit them across all other methods in our experiments.
We describe these settings in more detail in Appendix D.
To segment a test image, we extract its DDPM-based pixel-wise representations and use them to predict the pixel labels by the ensemble.
The ﬁnal prediction is obtained by majority voting.
E XPERIMENTS
This section experimentally conﬁrms the advantage of the DDPM-based representations for the semantic segmentation problem.
We start from a thorough comparison to the existing alternatives and then dissect the reasons for the DDPM success by additional analysis.
[Figure] [Figure]
Datasets.
In our evaluation, we mainly work with the “bedroom”, “cat” and “horse” categories from LSUN (Yu et al., 2015) and FFHQ-256 (Karras et al., 2019).
As a training set for each dataset
[Figure]
we consider several images for which the ﬁne-grained semantic masks annotations are collected following the protocol from (Zhang et al., 2021).
For each dataset, the professional assessor was hired to annotate train and test samples.
The semantic classes used to segment the images from each domain are listed in Appendix C.1.
Additionally, we consider the ADE-Bedroom dataset, which is a subset of the ADE20K dataset (Zhou et al., 2018), where we extracted only images of bedroom scenes with 30 most frequent classes.
In contrast to other datasets, ADE-Bedroom has publicly available annotations and a sizable evaluation set.
We denote the collected datasets as Bedroom-28, FFHQ-34, Cat-15, Horse-21, ADE-Bedroom30 , where the number corresponds to the number of semantic classes in the annotation.
We provide the details of these datasets in Table 1.
Methods.
In the evaluation, we compare our method (denoted as DDPM ) to several prior approaches, which tackle the few-shot semantic segmentation setup.
First, we describe the baselines that produce a large set of annotated synthetic images to train a segmentation model:
• DatasetGAN (Zhang et al., 2021) — this method exploits the discriminability of pixel-level features produced by GANs.
In more detail, assessors annotate a few GAN-produced images.
Then, the latent codes of these images are used to obtain the intermediate generator activations, which are considered as pixel-level representations.
Given these representations, a classiﬁer is trained to predict a semantic label for each pixel.
This classiﬁer is then used to label new synthetic GAN images, which, for their part, serve as a training set for the DeepLabV3 seg-
[Table]
mentation model (Chen et al
For each dataset, we increase the number of synthetic images until the performance on the validation set is not saturated.
According to (Zhang
et al.,
), we also remove 10% of synthetic samples with the most uncertain predictions.
• DatasetDDPM mirrors the DatasetGAN baseline with the only difference that GANs are replaced with DDPMs.
We include this baseline to compare the GAN-based and DDPM-based representations in the same scenario.
Note that our segmentation method described in Section 3.2 is more straightforward compared to
DatasetGAN and DatasetDDPM since it does not require auxiliary steps of the synthetic dataset generation and training the segmentation
model on it.
Then, we consider a set of baselines that allow extracting intermediate activations from the real
images
directly and use them as
pixel
level
representations similarly to our method.
In contrast to
DatasetGAN
and DatasetDDPM, these methods can potentially be
beneﬁcial
due to the
absence
of
the domain gap between real and synthetic images.
• SwAV (Caron et al., 2020) — one of the recent state-of-the-art self-supervised learning methods.
All models are trained on the corresponding LSUN datasets and FFHQ-256 on the same image resolution as DDPM.
• GAN Inversion employs the state-of-the-art GAN inversion method (Tov et al., 2021) to obtain the latent codes for real images.
In more detail, we map the annotated real images to the GAN latent space, which allows computing the intermediate generator activations and using them as pixel-level representations.
• GAN Encoder — while GAN Inversion struggles to reconstruct images from LSUN domains, we also consider the activations of the pretrained GAN encoder used for GAN Inversion.
• VDVAE
(Child, 2021) — state-of-the-art autoencoder model.
The intermediate activations are extracted from both encoder and decoder and concatenated.
While there are no pretrained models on the LSUN datasets, we evaluate this model only on the publicly available checkpointon FFHQ-256.
Note that VAEs are still signiﬁcantly inferior to GANs and DDPMs on LSUN.
• ALAE (Pidhorskyi et al., 2020) — another autoencoder model, which adopts StyleGANv generator and adds an encoder network to the adversarial training.
We extract features from the encoder model.
In our evaluation, we use publicly available models on the LSUN-Bedroom and FFHQ-1024 4 .
Generative Models.
In our experiments, we use the state-of-the-art StyleGAN2 (Karras et al., 2020) models for the GAN-based baselines and the state-of-the-art pretrained ADMs (Dhariwal & Nichol
2021) for our DDPM-based method
(Dhariwal & Nichol, 2021) does not provide a pretrained model for FFHQ-256, so we train it ourselves using the ofﬁcial implementation 5 .
Since there are no pretrained generative models on ADE20K, for evaluation on the ADE-Bedroom-30 dataset, we use the models (including the baselines) pretrained on LSUN-Bedroom b ac kg r oundp e r s on l e g m u zz l e t
h
i gh
b a rr e l h ea d t a il ea r n ec k s a dd l e b ac
k s hou
l d e r m a n e c h e s t l e gp r o t ec ti on f o r e l o c k hoo f e y
e
no s t r il b r i d l e
[Figure]
I o U
[Figure]
Main results.
The comparison of the methods in terms of the mean IoU measure is presented in
Table 2.
The results are averaged over 5 independent runs for different data splits.
We also report per class IoUs for Cat-15 and Horse-21 in Figure 6.
Additionally, we provide several qualitative examples of segmentation with our method in Figure 5.
Below we highlight several key observations:
• The proposed method based on the DDPM representations outperforms the alternatives on the more complex LSUN and ADE datasets, while on the simpler FFHQ dataset, it is comparable to DatasetGAN and VDVAE.
• DatasetDDPM outperforms its counterpart DatasetGAN on most benchmarks.
Note that both these methods use the DeepLabV3 network, trained on the same amount of synthetics produced by the corresponding generative model.
We attribute this superiority to the higher quality of DDPM synthetics, therefore, the smaller domain gap between synthetic and real data.
• On most datasets, DDPM outperforms the DatasetDDPM competitor.
We provide an additional experiment to investigate this in the discussion section below.
• The SwAV baseline underperforms compared to the DDPM-based segmentation.
We attribute this behavior to the fact that this baseline is trained in the discriminative fashion and can suppress the details, which are needed for ﬁne-grained semantic segmentation.
This result is consistent with the recent ﬁndings in (Cole et al., 2021), which shows that the state-of-the-art contrastive methods produce representations, which are suboptimal for ﬁne-grained problems.
• The GAN Inversion method performs poorly on the LSUN and ADE datasets.
For these relatively complex domains, this method cannot invert most real images reliably.
On the simpler FFHQ dataset, GAN Inversion is more competitive but is still inferior to most other methods.
• VDVAE demonstrates comparable performance on FFHQ-34, while ALAE is inferior to most other methods.
Note that it is not generally clear if autoencoders can reach GAN/DDPM generative performance on the complex LSUN datasets and hence provide strong segmentation performance on these domains.
Overall, the proposed DDPM-based segmentation outperforms the baselines that exploit alternative generative models and also the baseline trained in the discriminative fashion.
This result highlights the potential of using the state-of-the-art DDPM as strong unsupervised representation learners.
4. D ISCUSSION
The effect of training on real data .
The proposed DDPM method is trained on the annotated real images, while DatasetDDPM and DatasetGAN are trained on the synthetic ones, which are typically less natural, diverse, and can lack objects of particular classes.
Moreover, synthetic images are harder for human annotation since they might have some distorted objects that are difﬁcult to assign to a particular class.
In the following experiment, we quantify the performance drop caused by training on real or synthetic data.
Speciﬁcally, Table 3 reports the performance of the DDPM approach trained on real, DDPM-produced and GAN-produced annotated images.
As can be seen, training on real images is very beneﬁcial on the domains where the ﬁdelity of generative models is still relatively low, e.g., LSUN-Cat, which indicates that annotated real images are more reliable source of supervision.
Moreover, if the DDPM method is trained on synthetic images, its performance becomes on par with DatasetDDPM.
On the other hand, when trained on GAN-produced samples, DDPM signiﬁcantly outperforms DatasetGAN.
We attribute this to the fact that DDPMs provide more semantically-valuable pixel-wise representations compared to GANs.
[Table] [Table]
[Table]
to determinstic one.
Sample-efﬁciency.
In this experiment, we evaluate the performance of our method when it utilizes less annotated data.
We provide mIoU for four datasets in Table 4.
Importantly, DDPM is still able to outperform most baselines Table 2 using signiﬁcantly less supervision.
The effect of stochastic feature extraction.
Here, we investigate whether our method can beneﬁt from the stochastic feature extraction described in Section 3.2.
We consider the deterministic case when the noise (cid:15) ∼ N (0 , I ) is sampled once and used in (2) to obtain x t for all timesteps t during both training and evaluation.
Then, we compare it to the following stochastic options:
First, different (cid:15) t are sampled for different timesteps t and shared during the training and evaluation.
Second, one samples different noise for all timesteps at each training iteration; during the evaluation the method also uses unseen noise samples.
The results are provided in Table 5.
As one can see, the difference in the performance is marginal.
We attribute this behavior to the following reasons:
• Our method uses later t of the reverse diffusion process where the noise magnitude is low.
• Since we exploit the deep layers of the UNet model, the noise might not affect the activations from these layers signiﬁcantly.
C ONCLUSION
This paper demonstrates that DDPM can serve as representation learners for discriminative computer vision problems.
Compared to GANs, diffusion models allow for a straightforward computation of these representations for real images, and one does not need to learn an additional encoder, which maps images to the latent space.
This DDPM’s advantage and superior generative quality provide state-of-the-art performance in the few-shot semantic segmentation task.
The notable restraint of the
DDPM-based segmentation is a requirement of high-quality diffusion models trained on the dataset at hand, which can be challenging for complex domains, like Imagenet or MSCOCO.
However, given the rapid research progress on DDPM, we expect they will reach these milestones in the nearest future, thereby extending the range of applicability for the corresponding representations.
A E VOLUTION OF PREDICTIVE PERFORMANCE
[Figure]
[Figure]
B D ATASET DDPM & D ATASET GAN S ATURATION
[Figure]
[Table]
[Figure]
[Table] [Table]
[Figure]
C D ATASET D ETAILS
[Figure]
Bedroom-28 : [bed, footboard, headboard, side rail, carpet, ceiling, chandelier, curtain, cushion, ﬂoor, table, table top, picture, pillow, lamp column, lamp shade, wall, window, curtain rod, window frame, chair, picture frame, plinth, door, pouf, wardrobe, plant, table staff]
FFHQ-34 : [background, head, cheek, chin, ear, helix, lobule, botton lid, eyelashes, iris, pupil, sclera, tear duct, top lid, eyebrow, forehead, frown, hair, sideburns, jaw, moustache, inferior lip, oral comisure, superior lip, teeth, neck, nose, ala of nose, bridge, nose tip, nostril, philtrum, temple, wrinkles]
Cat-15 : [background, cat, back, belly, chest, leg, paw, head, ear, eye, mouth, tongue, tail, nose, whiskers, neck]
Horse-21 : [background, person, back, barrel, bridle, chest, ear, eye, forelock, head, hoof, leg, mane, muzzle, neck, nostril, tail, thigh, saddle, shoulder, leg protection]
ADE-Bedroom-30 : [wall, bed, ﬂoor, table, lamp, ceiling, painting, windowpane, pillow, curtain, cushion, door, chair, cabinet, chest, mirror, rug, armchair, book, sconce, plant, wardrobe, clock, light, ﬂower, vase, fan, box, shelf, television]
C. C LASS S TATISTICS
In Figure 9, we report the statistics of classes computed over annotated real images as well as annotated synthetic images produced by GAN and DDPM.
Figure 9 shows that GAN-based and
DDPM-based synthetics have comparable coverage of most semantic classes.
D T RAINING SETUP
The ensemble of MLPs consists of 10 independent models.
Each MLP is trained for ∼ 4 epochs using the Adam optimizer (Kingma & Ba, 2015) with 0 .
001 learning rate.
The batch size is 64 .
This setting is used for all methods and datasets.
MLP architecture.
We adopt the MLP architecture from (Zhang et al., 2021).
Speciﬁcally, we use an MLP with two hidden layers with ReLU nonlinearity and batch normalization.
The side of hidden layers are 128 and 32 for the dataset with a number of classes less than 30 , and 256 andfor others.
Also, we evaluate the performance of the proposed method for twice wider / deeper MLPs on the
Bedroom-28 and FFHQ-34 datasets and do not observe any noticeable difference, see Table 7.
[Table]
E R OBUSTNESS TO INPUT CORRUPTIONS
In this experiment, we investigate the robustness of DDPM-based representations.
First, we learn pixel classiﬁers on the clean images using the DDPM and SwAV representations on the Bedroom28 dataset.
Then, 18 diverse corruption types, adopted from (Hendrycks & Dietterich, 2019), are applied to test images.
Each corruption has ﬁve levels of severity.
In Figure 10, we provide mean
IoUs computed over all corruption types for 1 , 3 , 5 levels of severity, denoted as “weak”, “medium” and “strong”, respectively.
One can observe that the proposed DDPM-based method demonstrates higher robustness and preserves its advantage over the SwAV model even for severe image distortions.
[Figure]
DDPM demonstrates robustness
[Equation] [Figure]