Palette: Image-to-Image Diffusion Models
Google Research
Colorization
Inpainting
Uncropping
JPEG Restoration
I npu t
O u t pu t
O r i g i n a l
Abstract
We introduce Palette, a simple and general framework for image-to-image translation using conditional diffusion models.
Palette models trained on four challenging image-to-image translation tasks (colorization, inpainting, uncropping, and JPEG restoration) outperform strong GAN and regression baselines and bridge the gap with natural images in terms of sample quality scores.
This is accomplished without task-speciﬁc hyper-parameter tuning, architecture customization, or any auxiliary loss, demonstrating a desirable degree of generality and ﬂexibility.
We uncover the impact of an L 2 vs. L 1 loss in the denoising diffusion objective on sample diversity, and demonstrate the importance of self-attention through empirical architecture studies.
Importantly, we advocate a uniﬁed evaluation protocol based on ImageNet, with human evaluation and sample quality scores (FID, Inception Score, Classiﬁcation Accuracy of a pre-trained ResNet-50, and Perceptual Distance against original images).
We expect this standardized evaluation protocol to play a critical role in advancing image-to-image translation research.
Finally, we show that a generalist, multi-task Palette model performs as well or better than task-speciﬁc specialist counterparts.
Check out https://bit.ly/palette-diffusion for more details.
∗ Equal Contribution
Introduction
Many problems in vision and image processing can be formulated as image-to-image translation.
Examples include restoration tasks, like super-resolution, colorization, and inpainting, as well as pixel-level image understanding tasks, such as instance segmentation and the estimation of intrinsic images.
Many such tasks, like those in Fig.
1, are complex inverse problems, where multiple output images are consistent with a single input.
A natural approach to image-to-image translation is to learn the conditional distribution of output images given the input, using deep generative models that can capture multi-modal distributions in the high-dimensional space of images.
Generative Adversarial Networks (GANs) [Goodfellow et al., 2014, Radford et al., 2015] have emerged as the model family of choice for many image-to-image tasks [Isola et al., 2017a], as they are capable of generating high ﬁdelity outputs, are broadly applicable, and support efﬁcient sampling.
Nevertheless, GANs can be challenging to train Arjovsky et al.
[2017], Gulrajani et al.
[2017], and often drop modes in the output distribution Metz et al.
[2016], Ravuri and Vinyals [2019].
Autoregressive Models [van den Oord et al., 2016, Parmar et al., 2018], VAEs [Kingma and Welling, 2013, Vahdat and Kautz, 2020], and Normalizing Flows [Dinh et al., 2016, Kingma and Dhariwal, 2018] have also seen success in speciﬁc applications, but arguably, have not established the same level of sample quality and generality as GANs.
Diffusion and score-based models [Sohl-Dickstein et al., 2015, Song and Ermon, 2020, Ho et al., 2020] have received a surge of recent interest Cai et al.
[2020], Song et al.
[2021], Hoogeboom et al.
[2021], Vahdat et al.
[2021], Kingma et al.
[2021], Austin et al.
[2021], resulting in several key advances in modeling continuous data.
On speech synthesis, diffusion models have achieved human evaluation scores on par with SoTA autoregressive models [Chen et al., 2021a,b, Kong et al., 2021].
On the class-conditional ImageNet generation challenge, they have outperformed the strongest
GAN baselines in terms of FID scores [Dhariwal and Nichol, 2021, Ho et al., 2021].
On image superresolution, they have delivered impressive face enhancement results, outperforming GANs [Saharia et al., 2021a].
Despite these results, diffusion models have not been applied to a broad family of tasks, and it is not clear whether they can rival GANs in offering a versatile and general solution to the problem of image-to-image translation.
This paper investigates the general applicability of Palette , our implementation of image-to-image diffusion models, to a suite of distinct and challenging tasks, namely colorization, inpainting, uncropping ( a.k.a.
outpainting or extrapolation), and JPEG restoration ( a.k.a.
JPEG artifact removal) (see
Fig. 1, 2).
Our empirical results suggest that Palette, with no task-speciﬁc architecture customization, nor changes to hyper-parameters or the loss, delivers high-ﬁdelity outputs across all four tasks.
It outperforms task-speciﬁc baselines and our own strong regression model, which uses an identical neural architecture.
Importantly, we show that a single generalist Palette model, trained on colorization, inpainting and JPEG restoration, outperforms a task-speciﬁc JPEG restoration model and achieves competitive performance on the other tasks.
We study key elements of Palette, including the loss function and the neural net architecture.
We ﬁnd that while L 2 [Ho et al., 2020] and L 1 [Chen et al., 2021a] losses in the denoising objective yield similar sample-quality scores, L 2 leads to a higher degree of diversity in model samples, whereas
L 1 [Chen et al., 2021a] produces more conservative outputs.
We also ﬁnd that removing self-attention layers from the U-Net architecture of Palette, to build a fully convolutional model, hurts performance.
Finally, we advocate a standardized evaluation protocol for inpainting, uncropping, and JPEG restoration based on ImageNet [Deng et al., 2009] and report sample quality scores for several baselines.
We expect this benchmark to help advance image-to-image translation research.
In summary:
We demonstrate the versatility and applicability of conditional diffusion models to image-to-image translation.
Palette achieves SoTA in colorization and outperforms strong GAN and regression baselines in inpainting, uncropping, and JPEG restoration.
Palette outputs bridge the gap with natural images in sample quality scores, without using any task-speciﬁc tuning or customization.
We recognize the impact of the L 1 vs. L 2 loss in the denoising objective of diffusion models on sample diversity.
We also demonstrate the importance of self-attention for these translation tasks.
We propose a standardized evaluation protocol for image translation tasks based on ImageNet and report sample quality scores for multiple baselines including a strong regression baseline and prior work for each task.
We show that a multi-task Palette model performs as well or better than task-speciﬁc counterparts.
Related work
Our work is inspired by Pix2Pix [Isola et al., 2017a], which explored myriad of image-to-image translation tasks with GANs.
Other GAN-based techniques have been proposed for various imageto-image problems such as unpaired translation [Zhu et al., 2017a], unsupervised cross-domain generation
[Taigman et al., 2016], multi-domain translation [Choi et al., 2018], few shot translation [Liu et al., 2019] and many more.
Nevertheless, existing GAN models are sometimes unsuccessful in holistically translating images with consistent structural and textural regularity.
Diffusion models [Sohl-Dickstein et al., 2015] recently emerged with impressive results on image generation [Ho et al., 2020, 2021, Dhariwal and Nichol, 2021], audio synthesis [Chen et al., 2021a, Kong et al., 2020], and image super-resolution
[Saharia et al., 2021a, Kadkhodaie and Simoncelli, 2021], as well as unpaired image-to-image translation [Sasaki et al., 2021] and image editing [Meng et al., 2021, Sinha et al., 2021].
Our conditional diffusion models build on these recent advances, showing versatility on a suite of image-to-image translation tasks.
Early inpainting approaches Bertalmio et al.
[2000], Barnes et al.
[2009], He and Sun [2012], Hays and Efros [2007], Roth and Black [2005] work well on textured regions but often fall short in generating semantically consistent structure.
GANs are now widely used, but often require auxiliary objectives on structures, context, edges, contours and hand-engineered features Iizuka et al.
Yu et al.
[2018a, 2019], Nazeri et al.
[2019], Yi et al.
[2020], Liu et al.
[2020], Kim et al.
[2021b], and a lack of stochasticity and diversity in their outputs has been observed [Zheng et al., 2019, Zhao et al., 2021].
Generic diffusion models for image generation can be used for linear inverse tasks like inpainting [Sohl-Dickstein et al., 2015, Song et al., 2020, Kadkhodaie and Simoncelli, 2021], but we posit that conditional models trained for inpainting will be superior.
Image uncropping (a.k.a.
outpainting) is considered more challenging than inpainting as it entails generating open-ended content with less context.
Early methods relied on retrieval [Kopf et al., 2012, Wang et al., 2014,
Shan et al., 2014].
GAN-based methods are now predominant Teterwak et al.
[2019], but are often domain-speciﬁc Yang et al.
[2019b], Bowen et al.
[2021], Wang et al.
[2019a], Cheng et al.
Lin et al.
We show that conditional diffusion models trained on large datasets reliably address both inpainting and uncropping on a wide range of image domains.
Colorization is a well-studied image-to-image task Kumar et al.
[2021], Guadarrama et al.
Royer et al.
[2017], Ardizzone et al.
[2019], requiring a degree of scene understanding, which makes it a natural choice for self-supervised learning [Larsson et al., 2016].
Challenges include diverse colorization [Deshpande et al., 2017], respecting semantic categories [Zhang et al., 2016], and producing high-ﬁdelity color Guadarrama et al.
While some prior work makes use of specialized auxiliary classiﬁcation losses, we ﬁnd that generic image-to-image diffusion models work well without task-speciﬁc specialization.
JPEG restoration (aka.
JPEG artifact removal) is a nonlinear inverse problem for restoring realistic textures and removing compression artifacts.
Dong et al.
[2015] applied deep CNN architectures for JPEG restoration, and Galteri et al.
[2017, 2019] successfully applied GANs for artifact removal, but they have been restricted to quality factors of above 10.
We show the effectiveness of Palette in removing compression artifacts for quality factors as low as 5.
Multi-task training is a relatively under-explored area in image-to-image translation.
A few papers [Qian et al., 2019, Yu et al., 2018b] have focused on simultaneous training over multiple tasks, but they are primarily focused on enhancement tasks like deblurring, denoising, and super-resolution, and they use smaller, modular networks.
Several works have also dealt with simultaneous training over multiple degradations on a single task e.g., multi-scale super-resolution
[Kim et al., 2016], and JPEG restoration on multiple quality factors [Galteri et al., 2019, Liu et al., 2018b].
With Palette we take a ﬁrst step toward building multi-task image-to-image diffusion models for a wide variety of tasks.
Palette
Diffusion models [Sohl-Dickstein et al., 2015, Ho et al., 2020] convert samples from a satandard
Gaussian distribution into samples from an empirical data distribution through an iterative denoising process.
Conditional diffusion models [Chen et al., 2021a, Saharia et al., 2021b] make the denoising process conditional on an input signal.
Image-to-image diffusion models are conditional diffusion models of the form p ( y | x ) , where both x and y are images, e.g., x is a grayscale image and y is a color image.
These models have been applied to image super-resolution [Saharia et al., 2021a, Nichol and Dhariwal, 2021].
We study the general applicability of image-to-image diffusion models on a broad set of tasks.
For a detailed treatment of diffusion models, please see Appendix A. Here, we brieﬂy discuss the denoising loss function.
Given a training output image y , we generate a noisy version (cid:101) y , and train a neural network f θ to denoise (cid:101) y given x and a noise level indicator γ , for which the loss is
[Equation]
(cid:124) (cid:123)(cid:122)
[Equation]
Chen et al.
[2021a] and Saharia et al.
[2021a] suggest using the L 1 norm, i.e., p = 1 , whereas the standard formulation is based on the usual L 2 norm [Ho et al., 2020].
We perform careful ablations below, and analyze the impact of the choice of norm.
We ﬁnd that L 1 yields signiﬁcanlty lower sample diversity compared to L 2 .
While L 1 may be useful, to reduce potential hallucinations in some applications, here we adopt L 2 to capture the output distribution more faithfully.
Architecture.
Palette uses a standard U-Net architecture [Ho et al., 2020] with several modiﬁcations inspired by recent work [Song et al., 2021, Saharia et al., 2021a, Dhariwal and Nichol, 2021].
The network architecture is based on the 256 × 256 class-conditional U-Net model of Dhariwal and
Nichol [2021].
The two main differences between our architecture and theirs are (i) absence of class-conditioning, and (ii) additional conditioning of the source image via concatenation, following
Saharia et al.
[2021a].
Evaluation protocol
Evaluating image-to-image translation models is challenging.
Prior work on colorization [Zhang et al., 2016, Guadarrama et al., 2017, Kumar et al., 2021] has relied on FID scores and human evaluation for model comparison.
Tasks like inpainting [Yu et al., 2019, 2018a] and uncropping [Teterwak et al., 2019, Wang et al., 2019b] have often heavily relied on qualitative evaluation.
For many tasks, such as JPEG restoration [Dong et al., 2015, Liu et al., 2018b, Galteri et al., 2019], it has been common to use reference-based pixel-level similarity scores such as PSNR and SSIM.
It is also notable that many tasks lack a standardized dataset for evaluation, e.g., different test sets with method-speciﬁc splits are used for evaluation.
We propose a uniﬁed evaluation protocol for inpainting, uncropping, and JPEG restoration on
ImageNet [Deng et al., 2009], due to its scale, diversity, and public availability.
For inpainting and uncropping, existing work has primarily relied on Places2 dataset [Zhou et al., 2017] for evaluation.
Hence, we also use a standard evaluation setup on Places2 for these tasks.
Speciﬁcally, we advocate the use of ImageNet ctest10k split proposed by Larsson et al.
[2016] as a standard subset for benchmarking of all image-to-image translation tasks on ImageNet.
We also introduce a similar category balanced 10,950 image subset of Places2 validation set called places10k .
We further advocate the use of automated metrics that capture both image quality and diversity, in addition to controlled human evaluation.
We avoid the use of pixel-level metrics such as PSNR and SSIM since such metrics are not reliable measures of sample quality for difﬁcult tasks that require hallucination, like recent super-resolution work, where [Ledig et al., 2017, Dahl et al., 2017, Menon et al., 2020] observe that PSNR and SSIM tend to prefer blurry regression outputs which do not correlate well with human judgement.
We use four automated quantitative measures of sample quality for image-to-image translation:
Inception Score (IS) [Salimans et al., 2017]; Fréchet Inception Distance (FID) ; Classiﬁcation
Accuracy (CA) (top-1) of a pre-trained ResNet-50 classiﬁer; and a simple measure of Perceptual
Distance (PD) , i.e., Euclidean distance in Inception-v1 feature space.
In order to encourage and facilitate benchmarking on our proposed subsets, we release our model outputs on these benchmarks together with other data such as the inpainting masks used.
2 More details regarding our evaluation setup can be found in Appendix C.5.
For some tasks, we also assess sample diversity through pairwise SSIM and LPIPS scores between multiple model outputs.
Sample diversity is challenging and has been a key limitation of many existing GAN-based methods [Zhu et al., 2017b, Yang et al., 2019a].
The ultimate evaluation of image-to-image translation models is human evaluation ; i.e., whether or not humans can discriminate model outputs from natural images.
To this end we use 2-alternative forced choice (2AFC) trials to evaluate the perceptual quality of model outputs against the original images from which we obtained the test inputs ( c.f., the Colorization Turing Test
[Zhang et al., 2016]).
We summarize the results in terms of the fool rate , the percentage of human raters who select model outputs over the original natural images when they were asked “Which image would you guess is from a camera?”.
(See Appendix C for details.)
Experiments
We study the application of Palette to a suite of challenging image-to-image translation tasks:
Colorization transforms an input grayscale image to a plausible color image.
Inpainting ﬁlls in user-speciﬁed masked regions of an image with realistic content.
Uncropping extends an input image along one or more directions to enlarge the image.
JPEG restoration corrects for JPEG compression artifacts, restoring plausible image detail.
We evaluate Palette on these tasks without task-speciﬁc hyper-parameter tuning, architecture customization, or any auxiliary loss function.
Inputs and outputs for all tasks are represented as RGB 256 × 256 images.
Each of these tasks presents its own unique challenges.
Colorization entails some representation of objects, segmentation and layout, often with long-range image dependencies.
Inpainting is challenging with large masks and diverse image datasets comprising cluttered scenes.
Outpainting is widely considered even more challenging than inpainting as there is less surrounding context to constrain semantically meaningful generation.
While the other tasks are linear inverse problems, JPEG restoration is a non-linear inverse problem; it requires a good local model of natural image statistics to detect and correct compression artifacts.
While previous work has studied these problems extensively, it is rarely the case that a model with no task-speciﬁc engineering achieves strong performance in all tasks, beating strong task-speciﬁc GAN and regression baselines.
Palette uses an L 2 loss for the denoising objective, unless otherwise speciﬁed.
More implementation details can be found in Appendix B.
5.1 Colorization
While prior works [Zhang et al., 2016, Kumar et al., 2021] have adopted LAB or YCbCr color spaces to represent output images for colorization, we use the RGB color space to maintain generality across tasks.
Preliminary experiments indicated that Palette is equally effective in YCbCr and
RGB spaces.
We compare Palette with pix2pix [Isola et al., 2017b], PixColor [Guadarrama et al., 2017], and ColTran [Kumar et al., 2021].
Qualitative results are shown in Fig.
3, with quantitative scores in Table 1.
Palette establishes a new SoTA, outperforming existing works by a large margin.
Further, the performance measures (FID, IS, and CA) indicate that Palette outputs are close to being indistinguishable from the original images that were used to create the test greyscale inputs.
Surprisingly, our L 2 Regression baseline also outperforms prior task-speciﬁc techniques, highlighting the importance of modern architectures and large-scale training, even for a basic Regression model.
On human evaluation, Palette improves upon human raters’ fool rate of ColTran by more than 10%, approaching an ideal fool rate of 50%.
5.2 Inpainting
We follow Yu et al.
[2019] and train inpainting models on free-form generated masks, augmented with simple rectangular masks.
To maintain generality of Palette across tasks, in contrast to prior work, we do not pass a binary inpainting mask to the models.
Instead, we ﬁll the masked region with standard Gaussian noise, which is compatible with denoising diffusion models.
The training loss only considers the masked out pixels, rather than the entire image, to speed up training.
We compare
Palette with DeepFillv2 [Yu et al., 2019], HiFill [Yi et al., 2020], Photoshop’s Content-aware Fill ,
[Figure] [Figure]
[Table]
and Co-ModGAN [Zhao et al., 2021].
While there are other important prior works on inpainting such as [Liu et al., 2018a, 2020, Zheng et al., 2019]
[Table]
we were not able to compare Palette with all of them.
Qualitative and quantiative results are given in Fig.
4 and Table 2.
Palette exhibits strong performance across inpainting datasets and mask conﬁgurations, outperforming DeepFillv2, HiFill and
Co-ModGAN by a large margin.
Importantly, like the colorization task above, the FID scores for
Palette outputs in the case of 20-30% free-form masks, are extremely close to FID scores on the original images from which we created the masked test inputs.
See Appendix C.2 for more results.
[Table]
[Figure]
5.3 Uncropping
Recent works [Teterwak et al., 2019, Lin et al., 2021] have shown impressive visual effects by extending (extrapolating) input images along the right border.
We train Palette on uncropping in any one of the four directions, or around the entire image border on all four sides.
In all cases, we keep the area of the masked region at 50% of the image.
Like inpainting, we ﬁll the masked region
[Table]
with Gaussian noise
[Table]
and keep the unmasked region ﬁxed during inference.
We compare Palette with Boundless [Teterwak et al., 2019] and InﬁnityGAN [Lin et al., 2021].
While other uncropping methods exist (e.g., [Guo et al., 2020, Wang et al., 2019b])
[Table]
we only compare with two representative methods.
From the results in Fig.
5 and Table 3, one can see that Palette outperforms baselines on
ImageNet and Places2 by a large margin.
On human evaluation, Palette has a 40% fool rate, compared to 25% and 15% for Boundless and InﬁnityGAN (see Fig.
C.2 for details).
We further assess the robustness of Palette by generating panoramas through repeated application of uncropping (Fig. 2).
We observe that Palette is surprisingly robust, generating realistic and coherent outputs even after 8 repeated applications of uncrop.
We also generate zoom-out sequences by repeated uncropping around the entire border of the image with similarly appealing results 3 .
[Figure] [Figure]
[Table]
5.4 JPEG restoration
Finally, we evaluate Palette on the task of removing JPEG compression artifacts, a long standing image restoration problem [Dong et al., 2015, Galteri et al., 2019, Liu et al., 2018b].
Like prior work [Ehrlich et al., 2020, Liu et al., 2018b], we train Palette on inputs compressed with various quality
[Figure]
factors (QF).
While prior work has typically limited itself to a Quality Factor ≥ 10, we increase the difﬁculty of the task and train on Quality Factors as low as 5, producing severe compression artifacts.
Table 4 summarizes the ImageNet results
[Table]
with Palette exhibiting strong performance across all quality factors, outperforming the regression baseline.
As expected, the performance gap between Palette and the regression baseline widens with decreasing quality factor.
Figure 6 shows the qualitative comparison between Palette and our Regression baseline at a quality factor of 5.
It is easy to see that the regression model produces blurry outputs, while Palette produces sharper images.
[Table]
5.5 Self-attention in diffusion model architectures
Self-attention layers [Vaswani et al., 2017] have been an important component in recent U-Net architectures for diffusion models [Ho et al., 2020, Dhariwal and Nichol, 2021].
While self-attention layers build a direct form of global dependency into the model, they prevent generalization to unseen image resolutions.
Generalization to new resolutions at test time is convenient for many image-toimage translation tasks, and therefore previous works have relied primarily on fully convolutional architectures [Yu et al., 2019, Galteri et al., 2019].
We analyze the impact of these self-attention layers on sample quality for inpainting, one of the more difﬁcult image-to-image translation tasks.
In order to enable input resolution generalization for
Palette, we explore replacing global self-attention layers with different alternatives each of which represents a trade-off between large context dependency, and resolution robustness.
In particular, we experiment with the following four conﬁgurations:
Global Self-Attention : Baseline conﬁguration with global self-attention layers at 32 × 32, 16 ×and 8 × 8 resolutions.
Local Self-Attention : Local self-attention layers [Vaswani et al., 2021] at 32 × 32, 16 × 16 and 8 × 8 resolutions, at which feature maps are divided into 4 non-overlapping query blocks.
More ResNet Blocks w/o Self-Attention : 2 × residual blocks at 32 × 32, 16 × 16 and 8 ×resolutions allowing deeper convolutions to increase receptive ﬁeld sizes.
Dilated Convolutions w/o Self-Attention : Similar to 3.
ResNet blocks at 32 × 32, 16 × 16 and 8 × 8 resolutions with increasing dilation rates [Chen et al., 2017] allowing exponentially increasing receptive ﬁelds.
We train these models for 500K steps, and a batch size of 512.
Table 5 reports the performance of different conﬁgurations for inpainting.
Global self-attention offers much better performance than the fully-convolutional alternatives (even with 15% more parameters), re-afﬁrming the importance of self-attention layers for such architectures.
Surprisingly, local self-attention performs worse than fully-convolutional alternatives.
We leave more detailed analysis of local self-attention in image generation architectures to future work.
[Table] [Table]
I np a i n ti ng
C
o l o r i za ti on
U n c r opp i ng
5.6 Sample diversity
We next analyze sample diversity of Palette on two tasks, colorization and inpainting.
Speciﬁcally, we analyze the impact of the changing the diffusion loss function L simple
[Ho et al., 2020], and compare L 1 vs. L 2 on sample diversity.
While existing conditional diffusion models, SR3 [Saharia et al., 2021a] and WaveGrad [Chen et al., 2021a], have found L 1 norm to perform better than the conventional L 2 loss, there has not been a detailed comparison of the two.
To quantitatively compare sample diversity, we use multi-scale SSIM Guadarrama et al.
[2017] and the LPIPS diversity score
Zhu et al.
[2017b].
Given multiple generated outputs for each input image, we compute pairwise multi-scale SSIM between the ﬁrst output sample and the remaining samples.
We do this for multiple input images, and then plot the histogram of SSIM values (see Fig.
Following Zhu et al.
[2017b], we also compute LPIPS scores between consecutive pairs of model outputs for a given input image, and then average across all outputs and input images.
Lower SSIM and higher LPIPS scores imply more sample diversity.
The results in Table 6 thus clearly show that models trained with the L 2 loss have greater sample diversity than those trained with the L 1 loss.
Interestingly, Table 6 also indicates that L 1 and L 2 models yield similar FID scores (i.e., comparable perceptual quality), but that L 1 has somewhat lower Perceptual Distance scores than L 2 .
One can speculate that L 1 models may drop more modes than L 2 models, thereby increasing the likelihood that a single sample from an L 1 model is from the mode containing the corresponding original image, and hence a smaller Perceptual Distance.
Some existing GAN-based models explicitly encourage diversity; [Zhu et al., 2017b, Yang et al., 2019a]
propose methods for improving diversity of conditional GANs, and [Zhao et al., 2020, Han et al., 2019] explore diverse sample generation for image inpainting.
We leave comparison of sample diversity between Palette and other such GAN based techniques to future work.
5.7 Multi-task learning
Multi-task training for image-to-image translation is a challenging but largely underexplored area of research.
In prior work, the training paradigm is often restricted to multiple levels of corruptions
[Figure]
[Table]
within a single translation task.
For example, Yu et al.
[2019, 2018a] train inpainting models on different proportions of masked regions, Galteri et al.
[2019], Liu et al.
[2018b] train JPEG restoration models on multiple quality factors and [Zhang et al., 2017] train denoising models with a large range of noise level degradation.
Here we train a single generalist Palette model on multiple translation tasks simultaneously.
Speciﬁcally, we train Palette on tasks of JPEG restoration, inpainting, and colorization.
Table 7 indicates that multi-task generalist Palette outperforms the task-speciﬁc JPEG restoration specialist model, but slightly lags behind task-speciﬁc Palette models on inpainting and colorization.
Due to compute limitations
[Table]
we currently train multi-task Palette for the same number of training steps as task-speciﬁc models.
We expect its performance to improve with more training
[Table]
Conclusion
We present Palette, a simple and general framework for image-to-image translation.
Palette achieves strong results on four challenging image-to-image translation tasks (colorization, inpainting, uncropping, and JPEG restoration), outperforming strong GAN and regression baselines.
Unlike many
GAN models, Palette produces diverse outputs consistent with natural images.
This is accomplished without task-speciﬁc customization or optimization instability.
We also present a multi-task Palette model, that performs just as well or better over their task-speciﬁc counterparts.
Further exploration and investigation of multi-task diffusion models is an exciting avenue for future work.
This paper shows some of the potential of image-to-image diffusion models, but we look forward to seeing new applications.
Acknowledgement
We thank Jon Shlens, Doug Eck, Rif A. Saurous, Hossein Talebi and the Google Brain team for research discussions and technical assistance.
A Diffusion Models
Diffusion models comprise a forward diffusion process and a reverse denoising process that is used at generation time.
The forward diffusion process is a Markovian process that iteratively adds Gaussian noise to a data point y
≡ y over T iterations:
[Equation]
where α t are hyper-parameters of the noise schedule.
The forward process with α t is constructed in a manner where at t =
T , y T is virtually indistinguishable from Gaussian noise.
Note, we can also marginalize the forward process at each step:
[Equation]
where γ t = (cid:81) tt (cid:48) α (cid:48) t .
The Gaussian parameterization of the forward process also allows a closed form formulation of the posterior distribution of y t − 1 given ( y 0 , y t ) as
[Equation]
where µ = √ γ t − 1 (1 −
[Equation] [Equation]
y
[Equation]
and σ
[Equation]
=
[Equation]
This result proves to be very helpful during inference as shown below.
Learning: Palette learns a reverse process which inverts the forward process.
Given a noisy image (cid:101) y ,
[Equation]
the goal is to recover the target image y 0 .
We parameterize our neural network model f θ ( x, (cid:101) y , γ ) to condition on the input x , a noisy image (cid:101) y , and the current noise level γ .
Learning entails prediction of the noise vector (cid:15) by optimizing the objective
[Equation]
This objective, also known as L simple in Ho et al.
[2020], is equivalent to maximizing a weighted variational lower-bound on the likelihood [Ho et al., 2020].
Inference: Palette performs inference via the learned reverse process.
Since the forward process is constructed so the prior distribution p( y T ) approximates a standard normal distribution
N ( y T | 0 , I ) , the sampling process can start at pure Gaussian noise, followed by T steps of iterative reﬁnement.
Also recall that the neural network model f θ is trained to estimate (cid:15) , given any noisy image (cid:101) y , and y t .
Thus, given y t , we approximate y 0 by rearranging terms in equation 6 as
[Equation]
Following Ho et al.
[2020], we substitute our estimate ˆ y 0 into the posterior distribution of q ( y t − 1 | y 0 , y t ) in equation 5 to parameterize the mean of p θ ( y t − 1 | y t , x ) as
[Equation]
[Equation]
And we set the variance of p θ ( y t − 1 | y t , x ) to (1 − α t ) , a default given by the variance of the forward process Ho et al.
With this parameterization, each iteration of the reverse process can be computed as
[Equation]
where (cid:15) t ∼ N ( 0 , I ) .
This resembles one step of Langevin dynamics for which f θ provides an estimate of the gradient of the data log-density.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
do
[Table] [Table]
[Table]
for 6: return y
B Implementation Details
Training Details : We train all models with a mini batch-size of 1024 for 1M training steps.
We do not ﬁnd over ﬁtting to be an issue, and hence use the model checkpoint at 1M steps for reporting the ﬁnal results.
Consistent with previous works [Ho et al., 2020, Saharia et al., 2021a], we use standard Adam optimizer with a ﬁxed 1e-4 learning rate and 10k linear learning rate warmup schedule.
We use 0.9999 EMA for all our experiments.
We do not perform any task-speciﬁc hyper-parameter tuning, or architectural modiﬁcations.
Diffusion Hyper-parameters :
Following [Saharia et al., 2021a, Chen et al., 2021a] we use α conditioning for training Palette.
This allows us to perform hyper-parameter tuning over noise schedules and reﬁnement steps for
Palette during inference.
During training, we use a linear noise schedule of ( 1 e − 6 , 0 . 01 ) with 2000 time-steps, and use 1000 reﬁnement steps with a linear schedule of ( 1 e − 4 , 0 . 09 ) during inference.
Task Speciﬁc Details: We specify speciﬁc training details for each of the tasks below
: • Colorization : We use RGB parameterization for colorization.
We use the grayscale image as the source image and train Palette to predict the full RGB image.
During training, following [Kumar et al., 2021], we randomly select the largest square crop from the image and resize it to 256 × 256.
Inpainting : We train Palette on a combination of free-form and rectangular masks.
For free-form masks, we use Algorithm 1 in [Yu et al., 2019].
For rectangular masks, we uniformly sample between 1 and 5 masks.
The total area covered by the rectangular masks is kept between 10% to 40% of the image.
We randomly sample a free-form mask with 60% probability, and rectangular masks with 40% probability.
Note that this is an arbitrary training choice.
We do not provide any additional mask channel, and simply ﬁll the masked region with random Gaussian noise.
During training, we restrict the L simple loss function to the spatial region corresponding to masked regions, and use the model’s prediction for only the masked region during inference.
We train Palette on two types of 256 × 256 crops.
Consistent with previous inpainting works [Yu et al., 2019, 2018a, Yi et al., 2020], we use random 256 × 256 crops, and we combine these with the resized random largest square crops used in colorization literature [Kumar et al., 2021].
• Uncropping
: We train the model for image extension along all four directions, or just one direction.
In both cases, we set the masked region to 50% of the image.
During training, we uniformly choose masking along one side, or masking along all 4 sides.
When masking along one side, we further make a uniform random choice over the side.
Rest of the training details are identical to inpainting.
• JPEG Restoration : We train Palette for JPEG restoration on quality factors in (5, 30).
Since decompression for lower quality factors is a signiﬁcantly more difﬁcult task, we use an exponential distribution to sample the quality factor during training.
Speciﬁcally, the sampling probability of a quality range Q is set to ∝ e − Q 10 .
C Additional Experimental Results C.1 Colorization
Following prior work [Zhang et al., 2016, Guadarrama et al., 2017, Kumar et al., 2021], we train and evaluate models on ImageNet [Deng et al., 2009].
In order to compare our models with existing works in Table 1, we follow ColTran [Kumar et al., 2021] and use the ﬁrst 5000 images from ImageNet validation set to report performance on standard metrics.
We use the next 5000 images as the reference distribution for FID to mirror
ColTran’s implementation (as returned by TFDS [TFD] data loader).
For benchmarking purposes, we also report the performance of Palette on ImageNet ctest10k [Larsson et al., 2016] dataset in Table C.1.
Human Evaluation: The ultimate evaluation of image-to-image translation models is human evaluation; i.e., whether or not humans can discriminate model outputs from reference images.
To this end we use controlled human experiments.
In a series of two alternative forced choice trials, we ask subjects which of two side-by-side images is the real photo and which has been generated by the model.
In particular, subjects are asked “Which
Fool Rate % on Set-I (3 sec display)
[Figure]
Fool Rate % on Set-I (5 sec display
[Figure]
Fool Rate % on Set-I (3 sec display)
[Figure]
Fool Rate % on Set-I (5 sec display)
[Figure]
image would you guess is from a camera?
" Subjects viewed images for either 3 or 5 seconds before having to respond.
For the experiments we compare outputs from four models against reference images, namely, PixColor [Guadarrama et al., 2017], Coltran [Kumar et al., 2021], our Regression baseline, and Palette.
To summarize the result we compute the subject fool rate , i.e., the fraction of human raters who select the model outputs over the reference image.
We use a total of 100 images for human evaluation, and divide these into two independent subsets - Set-I and Set-II, each of which is seen by 50 subjects.
As shown in Figure C.1, the fool rate for Palette is close to 50% and higher than baselines in all cases.
We note that when subjects are given less time to inspect the images the fool rates are somewhat higher, as expected.
We also note the strength of our regression baseline, which also performs better than PixColor and Coltran.
Finally, to provide insight into the human evaluation results we also show several more examples of Palette output, with comparisons to benchmarks, in Figure C.3.
One can see that in several cases, Palette has learned colors that are more meaningful and consistent with the reference images and the semantic content of the images.
Figure C. also shows the natural diversity of Palette outputs for colorization model.
C.2 Inpainting
Comparison on 256 × 256 images : We report all inpainting results on 256 × 256 center cropped images.
Since the prior works we use for comparison are all trained on random 256 × 256 crops, evaluation on 256 × 256 center crops ensures fair comparison.
Furthermore, we use a ﬁxed set of image-mask pair for each conﬁguration for all models during evaluation.
Since HiFill [Yi et al., 2020] and Co-ModGAN [Yi et al., 2020] are primarily trained on 512 × 512 images, we use 512 × 512 center crops with exact same mask within the central 256 × 256 region.
This provides these two models with 4 × bigger inpainting context compared to DeepFillv2 and Palette.
We train two Palette models for Inpainting - i)
Palette (I) trained on ImageNet dataset, and ii) Palette (I+P) trained on mixture of ImageNet and Places2 dataset.
For Palette (I+P), we use a random sampler policy to sample from ImageNet and Places2 dataset with a uniform probability.
Table C.2 shows full comparison of Palette with existing methods on all inpainting conﬁgurations.
Based on the type of mask, and the area covered, we report results for the following categories - i) 10-20% free-form region, ii) 20-30% free-form region, iii) 30-40% free-form region and iv) 128 × 128 center rectangle region.
Palette consistently outperforms existing works by a signiﬁcant margin on all conﬁgurations.
Interestingly Palette (I) performs slightly better than Palette (I+P) on ImageNet indicating that augmentation with Places2 images during training doesn’t boost to ImageNet performance.
Furthermore, Palette (I) is only slightly worse compared to Palette (I+P) on Places2 even though it is not trained on Places2 images.
We observe a signiﬁcant drop in the performance of HiFill [Yi et al., 2020]
[Table] [Table]
with larger masks.
It is important to note that DeepFillv2 and HiFill are not trained on ImageNet, but we report their performance on ImageNet ctest10k primarily for benchmarking purposes.
C.3 Uncropping
Many existing uncropping methods [Cheng et al., 2021,
Teterwak et al., 2019] have been trained on different subsets of Places
[Zhou et al., 2017] dataset.
In order to maintain uniformity, we follow a similar setup as inpainting and train Palette on a combined dataset of Places2 and ImageNet.
While we train Palette to extend the image in all directions or just one direction, to compare fairly against existing methods we evaluate Palette on extending only the right half of the image.
For Table 3, we use ctest10k and places10k to report results on ImageNet and Places2 validation sets respectively.
We also perform category speciﬁc evaluation of Palette with existing techniques - Boundless
[Teterwak et al., 2019] and InﬁnityGAN [Lin et al., 2021].
Since Boundless is only trained on top-50 categories from Places dataset, we compare Palette with Boundless speciﬁcally on these categories from Places2 validation set in Table C.3.
Palette achieves signiﬁcantly better performance compared to Boundless re-afﬁrming the strength of our model.
Furthermore, we compare Palette with a more recent GAN based uncropping technique - InﬁnityGAN [Lin et al., 2021].
In order to fairly compare Palette with InﬁnityGAN, we speciﬁcally evaluate on the scenery categories from Places2 validation and test set.
We use the samples generously provided by Lin et al.
[2021], and generate outputs for Boundless, and Palette.
Table C.4 shows that Palette is signiﬁcantly better than domain speciﬁc model InﬁnityGAN on scenery images in terms of automated metrics.
Human Evaluation: Like colorization, we also report results from human evaluation experiments.
Obtaining high fool rates for uncropping is a signiﬁcantly more challenging task than colorization, because one half of the image area is fully generated by the model.
As a consequence there are more opportunities for synthetic artifacts.
Because the baselines available for uncropping are trained and tested on Places2, we run human evaluation experiments only on Places2.
Beyond the choice of dataset, all other aspects of experimental design are identical to that used above for colorization, with two disjoint sets of test images, namely, Set-I and Set-II.
[Table] [Table]
The results are characterized in terms of the fool rate, and are shown in Figure C.2.
higher rates on all human evaluation runs compared to existing methods, raters are more to inspect the rates worsen considerably on the hand approximately similar
C.4 JPEG Restoration
In order to be consistent with other tasks, we perform training and evaluation on ImageNet dataset.
Note that this is unlike most prior work [Dong et al., 2015, Liu et al., 2018b], which mainly use small datasets such as
DIV2K [Agustsson and Timofte, 2017] and BSD500 [Martin et al., 2001] for training and evaluation.
Recent works such as [Galteri et al., 2019] use a relatively larger MS-COCO dataset for training, however, to the best of our knowledge, we are the ﬁrst to train and evaluate JPEG restoration on ImageNet.
We compare Palette with a strong Regression baseline which uses an identical architecture.
We report results on JPEG quality factor settings of 5, 10 and 20 in Table 4.
C.5 Evaluation and Benchmarking Details
Several existing works report automated metrics such as FID, Inception Score, etc.
[Kumar et al., 2021, Lin et al., 2021, Yi et al., 2020]
but often lack key details such as the subset of images used for computing these metrics, or the reference distribution used for calculating FID scores.
This makes direct comparison with such reported metrics difﬁcult.
Together with advocating for our proposed benchmark validation sets, we also provide all the necessary details to exactly replicate our reported results.
We encourage future works to adopt a similar practice of reporting all the necessary evaluation details in order to facilitate direct comparison with their methods.
Benchmark datasets : For ImageNet evaluation, we use the 10,000 image subset from ImageNet validation set - ctest10k introduced by [Larsson et al., 2016].
While this subset has been primarliy used for evaluation in the colorization literature [Guadarrama et al., 2017, Su et al., 2020, Kim et al., 2021a], we extend its use for other image-to-image translation tasks.
Many image-to-image translation tasks such as inpainting, uncropping are evaluated on Places2 dataset [Zhou et al., 2017].
However, to the best of our knowledge, there is no such standardized subset for Places2 validation set used for benchmarking.
To this end, we introduce places10k , a 10,950 image subset of Places2 validation set.
Similar to ctest10k, we make places10k class balanced with 30 images per class (Places2 dataset has 365 classes/categories in total.).
Metrics : We report several automated metrics for benchmarking and comparison with existing methods.
Speciﬁ- cally, we report Fréchet Inception Distance (FID) , Inception Score , Perceptual Distance and Classiﬁcation
Accuracy for qualitative comparison.
When computing FID scores, the choice of the reference distribution is important, but is often not clariﬁed in existing works.
In our work, we use the full validation set as the reference distribution, i.e. 50k images from ImageNet validation set for computing scores on ImageNet subset ctest10k, and 36.5k images from Places2 validation set for computing scores on Places2 subset places10k.
For Perceptual
Distance, we use the Euclidean distance in the pool _ 3 feature space of the pre-trained InceptionV1 network (same as the features used
C.6 Limitations and Social Impact
While Palette achieves strong results on several image-to-image translation tasks demonstrating the generality and versatility of the emerging diffusion models, there are many important limitations to address.
Diffusion models generally require large number of reﬁnement steps during sample generation (e.g. we use 1k reﬁnement steps for Palette throughout the paper) resulting in signiﬁcantly slower inference compared to GAN based models.
This is an active area of research, and several new techniques [Nichol and Dhariwal, 2021, Watson et al., 2021, Jolicoeur-Martineau et al., 2021] have been proposed to reduce the number of reﬁnement steps signiﬁcantly.
We leave application of these techniques on Palette to future work.
Palette’s use of group-normalization and self-attention layers prevents its generalizability to arbitrary input image resolutions, limiting its practical usability.
Techniques to adapt such models to arbitrary resolutions such as ﬁne-tuning, or patch based inference can be an interesting direction of research.
Bias is a key issue in all generative models.
This includes conditional diffusion models like Palette, which generate samples from a posterior distribution over images conditioned on a task and an input image.
In some tasks, like the JPEG artifact removal, this entails the generation of local image structure to repair compression artifacts.
For other tasks, like inpainting and uncropping, relatively large regions of images are hallucinated, which are meant to be plausible and consistent with the surrounding image context.
Their ability to solve this task, in part, depende on the data they are trained with.
Our models have been primarily trained on ImageNet and Places2 data.
Salient objects that do not occur with sufﬁcient frequency in these training data will not be rendered well by Palette.
Examples may include images containing text, or images of people, especially faces, to which human vision is quite sensitive.
While our log-likelihood based objective is mode covering ( e.g., unlike some GAN-based objectives), we believe it is likely our diffusion-based models may drop modes and may produce somewhat biased results.
Hence, we recommend training the models on large datasets that are well suited to the test domain of interest before practical deployment.
Grayscale Input
PixColor
[Figure]
ColTran
[Figure]
Regression
Palette (Ours)
Original
[Figure]
Sample Sample
Original
[Figure] [Figure]
Sample Sample
Sample Sample
Original
[Figure] [Figure]
Sample Sample
Sample Sample
Original
[Figure]
Sample Sample
Sample Sample
Original
Input
Sample Sample
Sample Sample
Original
[Figure]
Sample Sample
Sample Sample
Original
[Figure]
Sample Sample
Original
[Figure]
Palette
Original