Pluralistic Image Completion with Gaussian Mixture Models
Abstract
Pluralistic image completion focuses on generating both visually realistic and diverse results for image completion.
Prior methods enjoy the empirical successes of this task.
However, their used constraints for pluralistic image completion are argued to be not well interpretable and unsatisfactory from two aspects.
First, the constraints for visual reality can be weakly correlated to the objective of image completion or even redundant .
Second, the constraints for diversity are designed to be task-agnostic , which causes the constraints to not work well.
In this paper, to address the issues, we propose an end-to-end probabilistic method.
Specifically, we introduce a unified probabilistic graph model that represents the complex interactions in image completion.
The entire procedure of image completion is then mathematically divided into several sub-procedures, which helps efficient enforcement of constraints.
The sub-procedure directly related to pluralistic results is identified, where the interaction is established by a Gaussian mixture model (GMM).
The inherent parameters of GMM are task-related , which are optimized adaptively during training, while the number of its primitives can control the diversity of results conveniently.
We formally establish the effectiveness of our method and demonstrate it with comprehensive experiments.
The implementation is available at https://github.com/tmllab/PICMM.
Introduction
Pluralistic image completion refers to the task of filling in the missing region of an incomplete image, so as to produce visually realistic and diverse image completion solutions [59–61].
Different from single image completion [39, 12, 55, 50] that learns a deterministic mapping from an incomplete image to a complete image, and produces a unique result, pluralistic image completion can generate various results with visually realistic contents.
Pluralistic image completion follows the fact that image completion is a highly subjective process and benefits a series of applications such as photo restoration [27], object removal [13] and transmission error concealment [37].
Pluralistic image completion is complicated and challenging.
The state-of-the-art methods encode images to latent features.
To constrain the visual reality of results, heuristic constraints are introduced, e.g. , the perceptual loss [27] and attention module [59, 44].
To constrain the diversity of results, the distribution of latent features is assumed to be a unimodal Gaussian distribution with predefined parameters [59].
Image completion is performed by sampling from the unimodal Gaussian distribution.
Although the above paradigm enjoys empirical successes of pluralistic image completion, it is not well interpretable and unsatisfactory in practice.
We detail the issues from two aspects.
First, the added constraints for the visual reality of results are based on general purposes .
These constraints may work well in some tasks.
Unfortunately, there is no clear understanding of what role they play in pluralistic image completion.
Inappropriate constraints are likely to have side effects.
It is not easy to finish the determination of added constraints as our desideratum, especially in complex image completion tasks.
Second, the diversity of image completion results is hard to be constrained in reason.
For a specific task, the parameters of the unimodal Gaussian distribution are directly related to the diversity.
Nevertheless, these parameters are designed to be prior knowledge and task-agnostic , which cause unreasonable pluralistic results .
[Figure]
In this paper, to address the above issues, we present an end-to-end probabilistic method.
The method is constructed with a unified probabilistic graph model (PGM) [17, 22] and translates the problem of pluralistic image completion into a structured mathematical representation.
Specifically, we suppose that pluralistic image completion can be represented with a directed acyclic graph that is designed reasonably.
The nodes of the graph denote different kinds of images and corresponding latent features.
The edges of the graph represent the interactions between nodes.
Based on this directed acyclic graph, the entire procedure of pluralistic image completion is divided into several sub-procedures.
Then, we perform different constraints for different subprocedures as our desideratum.
In particular, the diversity of image completion results is implemented by modeling the interaction between the latent features of the missing region and incomplete image with a Gaussian mixture model (GMM) [40, 64].
Compared with the unimodal Gaussian distribution, GMM has a larger capacity and is more competitive to meet the output diversity [56, 38].
For our method, the inherent parameters of GMM are task-related and optimized adaptively during training.
The different primitives of GMM represent the outputs with different patterns.
Additionally, the number of primitives can be chosen artificially with the needs for diversity in real-world applications.
Before delving into details, we clearly emphasize our contributions as follows:
• To our best knowledge, this paper is the first one that uses a PGM for pluralistic image completion.
Besides, the PGM is specially designed with justifications, rather than borrowing existing models.
With the designed PGM, the added constraints for visual reality and diversity are explainable.
• We propose to use GMM to increase result diversity.
One remarkable advantage is that the inherent parameters for diversity are task-related, leading to more reasonable results.
• We conduct a series of experiments on benchmark datasets to support our claims.
In both qualitative and quantitative comparisons with the state-of-the-art methods, our method achieves superior performance.
The generated contents for image completion are both visually realistic and diverse, such as those shown in Figure 1.
The rest of the paper is organized as follows.
In Section 2, we briefly review pluralistic image completion.
In Section 3, we discuss the proposed method step by step.
Experimental results are provided in Section 4.
Finally, we conclude the paper in Section 5.
Preliminaries
Problem Setting.
We first discuss image completion.
For the task of image completion, suppose we have an image, originally
I o , but degraded by some missing pixels to become a masked partial image I m .
The image that comprises the original missing pixels is called a complement partial image, which can be denoted by I c .
The goal of the image completion task is to reconstruct
I o by using I m .
The illustrations for different kinds of images are provided in Figure 2.
Traditional single image completion reconstructs the original image in a deterministic fashion, which results in only a single solution [12, 9].
Pluralistic image completion [59] is motivated by the fact that image completion is a highly subjective process.
Also, completed images by different experts may agree on high-level semantics, but have substantially different details [59].
Therefore, in pluralistic image completion, we need to generate multiple and diverse plausible results when presented with a masked partial image.
The problem of pluralistic image completion is challenging since we need to take care of the diversity and visual authenticity of results at the same time.
Prior Work.
Traditional methods on single image completion, e.g., diffusion-based methods [3, 1, 23] and patch-based methods [2, 4, 5], assume that image holes share similar content to visible regions.
Therefore, they choose to directly propagate the contextual appearances or realign the background patches to complete the image holes.
Benefiting from the strong power of deep neural networks, deep learning based single image completion methods achieve promising completion performance, e.g., employing shepard convolutional neural networks [39], restraining global and local consistency between image holes and visible regions [16], performing deep feature rearrangement [47], and learning a pyramid-context encoder network [53], etc .
In addition to these methods, multiple single image completion methods involve the generative adversarial networks [10] to learn the semantic of images [48, 49], e.g., [25, 51, 8, 6, 57, 12, 9].
Although these single-solution methods achieve outstanding performance in predicting deterministic result for image holes, they cannot generate various semantically meaningful results.
Existing works for pluralistic image completion usually rely on generative models [24], e.g. , CVAE [43, 13, 59], hierarchical VQ-VAE [37], PD-GAN [27], and BicycleGAN [63].
Generative pluralistic image completion methods can diversify meaningful completion results with different kinds of constraints.
However, such methods are not well interpretable as discussed.
The issues largely limit their applications in the real world.
Methodology
This section presents a probabilistic method for pluralistic image completion.
We first design a probabilistic graph model for our task (Section 3.1).
Then, based on the probabilistic graph model, we show how to divide the entire procedure of pluralistic image completion into several sub-procedures (Section 3.2).
Afterward, we discuss how to use GMM to diversify image completion results (Section 3.3).
Finally, we add reconstruction and adversarial losses to strengthen image completion, and summarize all algorithm flows (Section 3.4).
3.1 Probabilistic Graph Model Construction
Recall the notations in image completion, i.e. , I o , I m , and I c , they correspond to the latent features z o , z m , and z c respectively.
We suppose a probabilistic graph model for the image completion task, which is shown in Figure 3.
The probabilistic graph model is designed reasonably.
The graphical illustration of overall computational paths of the probabilistic graph model is presented in Figure 3(d).
Specifically, the images I m and I c form I o .
Their latent features z m
and z c can be exploited to form
I o .
Given z m , we can surmise z c .
We use the latent features here, since they are low-dimensional and informative [11, 21, 45, 46].
We further details Figure 3 as follows.
[Figure]
Inference.
We do not directly map I m
and I c to I o , since they are too high-dimensional and have information redundancy.
We exploit generative models.
The processes I m →
z m
and I c
→ z c denote that we encode images into corresponding latent features.
During training, we learn the encoder f
to finish the inference.
Dynamic Inference.
Given I m ( resp.
I c ), we can infer the latent feature z m ( resp.
z c ).
Afterward, z m (cid:57)(cid:57)(cid:75)
z c means that we use a variational distribution p ( z c | z m ) to approximate the distribution p ( z c | I c ) .
Here we denote this relationship using a dashed line, which is designed to be dynamic to diversify outputs.
Generation.
In image completion, the images I m
and I c can compose the image
I o .
Accordingly, for their latent features, we splice z m and z c to output I o .
During training, we learn the decoder g to finish the generation.
3.2 Image Completion Objective Decomposition
We cannot observe the underlying I c in the test procedure.
Based on the probabilistic graph model, we perform the following variational inference with the Kullback-Leibler (KL) divergence:
[Equation]
[Equation]
where q ψ ( ·|· ) and p ϕ ( ·|· ) are two conditioned distributions, with ψ and ϕ being the parameters of their corresponding functions.
The divergence (1) is minimized with respect to all parameters.
It should be noted that, in fact, the overall objective should be KL [ q ψ ( I o | I m )
∥ p ϕ ( I o | I m , I c )] =
KL [ E z m , z c [ q ψ ( I o , z m , z c | I m )]
∥ E z m , z c [ p ϕ ( I o , z m , z c | I m , I c )]] .
As the expectation in this formula is intractable , we need to deal with the 1-step Monte Carlo sampling objective, which is why the divergence (1) is formed.
Another advantage of using the divergence (1) is that constraining each sampling point can make the overall constraint tighter.
We further decompose the divergence (1) based on the probabilistic graph model as follows.
Proposition 1 Regarding the KL divergence (1), we show that the divergence can be decomposed as:
[Equation]
where q θ is the variational distribution, and the parameters θ are involved in divergence minimization.
The proof can be found in Appendix A.1.
We analyze three terms in Eq.
(2) combining the probabilistic graph model and our purpose:
• For a ⃝ , as discussed, we can generate I o with I m
and I c , or with their latent features z m and z c .
We minimize the distance about the generation of I o in the two ways.
The minimization can ensure that the generation from the images and latent features is consistent .
• For
b ⃝ , we do not limit the inference from I c to z c as a deterministic function.
Instead, to diversify outputs, we use the variational distribution to restrain q θ ( z c | z m ) .
The choice and discussion about the variational distribution will be provided in Section 3.3.
• For c ⃝ , it aims to keep that the inference using the variational distribution and the inference using the posterior is close , which also guarantees the reliability of sampling.
Besides, the variational distribution does not include I c to enable latent features to be used in test.
We employ the variational evidence lower bound (ELBO)
L ELBO [21, 15] to constrain c ⃝ .
It is because, suppose that p ( I m ) is a constant given I m , it is equivalent for maximizing the L ELBO and minimizing c ⃝ .
The details are provided in Appendix A.4.
3.3 Gaussian Mixture Model
As discussed, we need a suitable choice for the variational distribution q θ ( z c | z m ) to output pluralistic results.
In this work, we employ the Gaussian Mixture Model (GMM) [30, 31] for the dynamic mode q θ ( z c | z m ) , which is more flexible than a unimodal and can diversify outputs.
We denote the latent features of I c that are inferred from z m by ˆ z c , which is distinguished from z c achieved by the encoder.
For the latent features z , we denote the mean and covariance by µ z and Σ z .
Mathematically, we have
[Equation]
[Equation]
) where k denotes the number of primitives, α i denotes the mixing coefficient of the i -th primitive, and µ i , Σ i denotes the distribution parameters of the i -th primitive.
Note that the mixing coefficient α
i is the parameter of the Categorical distribution [31].
Intuitively, in Eq.
(3), we model z m with GMM, where the i -th primitive can be used to represent ˆ z ( i ) c .
In this way, we can obtain different ˆ z c to meet diversity requirements.
Besides, among all the parameters of GMM, except for k , the other parameters are called the inherent parameters of GMM.
In GMM (3), the number of primitives k can be determined artificially according to the need for output diversity.
The inherent parameters are optimized adaptively during training.
Specifically, for the optimization of the mixing coefficients α =
[ α 1 , . . .
, α k ] , we exploit the the frequency loss [38].
The objective is
[Equation]
For our task, with the frequency loss (4), the accumulated frequency approximate gradient is an asymptotically unbiased estimation of the true gradient.
We provide the detailed derivation and analysis in Appendix A.2.
The reason we mention the frequency loss is that, the metric for evaluating the performance of each primitive is adapted to KL divergence in our task, which is different from the original metric in [38].
For the optimization of µ and Σ of z m , we utilize the back-propogatemax-operation to reserve the distinguishable property of GMM (3).
For our problem, the objective of the back-propogate-max-operation is shown in the following proposition.
Proposition 2 By modeling the variational distribution with GMM (3), we have
[Equation]
where j = argmin i KL (cid:104) N ( z c | µ ˆ z c i , Σ
ˆ z c i ) ∥N ( z c | µ z c , Σ z c ) (cid:105) , and tr ( · ) denotes the trace of a matrix.
The derivation of Proposition 2 can be found in Appendix A.3.
For Proposition 2, if we minimize
L BM , we optimize µ and Σ of z m , which reduces the distance between ˆ z ( j ) c and z c .
For the term b ⃝ in Eq.
(2), we then have L GMM =
E z m ∼ q ψ ( z m | I m ) KL [ q θ ( z c | z m )
∥ p ϕ ( z c |
I c )]
≈ L F + L BM .
Discussion.
Note that the diversity of results is directly related to z m .
When GMM for z m is dominated by the i -th primitive ( α i ≈ 1 ), the diversity of image completion results is reduced.
Although this phenomenon did not appear in our experiments, it may exist in practice.
Actually, this phenomenon is reasonable.
From human cognition, if I m shows only one pattern , the inference for
I c will be deterministic [3].
For example, if the size of the missing region of an incomplete image is very small, we can infer the missing region both deterministically and reasonably, where the diversity of results is reduced.
Prior methods with task-agnostic constraints do not consider this fact, which may create unreasonable results .
For our method, the constraints for diversity are task-related, which can take this phenomenon into account.
ˆ I ( j ) o by g ∗ with ˆ I ( j )
o = g ∗ ( z m , ˆ z ( j ) c ) .
7: end for
Output: pluralistic image completion results { ˆ I ( j ) o } kj =1 .
3.4 Reconstruction and Adversarial Losses
To ensure the minimization of the term a ⃝ in Eq.
(2), we incorporate the use of reconstruction and adversarial losses [10].
The reconstruction loss is formulated as:
[Equation]
(6) where ˆ I o is generated from z m and z c , and ˆ I ( j ) m is generated from ˆ z ( j ) c .
Here, ˆ z ( j ) c can be inferred from z m by using Eq.
(3) and Eq.
The image ˆ I ( j ) m is obtained from ˆ I ( j ) o with the image mask that can degenerate I o to I m .
The image ˆ I ( j ) o is generated from z m and ˆ z ( j ) c .
The reconstruction loss (6) controls the visual rationality of image completion results.
Also, we efficiently avoid the deterministic fashion with GMM.
Furthermore, we leverage adversarial training [10] to make the generated images more realistic.
The adversarial loss L A is formulated as
[Equation]
where D is the discriminator optimized by the discriminator loss based on LSGAN [29].
We jointly train our encoder f and decoder g through the following combined loss:
[Equation]
where the weight λ
A is set to 0.05 in all experiments.
Then, the final objective of the loss function is the sum of three losses, i.e. , L FINAL =
L GMM + L ELBO + L C .
The final loss is to maximize the log-likelihood of the conditional data distribution.
Specifically, the adversarial loss uses the
Wasserstein distance to make generated images realistic given latent features.
The ELBO loss uses KL divergence and distribution log-likelihood to make variational posterior approximate real posterior.
Algorithm Flows.
For the convenience of following technical details, we provide the algorithm flows.
The algorithm flows of training and test stages can be found in Algorithm 1 and Algorithm 2.
The illustration is shown in Appendix B.
It should be noted that the number of primitives k does not mean that we are limited to generate only k diverse results given I m (in Steps 3 and 4 of Algorithm 2).
In fact, we can sample from the k primitives to obtain lots of image completion results.
Moreover, as GMM has a much larger capacity than the unimodal Gaussian distribution [31], we can achieve greater diversity.
We provide empirical observations in Section 4.3 and Appendix C.
Experiments
In this section, we conduct a series of experiments to justify our claims.
We first introduce the implementation of our method (Section 4.1).
The comprehensive experimental results and comparison with advanced methods are then provided and discussed (Section 4.2).
Finally, we conduct an analysis study to present and discuss our method in more detail (Section 4.3).
4.1 Implementation Details
Datasets.
We evaluated our proposed model on five popularly used datasets, i.e. , CelebA-HQ [18, 28],
FFHQ [19], Paris StreetView [7], Places2 [62], and ImageNet [41].
We verify the effectivness of our method with different types of mask regions, including both center and random masks [59, 44].
Network and Optimization.
The proposed method can be implemented efficiently.
The encoder and decoder of our pipeline are inspired by PIC [59] for comparison.
We apply average pooling and interpolation with convolutional layers to implement downsampling and upsampling respectively.
During optimization, we use the Adam optimizer [20].
The learning rate is fixed to 10 − 4 during the training procedure.
Baselines.
We compare the proposed method with the following state-of-the-art methods, which include (1) single image completion methods: DFv2 [52], EC [33], and MED [26]; (2) pluralistic image completion methods: PIC [59] and ICT [44].
We abbreviate our method ( P luralistic I mage
C ompletion with Gaussian M ixture M odels) as PICMM.
The methods are implemented by PyTorch and evaluated on NVIDIA Tesla A100 GPUs.
[Figure] [Figure]
[Figure] [Figure]
Measurement.
For measurement, we provide both qualitative and quantitative results.
For the metrics of quantitative results, we use common evaluation metrics such as the peak signal-to-noise ratio (PSNR), structural similarity (SSIM), mean absolute error (MAE), and fr ´ echet inception distance (FID) [14] to measure the similarity between the image completion result and ground truth.
Furthermore, pluralistic image completion is supposed to focus on generating diverse realistic results rather than merely approximating ground-truth ones [58].
To measure the result diversity , we add two perceptual quality metrics to quantitative results, which include LPIPS [59] and DIV-FID [44].
4.2 Comparison with Prior Methods 4.2.1 Qualitative Comparisons
We provide extensive qualitative comparison results to justify our claims.
First, we show the results on CelebA-HQ and FFHQ in Figure 4, which are images related to human faces.
As can be seen, the image completion results achieved by our method are realistic, following high image quality.
Also, compared with the baseline
PIC, our completion results are more diverse, e.g. , see the images in the second and last rows.
Second, we provide the results on Paris
StreetView and Places in Figures 5 and 6 respectively.
For the results on Paris StreetView, we can see that image completion by the baselines MED and PIC are somewhat distorted.
By contrast, our method can finish the image completion task better.
The details can be checked in the complements to walls.
Then, we turn the attention to the image completion results on Places2.
Our method still achieves superior performance compared with baselines.
Lastly, we provide the results on ImageNet in Figure 7.
Note that the sources of the data in ImageNet are very complicated.
The image completion task is rather challenging on this dataset, even for single image completion [59].
Different from some prior work [36, 16] that were trained on a 100k subset of training images of ImageNet, we directly the network on the original ImageNet training dataset with all images.
The results mean that our method can infer the content effectively.
Refined Result Analysis.
We provide some refined results in Figure 8 to analyze the image completion details of our method.
For the face images, we can find that the image completion results have very different facial expressions
[Table]
4.2.2 Quantitative Comparisons
[Table]
Common Metrics.
Quantitative evaluation with common metrics is difficult for pluralistic image completion, as our goal is to obtain diverse but reasonable solutions for a masked image [59].
To make a feasible comparison, as did in [59], we sample one completed image many times from k images to compare with the original image.
The results are provided in
Tables 1 and 2.
As can be seen, for
FFHQ and Places2, on both the settings of center and random masking, our method achieves superior performance.
On ImageNet, our method performs the best with the center masking consistently, while it is competitive with the random masking.
It should be noted that ICT is a very strong baseline and works better than our method in some cases.
It is because ICT is built with the image transformer [35], which is powerful in vision tasks.
Compared with ICT, the advantages of our method lies in not only better interpretability as discussed, but also faster inference speeds.
We will demonstrate this in inference time
[Table]
Diversity Metrics.
For pluralistic image completion, the diversity evaluations are significant.
We generate five completion images in each task and report the mean of the diversity metric.
The results are shown in Table 3, which represents that our method can generate more diverse completion images than baselines.
The results with common and diversity metrics support our claim very well.
That is to say, the image completion results of our method are both high-quality and pluralistic.
[Table]
Inference Time.
We report the inference time of our method, compared with PIC and ICT.
All inference runs on one NVIDIA Tesla A100 GPU for fairness.
The size of images is 256 × 256 .
We perform pluralistic image completion fordifferent images.
For each image, we generatecompletion results for it.
We report the total inference time for these images.
The results are shown in Table 4.
Note that as ICT is a transformer-based method, the calculated consumption of its inference is much heavier.
What’s worse, ICT needs iterative Gibbs sampling during inference.
The two issues make ICT have to face heavy computational consumption, which is also mentioned in [44].
In contrast, our method is more inference-efficiency, which is more than 750 times faster than ICT .
The merit could make our method easier to use in the real world.
4.3 More Analyses and Justifications
GMM Primitive Visualization.
We further stress the diversity of the completion results achieved by our method.
Specifically, we visualize the ˆ z c which is inferred from z m as discussed.
The visualization results are obtained with t-SNE [42].
Different ˆ z c are presented by the 2D-vectors in different colors, which are shown in Figure 9.
As can be seen, the vectors are scattered , which clearly demonstrates that the completion results of our method are diverse.
Refined Diversity Comparison.
We argue that, benefiting from that GMM has larger capacities than a unimodal Gaussian distribution, the completion results of our method would be more diverse.
To justify our claims, we provide the refined diversity comparison of our method with PIC and ICT, as shown in Figure 10.
Note that the six image completion results of PIC are obtained by sampling from one unimodal Gaussian distribution.
The results of our method are obtained by sampling from six primitives.
Clearly, our results are more diverse than PIC’s.
Besides, for ICT, it does not explicitly control the output diversity.
The diversity of pluralistic image completion is thus less interpretable than our method.
In addition, we state that our method is not limited to only generating k completion results before.
In fact, we can sample from k primitives of GMM to obtain lots of completion results.
Due to the limited page, we present the results and more analyses in Appendix C. Moreover, the richer completion result comparison is presented in Appendix D.
Conclusion
In this paper, we focus on the complicated and challenging problem of pluralistic image completion.
We propose a novel end-to-end probabilistic method for this challenging problem.
Based on the probabilistic graph model, our method divides the entire procedure of pluralistic image completion into several sub-procedures, where GMM is used to diversify outputs.
Experiments on a variety of datasets show that the image completion results by our method are both high-quality and pluralistic.
In future work, investigating the feasibility of the proposed method for other diverse decision-making scenarios might prove important.
Acknowledgements
Yibing Zhan was partially supported by the Major Science and Technology Innovation 2030 “
New
Generation Artificial Intelligence” key project (No. 2021ZD0111700) and the National Natural
Science Foundation of China (Grant No. 62002090).
Xiaobo Xia was supported by Australian
Research Council Projects DE-190101473 and Google PhD Fellowship.
Bo Han was supported by the RGC Early Career Scheme No. 22200720, NSFC Young Scientists Fund No. 62006202, and
Guangdong Basic and Applied Basic Research Foundation No. 2022A1515011652.
Tongliang Liu was partially supported by Australian Research Council Projects DP180103424, DE-190101473, IC190100031, DP-220102121, and FT-220100318.
The authors would give special thanks to Mingrui
Zhu (Xidian University), Zihan Ding (Princeton University), and Chenlai Qian (Southeast University) for helpful discussions.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] , (b) Did you describe the limitations of your work?
[Yes] , (c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] , 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Supplementary Material of “Pluralistic
Completion with
Mixture Models”
The appendix is organized as follows.
We detail theoretical analyses in Section A.
The related work is reviewed in Section B. Supplementary experimental results are provided in Section C and Section D.
A The Details of Theoretical Analyses A.1 The Proof of Proposition
As we do not have the access to the underlying I c , we can preform the following variational approximation with the Kullback-Leibler (KL) divergence:
[Equation]
The second term in Eq.
(9) is
[Equation] [Equation]
[Equation]
Combining Eq.
(9) and Eq.
(10), we complete the proof.
■
Note that the latent variables z m and z c are not codependent .
If z m and z c are codependent, the inference of z m ( resp.
z c ) must need z c ( resp.
z m ).
Then, in Eq.
(10), there should be q ψ ( z m | I m , z c ) and p ϕ
( z c | I c , z m ) , rather than q ψ ( z m | I m ) and p ϕ ( z c | I c ) .
As shown in the our Figure 3, z m and z c can be inferred without each other, but using I m
and I c .
Therefore, q ψ ( z m | I m ) and p ϕ ( z c | I c ) hold in Eq.
A.2 The Derivation of the Frequency Loss
As discussed, we exploit the frequency loss [38] for the Gaussian mixture model.
Here, we detail
The derivation of the frequency loss for our task, i.e. , pluralistic image completion.
Specifically, we use the KL divergence between each primitive N ( z c | µ ˆ z c i , Σ
ˆ z c i ) and N ( z c | µ z c , Σ z c ) as the metric of the performance:
[Equation]
Then, by comparing the KL divergence of each primitive, we can get the best primitive’s index j .
That is
[Equation]
We now can give a detailed derivation for Eq.
(4) below.
Specifically, for stochastic Gaussian mixture models, the gradient value of a single-instance sampling process from each primitive’s performance under the metric Eq.
(11), to parameters θ α that output the mixing coefficients
α
=
[ α 1 , . . .
, α k ] , can be estimated with a frequency approximate gradient .
Mathematically, we have
[Equation]
where δ i is the gradient of the performance metric Eq.
(11) for α i , ∇ θ i α i is the gradient of α i for parameters θ i , and 1 best i is the indicator function, where 1 best i = 1 if i = j and 1 best i = 0 otherwise.
The accumulated frequency approximate gradient is an asymptotically unbiased estimation of the true gradient for the sampling process from a categorical distribution α , with a batch of n → ∞ examples.
For δ i , we can detail it below:
[Equation]
Suppose a batch of examples with a number of n is applied, and the true probability of the primitive i to be the best primitive is p i .
The batch accumulated gradient will be
[Equation]
where the last formula indicates that n t
= np i when n → ∞ , since the true probability can be approximated by n t n in the limit case.
Also, as ∇ θ i α i is not always equal to 0 , the gradient equals to 0 if and only if α i
= p i .
Optimising with the above Eq.
(15) is the same as minimising the distance between α i and p i , with the optimal situation as α i =
p i when letting the last formula of Eq.
(15) be zero.
Based the above analyses, we therefore can build a frequency loss in this paper.
The derivation is completed.
■
A.3 The Proof of Proposition
We employ the mixture of Gaussian for the dynamic model q θ ( z c | z m ) , which is more flexible than a unimodal and can diversify the output.
More formally, we have
[Equation]
where k denotes the number of primitives, and α i denotes the weight of the i -th primitive.
Then the
KL divergence between q θ ( z c | z m ) and p ϕ ( z c | I c ) becomes
[Equation]
We further approximate Eq.
(17) by the back-propogate-max-operation with the frequency loss.
We will have
[Equation] [Equation]
where
[Equation]
=
[Equation]
We then have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation]
Suppose the covariance is a diagonal matrix [21]
[Equation]
we have
[Equation]
where tr ( · ) denotes the trace of a matrix.
Then
[Equation]
Eq.
(19) would be:
[Equation]
The proof is completed.
■
A.4 The Fully Optimization Expression of Proposition
As discussed in the main paper, we have
[Equation]
As Eq.
(22) shows, the item c ⃝ may be hard to implement, because we cannot know the real p ϕ
( z m | I m ) , which is used to restrain q ψ ( z m | I m ) outputted by a neural network.
However, inspired by [21]
[Equation]
we can employ a variational evidence lower bound (ELBO) to implement the item c ⃝ .
Firstly, we assume that there exists a ground truth model p ( I m ) , which is a constant given I m and does not depend on z m .
Then, we can derive the relationship between p ( I m ) and c ⃝ :
[Equation]
p
[Equation]
Thus
[Equation]
c ⃝ is equal to log p ( I m ) − ELBO .
Since log p ( I m ) is a constant, minimizing c ⃝ is equal to maximize the ELBO .
Accordingly
[Equation]
the fully optimization expression of Theorem 1 is below:
[Equation]
m
[Equation]
z c )
[Equation]
ψ
[Equation]
c |
[Equation]
m
[Equation]
KL
[Equation]
ψ
[Equation] [Equation]
] (cid:124) (cid:123)(cid:122) (cid:125) c ⃝ , (24) ■
B Advance of PICMM Over PIC
The illustration of our method is shown in Figure 11.
[Figure]
We also summarize the differences between PIC and our method as follows:
(1) Our method has better interpretability in pluralistic image completion than PIC, which is highlighted in this paper.
(2) GMM is used in our method, which better promotes diversity than PIC.
More importantly, the inherent parameters for diversity are task-related , rather than task-agnostic in PIC.
(3) Our method uses KL divergence for the whole objective.
We decompose the divergence and then minimize different decomposition terms, which supports our theory.
While, PIC only uses KL divergence as a single loss in its objective.
(4) Our method exploits the same CNN backbone as PIC.
In all quantitative comparisons, our method outperforms PIC.
C Supplementary Experimental Results on Diversity Analyses
In the main paper, we state that our method is not limited to only generate k image completion results.
Instead, we can sample from the k primitives of GMM to generate a different number of images.
The results are provided in Figure 12 and 13.
The images in the i -th row of Figure 12 and 13 are obtained by sampling from the i -th primitive of GMM.
As can be seen, the images in the same rows are less diverse than the images in the different rows, since one primitive has a limited capacity.
Multiple primitives can better meet the diversity needs.
We are able to sample from different primitives to achieve the better diversity in pluralistic image completion.
D Supplementary Pluralistic Completion Result Comparisons
In the main paper, we provide some comparisons of pluralistic completion results with state-of-the-art methods.
We provide more comparisons here.
The comparison methods include DFv2 3 , EC 4 , MED 5 ,
PIC 6 , and ICT 7 .
The experimental results on CelebA-HQ are provided in Figures 14 and 15.
The experimental results on FFHQ, Paris StreetView, and Places2 are shown in Figures 16, 17, andrespectively.
In addition, the experimental results on ImageNet are presented in Figure 19.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
E Ablation Study
To help understand the contribution of the different components in the proposed method, we provide ablation studies here.
Specifically, experiments are conducted on the dataset CelebA-HQ with the setting of center masking.
The number of primitives is set to k = 6 .
It should be noted that, without the loss L R , the image completion is unsupervised, resulting in rather unrealistic completed images.
The loss L R is hence added by default.
The results of ablation studies are provided in Table 5, which demonstrate the effectiveness of different components.
[Table]
F Results of Comparison with More Image Completion Methods
To make the experimental comparison more comprehensive and more convincing
[Table]
we add the baseline
VQ-VAE [37] and AOT-GAN [54].
The results of comparing our PICMM with the two baselines are shown in Table
[Table]
which verify the effectiveness of our method clearly
[Table]
G Evaluations on the Number of Primitives
We supplement the experiments with different number of primitives, i.e. , k = 1 , 2 , 3 , 4 , 5 .
The experiments are conducted on CelebA-HQ using center masks.
The implementation details keep the same as the details in the main paper.
We provide experimental results in Table 7.
The results show that the increase of k can diversify the completed images, without degenerating the performance with respect to common evaluation metrics.
[Table]
H Higher-Resolution Image Completion Results
Before this section, we resize the image to 256 × 256 for image completion.
Here, to show that our method can generate higher-resolution images, we set the image size to 512 × 512 .
Experiments are performed on the datasets FFHQ.
The results are provided in Figure 20.
I Possible Negative Impacts and Limitation
For possible negative impacts, since our PICMM is a method based on generative models, like a large number of methods of this kind, generated images with our method may danger to the security of
AI systems, with the form of deep fakes.
There are some advanced works targeting the issue, e.g. , [32, 34], which are helpful to handle the possible negative impacts.
For the limitation, our PICMM does not outperform ICT sometimes with respect to common metrics, since ICT uses a much larger transformer-based structure and iterative Gibbs sampling, which make
ICT’s inference slow.
For the people preferring the performance on common metrics, this would be a limitation of our PICMM.