Pluralistic Image Completion with Probabilistic Mixture-of-Experts
Anonymous Author(s)
AffiliationAddress email
Abstract
Pluralistic image completion focuses on generating both visually realistic anddiverse results for image completion.
Prior methods enjoy the empirical successesof this task.
However, their used constraints for pluralistic image completion areargued to be not well interpretable and unsatisfactory from two aspects.
First, theconstraints for visual reality can be weakly correlated to the objective of imagecompletion or even redundant .
Second, the constraints for diversity are designed tobe task-agnostic , which causes the constraints to not work well.
In this paper, toaddress the issues, we propose an end-to-end probabilistic method.
Specifically,we introduce a unified probabilistic graph model that represents the complexinteractions in image completion.
The entire procedure of image completion isthen mathematically divided into several sub-procedures, which helps efficientenforcement of constraints.
The sub-procedure directly related to pluralistic resultsis identified, where the interaction is established by a Gaussian mixture model(GMM).
The inherent parameters of GMM are task-related , which are optimizedadaptively during training, while the number of its primitives can control thediversity of results conveniently.
We formally establish the effectiveness of ourmethod and demonstrate it with comprehensive experiments.
Introduction
Pluralistic image completion refers to the task of filling in the missing region of an incomplete image,so as to produce visually realistic and diverse image completion solutions [58–60].
Different from single image completion [40, 13, 54, 49] that learns a deterministic mapping from an incompleteimage to a complete image and produces a unique result, pluralistic image completion can generatevarious results with visually realistic contents.
Pluralistic image completion follows the fact thatimage completion is a highly subjective process and benefits a series of applications such as photorestoration [27], object removal [14] and transmission error concealment [38].
Pluralistic image completion is complicated and challenging.
The state-of-the-art methods encodeimages to latent features.
To constrain the visual reality of results, heuristic constraints are introduced, e.g. , the perceptual loss [27] and attention module [58, 45].
To constrain the diversity ofresults, the distribution of latent features is assumed to be a unimodal Gaussian distribution withpredefined parameters [58].
Image completion is performed by sampling from the unimodal Gaussiandistribution.
Although the above paradigm enjoys empirical successes of pluralistic image completion, it is notwell interpretable and unsatisfactory in practice.
We detail the issues from two aspects.
First, theadded constraints for the visual reality of results are based on general purposes .
These constraintsmay work well in some tasks.
Unfortunately, there is no clear understanding of what role they playin pluralistic image completion.
Inappropriate constraints are likely to have side effects.
It is noteasy to finish the determination of added constraints as our desideratum, especially in complex image completion tasks.
Second, the diversity of image completion results is hard to be constrained inreason.
For a specific task, the parameters of the unimodal Gaussian distribution are directly relatedto the diversity.
Nevertheless, these parameters are designed to be prior knowledge and task-agnostic ,which cause unreasonable pluralistic results .
[Figure]
In this paper, to address the above issues, we presentan end-to-end probabilistic method.
The method isconstructed with a unified probabilistic graph model(PGM) [18, 23] and translates the problem of pluralistic image completion into a structured mathematical representation.
Specifically, we suppose that pluralistic image completion can be represented witha directed acyclic graph that is designed reasonably.
The nodes of the graph denote different kinds of images and corresponding latent features.
The edges ofthe graph represent the interactions between nodes.
Based on this directed acyclic graph, the entire procedure of pluralistic image completion is divided intoseveral sub-procedures.
Then, we perform differentconstraints for different sub-procedures as our desideratum.
In particular, the diversity of image completion resultsis implemented by modeling the interaction betweenthe latent features of the missing region and incomplete image with a Gaussian mixture model (GMM)[41, 63].
Compared with the unimodal Gaussian distribution, GMM has a larger capacity and is morecompetitive to meet the output diversity [55, 39].
Forour method, the inherent parameters of GMM are task-related and optimized adaptively during training.
The different primitives of GMM represent the outputs with different patterns.
Additionally, thenumber of primitives can be chosen artificially with the needs for diversity in real-world applications.
Before delving into details, we clearly emphasize our contributions as follows:
• To our best knowledge, this paper is the first one that uses a PGM for pluralistic image completion.
Besides, the PGM is specially designed with justifications, rather than borrowing existing models.
With the designed PGM, the added constraints for visual reality and diversity are explainable.
• We propose to use GMM to increase result diversity.
One remarkable advantage is that the inherentparameters for diversity are task-related, leading to more reasonable results.
• We conduct a series of experiments on benchmark datasets to support our claims.
In both qualitativeand quantitative comparisons with the state-of-the-art methods, our method achieves superiorperformance.
The generated contents for image completion are both visually realistic and diverse,such as those shown in Figure 1.
Codes will be released for the reproducibility of results.
The rest of the paper is organized as follows.
In Section 2, we briefly review pluralistic imagecompletion.
In Section 3, we discuss the proposed method step by step.
Experimental results areprovided in Section 4.
Finally, we conclude the paper in Section 5.
Preliminaries
Problem Setting.
We first discuss image completion.
For the task of image completion, suppose we have animage, originally I o , but degraded by some missingpixels to become a masked partial image I m .
Theimage that comprises the original missing pixels iscalled a complement partial image, which can be denoted by I c .
The goal of the image completion taskis to reconstruct I o by using I m .
The illustrations fordifferent kinds of images are provided in Figure 2.
Traditional single image completion reconstructsthe original image in a deterministic fashion, which results in only a single solution [13, 9].
Pluralistic image completion [58] is motivated by the fact that image completion is a highly subjectiveprocess.
Also, completed images by different experts may agree on high-level semantics, but havesubstantially different details [58].
Therefore, in pluralistic image completion, we need to generatemultiple and diverse plausible results when presented with a masked partial image.
The problem ofpluralistic image completion is challenging since we need to take care of the diversity and visualauthenticity of results at the same time.
Prior Work.
Traditional methods on single image completion, e.g., diffusion-based methods [3, 1, 24]and patch-based methods [2, 4, 5], assume that image holes share similar content to visible regions.
Therefore, they choose to directly propagate the contextual appearances or realign the backgroundpatches to complete the image holes.
Benefiting from the strong power of deep neural networks,deep learning based single image completion methods achieve promising completion performance,e.g., employing shepard convolutional neural networks [40], restraining global and local consistencybetween image holes and visible regions [16], performing deep feature rearrangement [46], andlearning a pyramid-context encoder network [52], etc .
In addition to these methods, multiple singleimage completion methods involve the generative adversarial networks [11] to learn the semanticof images [47, 48], e.g., [25, 50, 8, 6, 56, 13, 9].
Although these single-solution methods achieveoutstanding performance in predicting deterministic result for image holes, they cannot generatevarious semantically meaningful results.
Existing works for pluralistic image completion usually rely on generative models, e.g. , CVAE[44, 14, 58], hierarchical VQ-VAE [38], PD-GAN [27], and BicycleGAN [62].
Generative pluralisticimage completion methods can diversify meaningful completion results with different kinds ofconstraints.
However, such methods are not well interpretable as discussed.
The issues largely limittheir applications in the real world.
Methodology
This section presents a probabilistic method for pluralistic image completion.
We first design aprobabilistic graph model for our task (Section 3.1).
Then, based on the probabilistic graph model, weshow how to divide the entire procedure of pluralistic image completion into several sub-procedures(Section 3.2).
Afterward, we discuss how to use GMM to diversify image completion results (Section3.3).
Finally, we add reconstruction and adversarial losses to strengthen image completion, andsummarize all algorithm flows (Section 3.4).
3.1 Probabilistic Graph Model Construction
[Figure]
Recall the notations in image completion, i.e. , I o , I m , and I c , they correspond to the latent featuresz o , z m , and z c respectively.
We suppose a probabilistic graph model for the image completion task,which is shown in Figure 3.
The probabilistic graph model is designed reasonably.
The graphicalillustration of overall computational paths of the probabilistic graph model is presented in Figure 3(d).
Specifically, the images I m and I c form I o .
Their latent features z m
and z c can be exploited to form
I o .
Given z m , we can surmise z c .
We use the latent features here, since they are low-dimensionaland informative [12, 22].
We further details Figure 3 as follows.
Inference.
We do not directly map I m
and I c to I o , since they are too high-dimensional and haveinformation redundancy.
We exploit generative models.
The processes I m →
z m
and I c
→ z c denotethat we encode images into corresponding latent features.
During training, we learn the encoder f tofinish the inference.
Dynamic Inference.
Given I m ( resp.
I c ), we can infer the latent feature z m ( resp.
z c ).
Afterward,z m (cid:57)(cid:57)(cid:75) z c means that we use a variational distribution p ( z c | z m ) to approximate the distribution
Generation.
In image completion, the images I m
and I c can compose the image
I o .
Accordingly,for their latent features, we splice z m and z c to output I o .
During training, we learn the decoder g tofinish the generation.
3.2 Image Completion Objective Decomposition
We cannot observe the underlying I c in the test procedure.
Based on the probabilistic graph model,we perform the following variational inference with the Kullback-Leibler (KL) divergence:
[Equation]
where q ψ ( ·|· ) and p ϕ ( ·|· ) are two conditioned distributions, with ψ and ϕ being the parametersof their corresponding functions.
The divergence (1) is minimized with respect to all parameters.
It should be noted that, in fact, the overall objective should be KL [ q ψ ( I o | I m )
∥ p ϕ ( I o | I m , I c )] =
[Equation]
, z
[Equation]
[ q [Equation] ( I
[Equation]
, z
[Equation]
, z
[Equation]
| I
[Equation] [Equation] [Equation] [ p ϕ ( I o , z m , z c | I m , I c )]] .
As the expectation in this formulais intractable , we need to deal with the 1-step Monte Carlo sampling objective, which is why thedivergence (1) is formed.
Another advantage of using the divergence (1) is that constraining eachsampling point can make the overall constraint tighter.
We further decompose the divergence (1)based on the probabilistic graph model as follows.
Proposition 1 Regarding the KL divergence (1), we show that the divergence can be decomposed as:
[Equation]
where q θ is the variational distribution, and the parameters θ are involved in divergence minimization.
The proof can be found in Appendix A.1.
We analyze three terms in Eq.
(2) combining the probabilistic graph model and our purpose:
• For a ⃝ , as discussed, we can generate I o with I m
and I c , or with their latent features z m and z c .We minimize the distance about the generation of I o in the two ways.
The minimization can ensurethat the generation from the images and latent features is consistent .•
For b ⃝ , we do not limit the inference from I c to z c as a deterministic function.
Instead, to diversifyoutputs, we use the variational distribution to restrain q θ ( z c | z m ) .
The choice and discussion aboutthe variational distribution will be provided in Section 3.3.• For c ⃝ , it aims to keep that the inference using the variational distribution and the inference usingthe posterior is close , which also guarantees the reliability of sampling.
Besides, the variationaldistribution does not include I c to enable latent features to be used in test.
We employ the variationalevidence lower bound (ELBO)
L ELBO [22] to constrain c ⃝ .
It is because, suppose that p ( I m ) is aconstant given I m , it is equivalent for maximizing the L ELBO and minimizing c ⃝ .
The details areprovided in Appendix A.4.
3.3 Mixture-of-Experts Model
As discussed, we need a suitable choice for the variational distribution q θ ( z c | z m ) to output pluralisticresults.
In this work, we employ the Gaussian Mixture Model (GMM) [30, 31] for the dynamic modeq θ ( z c | z m ) , which is more flexible than a unimodal and can diversify outputs.
We denote the latent features of I c that are inferred from z m by ˆ z c , which is distinguished from z cachieved by the encoder.
For the latent features z , we denote the mean and covariance by µ z and Σ z .
Mathematically, we have
[Equation] [Equation]
where k denotes the number of primitives, α i denotes the mixing coefficient of the i -th primitive, andµ i , Σ i denotes the distribution parameters of the i -th primitive.
Note that the mixing coefficient α i is the parameter of the Categorical distribution [31].
Intuitively, in Eq.
(3), we model z m with GMM,where the i -th primitive can be used to represent ˆ z ( i ) c , i.e. , µ ˆ z c i = µ z m i and Σ ˆ z c i =
Σ z m i .
In this way,we can obtain different ˆ z c to meet diversity requirements.
Besides, among all the parameters of GMM, except for k , the other parameters are called the inherentparameters of GMM.
In GMM (3), the number of primitives k can be determined artificially accordingto the need for output diversity.
The inherent parameters are optimized adaptively during training.
Specifically, for the optimization of the mixing coefficients α =
[ α 1 , . . .
, α k ] , we exploit the thefrequency loss [39].
The objective is
[Equation]
For our task, with the frequency loss (4), the accumulated frequency approximate gradient is anasymptotically unbiased estimation of the true gradient.
We provide the detailed derivation andanalysis in Appendix A.2.
The reason we mention the frequency loss is that, the metric for evaluatingthe performance of each primitive is adapted to KL divergence in our task, which is different fromthe original metric in [39].
For the optimization of µ and Σ of z m , we utilize the back-propogatemax-operation to reserve the distinguishable property of GMM (3).
For our problem, the objective ofthe back-propogate-max-operation is shown in the following proposition.
Proposition 2 By modeling the variational distribution with GMM (3), we have
[Equation]
where j = argmin i KL (cid:104) N ( z c | µ ˆ z c i , Σ
ˆ z c i ) ∥N ( z c | µ z c , Σ z c ) (cid:105) , and tr ( · ) denotes the trace of a matrix.
The derivation of Proposition 2 can be found in Appendix A.3.
For Proposition 2, if we minimize
L BM , we optimize µ and Σ of z m , which reduces the distance between ˆ z ( j ) c and z c .
For the term b ⃝in Eq.
(2), we then have L GMM = E z
[Equation]
Discussion.
Note that the diversity of results is directly related to z m .
When GMM for z m isdominated by the i -th primitive ( α i ≈ 1 ), the diversity of image completion results is reduced.
Although this phenomenon did not appear in our experiments, it may exist in practice.
Actually, thisphenomenon is reasonable.
From human cognition, if I m shows only one pattern , the inference for
I c will be deterministic [3].
For example, if the size of the missing region of an incomplete image isvery small, we can infer the missing region both deterministically and reasonably, where the diversityof results is reduced.
Prior methods with task-agnostic constraints do not consider this fact, whichmay create unreasonable results .
For our method, the constraints for diversity are task-related, whichcan take this phenomenon into account.
3.4 Reconstruction and Adversarial Losses
To ensure the minimization of the term a ⃝ in Eq.
(2), we incorporate the use of reconstruction andadversarial losses [11].
The reconstruction loss is formulated as:
[Equation]
where ˆ I o is generated from z m and z c , and ˆ I ( j ) m is generated from ˆ z ( j ) c .
Here, ˆ z ( j ) c can be inferredfrom z m by using Eq.
(3) and Eq.
The image ˆ I ( j ) m is obtained from ˆ I ( j ) o with the image maskthat can degenerate I o to I m .
The image ˆ I ( j ) o is generated from z m and ˆ z ( j ) c .
The reconstructionloss (6) controls the visual rationality of image completion results.
Also, we efficiently avoid thedeterministic fashion with GMM.
Furthermore, we leverage adversarial training [11] to make thegenerated images more realistic.
The adversarial loss L A is formulated as
[Equation]
where D is the discriminator optimized by the discriminator loss based on LSGAN [29].
We jointlytrain our encoder f and decoder g through the following combined loss:
[Equation]
where the weight λ A is set to 0.05 in all experiments.
Then, the final objective of the loss functionis the sum of three losses, i.e. , L FINAL =
L GMM + L ELBO + L C .
The final loss is to maximizethe log-likelihood of the conditional data distribution.
Specifically, the adversarial loss uses the
Wasserstein distance to make generated images realistic given latent features.
The ELBO lossuses KL divergence and distribution log-likelihood to make variational posterior approximate realposterior.
Algorithm Flows.
For the convenience of following technical details, we provide the algorithm flows.
The algorithm flows of training and test stages can be found in Algorithm 1 and Algorithm 2.
Theillustration is shown in Appendix B.
It should be noted that the number of primitives k does not mean that we are limited to generateonly k diverse results given I m (in Steps 3 and 4 of Algorithm 2).
In fact, we can sample from the kprimitives to obtain lots of image completion results.
Moreover, as GMM has a much larger capacitythan the unimodal Gaussian distribution [31], we can achieve greater diversity.
We provide empiricalobservations in Section 4.3 and Appendix C.
Experiments
In this section, we conduct a series of experiments to justify our claims.
We first introduce theimplementation of our method (Section 4.1).
The comprehensive experimental results and comparisonwith advanced methods are then provided and discussed (Section 4.2).
Finally, we conduct an analysisstudy to present and discuss our method in more detail (Section 4.3).
4.1 Implementation Details
Datasets.
We evaluated our proposed model on five popularly used datasets, i.e. , CelebA-HQ [19, 28],
FFHQ [20], Paris StreetView [7], [-Places
 [55],-]Places2 [61], and ImageNet [42].
We verify the effectivness of ourmethod with different types of mask regions, including both center and random masks [58, 45].
Network and Optimization.
The proposed method can be implemented efficiently.
The encoderand decoder of our pipeline are inspired by PIC [58] for comparison.
We apply average pooling andinterpolation with convolutional layers to implement downsampling and upsampling respectively.
During optimization, we use the Adam optimizer [21].
The learning rate is fixed to 10 − 4 during thetraining procedure.
Baselines.
We compare the proposed method with the following state-of-the-art methods, whichinclude (1) single image completion methods: DFv2 [51], EC [34], and MED [26]; (2) pluralisticimage completion methods: PIC [58] and ICT [45].
We abbreviate our method ( P luralistic I mage
C ompletion with Probabilistic M ixture-of- E xperts) as PICME.
The methods are implemented by
PyTorch and evaluated on NVIDIA Tesla A100 GPUs.
Measurement.
For measurement, we provide both qualitative and quantitative results.
For themetrics of quantitative results, we use common evaluation metrics such as the peak signal-to-noiseratio (PSNR), structural similarity (SSIM), mean absolute error (MAE), and fr ´ echet inception distance (FID) [15] to measure the similarity between the image completion result and ground truth.
Furthermore, pluralistic image completion is supposed to focus on generating diverse realistic resultsrather than merely approximating ground-truth ones [57].
To measure the result diversity , we addtwo perceptual quality metrics to quantitative results, which include LPIPS [58] and DIV-FID [45].
[Figure]
[Figure] [Figure]
[Figure] [Figure]
4.2 Comparison with Prior Methods
4.2.1 Qualitative Comparisons.
We provide extensive qualitative comparison results tojustify our claims.
First, we show the results on CelebAHQ and FFHQ in Figure 4, which are images related tohuman faces.
As can be seen, the image completion results achieved by our method are realistic, following highimage quality.
Also, compared with the baseline PIC, ourcompletion results are more diverse, e.g. , see the imagesin the second and last rows.
Second, we provide the results on Paris StreetView and
Places in Figures 5 and 6 respectively.
For the results on
Paris StreetView, we can see that image completion bythe baselines MED and PIC are somewhat distorted.
Bycontrast, our method can finish the image completion task better.
The details can be checked in thecomplements to walls.
Then, we turn the attention to the image completion results on Places2.
Ourmethod still achieves superior performance compared with baselines.
Lastly, we provide the results on ImageNet in Figure 7.
Note that the sources of the data in ImageNetare very complicated.
The image completion task is rather challenging on this dataset, even for singleimage completion [58].
Different from some prior work [37, 16] that were trained on a 100k subsetof training images of ImageNet, we directly the network on the original ImageNet training datasetwith all images.
The results mean that our method can infer the content effectively.
Refined Result Analysis.
We provide some refined results in Figure 8 to analyze the image completion details of our method. For the face images, we can find that the image completion results havevery different facial expressions. [Table] 4.2.2 Quantitative Comparisons
[Table] Common Metrics. Quantitative evaluation with common metrics is difficult for pluralistic image completion,as our goal is to obtain diverse butreasonable solutions for a masked image [58]. To make a feasible comparison, as did in [58], we sample onecompleted image many times from kimages to compare with the originalimage.
The results are provided in Tables 1 and 2. As can be seen, for
FFHQ and Places2, on both the settings of center and random masking, our method achieves superiorperformance.
On ImageNet, our method performs the best with the center masking consistently,while it is competitive with the random masking.
It should be noted that ICT is a very strong baseline and works better than our method in somecases.
It is because ICT is built with the image transformer [36], which is powerful in vision tasks.
Compared with ICT, the advantages of our method lies in not only better interpretability as discussed,but also faster inference speeds.
We will demonstrate this in inference time.
[Table]
Diversity Metrics.
For pluralistic image completion, the diversity evaluations are significant. We generate fivecompletion images in each task andreport the mean of the diversity metric.
The results are shown in Table 3,which represents that our method cangenerate more diverse completion images than baselines. The results withcommon and diversity metrics support our claim very well. That is to say, the image completionresults of our method are both high-quality and pluralistic.
Method Inference time (minutes) ↓
PIC [58] 0.
ICT [45] 179. PICME † 0.
Inference Time. We report the inference timeof our method, compared with PIC and ICT. Allinference runs on one NVIDIA Tesla A100 GPUfor fairness. The size of images is 256 × 256 .
We perform pluralistic image completion for 100different images. For each image, we generate 6completion results for it. We report the total inference time for these images.
The results are shownin Table 4. Note that as ICT is a transformer-basedmethod, the calculated consumption of its inference is much heavier. What’s worse, ICT needs iterative Gibbs sampling during inference.
The two issues make ICT have to face heavy computationalconsumption, which is also mentioned in [45].
In contrast, our method is more inference-efficiency,which is more than 750 times faster than ICT . The merit could make our method easier to use inthe real world. 4.3 More Analyses and Justifications [Figure]
GMM Primitive Visualization. We further stress the diversity of the completion results achievedby our method. Specifically, we visualize the ˆ z c which is inferred from z m as discussed. Thevisualization results are obtained with t-SNE [43].
Different ˆ z c are presented by the 2D-vectors indifferent colors, which are shown in Figure 9. As can be seen, the vectors are scattered , which clearlydemonstrates that the completion results of our method are diverse. Refined Diversity Comparison. We argue that, benefiting from that GMM has larger capacities thana unimodal Gaussian distribution, the completion results of our method would be more diverse.
Tojustify our claims, we provide the refined diversity comparison of our method with PIC and ICT, asshown in Figure 10. Note that the six image completion results of PIC are obtained by sampling fromone unimodal Gaussian distribution. The results of our method are obtained by sampling from sixprimitives. Clearly, our results are more diverse than PIC’s.
Besides, for ICT, it does not explicitlycontrol the output diversity. The diversity of pluralistic image completion is thus less interpretablethan our method. In addition, we state that our method is not limited to only generating k completion results before. Infact, we can sample from k primitives of GMM to obtain lots of completion results.
Due to the limitedpage, we present the results and more analyses in Appendix C. Moreover, the richer completion resultcomparison is presented in Appendix D.
Conclusion
In this paper, we focus on the complicated and challenging problem of pluralistic image completion.
We propose a novel end-to-end probabilistic method for this challenging problem.
Based on theprobabilistic graph model, our method divides the entire procedure of pluralistic image completioninto several sub-procedures, where GMM is used to diversify outputs.
Experiments on a variety ofdatasets show that the image completion results by our method are both high-quality and pluralistic.
In future work, investigating the feasibility of the proposed method for other diverse decision-makingscenarios might prove important.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] ,
(b) Did you describe the limitations of your work?
[Yes] ,
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes] ,
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
or
(a) Did
include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Supplementary Material
Probabilistic
The appendix is organized as follows.
We detail theoretical analyses in Section A.
The related work isreviewed in Section B. Supplementary experimental results are provided in Section C and Section D.
A The Details of Theoretical Analyses
A.1 The Proof of Proposition 1
As we do not have the access to the underlying I c
[Equation]
we can preform the following variationalapproximation with the Kullback-Leibler (KL) divergence:
[Equation]
The second term in Eq.
(9) is
[Equation]
Combining Eq.
(9) and Eq.
(10), we complete the proof.
■
Note that the latent variables z m and z c are not codependent .
If z m and z c are codependent, theinference of z m ( resp.
z c ) must need z c ( resp.
z m ).
Then, in Eq.
(10), there should be q ψ ( z m | I m , z c )and p ϕ
( z c | I c , z m ) , rather than q ψ ( z m | I m ) and p ϕ ( z c | I c ) .
As shown in the our Figure 3, z m and z ccan be inferred without each other, but using I m
and I c .
Therefore, q ψ ( z m | I m ) and p ϕ ( z c | I c ) holdin Eq.
A.2 The Derivation of the Frequency Loss
As discussed, we exploit the frequency loss [39] for the Gaussian mixture model.
Here, we detail
The derivation of the frequency loss for our task, i.e. , pluralistic image completion.
Specifically, weuse the KL divergence between each primitive N ( z c | µ ˆ z c
i , Σ
ˆ z c i ) and N ( z c | µ z c , Σ z c ) as the metric ofthe performance:
[Equation]
Then, by comparing the KL divergence of each primitive, we can get the best primitive’s index j .
That is
[Equation]
We now can give a detailed derivation for Eq.
(4) below.
Specifically, for stochastic mixture-ofexperts, the gradient value of a single-instance sampling process from each primitive’s performanceunder the metric Eq.
(11), to parameters θ α that output the mixing coefficients
α
=
[ α 1 , . . .
, α k ] ,can be estimated with a frequency approximate gradient .
Mathematically, we have
[Equation]
where δ i is the gradient of the performance metric Eq.
(11) for α i , ∇ θ i α i is the gradient of α
{+i forparameters θ i , and 1 best i is the indicator function, where 1 best i = 1 if i = j and 1 best i = 
+}
otherwise.
The accumulated frequency approximate gradient is an asymptotically unbiased estimation of thetrue gradient for the sampling process from a categorical distribution α , with a batch of n → ∞examples.
For δ i , we can detail it below:
[Equation]
Suppose a batch of examples with a number of n is applied, and the true probability of the primitive ito be the best primitive is p i .
The batch accumulated gradient will be
[Equation]
where the last formula indicates that n t
= np i when n → ∞ , since the true probability can beapproximated by n t n in the limit case.
Also, as ∇ θ i α i is not always equal to 0 , the gradient equals to0 if and only if α i = p i .
Optimising with the above Eq.
(15) is the same as minimising the distancebetween
α i and p i , with the optimal situation as α i = p i when letting the last formula of Eq.
(15) bezero.
Based the above analyses, we therefore can build a frequency loss in this paper.
The derivation is completed.
■
A.3 The Proof of Proposition 2
We employ the mixture of Gaussian for the dynamic model q θ ( z c | z m ) , which is more flexible than aunimodal and can diversify the output.
More formally, we have
[Equation]
(16) where k denotes the number of primitives, and α i denotes the weight of the i -th primitive.
Then the
KL divergence between q θ ( z c | z m ) and p ϕ ( z c | I c ) becomes
[Equation]
We further approximate Eq.
(17) by the back-propogate-max-operation with the frequency loss.
Wewill have
[Equation]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Table]
[Equation]
[Figure]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Figure] [Equation]
[Figure] [Table] [Equation] [Figure]
[Equation] [Table]
[Figure] [Equation] [Figure] [Equation]
[Table]
[Figure] [Equation] [Figure]
[Equation] [Table]
[Figure] [Table]
[Figure] [Equation] [Figure] [Equation]
Suppose the covariance is a diagonal matrix [22] [Equation] we have [Equation]
[Equation]
where tr ( · ) denotes the trace of a matrix.
Then [Equation] Eq.
(19) would be: [Equation] The proof is completed.
■ A.4 The Fully Optimization Expression of Proposition As discussed in the main paper, we have [Equation]
[Equation] As Eq.
(22) shows, the item c ⃝ may be hard to implement, because we cannot know the realp ϕ
( z m | I m ) , which is used to restrain q ψ ( z m | I m ) outputted by a neural network. However, inspired by [22], we can employ a variational evidence lower bound (ELBO) to implementthe item c ⃝ . Firstly, we assume that there exists a ground truth model p ( I m ) , which is a constantgiven I m and does not depend on z m . Then, we can derive the relationship between p ( I m ) and c ⃝ :
[Equation]
Thus, c ⃝ is equal to log p ( I m ) − ELBO . Since log p ( I m ) is a constant, minimizing c ⃝ is equal tomaximize the ELBO .
Accordingly [Equation] the fully optimization expression of Theorem 1 is below: [Equation]
m , I c )] = E ( z m , z c ) ∼ q ψ ( z m , z c | I m ) KL [ q ψ ( I o | z m , z c ) ∥
p ϕ ( I o | I m , I c )] (cid:124) (cid:123)(cid:122) (cid:125) a ⃝ + E z m ∼ q ψ ( z m | I m ) KL [ q θ ( z c | z m )
∥ p ϕ ( z c | I c )] (cid:124) (cid:123)(cid:122) (cid:125) b ⃝ − E z m ∼ q ψ ( z m | I m )
[log p ( I m | z m )]
+ KL [ q ψ ( z m | I m ) || p z m )] (cid:124) (cid:123)(cid:122) (cid:125) c ⃝ , (24) ■
The illustration of our method is shown in Figure 11. [Figure] We also summarize the differences between PIC and our method as follows:
(1) Our method has better interpretability in pluralistic image completion than PIC, which is high- lighted in this paper.
(2) GMM is used in our method, which better promotes diversity than PIC.
More importantly, the inherent parameters for diversity are task-related , rather than task-agnostic in PIC. (3) Our method uses KL divergence for the whole objective.
We decompose the divergence andthen minimize different decomposition terms, which supports our theory. While, PIC only uses KLdivergence as a single loss in its objective.(4) Our method exploits the same CNN backbone as PIC.
In all quantitative comparisons, our methodoutperforms PIC. C Supplementary Experimental Results on Diversity Analyses In the main paper, we state that our method is not limited to only generate k image completion results. Instead, we can sample from the k primitives of GMM to generate a different number of images.
Theresults are provided in Figure 12 and 13.
The images in the i -th row of Figure 12 and 13 are obtainedby sampling from the i -th primitive of GMM.
As can be seen, the images in the same rows are lessdiverse than the images in the different rows, since one primitive has a limited capacity.
Multipleprimitives can better meet the diversity needs.
We are able to sample from different primitives toachieve the better diversity in pluralistic image completion. With each primitive, six diverse images are generated. With each primitive, six diverse images are generated D Supplementary Pluralistic Completion Result Comparisons
In the main paper, we provide some comparisons of pluralistic completion results with state-of-the-artmethods.
We provide more comparisons here.
The comparison methods include DFv2 1 , EC 2 , MED 3 , PIC 4 , and ICT 5 . The experimental results on CelebA-HQ are provided in Figures 14 and 15. Theexperimental results on FFHQ, Paris StreetView, and Places2 are shown in Figures 16, 17, and 18respectively.
In addition, the experimental results on ImageNet are presented in Figure 19.
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure] [Figure] E Ablation Study To help understand the contribution of the different components in the proposed method, we provideablation studies here.
Specifically, experiments are conducted on the dataset CelebA-HQ with thesetting of center masking.
The number of primitives is set to k = 6 .
It should be noted that, withoutthe loss L R , the image completion is unsupervised, resulting in rather unrealistic completed images.
The loss L R is hence added by default.
The results of ablation studies are provided in Table 5, whichdemonstrate the effectiveness of different components.
[Table]
F Results of Comparison with More Image Completion Methods
To make the experimental comparison more comprehensive and more convincing, we add the baseline
VQ-VAE [38] and AOT-GAN [53].
The results of comparing our PICME with the two baselines areshown in Table 6, which verify the effectiveness of our method clearly.
Method
[Table]
VQ-VAE [38]
[Table]
AOT-GAN [53]
[Table]
[Table] [Table] G Evaluations on the Number of Primitives We supplement the experiments with different number of primitives, i.e. , k = 1 , 2 , 3 , 4 , 5 .
Theexperiments are conducted on CelebA-HQ using center masks.
The implementation details keep thesame as the details in the main paper. We provide experimental results in Table 7. The results showthat the increase of k can diversify the completed images, without degenerating the performance withrespect to common evaluation metrics.k PSNR
[Table]
H Higher-Resolution Image Completion Results
Before this section, we resize the image to 256 × 256 for image completion.
Here, to show that ourmethod can generate higher-resolution images, we set the image size to 512 × 512 . Experiments areperformed on the datasets FFHQ. The results are provided in Figure 20. I Why “Mixture-of-Experts” is used?
Notice that, recent works exploit the term “Mixture-of-Experts (
MoE
)”, e.g. , [33, 10].
Althoughthe works are famous nowadays, we think our design is still a kind of the MoE structure underthe definition proposed in [17].
In more detail, [17] calls the structure with experts and a gatingfunction switching the expert as the MoE structure.
In our design, we consider the component
Gaussians as experts and the Categorical distribution as the gating function.
Therefore, the term“Mixture-of-Expert” is used in this paper.
J Possible Negative Impacts and Limitation
For possible negative impacts, since our PICME is a method based on generative models, like a largenumber of methods of this kind, generated images with our method may danger to the security of
AI systems, with the form of deep fakes.
There are some advanced works targeting the issue, e.g. ,[32, 35], which are helpful to handle the possible negative impacts. For the limitation, our PICME does not outperform ICT sometimes with respect to common metrics,since ICT uses a much larger transformer-based structure and iterative Gibbs sampling, which make ICT’s inference slow. For the people preferring the performance on common metrics, this would be alimitation of our PICME.