Pluralistic Image Completion with Probabilistic Mixture-of-Experts
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Pluralistic image completion focuses on generating both visually realistic anddiverse results for image completion.
Prior methods enjoy the empirical successesof this task.
However, their used constraints for pluralistic image completion areargued to be not well interpretable and unsatisfactory from two aspects.
First, theconstraints for visual reality can be weakly correlated to the objective of imagecompletion or even redundant .
Second, the constraints for diversity are designed tobe task-agnostic , which causes the constraints to not work well.
In this paper, toaddress the issues, we propose an end-to-end probabilistic method.
Speciﬁcally,we introduce a uniﬁed probabilistic graph model that represents the complexinteractions in image completion.
The entire procedure of image completion isthen mathematically divided into several sub-procedures, which helps efﬁcientenforcement of constraints.
The sub-procedure directly related to pluralistic resultsis identiﬁed, where the interaction is established by a Gaussian mixture model(GMM).
The inherent parameters of GMM are task-related , which are optimizedadaptively during training, while the number of its primitives can control thediversity of results conveniently.
We formally establish the effectiveness of ourmethod and demonstrate it with comprehensive experiments.
Introduction
Pluralistic image completion refers to the task of ﬁlling in the missing region of an incomplete image,so as to produce visually realistic and diverse image completion solutions [52–54].
Different fromsingle image completion [35, 12, 48, 44] that learns a deterministic mapping from an incompleteimage to a complete image and produces a unique result, pluralistic image completion can generatevarious results with visually realistic contents.
Pluralistic image completion follows the fact thatimage completion is a highly subjective process and beneﬁts a series of applications such as photorestoration [25], object removal [13] and transmission error concealment [33].
Pluralistic image completion is complicated and challenging.
The state-of-the-art methods encodeimages to latent features.
To constrain the visual reality of results, heuristic constraints are introduced, e.g. , the perceptual loss [25] and attention module [52, 40].
To constrain the diversity ofresults, the distribution of latent features is assumed to be a unimodal Gaussian distribution withpredeﬁned parameters [52].
Image completion is performed by sampling from the unimodal Gaussiandistribution.
Although the above paradigm enjoys empirical successes of pluralistic image completion, it is notwell interpretable and unsatisfactory in practice.
We detail the issues from two aspects.
First, theadded constraints for the visual reality of results are based on general purposes .
These constraintsmay work well in some tasks.
Unfortunately, there is no clear understanding of what role they playin pluralistic image completion.
Inappropriate constraints are likely to have side effects.
It is noteasy to ﬁnish the determination of added constraints as our desideratum, especially in complex image completion tasks.
Second, the diversity of image completion results is hard to be constrained inreason.
For a speciﬁc task, the parameters of the unimodal Gaussian distribution are directly relatedto the diversity.
Nevertheless, these parameters are designed to be prior knowledge and task-agnostic ,which cause unreasonable pluralistic results .
[Figure]
In this paper, to address the above issues, we presentan end-to-end probabilistic method.
The method isconstructed with a uniﬁed probabilistic graph model[16, 21] and translates the problem of pluralistic image completion into a structured mathematical representation.
Speciﬁcally, we suppose that pluralisticimage completion can be represented with a directedacyclic graph that is designed reasonably.
The nodesof the graph denote different kinds of images and corresponding latent features.
The edges of the graphrepresent the interactions between nodes.
Based onthis directed acyclic graph, the entire procedure of pluralistic image completion is divided into several subprocedures.
Then, we perform different constraintsfor different sub-procedures as our desideratum.
In particular, the diversity of image completion resultsis implemented by modeling the interaction betweenthe latent features of the missing region and incomplete image with a Gaussian mixture model (GMM)[36, 57].
Compared with the unimodal Gaussian distribution, GMM has a larger capacity and is morecompetitive to meet the output diversity [49, 34].
Forour method, the inherent parameters of GMM are taskrelated and optimized adaptively during training.
The different primitives of GMM represent theoutputs with different patterns.
Additionally, the number of primitives can be chosen artiﬁcially withthe needs for diversity in real-world applications.
Before delving into details, we clearly emphasize our contributions as follows:
• To our best knowledge, this paper is the ﬁrst one that uses a PGM for pluralistic image completion.
Besides, the PGM is specially designed with justiﬁcations, rather than borrowing existing models.
With the designed PGM, the added constraints for visual reality and diversity are explainable.
• We propose to use GMM to increase result diversity.
One remarkable advantage is that the inherentparameters for diversity are task-related, leading to more reasonable results.
• We conduct a series of experiments on benchmark datasets to support our claims.
In both qualitativeand quantitative comparisons with the state-of-the-art methods, our method achieves superiorperformance.
The generated contents for image completion are both visually realistic and diverse,such as those shown in Figure 1.
Codes will be released for the reproducibility of results.
The rest of the paper is organized as follows.
In Section 2, we brieﬂy review pluralistic imagecompletion.
In Section 3, we discuss the proposed method step by step.
Experimental results areprovided in Section 4.
Finally, we conclude the paper in Section 5.
Preliminaries
Problem Setting.
We ﬁrst discuss image completion.
For the task of image completion, suppose we have animage, originally I o , but degraded by some missingpixels to become a masked partial image I m .
Theimage that comprises the original missing pixels iscalled a complement partial image, which can be denoted by I c .
The goal of the image completion taskis to reconstruct I o by using I m .
The illustrations fordifferent kinds of images are provided in Figure 2.
Traditional single image completion reconstructsthe original image in a deterministic fashion, which results in only a single solution [12, 9].
Pluralistic image completion [52] is motivated by the fact that image completion is a highly subjectiveprocess.
Also, completed images by different experts may agree on high-level semantics, but havesubstantially different details [52].
Therefore, in pluralistic image completion, we need to generatemultiple and diverse plausible results when presented with a masked partial image.
The problem ofpluralistic image completion is challenging since we need to take care of the diversity and visualauthenticity of results at the same time.
Prior Work.
Traditional methods on single image completion, e.g., diffusion-based methods [3, 1, 22]and patch-based methods [2, 4, 5], assume that image holes share similar content to visible regions.
Therefore, they choose to directly propagate the contextual appearances or realign the backgroundpatches to complete the image holes.
Beneﬁting from the strong power of deep neural networks,deep learning based single image completion methods achieve promising completion performance,e.g., employing shepard convolutional neural networks [35], restraining global and local consistencybetween image holes and visible regions [15], performing deep feature rearrangement [41], andlearning a pyramid-context encoder network [47], etc .
In addition to these methods, multiple singleimage completion methods involve the generative adversarial networks [10] to learn the semanticof images [42, 43], e.g., [23, 45, 8, 6, 50, 12, 9].
Although these single-solution methods achieveoutstanding performance in predicting deterministic result for image holes, they cannot generatevarious semantically meaningful results.
Existing works for pluralistic image completion usually rely on generative models, e.g. , CVAE [39, 13,52], hierarchical VQ-VAE [33], PD-GAN [25], and BicycleGAN [56].
Generative pluralistic imagecompletion methods diversiform meaningful completion results with different kinds of constraints.
However, such methods are not well interpretable as discussed.
The issues largely limit theirapplications in the real world.
Methodology
This section presents a probabilistic method for pluralistic image completion.
We ﬁrst design aprobabilistic graph model for our task (Section 3.1).
Then, based on the probabilistic graph model, weshow how to divide the entire procedure of pluralistic image completion into several sub-procedures(Section 3.2).
Afterward, we discuss how to use GMM to diversify image completion results (Section3.3).
Finally, we add reconstruction and adversarial losses to strengthen image completion, andsummarize all algorithm ﬂows (Section 3.4).
3.1 Probabilistic Graph Model Construction
[Figure]
Recall the notations in image completion, i.e. , I o , I m , and I c , they correspond to the latent featuresz o , z m , and z c respectively.
We suppose a probabilistic graph model for the image completion task,which is shown in Figure 3.
The probabilistic graph model is designed reasonably.
The graphicalillustration of overall computational paths of the probabilistic graph model is presented in Figure 3(d).
Speciﬁcally, the images I m and I c form I o .
Their latent features z m
and z c can be exploited to form
I o .
Given z m , we can surmise z c .
We use the latent features here, since they are low-dimensionaland informative [11, 20].
We further details Figure 3 as follows.
Inference.
We do not directly map I m
and I c to I o , since they are too high-dimensional and haveinformation redundancy.
We exploit generative models.
The processes I m !
z m
and I c !
z c denote 131 that we encode images into corresponding latent features.
During training, we learn the encoder f toﬁnish the inference.
Dynamic Inference.
Given I m ( resp.
I c ), we can infer the latent feature z m ( resp.
z c ).
Afterward,z m 99K z c means that we use a variational distribution p ( z c | z m ) to approximate the distribution 135 p ( z c |
Generation.
In image completion, the images I m
and I c can compose the image
I o .
Accordingly,for their latent features, we splice z m and z c to output I o .
During training, we learn the decoder g toﬁnish the generation.
3.2 Image Completion Objective Decomposition
We cannot observe the underlying I c in the test procedure.
Based on the probabilistic graph model,we perform the following variational inference with the Kullback-Leibler (KL) divergence:
[Equation]
where q ( ·|· ) and p  ( ·|· ) are two conditioned distributions, with and  being the parametersof their corresponding functions.
The divergence (1) is minimized with respect to all parameters.
It should be noted that, in fact, the overall objective should be KL [ q ( I o | I m )
k p  ( I o | I m , I c )] =
KL [ z
[Equation]
, z
[Equation]
[ q ( I o , z m , z
[Equation]
I
[Equation]
, z
[Equation]
[ p  ( I o , z m , z c | I m , I c )]] .
As the expectation in this formula 147 is intractable , we need to deal with the 1-step Monte Carlo sampling objective, which is why thedivergence (1) is formed.
Another advantage of using the divergence (1) is that constraining eachsampling point can make the overall constraint tighter.
We further decompose the divergence (1)based on the probabilistic graph model as follows.
Proposition 1 Regarding the KL divergence (1), we show that the divergence can be decomposed as: [Equation]  ( I o | I m , I c )] [Equation]
|
[Equation]
|
[Equation] } c  , [Equation] where q ✓ is the variational distribution, and the parameters ✓ are involved in divergence minimization.
The proof can be found in Appendix A.1.
We analyze three terms in Eq.
(2) combining the probabilistic graph model and our purpose:
• For a  , as discussed, we can generate I o with I m
and I c , or with their latent features z m and z c . We minimize the distance about the generation of I o in the two ways.
The minimization can ensure that the generation from the images and latent features is consistent .
• For b  , we do not limit the inference from I c to z c as a deterministic function.
Instead, to diversifyoutputs, we use the variational distribution to restrain q ✓ ( z c | z m ) .
The choice and discussion aboutthe variational distribution will be provided in Section 3.3. • For c  , it aims to keep that the inference using the variational distribution and the inference using the posterior is close , which also guarantees the reliability of sampling.
Besides, the variational distribution does not include I c to enable latent features to be used in test.
We employ the variational evidence lower bound (ELBO)
L ELBO [20] to constrain c  .
It is because, suppose that p ( I m ) is aconstant given I m , it is equivalent for maximizing the L ELBO and minimizing c  .
The details are provided in Appendix A.4.
3.3 Mixture-of-Experts Model
As discussed, we need a suitable choice for the variational distribution q ✓ ( z c | z m ) to output pluralisticresults.
In this work, we employ the Gaussian Mixture Model (GMM) [28, 29] for the dynamic modeq ✓ ( z c | z m ) , which is more ﬂexible than a unimodal and can diversify outputs.
We denote the latent features of I c that are inferred from z m by ˆ z c , which is distinguished from z cachieved by the encoder.
For the latent features z , we denote the mean and covariance by µ z and ⌃ z .
Mathematically, we have
[Equation]
where k denotes the number of primitives, ↵ i denotes the mixing coefﬁcient of the i -th primitive, andµ i , ⌃ i denotes the distribution parameters of the i -th primitive.
Note that the mixing coefﬁcient ↵ i is the parameter of the Categorical distribution [29].
Intuitively, in Eq.
(3), we model z m with GMM,where the i -th primitive can be used to represent ˆ z ( i ) c , i.e. , µ ˆ z c i = µ z m i and ⌃ ˆ z c i =
⌃ z m i .
In this way,we can obtain different ˆ z c to meet diversity requirements.
Besides, among all the parameters of GMM, except for k , the other parameters are called the inherentparameters of GMM.
In GMM (3), the number of primitives k can be determined artiﬁcially accordingto the need for output diversity.
The inherent parameters are optimized adaptively during training.
Speciﬁcally, for the optimization of the mixing coefﬁcients ↵ =
[ ↵ 1 , . .
. , ↵ k ] , we exploit the thefrequency loss [34].
The objective is
[Equation]
For our task, with the frequency loss (4), the accumulated frequency approximate gradient is anasymptotically unbiased estimation of the true gradient.
We provide the detailed derivation andanalysis in Appendix A.2.
The reason we mention the frequency loss is that, the metric for evaluatingthe performance of each primitive is adapted to KL divergence in our task, which is different fromthe original metric in [34].
For the optimization of µ and ⌃ of z m , we utilize the back-propogatemax-operation to reserve the distinguishable property of GMM (3).
For our problem, the objective ofthe back-propogate-max-operation is shown in the following proposition.
Proposition 2 By modeling the variational distribution with GMM (3), we have
[Equation]
[Equation]
where j = argmin i KL h N ( z [Equation] | µ [Equation]
i , ⌃
[Equation] z [Equation] , ⌃
[Equation]
) i , and tr ( · ) denotes the trace of a matrix.
The derivation of Proposition 2 can be found in Appendix A.3.
For Proposition 2, if we minimize
L BM , we optimize µ and ⌃ of z m , which reduces the distance between ˆ z ( j ) c and z c .
For the term b in Eq.
(2), we then have L GMM =
[Equation]
Discussion.
Note that the diversity of results is directly related to z m .
When GMM for z m isdominated by the i -th primitive ( ↵ i ⇡ 1 ), the diversity of image completion results is reduced.
Although this phenomenon did not appear in our experiments, it may exist in practice.
Actually, thisphenomenon is reasonable.
From human cognition, if I m shows only one pattern , the inference for
I c will be deterministic [3].
For example, if the size of the missing region of an incomplete image isvery small, we can infer the missing region both deterministically and reasonably, where the diversityof results is reduced.
Prior methods with task-agnostic constraints do not consider this fact, whichmay create unreasonable results .
For our method, the constraints for diversity are task-related, whichcan take this phenomenon into account.
3.4 Reconstruction and Adversarial Losses
To ensure the minimization of the term a  in Eq.
(2), we incorporate the use of reconstruction andadversarial losses [10].
The reconstruction loss is formulated as:
[Equation]
where ˆ I o is generated from z m and z c , and ˆ I ( j ) m is generated from ˆ z ( j ) c .
Here, ˆ z ( j ) c can be inferredfrom z m by using Eq.
(3) and Eq.
The image ˆ I ( j ) m is obtained from ˆ I ( j ) o with the image maskthat can degenerate I o to I m .
The image ˆ I ( j ) o is generated from z m and ˆ z ( j ) c .
The reconstructionloss (6) controls the visual rationality of image completion results.
Also, we efﬁciently avoid thedeterministic fashion with GMM.
Furthermore, we leverage adversarial training [10] to make thegenerated images more realistic.
The adversarial loss L A is formulated as
[Equation]
where D is the discriminator optimized by the discriminator loss based on LSGAN [27].
We jointlytrain our encoder f and decoder g through the following combined loss:
[Equation]
where the weight  A is set to 0.05 in all experiments.
Then, the ﬁnal objective of the loss functionis the sum of three losses, i.e. , L FINAL =
L GMM + L ELBO + L C .
The ﬁnal loss is to maximizethe log-likelihood of the conditional data distribution.
Speciﬁcally, the adversarial loss uses the
Wasserstein distance to make generated images realistic given latent features.
The ELBO lossuses KL divergence and distribution log-likelihood to make variational posterior approximate realposterior.
Algorithm Flows.
For the convenience of following technical details, we provide the algorithm ﬂows.
The algorithm ﬂows of training and test stages can be found in Algorithm 1 and Algorithm 2.
Theillustration is shown in Appendix B.
It should be noted that the number of primitives k does not mean that we are limited to generateonly k diverse results given I m (in Steps 3 and 4 of Algorithm 2).
In fact, we can sample from the kprimitives to obtain lots of image completion results.
Moreover, as GMM has a much larger capacitythan the unimodal Gaussian distribution [29], we can achieve greater diversity.
We provide empiricalobservations in Section 4.3 and Appendix C.
Experiments
In this section, we conduct a series of experiments to justify our claims.
We ﬁrst introduce theimplementation of our method (Section 4.1).
The comprehensive experimental results and comparisonwith advanced methods are then provided and discussed (Section 4.2).
Finally, we conduct an analysisstudy to present and discuss our method in more detail (Section 4.3).
4.1 Implementation Details
Datasets.
We evaluated our proposed model on ﬁve popularly used datasets, i.e. , CelebA-HQ [17, 26],
FFHQ [18], Paris StreetView [7], {+Places
 [55],+} and ImageNet [37].
We verify the effectivness of ourmethod with different types of mask regions, including both center and random masks [52, 40].
Network and Optimization.
The proposed method can be implemented efﬁciently.
The encoderand decoder of our pipeline are inspired by PIC [52] for comparison.
We apply average pooling andinterpolation with convolutional layers to implement downsampling and upsampling respectively.
During optimization, we use the Adam optimizer [19].
The learning rate is ﬁxed to 10  4 during thetraining procedure.
Baselines.
We compare the proposed method with the following state-of-the-art methods, whichinclude (1) single image completion methods: DFv2 [46], EC [30], and MED [24]; (2) pluralisticimage completion methods: PIC [52] and ICT [40].
We abbreviate our method ( P luralistic I mage
C ompletion with Probabilistic M ixture-of- E xperts) as PICME.
The methods are implemented by
PyTorch and evaluated on NVIDIA Tesla A100 GPUs.
Measurement.
For measurement, we provide both qualitative and quantitative results.
For themetrics of quantitative results, we use common evaluation metrics such as the peak signal-to-noiseratio (PSNR), structural similarity (SSIM), mean absolute error (MAE), and fr ´ echet inception distance (FID) [14] to measure the similarity between the image completion result and ground truth.
Furthermore, pluralistic image completion is supposed to focus on generating diverse realistic resultsrather than merely approximating ground-truth ones [51].
To measure the result diversity , we addtwo perceptual quality metrics to quantitative results, which include LPIPS [52] and DIV-FID [40].
[Figure]
[Figure] [Figure]
[Figure] [Figure]
4.2 Comparison with Prior Methods
4.2.1 Qualitative Comparisons.
We provide extensive qualitative comparison results tojustify our claims.
First, we show the results on CelebAHQ and FFHQ in Figure 4, which are images related tohuman faces.
As can be seen, the image completion results achieved by our method are realistic, following highimage quality.
Also, compared with the baseline PIC, ourcompletion results are more diverse, e.g. , see the imagesin the second and last rows.
Second, we provide the results on Paris StreetView and
Places in Figures 5 and 6 respectively.
For the results on
Paris StreetView, we can see that image completion bythe baselines MED and PIC are somewhat distorted.
Bycontrast, our method can ﬁnish the image completion task better.
The details can be checked in thecomplements to walls.
Then, we turn the attention to the image completion results on Places2.
Ourmethod still achieves superior performance compared with baselines.
Lastly, we provide the results on ImageNet in Figure 7.
Note that the sources of the data in ImageNetare very complicated.
The image completion task is rather challenging on this dataset, even for singleimage completion [52].
Different from some prior work [32, 15] that were trained on a 100k subsetof training images of ImageNet, we directly the network on the original ImageNet training datasetwith all images.
The results mean that our method can infer the content effectively.
Reﬁned Result Analysis.
We provide some reﬁned results in Figure 8 to analyze the image completion details of our method.
For the face images, we can ﬁnd that the image completion results havevery different facial expressions.
[Table]
4.2.2 Quantitative Comparisons
[Table]
Common Metrics.
Quantitative evaluation with common metrics is difﬁcult for pluralistic image completion,as our goal is to obtain diverse butreasonable solutions for a masked image [52].
To make a feasible comparison, as did in [52], we sample onecompleted image many times from kimages to compare with the originalimage.
The results are provided in
Tables 1 and 2.
As can be seen, for
FFHQ and Places2, on both the settings of center and random masking, our method achieves superiorperformance.
On ImageNet, our method performs the best with the center masking consistently,while it is competitive with the random masking.
It should be noted that ICT is a very strong baseline and works better than our method in somecases.
It is because ICT is built with the image transformer [31], which is powerful in vision tasks.
Compared with ICT, the advantages of our method lies in not only better interpretability as discussed,but also faster inference speeds.
We will demonstrate this in inference time.
[Table]
Diversity Metrics.
For pluralistic im- [Table] age completion, the diversity evaluations are signiﬁcant.
We generate ﬁvecompletion images in each task andreport the mean of the diversity metric.
The results are shown in Table 3,which represents that our method cangenerate more diverse completion images than baselines.
The results withcommon and diversity metrics support our claim very well.
That is to say, the image completionresults of our method are both high-quality and pluralistic.
[Table]
Inference Time.
We report the inference timeof our method, compared with PIC and ICT.
All [Table] inference runs on one NVIDIA Tesla A100 GPUfor fairness.
The size of images is 256 ⇥ 256 .
We perform pluralistic image completion for 100different images.
For each image, we generate 6completion results for it.
We report the total inference time for these images.
The results are shownin Table 4.
Note that as ICT is a transformer-basedmethod, the calculated consumption of its inference is much heavier.
What’s worse, ICT needs iterative Gibbs sampling during inference.
The two issues make ICT have to face heavy computationalconsumption, which is also mentioned in [40].
In contrast, our method is more inference-efﬁciency,which is more than 750 times faster than ICT .
The merit could make our method easier to use inthe real world.
4.3 More Analyses and Justiﬁcations
[Figure]
GMM Primitive Visualization.
We further stress the diversity of the completion results achievedby our method.
Speciﬁcally, we visualize the ˆ z c which is inferred from z m as discussed.
Thevisualization results are obtained with t-SNE [38].
Different ˆ z c are presented by the 2D-vectors indifferent colors, which are shown in Figure 9.
As can be seen, the vectors are scattered , which clearlydemonstrates that the completion results of our method are diverse.
Reﬁned Diversity Comparison.
We argue that, beneﬁting from that GMM has larger capacities thana unimodal Gaussian distribution, the completion results of our method would be more diverse.
Tojustify our claims, we provide the reﬁned diversity comparison of our method with PIC and ICT, asshown in Figure 10.
Note that the six image completion results of PIC are obtained by sampling fromone unimodal Gaussian distribution.
The results of our method are obtained by sampling from sixprimitives.
Clearly, our results are more diverse than PIC’s.
Besides, for ICT, it does not explicitlycontrol the output diversity.
The diversity of pluralistic image completion is thus less interpretablethan our method.
In addition, we state that our method is not limited to only generating k completion results before.
Infact, we can sample from k primitives of GMM to obtain lots of completion results.
Due to the limitedpage, we present the results and more analyses in Appendix C. Moreover, the richer completion resultcomparison is presented in Appendix D.
Conclusion
In this paper, we focus on the complicated and challenging problem of pluralistic image completion.
We propose a novel end-to-end probabilistic method for this challenging problem.
Based on theprobabilistic graph model, our method divides the entire procedure of pluralistic image completioninto several sub-procedures, where GMM is used to diversify outputs.
Experiments on a variety ofdatasets show that the image completion results by our method are both high-quality and pluralistic.
In future work, investigating the feasibility of the proposed method for other diverse decision-makingscenarios might prove important.
[-i forparameters θ i , and 1 best i is the indicator function, where 1 best i = 1 if i = j and 1 best i = 
-]