T OWARDS
C ONTROLLABLE
AND
I
NTERPRETABLE
F ACE
C OMPLETION
VIA
S TRUCTURE -A
WARE
AND
F REQUENCY -O RIENTED A TTENTIVE GAN S
Anonymous authors
Paper under double-blind review
A BSTRACT
Face completion is a challenging conditional image synthesis task.
This paper proposes controllable and interpretable high-resolution and fast face completion by learning generative adversarial networks (GANs) progressively from low resolution to high resolution.
We present structure-aware and frequency-oriented attentive GANs.
The proposed structure-aware component leverages off-the-shelf facial landmark detectors and proposes a simple yet effective method of integrating the detected landmarks in generative learning.
It facilitates facial expression transfer together with facial attributes control, and helps regularize the structural consistency in progressive training.
The proposed frequency-oriented attentive module (FOAM) encourages GANs to attend much more to ﬁner details in the coarse-to-ﬁne progressive training, thus enabling progressive attention to face structures.
The learned FOAMs show a strong pattern of switching their attention from low-frequency to high-frequency signals.
In experiments, the proposed method is tested on the CelebA-HQ benchmark.
Experiment results show that our approach outperforms state-of-the-art face completion methods.
The proposed method is also fast with mean inference time of 0 .
54 seconds for images at 1024 × 1024 resolution (using a Titan Xp GPU).
I NTRODUCTION
Conditional image synthesis aims to learn the underlying distribution governing the data generation with respect to the given conditions/context, which is also able to synthesize novel content.
Much progress (Iizuka et al., 2017; Yeh et al., 2017; Li et al., 2017; Yang et al., 2016; Denton et al., 2016;
Pathak et al., 2016; Yu et al., 2018; Liu et al., 2018; Brock et al., 2018;
Karras et al., 2018) has been made since the generative adversarial networks (GANs) were proposed (Goodfellow et al., 2014).
Despite the recent remarkable progress, learning controllable and interpretable GANs for high-ﬁdelity image synthesis at high resolutions remain an open problem.
We are interested in controllable and interpretable GANs.
We take a step forward by focusing on high-resolution and fast face completion tasks in this paper.
Face completion is to replace target regions, either missing or unwanted, of face images with synthetic content so that the completed images look natural, realistic, and appealing.
State-of-the-art face completion approaches using
GANs largely focus on generating random realistic content.
However, users may want to complete the missing parts with certain properties (e.g. expressions).
Controllability is entailed.
Existing face completion approaches are usually only able to complete faces at relatively low resolutions (e.g. 176 × 216 (Iizuka et al., 2017) and 256 × 256 (Yu et al., 2018)).
To facilitate high-resolution image synthesis, the training methodology of growing GANs progressively (Karras et al., 2017) is widely used.
For face completion tasks, one issue of applying progressive training is how to avoid distorting the learned coarse structures when the network is growing to a higher resolution.
Interpretability is thus entailed to guide GANs in the coarse-to-ﬁne pipeline.
In addition, most existing approaches (Iizuka et al., 2017; Yeh et al., 2017; Li et al., 2017) require post-processing (e.g. Poisson Blending (P´erez et al., 2003)), complex inference process (e.g. thousands of optimization iterations (Yeh et al., 2017) or repeatedly feeding an incomplete image to CNNs at multiple scales (Yang et al., 2016)) during test.
We present structure-aware and frequency-oriented attentive GANs that are progressively trained for high-resolution and fast face completion using a fast single forward step in inference without any post-processing.
By controllable, it means that the completed face images can have different facial attributes (e.g., smiling vs not smiling) and/or facial expressions transferred from a given source actor.
By interpretable, it means that the coarse-to-ﬁne generation process in progressive training is rationalized.
We utilize facial landmarks as backbone guidance of face structures and propose a straightforward method of integrating them in our system.
We design a novel Frequency-Oriented
Attention Module (FOAM) to induce the model to attend to ﬁner details (i.e. higher-frequency content, see Fig. 1).
We observe signiﬁcant improvement of the completion quality by the FOAM against the exactly same system only without FOAM.
A conditional version of our network is designed so that the appearance properties (e.g. male or female), and facial expressions of the synthesized faces can be controlled.
Moreover, we design a set of loss functions inducing the network to blend the synthesized content with the contexts in a realistic way.
Our method was compared with state-of-the-art approaches on a high-resolution face dataset CelebA-HQ (Karras et al., 2017).
Both the evaluations and a pilot user study showed that our approach completed face images signiﬁcantly more naturally than existing methods.
R ELATED W ORK
Recent learning based methods have shown the capability of CNNs to complete large missing content.
Based on existing GANs, the Context Encoder (CE) (Pathak et al., 2016) encodes the contexts of masked images to latent representations, and then decodes them to natural content images, which are pasted into the original contexts for completion.
However, the synthesized content of CE is often blurry and has inconsistent boundaries.
Given a trained generative model, Yeh et al.
(Yeh et al., 2017) propose a framework to ﬁnd the most plausible latent representations of contexts to complete masked images.
The Generative Face Completion model (GFC) (Li et al., 2017) and the Global and
Local Consistent model (GL) (Iizuka et al., 2017) use both global and local discriminators, combined with post-processing, to complete images more coherently.
Built on GL, Yu et al.
(Yu et al.,
2018) design a contextual attention layer (CTX) to help the model borrow contextual information from distant locations.
Liu et al.
(Liu et al., 2018) incorporates partial convolutions to handle irregular masks.
Unfortunately, these approaches can only complete face images in relatively low resolutions (e.g. 176 × 216 (Iizuka et al., 2017) and 256 × 256 (Yu et al., 2018)).
Yang et al.
(Yang et al., 2016) combine a global content network and a texture network, and the networks are trained at multiple scales repeatedly to complete high-resolution images ( 512 × 512 ).
But, they assume that the missing content always shares some similar textures with the context, which is improbable for the face completion task.
T HE P ROPOSED M ETHOD 3. P ROBLEM F ORMULATION
[Figure]
Denote by Λ an image lattice (e.g., 1024 ×pixels).
Let I Λ be a face color image deﬁned on the lattice Λ .
Denote by Λ t and Λ ctx the target region to complete and the remaining context region respectively (note that the target region is not necessarily a single connected component, and the two parts form a partition of the lattice).
I Λ t is masked out with the same gray pixel value.
Let M Λ be a binary mask image with all pixels in M Λ t being 1 and pixels in M Λ ctx being 0 .
For simplicity, we will omit the subscripts Λ , Λ t and Λ ctx when the text context is clear.
Unlike existing approaches (Pathak et al., 2016; Li et al., 2017; Iizuka et al., 2017) which ﬁrst utilize unconditional image synthesis to generate the target region image and then blend them with context using using sophisticated post-processing, we address the completion problem as a coherent conditional image generation process.
As illustrated in Fig. 2, given an observed image I obs with the target region I obs Λ t masked out from a ground-truth uncorrupted image I gt , the objective of the proposed face completion is to synthesize an image I syn that looks natural and realistic, and to enable a controllable generation process in terms of a given facial attribute vector, denoted by A (such as male vs female, and smiling vs not smiling and for simplicity we use binary attribute vector in this paper) and/or a given facial expression encoded by facial landmark, denoted by L .
Denote by X G
=
( I obs , M, A, L ) the input of the generator G ( · ) that realizes the completion.
We have,
[Equation]
subject to
[Equation]
where θ G collects all parameters of the generator and ≈ represents that the two context regions I syn Λ ctx
and I obs Λ ctx need to be kept very similar.
Structure-Aware Completion.
As illustrated in Fig.
3 (left), to enable transferring facial expressions in completion, we leverage the off-the-shelf state-of-the-art facial landmark detector, Face
[Figure]
Alignment Network (FAN) (Bulat & Tzimiropoulos, 2017) which achieved very good results for faces in the wild.
Motivated by this, we also want to integrate the landmark information in completion for faces without facial expression transfer required.
Recent works (Isola et al., 2016; Wang et al., 2017; Zhu et al., 2017; Sangkloy et al., 2017; Xian et al., 2017; Chen & Hays, 2018) have shown the capability of GANs to translate sketches to photo-realistic images.
We choose facial landmarks as an abstract representation of face structures in general.
As illustrated in Fig.
3 (right), we ﬁrst train a simple face completion model at the resolution of 256 × 256 using reconstruction loss (Section 3.3) only.
Given an image 1 , we use the trained model to generate a blurry completed image from which the landmarks are extracted with FAN (we observed that FAN can compute sufﬁciently good landmarks from blurry completed images).
Not only can this unify the generation process for different controllable settings (since the inputs to the generator are kept the same between with and without facial expression transfer), but it also makes the completion process structure-aware.
Since faces have very regular structures (e.g. the eyes are always above a nose), when some facial components are occluded, it is possible to predict which parts are missing.
Given a corrupted image, the quality of synthesized image can be further improved if the model is able to “draw” a sketch of the face ﬁrst, which provides backbone guidance for image completion.
3. L EARNING WITH THE FOAM BETWEEN P ROGRESSIVE S TAGES
On top of GANS (Goodfellow et al., 2014), the framework of Context Encoder (CE) (Pathak et al., 2016) is adopted, so the generation process of our model is conditioned on the contextual information.
The framework of training GANs progressively (Karras et al., 2017) is also adopted to facilitate a high-resolution completion model.
This starts with the lowest resolution (such as 4 × 4 ).
After running a certain number of iterations, higher resolution layers are added to both the generator and discriminator simultaneously until the network is grown to a desired resolution (such as 1024 × 1024).
We present details of the proposed FOAM to stabilize and rationalize the progressive training.
Denote by G r and D r the generator and discriminator at a resolution level r , respectively, where r ∈ { 1 , · · · , R } is the index of resolution (e.g., r = 1 represents 4 × 4 and r =
R = 9 represents 1024 × 1024 ).
The ﬁnal stage generator G R () will be used as the generator G in Eqn. 1 in testing.
The observed masked image, its corresponding binary mask, and the facial landmarks are re-sized to I obs r , M r and L r for each resolution respectively.
In our model, both G r and D r are conditioned on facial landmarks.
We attach the resolution index to the input and rewrite Eqn. 1 as,
I syn r
[Equation]
r ) , subject to I
syn r
[Equation]
Λ ctx
≈ I obs [Equation] , (2) where X G r
=
( I obs r , M r , A, L r ) .
For the discriminator D r , its input is X D r =
( I r , L r ) where I r represents either an uncorrupted image or a image synthesized by G r .
D r has two branches which share a common backbone and predict the fake vs real classiﬁcation and the attribute estimation ˆ A respectively.
The loss functions for training are deﬁned in Section 3.3.
During progressive training, to avoid sudden changes to the trained parameters of G r − 1 , the added layers (i.e. the higher resolution components) need to be faded into the networks smoothly during a growing stage .
Since the parameters of added layers are initialized randomly, these layers may generate noise that distorts the coarser structures learned by G r − 1 if they are merged with G r − 1 directly.
To reduce this effect, Karras et al.
(Karras et al., 2017) use a linear combination of the higher and lower resolution branches.
The synthesized image ˆ I syn is computed by
[Equation]
in which I syn r and ˜ I syn r − 1 are the output images from the higher and lower resolution branches respectively ( ˜ I syn r − 1 is up-sampled from I syn r − 1 to match the resolution of r ).
α is a weight increasing linearly from zero to one during the growing stage.
Therefore, at the beginning, the added layers have no impact on the network.
During training, the inﬂuence of the higher resolution branch increases linearly while the weight of the lower-resolution branch decreases.
In the end when α = 1 , the synthesized image depends only on the higher resolution branch (i.e. ˆ I syn =
I syn r ) and the lower resolution branch can simply be removed.
Because of this, once the training is complete, a cor-
T o F ilt e r
D o w n 64 X32 X+ +
U p
Skip 64 X
ToFilter ToRGB ToRGB D o w n     −       −   −    −  ̂   ̂   ̂  −     −   rupted image only needs to be fed to a single branch for image completion, and this process does not depend on any inputs or networks of lower resolutions.
The FOAM.
Eqn. 3 is equivalent to applying “all-pass ﬁlters” to the higher and lower resolution branches, since all the pixels in images are assigned the same weight (i.e. α or 1 − α ) regardless of their locations.
Although this linear combination (Eqn. 3) has been shown effective for reducing the impact of noise generated during the growing stage, we observe that it does not work well for highresolution face completion, as shown in Fig. 4.
The coarse structures that have been learned well at lower resolutions are still vulnerable to being distorted during the joint training (i.e., 0 < α < 1 ).
The intuitive idea of the proposed FOAM is to encourage the generator to focus more on learning ﬁner details during the growing stage, which is enabled by changing the “all-pass ﬁlters” reﬂected in
Eqn. 3 to attentive “band-pass ﬁlters” that learn to protect what has been learned well in the previous stages and to update ﬁner details as needed under the guidance of the loss functions.
Existing approaches (Gregor et al., 2015; Yu et al., 2018) use spatial attention mechanisms to encourage networks to attend to selected parts of images (e.g. a rectangular region).
As illustrated in Fig.
1, we observe that the FOAM ﬁlters indeed act like “band-pass ﬁlters” and show a strong pattern of switching its attention from coarse structures (i.e. the low-frequency information) to ﬁner details (i.e. the high-frequency information) as the resolution increases.
But, we note that different from regular band-pass ﬁlters, the ﬁlters learned by FOAM are predicted based on image semantics through the objective function (see Equation 14).
This makes them sensitive to locations inferred on-the-ﬂy in a coarse-to-ﬁne manner.
For instance, the model learns to pay more attention to eye regions where the rich details aggregate, especially at high resolutions.
With the help of FOAM, the model is capable of learning meaningful and interpretable ﬁlters automatically.
As illustrated in Fig.
5, the proposed FOAM consists of a read and a write operation.
In the read operation, only information that is important in I obs r
but does not exist in I obs r − 1 will be allowed to enter the network.
Similarly, in the write operation, only when the added layers produce information that can help reduce the overall loss, will it be allowed to add to the synthesized image
ˆ I syn .
The read and write operations, which are like two gates in a circuit, are controlled by the read and write ﬁlters learned by our model, respectively (denoted by F read and F write ).
F read is predicted from the lower resolution branch and computed by,
[Equation]
using a trained generator G ﬁxed r − 1 with ﬁxed weights and a small trainable network ToFilter .
Similarly, F write is predicted from the last feature maps of the higher resolution branch.
The value in the ﬁlters represents the weight.
F read helps extract the most valuable information in the contexts of I obs r and
I obs r − 1 .
The read operation is implemented by,
[Equation]
where (cid:12) denotes element-wise multiplication.
˜
I obs r − 1 is up-sampled from I obs r − 1 to match the resolution of level r .
Similar to Eqn. 3, F read and (1 − F read ) are assigned to the higher and lower resolution branches, respectively.
The write ﬁlter F write combines the outputs from two branches (i.e. I syn r and the up-sampled ˜
I syn r − 1 ) to generate the ﬁnal completed image ˆ I syn r .
F write helps extract the most valuable information in the contexts of I syn r and ˜
I syn r − 1 .
The write operation is deﬁned by, ˆ I syn r
[Equation]
so, only the target region of ˆ I syn r is controlled by F write .
The context region is a linear combination of the contexts of I syn r and ˜
I syn r − 1 .
To facilitate fast face completion in testing, we further design transformation functions to adjust the value ranges of F read and F write , so the lower resolution branches and FOAMs can both be safely removed when the growing process is done.
Similar to the vanilla progressive training method, a testing image only needs to go through the ﬁnal stage for completion.
To that end, a transformation function (Eqn. 7) is used to adjust the upper and lower bounds of the dynamic value ranges of the read and write ﬁlters.
For instance, the transformed ˆ F read starts as an all-zero ﬁlter, is adjusted by a trainable ToFilter at the growing stages, and eventually increased to all ones.
The transformed ﬁlters ˆ
F read and ˆ F write are deﬁned by,
[Equation]
where the parameters are computed by
[Equation]
in which α is a weight increasing linearly from zero to one proportional to the number of seen images during growing.
Eqn.
7 will be actually used in the read operation, Eqn. 5 and the write operation, Eqn. 6.
3. L OSS F UNCTIONS
To induce high-ﬁdelity face completion, we utilize the loss functions as follows.
Adversarial Loss Given an uncorrupted ground-truth image I gt , its attribute vector A , a mask M , landmarks L , and the corresponding corrupted image I obs , we deﬁne the loss by,
[Equation]
where I syn =
G ( I obs , M, A, L ) and D cls represents the classiﬁcation branch of the discriminator.
Attribute Loss Similar to the InfoGAN models (Chen et al., 2016; Choi et al., 2017), for the attribute prediction head classiﬁer in the discriminator, we deﬁne the attribute loss based on crossentropy between the predicted attribute vectors, ˆ A gt =
D attr ( I gt , L ) and ˆ A obs
= D attr ( I obs , L ) , and the corresponding input attribute vectors A for both a ground-truth image and a synthesized image, l
[Equation]
) where D attr represents the attribute prediction branch of the discriminator.
Reconstruction Loss Since our method generates an entire completed face, we deﬁne a weighted reconstruction loss l rec to rebuild both the content and the context regions,
[Equation]
where I diff =
I gt − I syn and κ is the trade-off parameter.
Feature Loss
In additional to the reconstruction loss, we also expect a synthesized image to have similar feature representations (Johnson et al., 2016) to a ground-truth image.
Let φ be a pre-trained deep neural network and φ j be the activation of the j th layer of φ , the feature loss is deﬁned by,
[Equation]
In our experiments, φ j is the relu2 2 layer of a 16-layer VGG network (Simonyan & Zisserman, 2014) pre-trained on the ImageNet dataset (Russakovsky et al., 2015).
Boundary Loss To make the generator learn to blend the synthesized target region with the original context region seamlessly, we further deﬁne a close-up reconstruction loss along the boundary of the mask.
Similar to (Yeh et al., 2017), we ﬁrst create a weighted kernel w based on the mask image
M .
w is computed by blurring the mask boundary in M with a mean ﬁlter so that the pixels closer to the mask boundary are assigned larger weights.
The kernel size of the mean ﬁlters is seven in our experiments.
We have,
[Equation]
Our model is trained end-to-end by integrating the expected loss of the loss functions deﬁned above under the minimax two-player game setting.
The full objective is,
[Equation]
where λ i ’s are trade-off parameters between different loss terms.
Fig.
4 shows an ablation study of the importance of the loss functions.
Note that the ablation study was run at 256 × 256 .
Since the training of high-resolution models was very time consuming, an ablation study for 1024 ×networks is left for future work.
E XPERIMENTS
Datasets and Experiment Settings.
We used the CelebA-HQ (Karras et al., 2017) dataset for evaluation.
It contains 30,000 aligned face images at 1024 × 1024 resolution.
The dataset is split randomly while ensuring there is no identity overlap between test/training sets: 3,009 images for testing, and 26, for training.
There were two types of masks in training: center and random.
The center mask was a square region in the middle of the image with a side length of half the size of the image.
The random masks, generated in a similar way to previous methods (Iizuka et al., 2017; Yu et al., 2018), were rectangular regions with random width-to-height ratios, sizes and locations covering about 5% to 25% of the original images.
Network architectures, hyper-parameters and more results are provided in the Appendix.
Completion without Attribute Control.
We ﬁrst trained a highresolution ( 1024 × 1024 ) model with center masks (examples shown in Fig. 6) to test whether our model is capable of learning high-level semantics and structures of faces and synthesizing large missing regions.
The second model was trained with random rectangular masks, but was able to handle various shaped masks (e.g. irregular hand-drawn masks).
Fig.
6 shows that our model was able to capture the anatomical structures of faces and generate content that is consistent with the holistic semantics.
Completion with Attribute Control.
Two attributes (“Male vs Female” and “Smiling vs Not Smiling”) were chosen.
This model was trained from scratch and the result was run at a 512 ×resolution.
Fig.
7 shows that the attributes of synthesized images were controlled by our model explicitly.
Fig.
8 shows the facial expression transfers together with attribute control.
Quantitative Evaluation.
In current literature (Yeh et al., 2017; Yu et al., 2018), reconstruction metrics such as mean L1 , L2 errors and peak signal-to-noise ratio (PSNR) are commonly used for the evaluation of in-painting methods.
We show the comparison between our method and state- of-the-art models at their reported resolutions respectively (Table 1).
The result shows that our model outperformed state-of-the-art approaches.
In addition to comparing how well each model
[Table]
reconstructs the missing regions, it is also important to evaluate the naturalness of synthesized images since image completion aims to generate realistic and plausible content rather than restoring the original images perfectly.
Due to the lack of good metrics for naturalness, we ran a pilot user study, which is considered the “gold standard” to evaluate GAN models.
Our method obtains signiﬁcantly better results (Fig. 9, see detail in the Appendix) .
Computation Time.
We tested our model with a Titan Xp GPU by processing 3000 1024 × 1024 images with 512 × 512 holes.
The mean completion time is 0.54 second per image.
It takes about one minute for the model of Yang et al.
(Yang et al., 2016) to complete a 512 ×scene image with a Titan X GPU.
Limitations Though our method has low inference time, the training time is long due to the progressive growing of networks.
In our experiment, it takes about three weeks to train a 1024 × 1024 model on a Titan Xp GPU.
By carefully zooming in our results, we ﬁnd that our high-resolution model fails to learn lowlevel skin textures, such as furrows and sweat holes.
Moreover, the model could generate distorted content when removing large parts (e.g. hats) or synthesize some plausible but unnatural faces (Fig. 10).
Furthermore, for facial expression transfer, our method requires that the head poses of the source and target faces are similar.
These issues are left for future work.
C ONCLUSION
We propose a progressive GAN with frequency-oriented attention modules (FOAM) for high resolution and fast controllable and interpretable face completion, which learns face structures from coarse to ﬁne guided by the
FOAM.
By consolidating information across all scales, our model not only outperforms state-of-the-art methods by generating sharper images in low resolution (such as 256 × 256 ), but is also able to synthesize faces in higher resolutions (such as 512 × 512 and 1024 × 1024 ) than existing techniques.
Our attribute and expression controller allows users to manipulate the appearance and facial expressions of generated images explicitly with attribute vectors and landmarks.
Our system is designed in an end-to-end manner, in that it learns to generate completed faces directly and more efﬁciently.
A A PPENDIX
In this section we ﬁrst show more results of high resolution face completion.
Then we present detail of the user study.
We also provide detail of the network architectures and hyper-parameters used in training.
A. M ORE R ESULTS OF F ACE C OMPLETION
Fig.
11, Fig.
12 and Fig.
13 show more results of high resolution face completion with masks of various shape.
Our model is capable of learning high-level semantics and structures of faces and handling challenging mask types that were not included in the training set (e.g. hand-drawn masks in Fig. 13).
Fig.
14 and Fig.
15 show more results of hard and soft attribute control respectively.
Fig.
16 show more results of joint attribute and expression control.
A. D ETAIL OF U SER S TUDY
We compared our method with CTX (Yu et al., 2018), which is a state-of-the-art CNN-based face completion approach capable of completing face images at 256 × 256 resolution, with a pilot user study at 256 × 256 resolution with random masks.
27 subjects (15 male and 12 female participants, with ages from 22 to 32) volunteered to participate.
There were four sessions in the experiments.
For each trial, a user was shown two images and asked to choose the more realistic one.
In the ﬁrst session, two images completed from the same image by different methods were chosen (one by our model and another by CTX).
In sessions two to four, a real image and a corresponding synthesized image were shown.
In the ﬁrst session, time was unlimited.
In session two to four, images were on display for 250ms, 1000ms, 4000ms respectively.
We randomly chose 100 pairs of images [Ours, CTX] from the test set for session one and another 100 groups of images [Ours, CTX, GT] for session two to four.
The display order of images was randomized.
The result (Fig. 9) shows that there was a signiﬁcantly higher chance that images completed by our model looked more realistic than those completed by CTX in session one.
While comparing with the ground-truth images in session two to four, the highest possible percentage that a method can achieve is 50%, which is a random guessing.
There was a probability of about 40% that our method could fool a human observer when the display time was 250 ms.
As the display time increased, users started to notice more detailed artifacts, so all the percentages dropped.
Regardless, our model always signiﬁcantly outperformed CTX.
Statistical analysis was performed to conﬁrm the signiﬁcant differences between our method and CTX.
The details of analysis are listed in the supplemental materials.
Some of the most-frequently-picked images by participants are shown in Fig.
Overall, our approach generated sharper images with more details and fewer distortions.
Sometimes, the synthesized faces looked even more natural than the ground-truth images.
[Table] [Table]
079) =
[Table] [Table]
Time
F(1.283,
[Table]
346) = 1. p=0.
Statistical Analysis of the User Study.
In order to conﬁrm the intuition of our comparison results, we tested for statistical signiﬁcance.
To do this, we ﬁrst collapsed each participant’s rankings into a frequency list.
Once frequency lists were built for all participants, the frequencies for each method were again averaged over the 27 participants to produce a ﬁnal list of averages from n = 27 samples.
For session one, we performed the paired samples t-test to compare the means of frequencies for these two methods: t (26) = 46 .
368 , p < 0 .
The results conﬁrmed that our method was favored signiﬁcantly more often than CTX.
For session two through four, a two-way repeated measures analysis of variance (ANOVA) was used because there were two factors:
Method (ours and CTX) and Time (250 ms, 1000 ms, and 4000 ms).
Since the sphericity assumption was not met in our data, we used the correction of Huynh-Feldt for the Method and Time factors, and the GreenhouseGeisser correction for the Method × Time interactions.
Not surprisingly, results (Table 2) showed a signiﬁcant difference in means of different method groups for a standard α = 0 . 05 , which denoted that there was a signiﬁcantly higher probability that images completed by our method were classiﬁed as real samples versus those generated by CTX.
A. N ETWORK A RCHITECTURES AND H YPER -P ARAMETERS
The generator G in our model is implemented by a U-shape network architecture consisting of the ﬁrst component G enc transforming the observed image and its mask to a latent vector and the second component G dec transforming the concatenated vector (latent code and input attributes) to a completed image.
There are residual connections between layers in G enc and the counterpart in
G dec similar in the spirit to the U-Net (Ronneberger et al., 2015) and the Hourglass network (Newell et al., 2016) to consolidate information across multiple scales.
Fig.
19 illustrates the two structures of a layer in the generator for training without and with the attribute controller respectively, which are adapted from the U-Net and Hourglass network.
Every convolutional layer (Conv) is followed by an Instance Normalization (InsNorm) and a
LeakyReLU layer, except that the Conv before the latent vector (i.e. the second Conv layer in
Table 3) is not followed by an InsNorm.
Additionally, the there are no InsNorms or LeakyReLUs after the last Convs of both D cls and D attr .
All Convs used in the residual block of the skip connections of our conditional model have a kernel size of three and a stride of one.
Since we use Instance Normalization rather than Batch Normalization, the batch size is not an important hyper-parameter.
Technically, for faster computation, we use as large a batch size as possible so long as it does not exceed the GPU memory limit.
Tables 3 and 5 demonstrate the architecture of the components of the generator G while Tablesshows the components of the discriminator D .
In Table 6, depending on the operation of the skip connection (Skip), the number of ﬁlters is either doubled (for a concatenation operation) or remains the same (for an addition operation).
The progressive training process is illustrated in Fig.
At a resolution lower than 1024 × 1024 , the input face images, masks, landmarks and real images are all down-sampled with average pooling to ﬁt the given scale.
One of the major challenges of generating high resolution images is the limitation of Graphics Processing Unit (GPU) memory.
Most completion networks use Batch
Normalization (Ioffe & Szegedy, 2015) to avoid covariate shift.
However, with the limited GPU memory, only a small number of batch sizes are supported at high resolution, resulting in low quality of generated images.
We use the Instance Normalization (Ulyanov et al., 2016), similar to Zhu et al.
(Zhu et al., 2017), and update D with a history of completed images instead of the latest generated one (Shrivastava et al., 2016) to stabilize training.
At the growing stage, new layers are added for both D and G and these layers are faded in with current networks smoothly.
After the fade-in process, the network is trained on more images for stabilization.
We used 300K, and 150K training images for resolution [ 8 × 8 to 256 × 256 ] and [ 512 × 512 , 1024 × 1024 ] respectively at growing stage, and 600K, 430K images for 4 × 4 and [ 8 × 8 to 1024 × 1024 ] at stabilizing stage respectively.
In the experiments, the reconstruction trade-off parameter was set to κ = 0 .
7 to focus more on the target region.
To balance the effects of different objective functions, we used λ attr = 2 , λ rec = 500 , λ feat = 8 , and λ bdy = 5000 .
The Adam solver (Kingma & Ba, 2014) was employed with a learning rate of 0.0001.
[Table] [Table]
[Table] [Table]