[Re]
VCNet: A Robust Approach to Blind Image Inpainting
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility
Summary
In this study we report and reproduce, on a large scale, the results of the article of a novel blind image inpainting architecture, namely VCNet, which jointly-controls the mask prediction and the blind inpainting modules.
We have implemented this architecture from scratch in PyTorch and then have conducted our experiments and blind inpainting evaluation on all datasets, as described in the article.
We have achieved to reproduce the results qualitatively andquantitatively in most cases
Scope of Reproducibility
In the scope of this study, we validate the qualitative and quantitative results of VCNet on robust blind image inpainting.
Methodology
The original study achieves robust blind image inpainting by exploiting the mask prediction network (MPN) for guidingthe blind inpainting branch (RIN).
The paper has been implemented from scratch in PyTorch.
We have conductedthe synthetic data experiments on FFHQ and Places2 datasets, and also tried to achieve some of the blind inpaintingevaluation tasks, as stated in the paper.
Experiments have been completed on 1x RTX 2080 Ti in 4 to 6 days for each,and do not require any other signiﬁcant resources, but GPU memory.
Results
We have achieved to reproduce the results qualitatively and quantitatively on a large scale.
Qualitatively, MPN is able tolearn the corrupted areas in input images in earlier steps reported in the paper, and RIN produces visually plausibleoutputs after extensive hyper-parameter tuning.
Moreover, we measured the quantitative performance of the reproducedmodel with the metrics reported in the paper.
What was easy
The paper is well-written.
The main components of VCNet, except Probabilistic Context Normalization, are composedof the common layers in PyTorch.
Therefore, the general implementation is straightforward.
We have no issues withimplementing the loss functions, and all hyper-parameters are precisely indicated in the paper.
What was difﬁcult
Due to the lack of computational resources, we could not ﬁt the data with the batch size reported in the paper and thus we need to adjust the learning rates and the number of training steps according to the appropriate batch size in our settings.
GAN training is quite unstable while trying to tune hyper-parameters in the case of any change.
Mask smoothing procedure and the input format for RIN are the issues that we ﬁxed after communicating with the authors.
For the experiments of face-swapping, we could not achieve to reproduce the reported results by applying exactly the same technique mentioned in the paper, even after ﬁne-tuning trials .
The failure cases can be found in this paper
Communication with original authors
We were in contact with the authors since the beginning of the challenge.
They swiftly answered our questions and some important missing points in training scheme and the architecture
Introduction
The paper [1] proposes the versatile blind image inpainting pipeline that avoids the over-simpliﬁcation of the assumptionsin the previous blind inpainting studies.
The previous approach for specifying the corrupted areas was to ﬁll them withsome constant values ( i.e. black or white pixels) or simple data distributions [2,3,4,5,6,7].
However, such patterns,which are easily and almost perfectly identiﬁed by the network, can be considered as features to learn, instead of thecontextual semantics.
This may signiﬁcantly degrade the overall performance of the network, especially when thecorrupted area contains some unknown content.
In the paper, this assumption is relaxed as the training data is generatedby using similar natural images for ﬁlling the corrupted area in the original images with random strokes.
Secondly, the network architecture, namely VCNet , has two stages of mask prediction and inpainting with the guideof predicted masks representing the corrupted areas.
This design has the capacity of inpainting the corrupted imageswithout requiring any mask annotations during inference.
To neutralize the negative effect of mask prediction errors onthe generation part, VCNet utilizes the probabilistic context normalization ( PCN ), which mainly transfers the contextualinformation in different layers depending on the mask prediction probabilities.
After training on well-known datasets,
VCNet is able to inherently learn to inpaint the corrupted images in visually plausible and semantically coherentway, and it has slightly better or on-par quantitative results when compared to the other methods.
Several differentexperiments for VCNet have been conducted to validate the blind inpainting results ( e.g. face-swapping, different noisesources).
In this reproducibility report, we studied the new idea of generating training data and VCNet model architecture indetail, which contains implementing the architecture described in the paper, running the experiments, reporting theimportant details about certain issues encountered during reproducing, and comparing the obtained results with the onesreported in the original paper.
Moreover, we discuss the further improvements for VCNet to enhance the quality ofinpainting results.
Scope of reproducibility
The main contribution of the paper is to propose a robust blind image inpainting system which jointly trains a maskprediction module and an inpainting module.
This system is also able to prevent the degradation of the performance ofinpainting module due to the error propagation of mask prediction module by employing a new spatial normalizationlayer.
To validate the qualitative and quantitative results of the paper, we mainly focus on answering the followingquestions: (1) Can we predict the noise mask by only using the corrupted images?
(2) Can we validate the reportedinpainting quality measured by the common inpainting metrics? (3) Can we achieve the qualitative performancedemonstrated in the original paper? (4) What are the critical issues faced during the reproduction study, which maydirectly affect the performance?
(5) Can we reproduce blind inpainting evaluation studies mentioned in the paper (e.g.,different noise sources, raindrop removal and face-swapping)?
Methodology
Since the paper proposes a new model architecture, namely VCNet , we mainly focused on implementing this newarchitecture and its novel components within the details described in the paper.
Moreover, we replicated the new trainingdata generation mechanism where an image, as a noise source, ﬁlls the missing areas in the original corrupted image.
At this point, we found that the paper was well-written, and contains the most of the details required to reproduce theresults.
Initially, we wrote the code from scratch in PyTorch [8] since the original repository was not publicly available.
The code writing process was not as challenging as we anticipated, but we still had some missing points not included inthe paper.
After communicating with the authors, we had a chance to access the private repository of the paper, andchecked these points accordingly.
In this section, we introduce the implementation details of VCNet, the points in thepaper which were important for reproduction, and our experimental setup.
3.1 VCNet
The corrupted image I in this setting is formulated as
[Equation]
where O ∈ R h × w × 3 is the original RGB image, which is sampled from FFHQ and Places2 datasets, N ∈ R h × w × 3 isthe noise source, which is correspondingly sampled from CelebA-HQ and ImageNet datasets, and M ∈ R h × w × 1 is the randomly-generated binary mask composing O and N .
The strategy of using real-world image patches for generatingthe noise source enforces the inpainting model to focus on the contextual information in the uncorrupted parts of images.
This favorably alters the inpainting process in such ways that (1) the model generalizes better in real-world scenarios,and (2) the prior introduced by free-form strokes is moderated.
Moreover, directly blending the ground truth O and thebinary mask M leads to an issue for stable training, as we experienced and also mentioned in the paper.
Therefore, wehave to smooth the contact regions between O and M by applying iterative Gaussian smoothing ( k = 15 , σ = 4 ) to Min order to overcome this issue.
The proposed architecture for robust blind image inpainting, namely VCNet, is composed of two components introducedas Mask Prediction Network (MPN) and Robust Inpainting Network (RIN) (see Figure 1).
The former predicts the noisysignals, ( i.e. , binary masks) in the corrupted images, while the latter is responsible for inpainting the corrupted areas byutilizing the predicted noisy signals.
During training, both networks are jointly trained after a certain number of steps.
By doing so, RIN is able to locate the corrupted areas without any annotated supervision, and MPN is regularized by
RIN due to the effort of preserving the contextual information.
MPN is composed of an encoder-decoder structure with residual blocks.
The objective function is designed in selfadaptive manner where the effect of being positive or negative is adjusted by the densities in the output, and it can beseen as follows:
[Equation]
where ˆ M is the predicted binary mask, p is the number of pixels, and τ represents the density of uncorrupted pixels inthe ground truth mask.
Note that although the objective function leads MPN to detect all corrupted parts, after startingjointly training with RIN, it eventually learns to ignore some corrupted parts, consistent with the contextual information.
Similar to MPN, RIN is composed of an encoder-decoder structure with probabilistic contextual blocks (PCB), which isa residual block variant spatially-guided by the predicted binary mask.
This block comes up with an advanced solutionto propagate the contextual information throughout the layers.
Probabilistic context normalization (PCN) moduleappeared in these blocks transfers the contextual information in different layers for the corrupted areas without ignoringthe features in the uncorrupted areas.
The formula of PCN can be seen as follows:
[Equation]
where T ( · ) is responsible for transferring the internal statistics of the instances, and formulated as follows
[Equation]
[Table]
where X is the input for PCN, H is the down-sampled binary mask, X P = X
· H , X Q = X · ( 1 − H ) , and µ ( · , · ) isthe weighted average and σ ( · , · ) is the standard deviation of the composite feature maps governed by the binary mask.
The objective function of RIN is the weighted combination of pixel-wise reconstruction loss, semantic consistency loss,texture consistency loss and adversarial loss
[Equation]
(5) where V l O and V l ˆO represent the feature maps of the ground truth O and the output ˆO at level l , extracted from pre-trained
VGG-19 network V .
L mrf is ID-MRF loss [9, 10], which computes the sum of the patch-wise difference between thegenerated content and its corresponding output patch, and enhances the details in the generated content.
L adv represents WGAN-GP loss [11, 12] for adversarial training. The coefﬁcients for all components of the objective function arepresented in Table 1. 3.2 Implementation Details
In our implementation, we have completely followed the instructions in the original paper.
According to our observations,there are some critical points, which directly affects the training behavior and the overall performance of VCNet.
• All layers in VCNet are initialized with normal distribution with gaining of 0.02 (optional scaling factor).
• In the paper, the details about the discriminator is missing.
According to the repository provided by the authors,
discriminator is fundamentally designed as in SN-PatchGAN [17].
• Changing only the batch size to ﬁt the data to the appropriate resources ruins the training behavior of themodel.
To overcome this, we adjusted the learning rate, and then the number of iterations required for theconvergence has increased. • Conﬁdence-driven mask smoothing is a must to have a stable training.
As mentioned in the original paper,applying smoothing driven by estimated mask conﬁdence to the mask before feeding to PCN leads to minimizethe mask error propagation in deeper layers and improve the contextual modeling.
In practice, without this, thetraining diverges in earlier steps, and MPN eventually interprets any noise signals as a part of the main context( i
e. predicts as all-zero binary mask ).
• During training on FFHQ, mixing the noise sources ( i
e. using CelebA-HQ and ImageNet together ) signiﬁcantlyimproves the overall performance, and avoids any bias on facial dataset. • Before feeding the images to RIN, we applied alpha blending to them with the binary mask predicted by MPN.This is represented with a parameter called embrace in the original code.
3.3 Hyperparameters
Since the main objective of this study is to achieve the reproduction of the original work qualitatively and quantitatively,we employed the default parameters reported in the paper, except batch size and learning rate, due to the computationalconstraint.
At this point, we could not ﬁt the data to our resources with the default batch size.
After using the bestpossible batch size for our resources, we realized that the learning curve does not improve at all, or improve very slowly,during training with the default learning rate and our batch size.
Therefore, we adjusted the learning rate and the numberof steps accordingly with a number of trials.
3.4 Experimental setup
In this study, we have followed the same protocol described in the original paper for our all experiments.
Theseexperiments have been conducted in 1 x RTX 2080Ti for approximately 4-6 days ( i.e. 4 days for FFHQ, 6 days for
Places2, due to the more iterations required to converge ), and do not require any other signiﬁcant resources, but GPUmemory.
To generate the training data, we basically employed two dataloaders for the original images and noise sources, andblend them on-the-ﬂy with the randomly-generated binary masks before feeding to the network.
The parameters forgenerating random stroke brushes, as a binary mask, can be found in the conﬁguration ﬁle in our GitHub repository.
Our implementation and the trained weights are open-sourced, and can be found at https://anonymous.4open.science/r/7164e419-4bc7-48a8-abdf-fd116596fed9/ .
3.5 Datasets
Following the paper, we have conducted our experiments on four well-known datasets: CelebA-HQ [13], FFHQ [14],
Places
[15] and ImageNet [16].
Due to the computational and time constraints, we picked 50K images from 10 classes( auto showroom, bridge, canyon, house, mausoleum, ocean, park, rainforest, sky, street ) of Places2 dataset, and alsoemployed the validation set of ImageNet for training.
The image size is set to 256 × 256 where resizing is applied to
FFHQ and CelebA-HQ, random cropping and padding is applied to ImageNet and Places2.
We have conﬁgured ourexperimental setup with two settings where training images are picked from FFHQ and Places2, and corrupted by theircorresponding noise sources (CelebA-HQ & ImageNet for FFHQ, only ImageNet for Places2).
3.6 Computational requirements
VCNet has U-Net-like structure with residual blocks in both MPN and RIN, and the components of its objectivefunctions require to obtain some information from a VGG-like model.
Considering this, training VCNet requires∼ 10 GB GPU memory with the batch size of 4, while a single inference only requires ∼ 1 GB memory, and it took 225milliseconds in RTX 2080Ti GPU.
Results
In this study, we mainly aim to reproduce the results in [1].
In general, we have achieved to reproduce quantitativeand qualitative results of the experiments done on FFHQ and Places datasets.
Following the paper, we measured theperformance of our VCNet implementation by employing binary cross entropy (BCE) for mask predictions, peaknoise-to-signal ratio (PSNR) and structural similarity index measurement (SSIM) for the inpainting quality.
4.1 Mask prediction by only using the corrupted images
As shown in Table 2, our VCNet implementation has on-par performance in mask prediction task on FFHQ dataset interms of BCE, compared to the reported results in the paper.
Also, it achieves superior performance on FFHQ dataset,compared to the most of the other methods mentioned in the paper, except Partial Convolutions.
Even though thepredicted masks are mostly near to be identical to the ground truth in the view of visualization (see Figure 3), there isstill a gap between the performance of our implementation and the one reported in the original paper on Places2 dataset.
[Table] [Table]
[Table]
[Table]
VCNet (ours) 0.
4.2 Inpainting performance
In this study, the main aim is to reproduce the inpainting results reported in the paper.
Recall that we have conductedthe experiments in such settings where the noise sources are taken from CelebA-HQ and ImageNet validation set forthe original images from FFHQ, and ImageNet validation set for the original images from Places2.
For both datasets,we were able to reproduce the results reported in the paper in terms of quantitative metrics for the inpainting quality( PSNR, SSIM ).
Particularly on FFHQ, we obtain better PSNR values ( ∼ 4 dB) than the original results.
The main claim in the paper can be summarized as VCNet is able to produce visually more convincing and lessdegraded blind inpainting results since PCN resolves the problem of degradation in the output induced by the unknowncontamination areas.
In the scope of the reproducibility of the paper, our qualitative results show that it is possibleto generate realistic inpainting results with the help of predicted binary masks, as plausible as the ones reported.
Inthis regard, we can state that our qualitative results support this claim.
Although the corrupted images may containsimilar context with different priors, or even dissimilar context, VCNet achieves, ﬁrst, to predict the binary mask ofnoisy signals from these images, and then inpainting the corrupted areas without any degrading yielded by the noisesources.
The results of our VCNet implementation are shown in Figure 2 ( FFHQ ) and Figure 3 ( Places2 ).
4.3 Blind inpainting evaluation studies: different noise sources, raindrop removal and face-swapping
One of the claims in the paper is that VCNet can deal with noise sources unseen during training, like Gaussian noise orsome constant colors .
To support this claim, and to show the effectiveness of new training data generation methodologyfollowed in the training scheme of VCNet, the results for a corrupted image with different noise sources that are notincluded to the training are presented in the paper.
Following the paper, we demonstrate the visual results of ourimplementation of VCNet model on FFHQ dataset corrupted by different noise sources in Figure 4.
We can concludethat VCNet is robust to detect different noisy signals, even the ones not included to the training , and then to ﬁll themissing parts in visually plausible way.
Secondly, the other claim in the paper is that the learned visual consistency ability on blind image inpainting task can betransferred into the other similar removal tasks with a few training data.
To support this claim, raindrop removal task hasbeen studied by employing VCNet with pre-trained weights trained on Places2 dataset.
At this point, we re-designed our training setup to be compatible with raindrop removal model training ( i.e. the combination of blind image inpaintingand image-to-image translation settings ), as described in the paper.
We used the ﬁrst 20 training images of the raindropremoval dataset, and fed them into VCNet with pre-trained weights for only 2,000 iterations.
As shown in Figure 5, ourimplementation of VCNet has achieved to reproduce the raindrop removal task, so we can conclude that VCNet has thegeneralization property on the other similar tasks, even by training a few instances for a small amount of time .
Lastly, image blending on user-fed visual materials with the given image is the other potential application of this blindimage inpainting system.
The paper claims that VCNet can be used for ﬁlling the user-fed visual contents to edit theoriginal images.
We have tried to achieve the face-swapping task to support this claim, ﬁrst as described in the paper.
However, we could not execute the blind image inpainting process since MPN cannot detect any noisy signals in thecorrupted images, and it considers all pixels pertain to the contextual information of the images.
Thereafter, we changedthe strategy of generating the corrupted images, and employed randomly-generated binary masks half-resized andcenter-pasted to the blank masks for being able to execute the second stage ( i.e. RIN) of inpainting process.
Despite allthese attempts, we could not achieve to reproduce the results of the face-swapping task, and the failure cases can beseen in Figure 6.
Discussion
We clearly state that the paper we reproduced was well-written.
Even though there are a few missing implementationdetails ( e.g., the input format for RIN, the discriminator architecture and weights initialization procedure ), we haveachieved to reproduce the visual results reported in the paper on a large scale.
Quantitatively, our implementation of
VCNet has on-par performances with the reported results for all metrics, and particularly, slightly higher PSNR valuewith the model trained on FFHQ dataset.
Therefore, it is shown that the main claim and the most of the blind inpaintingstudies in the paper are supported by our experiments.
What was easy: The paper is well-written.
The main components of VCNet, except Probabilistic Context Normalization, are composed of the common layers in PyTorch.
Therefore, the general implementation is very straightforward.
We have no issues on implementing the loss functions, and all hyper-parameters are precisely given in the paper.
What was difﬁcult: Due to the lack of computational resources, we could not ﬁt the data with the batch size reportedin the paper, and thus we need to adjust the learning rates and the number of training steps according to the appropriatebatch size in our settings.
GAN training is quite unstable while trying to tune hyper-parameters if any change is required.
Moreover, mask smoothing procedure and the input format for RIN are the issues that we ﬁxed after communicatingwith the authors.
For the experiments of face-swapping task, we could not achieve to reproduce the reported results byapplying exactly the same technique, even after ﬁne-tuning trials .
The failure cases are shown in Figure 6.
Communication with original authors: We were in contact with the authors since the beginning of the challenge.
They swiftly
After a while, they have shared the private repository with us to make it possible to double check our code, and it wasvery helpful.