W HAT ’ S B EHIND THE M ASK : E STIMATING U NCERTAINTY IN I MAGE - TO -I MAGE P ROBLEMS
Anonymous authors
Paper under double-blind review
A BSTRACT
Estimating uncertainty in image-to-image networks is an important task, particularly as such networks are being increasingly deployed in the biological and medical imaging realms.
In this paper, we introduce a new approach to this problem based on masking.
Given an existing image-to-image network, our approach computes a mask such that the distance between the masked reconstructed image and the masked true image is guaranteed to be less than a speciﬁed threshold, with high probability.
The mask thus identiﬁes the more certain regions of the reconstructed image.
Our approach is agnostic to the underlying image-to-image network, and only requires triples of the input (degraded), reconstructed and true images for training.
Furthermore, our method is agnostic to the distance metric used.
As a result, one can use L p -style distances or perceptual distances like
LPIPS, which contrasts with interval-based approaches to uncertainty.
Our theoretical guarantees derive from a conformal calibration procedure.
We evaluate our mask-based approach to uncertainty on image colorization, image completion, and super-resolution tasks, demonstrating high quality performance on each.
I NTRODUCTION
Deep Learning has been very successful in many applications, spanning computer vision, speech recognition, natural language processing, and beyond.
For many years, researchers were mainly content to develop new techniques to achieve unprecedented accuracy, without concern for understanding the uncertainty implicit in such models.
More recently, however, there has been a concerted effort within the research community to understand and quantify the uncertainty of deep models.
This paper addresses the problem of estimating uncertainty in the realm of image-to-image (sometimes referred to as image reconstruction) tasks.
Such tasks include super-resolution, deblurring, colorization, and image completion, amongst others.
Computing the uncertainty is important generally, but is particularly so in application domains such as biological and medical imaging, in which ﬁdelity to the ground truth is paramount.
If there is an area of the reconstructed image where such ﬁdelity is unlikely or unreliable due to high uncertainty, this is crucial to convey.
Our approach to uncertainty estimation is based on masking.
Speciﬁcally, we are interested in the possibility of computing a mask such that the uncertain regions in the image are masked out.
More formally, we would like the distance between the masked reconstructed image and the masked true image to be small in expectation.
Ideally, the method should be agnostic to the choice of distance function, which should be dictated by the application.
A high level overview of our approach is illustrated in Figure 1.
We show a direct connection between the mask and a theoretically well-founded deﬁnition of uncertainty that matches image-to-image tasks.
We then derive an algorithm for computing such a mask which can apply to any existing (i.e. pre-trained) image-to-image network, and any distance function between image pairs.
All that is required for training the mask network is triplets of the input (degraded), reconstructed and true images.
Using a procedure based on conformal prediction (Angelopoulos & Bates, 2021a), we can guarantee that the masks so produced satisfy the following criterion: the distance between the masked reconstructed image and the masked true image is guaranteed to be less than a speciﬁed threshold, with high probability.
We demonstrate the power of the method on image colorization, image completion, and super-resolution tasks.
[Figure]
Colorization Model
ˆ y
M a s k
i ng M od e l m y (cid:12) m =
y (cid:12) m ˆ y (cid:12) m
= ˆ y (cid:12) m
Our contributions are as follows:
• We present an approach to uncertainty computation based on masking.
We show that moving from binary to continuous masks preserves the connection with a theoretically wellfounded deﬁnition of uncertainty that is relevant for image-to-image tasks.
• We derive an algorithm for computing the masks which works for arbitrary image-to-image networks, and any distance function between image pairs.
The masks are guaranteed to yield a small distance between masked ground truth and reconstructed images, with high probability.
• We demonstrate the effectiveness of the method on image colorization, image completion, and super-resolution tasks attaining high quality performance on each.
N OTATIONS
Throughout the paper we represent an image as a column-stacked vector x ∈ R n .
We use x k to denote the k th image of a collection, while x ( i ) marks the i th entry of x .
We deﬁne #» 0 ∈ R n and #» 1 ∈ R n to be vectors of all zeros and ones respectively.
The operation x (cid:12) y stands for the Hadamard (point-wise) product between x and y .
For p ≥ 1 , we denote the (cid:96) p -norm of x by (cid:107) x (cid:107) pp (cid:44) (cid:80) ni
=1 | x ( i ) | p .
The symbol d ( · , · ) represents a general distortion measure between two images.
We deﬁne a continuous mask as a vector m ∈ [0 , 1] n , where the average size of a mask equals (cid:13)(cid:13) #» 1 − m (cid:13)(cid:13) 1 /n such that the mask #» 1 (no-masking) has size 0 while the mask #» 0 has size of 1.
Given a mask m , we deﬁne d m ( x, y ) (cid:44) d ( m (cid:12) x, m (cid:12) y ) as the distortion measure between the masked versions of images x and y .
For a natural number n ≥ 1 we deﬁne the set [ n ] (cid:44) { 1 , ..., n } .
R ELATED W ORK
Bayesian Uncertainty Quantiﬁcation The Bayesian paradigm deﬁnes uncertainty by assuming a distribution over the model parameters and/or activation functions.
The most prevalent approach is
Bayesian neural networks (MacKay, 1992; Valentin Jospin et al., 2020; Izmailov et al., 2020), which are stochastic models trained using Bayesian inference.
Yet, as the number of model parameters has grown rapidly, computing the exact posteriors has became computationally intractable.
This shortcoming has led to the development of approximation methods such as Monte Carlo dropout (Gal & Ghahramani, 2016; Gal et al., 2017a), stochastic gradient Markov chain Monte Carlo (Salimans et al., 2015; Chen et al., 2014), Laplacian approximations (Ritter et al., 2018) and variational inference (Blundell et al., 2015; Louizos & Welling, 2017; Posch et al., 2019).
Alternative Bayesian techniques include deep Gaussian processes (Damianou & Lawrence, 2013), deep ensembles (Ashukha et al., 2020;
Hu et al., 2019), and deep Bayesian active learning (Gal et al., 2017b), to name just a few.
For a recent comprehensive review on Bayesian uncertainty quantiﬁcation we refer the readers to Abdar et al.
Distribution-Free Methods and Conformal Prediction Unlike Bayesian methods
, the frequentist approach assumes the true model parameters are ﬁxed with no underlying distribution.
Examples of such distribution-free techniques are model ensembles (Lakshminarayanan et al., 2017; Pearce et al., 2018), bootstrap (Kim et al., 2020; Alaa & Van Der Schaar, 2020), interval regression (Pearce et al., 2018;
Kivaranovic et al., 2020; Wu et al., 2021) and quantile regression (Gasthaus et al., 2019;
Romano et al., 2019).
An important distribution-free technique which is most relevant to our work is conformal prediction (Angelopoulos & Bates, 2021b; Shafer & Vovk, 2008).
The conformal approach relies on a labeled calibration dataset to convert point estimations into prediction regions.
Conformal methods can be used with any estimator, require no retraining, are computationally efﬁcient and provide coverage guarantees in ﬁnite samples (Lei et al., 2018).
Recent development includes conformalized quantile regression (Romano et al., 2019; Sesia & Cand ` es, 2020;
Angelopoulos et al., 2022b), conformal risk control (Angelopoulos et al., 2022a; Bates et al., 2021; Angelopoulos et al., 2021) and semantic uncertainty intervals for generative adversarial networks (Sankaranarayanan et al., 2022).
Sun (2022) provides an extensive survey on distribution-free conformal prediction methods.
Uncertainty for Image-to-image Regression Tasks
A work that stands out in this realm is (Angelopoulos et al., 2022b), which experiments with the conformal prediction technique.
They evaluate and compare several models that address both the prediction of a value per each pixel, and an accompanying conﬁdence interval that should contain the true value.
After calibration, a guarantee is provided such that with high probability the true values of the pixels are indeed within these predicted conﬁdence intervals.
They ﬁnd that quantile regression gives the best results among the four methods considered.
As our work addresses the same breed of image-to-image problems, we provide extensive comparison to this work.
M ASK -B ASED U NCERTAINTY E STIMATION 3. P ROBLEM F ORMULATION
We consider the problem of recovering an image y ∈ [0 , 1] n from its observations x ∈ [0 , 1] n x and denote by ˆ y ( x ) ∈ [0 , 1] n a given estimator of y .
We deﬁne the uncertainty of ˆ y given x as follow:
[Equation]
where d ( · , · ) denotes an arbitrary distortion measure between the ground truth image y and its estimation ˆ y , and E is the expectation over the random variable y | x .
Ideally, we would like to design ˆ y such that
[Equation]
where 0 ≤ α and 0 ≤ β ≤ 1 are user-deﬁned, and the probability P ( · ) is computed over the joint probability of the problem pairs ( y, x ) .
However, in practice, condition (2) might not hold.
Hence, our goal is to identify areas within ˆ y which have the most adverse effect on the uncertainty and lead to the violation of (2).
Formally, we aim to construct an uncertainty map given by a continuous mask m ∈ [0 , 1] n such that
[Equation]
where d m ( y, ˆ y ) (cid:44) d ( m (cid:12) y, m (cid:12) ˆ y ) .
Notice we must avoid trivial and undesired solutions, e.g. m = 0 , which satisfy (3) but provide no true information regarding the uncertainty.
To that end, we consider solving the following problem
[Equation]
In words, we wish to ﬁnd the minimal masking required to satisfy the distortion constraint.
Notice that Problem (P1) does not depend on β as we consider an instance regression problem for a given x and an originating y .
We bring back the probabilistic condition using β in the next section, when addressing the whole image manifold.
Under certain conditions, the optimal solution is readily given in a closed-form expression as shown by the next theorem.
Theorem 1.
Consider Problem (P1) with the distortion measure d ( y, ˆ y )
= (cid:107) y − ˆ y (cid:107) pp where p > 1 .
Then, for sufﬁciently small α , the optimal mask m ∗ admits a closed-form solution given by
[Equation]
where for all j ∈ [ n ] q ( j )
[Equation]
Proof.
See Appendix C.
Note that the above implies the following:
Corollary 1.
The optimal mask, as deﬁned by the solution of (P1), is directly related to the uncertainty measure in Equation (1) via
[Equation]
Proof.
[Equation]
Thus, the above establishes a correspondence between the optimal mask and uncertainty as deﬁned in (1).
Furthermore, it can be shown that the result of Theorem 1 can be extended to p = 1 , leading to a binary mask (see Appendix A).
The result of Theorem 1 serves only as a general motivation in our work, since obtaining the optimal mask directly via the above equations is impractical for a number of reasons.
First, α may vary depending on the user and the application.
Second, for arbitrary distortion measure Problem (P1) does not admit a closed-form solution.
Last and most importantly, at inference time the ground truth y is unknown.
Hence, our next step is to provide an estimation for the mask which requires no knowledge of y , can be computed for any differentiable distortion measure, and is easily adapted for various values of α and β .
3. M ETHOD
Here, we aim to design an estimator for the the uncertainty mask deﬁned by Problem (P1).
We start by modeling the mask as a neural network m θ ( x, ˆ y ) with parameters θ , which we refer to as the masking model (see details in Section 4.2).
Similar to Problem (P1), we deﬁne the following optimization problem
[Equation]
Ignoring (for now) the second constraint, the Lagrangian of Problem (P2) is given by
[Equation]
where µ > 0 is the dual variable which is considered as an hyperparameter.
Given µ , the optimal mask can be found by minimizing L ( θ, µ ) with respect to θ , which is equivalent to minimizing
[Equation]
since α does not depend on θ .
Thus, motivated by the above, we train our masking model using the following loss function:
[Equation]
where D (cid:44) { ( x k , ˆ y k , y k ) } k is a dataset of triplets of the degraded input, recovered output and the true image respectively.
To enforce that the mask is within [0 , 1] we simply clip the output of masking model.
Note that the proposed approach facilitates the use of any differentiable distortion measure.
Furthermore, at inference time, our masking model provides an approximation of the optimal uncertainty mask – the solution of (P1) – without requiring y .
Following the above, notice that the loss function is independent of α and β , hence, we cannot guarantee that m θ satisﬁes condition (3).
To overcome this, we follow previous work on conformal prediction and consider the output of our masking model as an initial estimation of the uncertainty which can be calibrated to obtain strong statistical guarantees.
In general, the calibrated mask m λ follows the form
[Equation]
where F ( · ; λ ) is a monotonic non-decreasing function, parameterized by a global scalar which is tuned throughout the calibration process.
As our goal is to reduce the distortion below a predeﬁned ﬁxed level, the calibrated mask need to be inversely proportional to the distortion.
Since our predicted mask is proportional to the distortion we use the following formula 1 ,
[Equation]
which was found empirically to perform well in our experiments.
To set the value of λ we assume a given calibration dataset C
= { ( x k , ˆ y k , y k ) } k and perform the following calibration procedure:
For each pair ( x k , ˆ y k ) we predict a mask m k
= m θ ( x k , ˆ y k ) .
We deﬁne a calibrated mask m λ k according to (8) where λ k is the maximal value such that
[Equation]
Given all { λ k } , we set λ to be the 1 − β quantile of { λ k } ,
i
e. the maximal value for which at least β fraction of the calibration set satisﬁes condition (P1).
The ﬁnal mask, calibrated according to α and β , is guaranteed to satisfy condition (3) above.
The beneﬁts of our masking approach are as follows.
First, it provides a measure of uncertainty which can be easily interpreted.
In addition, our training process can accept any distortion measure and learns the relationships and correlations between different pixels in the image.
Note thatour model is trained only once, irrespective of the values of α and β .
Thus, our model can be easily adapted for different values of α and β via our simple calibration process.
Finally, at inference, we can produce our uncertainty map without the knowledge of the ground truth while still providing strong statistical guarantees on our output mask.
A formal description of our algorithm can be seen in Appendix B
E XPERIMENTS 4. D ATASETS AND T ASKS
Datasets Two data-sets are used in our experiments:
Places365 (Zhou et al., 2017):
A large collection of 256x256 images from 365 scene categories.
We use 1,803,460 images for training and 36,500 images for validation/test.
Rat Astrocyte Cells (Ljosa et al., 2012):
A dataset of 1,200 uncompressed images of scanned rat cells of resolution 990 × 708.
We crop the images into 256 × 256 tiles, and randomly split them into train and validation/test sets of sizes 373,744 and 11,621 respectively.
The tiles are partially overlapped as we use stride of 32 pixels when cropping the images.
Tasks We consider the following image-to-image tasks:
Completion: Using grayscale version of Places365, we remove a middle vertical and horizontal stripe of 32 pixel width, and aim to reconstruct the missing part.
Super Resolution: We experiment with this task on the two data-sets.
The images are scaled down to 64 × 64 images and the goal is to reconstruct the original image.
Colorization : We convert the Places365 images to grayscale and aim to recover their colors.
Figure 4 shows a sample input and output for each of the above tasks.
4. E XPERIMENTAL D ETAILS AND S ETTINGS
Image-to-Image Models We start by training models for the above three tasks.
Note that these models are not intended to be state of the art, but rather used to demonstrate the uncertainty estimation technique proposed in this work.
We use the same model architecture for all tasks: an 8 layer
U-Net.
For each task we train two versions of the network: (i)
A simple regressor; and (ii)
A conditional GAN, where the generator plays the role of the reconstruction network.
For the GAN, the discriminator is implemented as a 4 layer CNN.
We use the L1 loss as the objective for the regressor, and augment it with an adversarial loss in the conditional GAN ( λ = 20 ), as in Isola et al.
All models are trained for two epochs using the Adam optimizer with a learning rate equal to 1 eand a batch size equal to 50 .
Mask Model
For our masking model we also use an 8 layer U-Net architecture.
This choice was made for simplicity and compatibility with previous work.
The input for the mask model is the concatenation of the degraded image and the predicated image on the channel axis.
It’s output is a mask having the same shape as the predicted image, with values in the range [0 , 1] .
The masking model is trained using the loss function described in Section 3.2 with µ = 2 , learning rate of 1 eand a batch size of 25.
Experiments We consider the L1, L2, SSIM and LPIPS distances.
We set aside 1 , 000 samples from each validation set for calibration and use the remaining samples for evaluation.
We demonstrate the ﬂexibility of our approach by conducting experiments on a variety of 12 settings: (i) Image
Completion: { Regressor, GAN } × { L1, LPIPS } ; (ii) Super Resolution: { Regressor, GAN } × { L1,
SSIM } ; and (iii) Colorization: { Regressor, GAN } × { L1, L2 } .
Thresholds Recall that given a predicted image, our goal is to ﬁnd a mask that, when applied to both the prediction and the (unknown) reference image, reduces the distortion between them to a predeﬁned threshold with high probability β .
Here we ﬁx β = 0 .
9 and choose the threshold to be the 0 .
1 -quantile of each measure computed on a random sample from the validation set, i.e. roughly 10% of the predictions are already considered sufﬁciently good and do not require masking at all.
4. C OMPETITOR T ECHNIQUES
Quantile – Interval-Based Technique We compare our method to the quantile regression option presented in (Angelopoulos et al., 2022b).
As described in Section 2, their uncertainty is formulated by pixel-wise conﬁdence intervals, calibrated so as to guarantee that with high probability the true value of the image is indeed within the speciﬁed range.
While their predicted conﬁdence intervals are markedly different from the expected distortion we consider, we can use these intervals as a heuristic mask.
For completeness, we also report the performance of the quantile regression even when it is less suitable, i.e. when the underlying model is a GAN and when the distortion loss is different from L1.
We note again that for the sake of a fair comparison, our implementation of the masking model uses exactly the same architecture as the quantile regressor.
Opt – Oracle We also compare our method with an oracle, denoted Opt, which computes an optimal mask using the following optimization procedure that has access to the reference image y :
[Equation]
This minimization is obtained via gradient descent, using the Adam optimizer with a step-size 0 .
01 , iterating until the destination distortion value is attained.
Clearly, this approach does not require calibration, as the above procedure ensures that all the masked samples are below the target distortion threshold.
4. R ESULTS AND D ISCUSSION
We now show a series of results that demonstrate our proposed uncertainty masking approach, and its comparison with Opt and Quantile.
2 We begin with a representative visual illustration of our proposed mask for several test cases, see Figure 2.
As can be seen, the mask aligns well with the true prediction error, while still identifying sub-regions of high certainty.
In Figure 3 we explore the performance of the three compared methods with respect to (i) the probabilistic guarantee, (ii) the size of the masks obtained, and (iii) the correlation between mask sizes – [Table] [Table] [Table]
[Table]
[Table]
Quantile and Ours versus Opt.
As can be seen from the top row, all three methods meet the distortion threshold with fewer than 10% exceptions, as required.
Naturally, Opt does not have outliers since each mask is optimally calibrated by its computation.
The spread of loss values tends to be higher with Quantile, indicating weaker performance.
The colorization results, here and below, seem to be more challenging, with a smaller performance increase for our method.
The above probabilistic results are not surprising as they are the outcome of the calibration stage.
The main question remaining relates to the size of the mask created so as to meet this threshold distortion.
The middle row in Figure 3 presents histograms of these sizes, showing that our approach tends to produce masks that are close in size to those of Opt; while Quantile produces larger, and thus inferior, masked areas.
Again, our colorization results show smaller if any improvement.
The bottom row in Figure 3 shows the correlation between mask sizes, and as can be observed, a high such correlation exists between our method and Opt, while Quantile produces less correlated and much heavier masks.
Quantitative results are presented in Table 1 for the completion, super resolution, and colorization tasks.
First and foremost, the results obtained by our method surpass those of Quantile across the board.
Our method exhibits smaller mask size (cid:107) M (cid:107) , aligned well with the masks obtained by Opt.
Quantile, as expected, produces larger masks.
In terms of the correlation between the obtained mask-size and the unmasked distortion value C ( M , D ) , our method shows high such agreement, while Quantile lags behind.
This correlation indicates a much desired adaptivity of the estimated mask to the complexity of image content and thus to the corresponding uncertainty.
The correlation
C ( M , M opt ) between Opt’s mask size and the size of the mask obtained by our method or Quantile show similar behavior, aligned well with behavior we see in the bottom of Figure 3. [Figure]
C ONCLUSIONS
Uncertainty assessment in image-to-image regression problems is a challenging task, due to the implied complexity, the high dimensions involved, and the need to offer an effective and meaningful visualization of the estimated results.
This work proposes a novel approach towards these challenges by constructing a continuous mask that highlights the trust-worthy regions in the estimated image.
This mask provides a measure of uncertainty accompanied by an accuracy guarantee, stating that with high probability, the distance between the original and the estimated images over the nonmasked regions is below a user-speciﬁed value.
The presented paradigm is ﬂexible, being agnostic to the choice of distance function used and the regression method employed, while yielding masks of small area.
A T ASK I LLUSTRATIONS
B C ALIBRATION A LGORITHM
Algorithm 1 Finding λ
i def scale ( lambda_ , mask , eps =1e-6): return ( mask * lambda_ / ( eps + mask
(1 - mask ))).
clip (0, 1) def find_lambda ( mask , y , y_hat , eps =1e-4): l , h , lambda_
= eps , 1 / eps ,while h - l > eps : lambda_
= ( h + l ) /loss
=
loss_fn ( y * scale ( a , mask ), y_hat * scale ( a , mask ))
if loss > alpha : h = a else: l = a return lambda_
C P ROOF OF T HEOREM
Recall that we have deﬁned the following optimization problem as deﬁning the optimal mask: max m (cid:107) m (cid:107)subject to E y | x [ d m ( y, ˆ y )] ≤ α, #» 0 ≤ m ≤ #» 1 .
(P1)
Theorem 1: Consider Problem (P1) with the distortion measure d ( y, ˆ y )
= (cid:107) y − ˆ y (cid:107) pp where p > 1 .
Then, for sufﬁciently small α , the optimal mask m ∗ admits a closed-form solution given by
[Equation] where for all j ∈ [ n ] q ( j ) [Equation] y
[Equation] [Equation] y (
[Equation] Proof.
We start with reformulating Problem (P1) as follows:
[Equation]
subject to E
[Equation]
[Equation]
Next, we deﬁne the Lagrangian of the above problem,
[Equation]
where µ > 0 is the dual variable, and we invoke the chosen metric d ( y, ˆ y )
= (cid:107) y − ˆ y (cid:107) pp for p > 1 .
According to the Karush–Kuhn–Tucker (KKT) conditions, the optimal solution ( m ∗ , µ ∗ ) must satisfy
[Equation]
Furthermore, notice that the Lagrangian is separable with respect to the coordinates of m , thus, we can write
[Equation]
In the above we assume that m ( i ) ≥ 0 , a fact that should be veriﬁed once a solution is formed.
Hence, we obtain
[Equation]
To simplify our derivation, we deﬁne
[Equation]
Thus, we can rewrite the above result equivalently as m p − 1 ( i ) =
( µpd ( i ) ) − 1 or m ( i ) =
( µp ) − 1 / ( p − 1) · q ( i ) . Now, consider the second KKT condition:
[Equation]
Therefore
[Equation]
the optimal solution m ∗ is given by
[Equation]
Note that as assumed,
the obtained solution m ∗ ( i ) is non-negative.
Finally, assuming α is sufﬁciently small such that
[Equation]
the solution satisﬁes the second constraint, m ∗ ( i ) ≤ 1 , which completes the proof.
We now turn to present a closely related result, referring to the case d ( y, ˆ y )
= (cid:107) y − ˆ y (cid:107) 1 .
Theorem 2: Consider Problem (P1) with the distortion measure d ( y, ˆ y )
= (cid:107) y − ˆ y (cid:107) 1 .
Then the optimal mask m ∗ admits a closed-form solution given by
[Equation]
where d ( i ) (cid:44) E y | x [ | y ( i ) − ˆ y
( i ) | ] and d ∗ is deﬁned by [Equation] i
[Equation]
Proof.
Consider the Lagrangian of the above problem when d ( y, ˆ y ) =
[Equation] y [Equation]
[Equation]
where µ > 0 is the dual variable.
As before, the problem is separable with respect to the coordinates of m , leading to the following one-dimensional problem
[Equation]
where d ( i ) (cid:44) E y | x [ | y ( i ) − ˆ y
( i ) | ] .
Thus,
[Equation]
Observe that this result is intuitive in retrospect, as smaller distances d ( i ) correspond to un-masked areas.
In addition, regardless of µ , the choice of the mask value is dictated by the distance, a fact that we leverage hereafter.
Recall that the solution should satisfy
[Equation]
where m ( i ) are 1-es for the smaller values of d ( i ) .
Thus, (cid:88) i m ( i ) d ( i ) = (cid:88) i : d ( i ) <
d
[Equation]
d ( i
[Equation]
= [Equation] Hence, the above provides a deﬁnition for the threshold value d ∗ , and µ = 1 /d ∗ , which completes the proof.
D M ORE R ESULTS
In this appendix we bring the results corresponding to all 12 settings explored in our work, corresponding to three inverse problems, two regression techniques and two metrics per each, along the following breakdown:
• Image Completion: { Regressor, GAN } × { L1, LPIPS } ; • Super Resolution: { Regressor, GAN } × { L1, SSIM } ; and • Colorization: { Regressor, GAN } × { L1, L2 } .
We start in Figures 5-7 with the obtained distributions of masked distortion values for Opt, Ours and
Quantile.
The goal here is to show that all three methods meet the required distortion threshold with exceptions that do not surpass the destination probability β
= 0 .
Figures 8-10 present histograms of the obtained mask sizes for the three methods.
The goal is to get minimal areas in these masks, so as to keep most of the image content unmasked.
Figures 11-13 conclude these results by bringing graphs showing the inter-relation between the Opt mask size and the ones given by Ours and Quantile.
Analysis and conclusions drawn from these graphs are brought in the experimental part of the paper.
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]