Perceptual Attacks of No-Reference Image Quality Models with Human-in-the-Loop
Anonymous Author(s)
AffiliationAddress email
Abstract
No-reference image quality assessment (NR-IQA) aims to quantify how humansperceive visual distortions of digital images without access to their undistorted references.
NR-IQA models are extensively studied in computational vision, and arewidely used for performance evaluation and perceptual optimization of man-madevision systems.
Here we make one of the first attempts to examine the perceptualrobustness of NR-IQA models.
Under a Lagrangian formulation, we identify insightful connections of the proposed perceptual attack to previous beautiful ideasin computer vision and machine learning, including the Carlini-Wagner attack,the maximum a posterior estimation and more.
We test one knowledge-drivenand three data-driven NR-IQA methods under four full-reference IQA models (asapproximations to human perception of just-noticeable difference).
Through carefully designed psychophysical experiments, we find that all four NR-IQA modelsare vulnerable to the proposed perceptual attack.
More interestingly, we observethat the generated counterexamples are not transferable, manifesting themselves asdistinct design flows of respective NR-IQA methods.
Introduction
Over the past decade, deep neural networks (DNNs) have revolutionized a wide range of computervision and machine learning applications.
For example, ResNet [1] reduced the object recognitionerror rate on ImageNet [2] to 3.56%, surpassing the performance of informed humans.
A similar storyis also written in the field of image quality assessment (IQA) where DNN-based IQA models [3, 4, 5,6 ] exhibit very high correlations with human opinions of perceptual quality on various subject-rateddatasets [7, 8, 9, 5].
IQA models can be roughly classified into full-reference IQA (FR-IQA) andno-reference IQA (NR-IQA) ones depending on the availability of the pristine undistorted image as areference.
Representative FR-IQA methods include the Minkowski distance ( i .
e ., the ℓ p -norm inducedmetric), the structural similarity (SSIM) index [10], the learned perceptual image patch similarity(LPIPS) method [3], and the deep image structure and texture similarity (DISTS) model [4], whichare widely used for measuring signal fidelity and quality in various vision applications.
NR-IQAmodels [11] are extensively studied in computational vision, which mimics the human ability to judgethe perceptual quality of a test image without comparison to any reference image.
NR-IQA plays anindispensable role in the design and optimization of real-world image processing algorithms.
Despite the remarkable achievements of DNN-based models, recent work has identified their vulnerability to adversarial perturbations [13, 14].
For example, in natural image classification, a visuallyindistinguishable perturbation added to a natural image would mislead “top-performance” classifiers.
This imperceptible perturbation is called below the just-noticeable difference (JND) of human perception in psychophysics, and is less relevant in image classification [15].
This is because for themajority of practical adversarial attacks [16] considered in image classification ( e .
g ., the ℓ ∞ -norm
[Figure]
constrained attack), the allowable perturbations, even if above JNDs, do not alter the image semantics
:they are label-preserving.
In the context of IQA, it becomes highly nontrivial to craft label-preservingattacks for two main reasons.
First, for adversarial perturbations that are above JNDs, they are highlylikely to be perceived as some form of visual distortions that lead to quality degradation (see Fig. 1).
Second, computational prediction of JNDs for natural images [17] remains an open research problemas it depends on the combination of the image content and the perturbation type, constrained by thepsychophysical experimental conditions ( e .
g ., the maximum time of visual inspection).
In this work, we take initial steps to examine the perceptual robustness of NR-IQA models.
Our maincontributions are threefold.
• We propose a two-step perceptual attack for NR-IQA methods with human-in-the-loop.
First,
we define the objective of the perceptual attack as a Lagrangian function of an FR-IQA
model (as the “perceptual” constraint) and the NR-IQA model (to be examined).
By varying
the Lagrange multiplier, we generate a series of perturbed images of different distortion
visibility.
Second, we ask human observers in a well-controlled psychophysical experiment
to determine whether the perturbation of each image is discriminable.
The output of our
perceptual attack is a computer-generated counterexample, that is below the JND while
leading to the most significant change in quality prediction.
• We draw connections between the proposed perceptual attack and conventional wisdom,
including the Carlini-Wagner
attack [18] in classification, maximum a posterior (MAP)estimation, maximum differentiation (MAD) competition [19], and eigen-distortion analysis [20].•
We conduct an extensive experiment to examine four NR-IQA models, the knowledge-driven
BRISQUE [12], the shallow learning-based CORNIA [21], and the deep learning-based
Ma19 [22] and UNIQUE [6] under four FR-IQA models, the Chebyshev distance ( i .
e .,the ℓ ∞ -norm induced metric), SSIM, LPIPS, and DISTS (as approximations to humanperception of JND).
We arrive at several important observations, among which the mostinteresting one is that the proposed perceptual attack succeeds in fooling all four NR-IQAmodels, but the generated counterexamples are not transferable, manifesting themselves asdistinct design flows of respective NR-IQA methods.
Related Work
NR-IQA models, summarize representative adversarial attacks in discuss them in a broader context of “analysis by synthesis”
2.1 NR-IQA Models
Early NR-IQA models were designed to deal with specific distortion types, e .
g ., JPEG compression [25] and JPEG2000 compression [26].
In the past decade, research on general-purpose NR-IQAbecome popular with the proposal of a variety of quality-aware features based on natural scene statistics (NSS) [12, 11, 27].
Unsupervised feature learning such as codebook construction [21] was alsoexplored.
Since the work of [28], DNN-based methods began to revolutionize the field of NR-IQA.
Towards developing more accurate NR-IQA models, recent research includes joint learning frommultiple databases [6], active learning for improved generalizability [29], patch-to-picture learningfor local quality prediction [30], and continual learning for handling streaming visual data [31].
2.2 Adversarial Attacks in Classification
Machine learning models are, for a long time, known to be vulnerable to adversarial examples [13,32, 33, 34]: data samples that have been modified very slightly but enough to falsify a machinelearning model.
Most frequently, adversarial examples are automatically generated by projectedgradient-based methods.
Szegedy et al .
[14] used a box-constrained L-BFGS method to discover theprevalence of adversarial examples in DNN-based classifiers.
Goodfellow et al .
[35] introduced a fastgradient sign method (FGSM), which modifies samples along the steepest descent direction under theℓ ∞ -norm constraint.
Kurakin et al .
[36] provided an extension to FGSM, which iteratively generatesadversarial examples with a small step size.
Madry et al .
[37] formulated adversarial training asrobust optimization [38], where adversarial example generation corresponds to solving the innermaximization problem by projected gradient descent (PGD).
The authors recommended a practicaltrick to construct more transferable adversarial examples: adding a small amount of uniform noise toeach pixel of the initial image as a form of dequantization.
Moosavi et al .
[39] defined the adversarialattack as the process to find the minimal perturbation that leads to erroneous model behavior.
Closeto the present work, Carlini and Wagner [18] considered a Lagrangian relaxation of the ℓ p -normconstrained optimization problem to search for adversarial examples.
Laidlaw et al .
[40] switchedto a more perceptual FR-IQA model, LPIPS [3], under which adversarial training gives improvedrobustness to unseen attacks.
Here, we consider the Lagrangian formulation in a different context( i .
e ., NR-IQA), for a different purpose ( i .
e ., encouraging generating samples below JNDs), and underdifferent FR-IQA models ( i .
e ., the Chebyshev distance, SSIM, LPIPS, and DISTS).
We conclude this section by putting adversarial attacks in a broader context of “analysis by synthesis”,which is a core idea in the pattern theory by [24].
Analysis by synthesis suggests to test a machinelearning model in generative rather than discriminative ways, which is well demonstrated in thefield of texture modeling [41].
Adversarial attacks can be seen as a form of analysis by synthesis,where the machine learning model is tested on the automatically generated worst-case data samples,which are less likely to be included in the fixed and excessively reused test sets [42].
In the contextof computational vision, the MAD competition [19], the eigen-distortion analysis [20], and thecontroversial stimuli synthesis [43] all fall into the category of analysis by synthesis.
As one of ourmain contributions, we will make insightful connections of the proposed perceptual attack to thesemethods.
Perceptual Attacks on NR-IQA Models
110 We first formulate the perceptual attack in NR-IQA, and then put it in proper context by makingconnections to previously related techniques from different engineering fields.
3.1 Problem Formulation
An adversarial attack aims to modify a benign data sample x 0 to x ⋆ such that the prediction of amachine learning model f w ( · ) , parameterized by a vector w , undesirably deviates from the groundtruth label c
= f ( x 0 ) , where f ( · ) denotes the underlying true hypothesis.
This problem can begenerally formulated as
[Equation]
where L ( · , · ) denotes the loss function of a particular machine learning task, D ( · , · ) is a signal fidelitymetric to define the feasible set of perturbations, and T constrains the maximum magnitude of allpossible perturbations.
In image classification, one may adopt the cross-entropy function as L (orsimply the negative logit of the ground-truth category for untargeted attacks and the logit differenceof the specified and the ground-truth categories for targeted attacks).
ℓ p -norm induced metrics arefrequently used to implement D .
We refer readers to [44, 45, 46, 47, 48, 49] for instantiations of
Eq.
(1) for attacking other vision tasks.
It is tempting to reuse Eq.
(1) to instantiate the adversarial attack on NR-IQA models, wheref w : R M
(cid:55)→ R takes an image x ∈ R M as input, and computes a real number as the quality estimate.
Without loss of generality, we assume a larger f w ( x ) indicates higher predicted quality.
The trueperceptual quality, f ( x ) , referred to as the mean opinion score (MOS) of x , can be collected via astandard psychophysical experiment.
As a regression task, we may specify L as some discrepancyfunction between the quality prediction of the perturbed image, f w ( x ) , and the MOS of the initialimage f ( x 0 ) , e .
g ., the mean squared error (MSE), ( f w ( x ) − f ( x 0 )) 2 .
D can be implemented by any“perceptual” FR-IQA model.
We point out three caveats of this constrained optimization.
First, existing FR-IQA models onlyprovide a rough account for human perception of image quality.
For example, the α -level setof D w.r.t.
x 0 , i .
e ., { x | D ( x, x 0 )
= α } , may contain two images of drastically different quality.
Computational models that are specifically designed for measuring JNDs [17, 50, 51, 52, 53, 54, 55],are only tested under simple perturbations ( e .
g ., noise shaping and compression), and may have theirown adversarial examples.
As a consequence, the feasible set D ( x, x 0 ) ≤ T is little likely to havethe quality-preserving property.
Second, working with an imperfect FR-IQA model requires settinga content-dependent threshold T ( x 0 ) to ensure a quality-preserving feasible set.
However, this is ahighly non-trivial task, and may result in a trivial solution ( e .
g .,
T ( x 0 ) →
0 due to the inaccuracyof D ).
Third, solving the constrained optimization problem using PGD (even with careful step sizescheduling) appears to converge very slowly (due to the highly nonlinearity and nonconvexity of D ).
In addition, PGD tends to make the constraint tight with equality and thus is less likely to promoteimages that are below JNDs.
All these motivate us to consider a Lagrangian relaxation of Problem (1):
[Equation]
where λ is the Lagrange multiplier.
An immediate advantage of Eq.
(2) is that it allows simultaneousmaximization of the discrepancy term measured by the MSE and minimization of the FR-IQA termas the fidelity constraint, which encourages perturbed images to be below JNDs.
An alternative wayof viewing Eq.
(2) is that we perturb x 0 in the quality increase and decrease directions specified bythe NR-IQA model f w , respectively, under the constraint of D , and we choose to optimize along thedirection that leads to the best discrepancy-fidelity trade-off.
Empirically, we find that, if x 0 is ofpoor quality, it is easier to expose counterexamples in the direction of quality improvement, and viceversa.
Meanwhile, it is noteworthy that our discrepancy term relies on the MOS of the initial image,f ( x 0 ) , which permits naturally occurring failure examples.
This is the case when f w makes a poorquality prediction of x 0 , leading to large discrepancy and high fidelity.
Nevertheless, f ( x 0 ) can bereplaced by f w ( x 0 ) if they are close.
3.2 Perceptually Imperceptible Counterexample Generation
We describe a variant of the steepest ascent method to solve Problem (2).
Given the initial image x 0 ,we first add to each pixel some noise sampled uniformly from a discrete set {− 1 / 255 , 0 , 1 / 255 } totrigger the optimization [37].
We then compute the steepest ascent direction g of the objective w.r.t.x , and take a step of γ along the direction of ℓ p -normalized g for faster convergence.
The pixel valuesof the intermediate image are clamped into the valid range, i .
e ., [0 , 1] , whenever necessary.
After themaximum number of iterations is reached, we quantize it to obtain a 24-bit color image as the output.
By varying the Lagrange multiplier λ , we are able to generate a sequence of images with differentdistortion visibility.
We next design a psychophysical experiment to identify the perceptuallyimperceptible counterexample, giving careful treatment to two subtleties.
First, the distortionvisibility generally reduces with the increase of λ and the decrease of D (see Eq.
(2)), but there areexceptions due to the high dimensionality and non-convexity of the optimization problem.
Therefore,standard psychophysical staircase methods [56] that assume monotonicity w.r.t.
the stimulus intensitymay not be straightforwardly applied.
Second, ideally, we would like to measure the just-noticeabledistortion 1 , but practically, it is difficult to collect such human thresholds in a reliable way unlessexcessive instructions are given.
This is especially true when x 0 has already been severely distorted( e .
g ., blurred or compressed), and it is no easy task to determine whether the added perturbationfurther degrades the image quality.
We thus choose the yes-no task for screening perturbed images below JNDs [57].
Specifically, humanparticipants undergo a series of trials, each consisting of a pair of the perturbed image (corresponding
Require:
An NR-IQA model f w ( · ) , an FR-IQA model D ( · , · ) as the fidelity measure, a set of K hyperparameter values { λ i } Ki =1 , and the step size γ
Input: An initial image x
Output: perceptually imperceptible counterexample do 2: randomly sampled
} 3: while maximum iteration is reached do 4: λ ← λ i 5:
Compute the objective value J ← − D ( x, x 0 ) + λ ( f w ( x ) − f ( x 0 ))6:
Compute the steepest ascent direction g ← arg max v {∇ x J T v | ∥ v ∥ p = 1 } 7:
Update the image x ← x + γ · g/ ∥ g ∥ p 8:
Clamp x into
0 , 1] 9: y i ← x 10:
Quantize y i to a 24-bit color image 11: Identify the perceptually imperceptible y i }
= in a psychophysical experiment to a certain λ ) and the initial image.
For each trial, they must judge whether the two images lookidentical.
A perturbed image is said to be below the JND if participants fail to distinguish it fromthe initial image 75% of the time.
We identify the perceptually imperceptible counterexample as theperturbed image that 1) is below the JND and 2) causes the largest change in quality prediction.
More details about the psychophysical experiment can be found in Sec. 4.1, and the procedure ofcounterexample generation of NR-IQA models is summarized in Algorithm 1.
[Figure]
, and (e) UNIQUE, respectively.
3.3 Connections to Previous Work
In this subsection, we make insightful connections to several beautiful ideas in signal processing,computational vision, and machine learning.
Carlini-Wagner attack [18] has the closest relationship to the proposed perceptual attack on NRIQA models with three differences.
First, Carlini and Wagner chose to work with the Lagrangianformulation mainly from the perspective of computational convenience , while we arrive at the sameformulation from a different perceptual perspective: no computational models exist to reliablycompute JNDs.
Second, for classification, there are C − 1 directions for the Carlini-Wagner attack tolook for adversarial examples, where C is the number of categories.
For NR-IQA as regression, thereare two such directions.
Third, the Carlini-Wagner attack suffices to employ the binary search to findthe desired Lagrange multiplier due to the label-preserving property of the majority perturbationsin image classification.
In contrast, the proposed perceptual attack may have to rely on carefullydesigned psychophysical experiments to determine the value of λ in NR-IQA, which is commonpractice in computational vision.
MAP estimation [58, 59, 60, 61] seeks to estimate a latent image x
⋆ given a corrupted observationx 0 by maximizing the posterior probability p ( x | x 0 ) :
[Equation]
where log p ( x 0 | x ) is the data fidelity term implemented by an FR-IQA model ( e .
g ., the MSE)
andlog p ( x ) is the image prior term to measure the naturalness of the image.
A better prior leads toimproved estimation performance.
The added hyperparameter λ is often manually tuned, whereλ = 1 makes Eq.
(3) hold.
It is widely acknowledged that an ideal NR-IQA model must rely solelyon knowledge of the appearance of natural undistorted images.
This suggests that the NR-IQAmethod should embody an image prior model, and perhaps even that the quality predictions shouldbe monotonically related to probability densities [22].
With such conceptual equivalence between
NR-IQA methods and natural image priors
, we see that the proposed perceptual attack differs onlyfrom MAP estimation in the choice of λ .
Specifically, our attack sets a λ to produce the imagebelow the JND relative to x 0 while causing the most significant change of f w .
In contrast, MAPestimation tests p ( x ) using a λ that leads to the best estimation performance.
More importantly, weshall come to a conclusion: in order for NR-IQA models to work in MAP estimation, they must firstsurvive the proposed perceptual attack as an easier task.
Taking a step further, we may set λ = + ∞ ,which corresponds to the maximization of f w alone as a way of performing image enhancement.
Not surprisingly, all state-of-the-art NR-IQA models fail this task, generating images with annoyingdistortions but higher predicted quality scores (see Fig. 2).
MAD competition [19] is an efficient methodology for comparing computational methods of perceptually discriminable quantities ( e .
g ., image quality).
In the context of comparing two IQA models, f 1and f 2 , MAD automatically synthesizes an image pair ( x ⋆ , y ⋆ ) that are likely to falsify at least one
IQA model in competition by solving
[Equation]
where x 0 is the initial image and f 2 ( x 0 ) specifies a quality level.
By doing so, we obtain a pairof counterexamples, on which f 1 and f 2 hold contradictory opinions: f 1 predicts x
⋆ to have muchbetter quality than y ⋆ , while f 2 treats them as images of identical quality.
Similar as the proposedperceptual attack, psychophysical testing on ( x ⋆ , y ⋆ ) is required to declare which model is the winner.
In contrast to our attack that is constrained in the set of images that look identical to x 0 according tohuman perception (approximated by an FR-IQA model), MAD operates at the level set of another
IQA model f 2 that contains images of the same predicted quality as x 0 .
Moreover, MAD generatesthe best-quality and worst-quality images in terms of f 1 to compare the relative performance of two
IQA models, while we only tests the perceptual robustness of a single NR-IQA model.
Eigen-distortion analysis [20] is a computational method for comparing perceptual image representations, f : R M (cid:55)→ R N .
It computes the eigenvectors of the Fisher information matrix 2 withthe largest and smallest eigenvalues to account for the model-predicted most- and least-noticeabledistortion directions, respectively.
A psychophysical experiment based on a standard staircase methodis necessary to measure the ratio of the visibility thresholds induced by the two directions.
When
N = 1 ( i .
e ., f can be an NR-IQA method), the most-noticeable direction reduces to the gradient of fw.r.t.
the input image x , ∂f/∂x .
Correspondingly, the least-noticeable direction (with an eigenvalueof zero) can be any M -dim unit vector living in the M − 1 -dim subspace orthogonal to ∂f/∂x .
Taking consecutive small steps along least-noticeable directions creates a predicted imperceptibleperturbation, which is subject to human verification.
In this sense, the proposed perceptual attack aimsto tackle a “dual” problem, generating a true imperceptible perturbation to maximize the predictiondiscrepancy.
Experiments
In this section, we first set up the experiments, including descriptions of NR-IQA and FR-IQA models,details of the psychophysical experiment, and new evaluation metrics for measuring the perceptualrobustness of NR-IQA models.
We then present the quantitative and qualitative results accompaniedby an in-depth analysis.
4.1 Experimental Setups
Choice of NR-IQA Models .
We test four NR-IQA models that are believed to be representative inthe field: BRISQUE [12], CORNIA [21], Ma19 [22], and UNIQUE [6].
BRISQUE is a knowledge-
[Table]
[Table]
d e fi n e d i n E q .
[Table]
driven model, extracting NSS from mean-subtracted contrast-normalized pixel values.
CORNIA
[21]relies on unsupervised learning of a visual codebook from image patches, followed by soft-assignmentcoding and max pooling, to obtain image representations.
As feature engineering is not involved,
CORNIA can be seen as a shallow learning-based method.
Ma19 [22] is a four-layer DNN withgeneralized divisive normalization [62] as nonlinear activation, which is trained from a large numberof corrupted image pairs without reliance on MOSs.
UNIQUE [6] trains a variant of ResNet-34 [1] onmultiple IQA datasets to handle both synthetic and realistic camera distortions.
We use the trainingcodes provided by the original authors to re-train BRISQUE and CORNIA on LIVE [7], Ma19 [22]on our own collected dataset 3 , and UNIQUE on six human-rated IQA databases [7, 63, 64, 9, 5, 65].
There is no overlap between all training sets and the initial images used to generate counterexamples.
As suggested by the Video Quality Experts Group [66], a four-parameter logistic function can beadopted to compensate for the prediction nonlinearity, making different NR-IQA more comparable:
[Equation]
β 1 to β 4 are fitting parameters, where β 1 and β 2 determine the maximum and minimum mappingvalues.
Empirically, we find that, for different NR-IQA models, the estimated β 1 and β 2 can be quitedifferent.
Thus, we choose to manually enforce β 1 = 10 and β 2 = 0 .
We consider the learned q aspart of the NR-IQA model.
Choice of FR-IQA Models .
We consider four FR-IQA models to approximate the perceptual distancebetween the initial and perturbed images: the Chebyshev distance, SSIM 4 [10], LPIPS [3], and
DISTS [4].
The Chebyshev distance constrains the maximum pixel difference within an ℓ ∞ -ball.
SSIM [10] is arguably the most successful “perceptual” metric that compares luminance, contrast,and structure, separately.
LPIPS [3] computes the Euclidean distance between deep representationsof two images, which shows reasonable effectiveness in explaining image quality.
DISTS [4] is thefirst FR-IQA method that unifies structure and texture similarity, and is competitive in perceptualoptimization of various image processing tasks [67].
We apply the Chebyshev distance to each colorchannel, and take the maximum distance across three channels.
We enable SSIM to be aware of colorinformation by treating the color-to-grayscale conversion as a fixed differentiable front-end, whichallows the gradient to be back-propagated to the input color image.
Both LPIPS and DISTS are basedon variants of VGG-16 [68].
Details of the Psychophysical Experiment .
We collect twelve images as initializations from thepublicly available LIVE IQA database [7] with different perceptual quality and different distortiontypes (see Fig.
5 in the Appendix).
For each of sixteen combinations of NR-IQA and FR-IQA models,and each of the twelve initial images, we set λ to 32 values, and optimize the objective in Eq.
(2) togenerate 32 perturbed images.
For the Chebyshev distance, we use the steepest ascent direction inℓ ∞ -norm, and for the remaining three FR-IQA models, we choose the steepest ascent direction inℓ 2 -norm, namely the gradient direction.
We set the step size γ to 10 − 3 and the maximum number ofiterations to 200 , respectively.
We recruit fifteen human subjects (with normal or corrected-to-normal vision) to participate in thepsychophysical experiment, viewing the image pairs from a fixed distance of twice the screen height.
A training session is performed to familiarize each subject with the study.
For each yes-no trial,subjects are shown (for one second each with a half-second gray screen between images, and inrandomized order) a perturbed image and the corresponding initial image, and then asked to determinewhether the two images look visually different.
All image pairs are displayed on a 24 ′′ LCD monitorat a resolution of 1920 × 1080 (in an office environment with normal illumination).
To avoid thefatigue effect, subjects are required to take a break during the experiment.
In total, we generate4 × 4 × 12 = 192 counterexamples to test the robustness of NR-IQA models.
Evaluation Metrics .
Unlike image classification, it is generally difficult (or even pointless) for aregression task to define the success of an adversarial attack on a per-image basis.
As image qualityis a relative perceptual quantity, we adopt the Spearman rank-order correlation coefficient (SRCC) tomeasure the prediction monotonicity between model predictions and MOSs of 24 images ( 12 initialimages and 12 corresponding counterexamples).
In addition, we define the average ratio of maximumallowable change in quality prediction to actual change over counterexamples in a logarithmic scale:
[Equation]
where S is the number of initial images and x i denotes the i -th one.
A larger R means better stability( i .
e ., robustness), but not necessarily better quality prediction performance as the MOS is not involvedin the computation.
4.2 Main Results
We summarize the SRCC and average ratio results of NR-IQA models under intra-model attacks( i .
e ., using counterexamples generated by the proposed attack with different FR-IQA models) andinter-model attacks ( i .
e ., using counterexamples originally spotted to falsify another NR-IQA model)in
Table 1 and Table 2, respectively.
The results without perceptual attacks are also presented asreferences.
From the tables, we have a number of insightful observations.
First, the intra-attacks are capable of falsifying the corresponding NR-IQA models, no matter which
FR-IQA model is adopted.
This is evidenced by a catastrophic SRCC drop relative to the originalperformance.
We then conclude that none of the evaluated NR-IQA design philosophies ( i .
e ., NSSbased, codebook-based, and DNN-based approaches), are inherently perceptually robust.
Second,compared with intra-model attacks, NR-IQA methods appear much more robust to inter-model attacks.
[Figure] [Figure]
This manifests the poor transferability of perceptually imperceptible counterexamples, which fail tocraft black-box attacks in NR-IQA.
On the positive side, the generated examples by the proposedperceptual attack are informative to reveal distinct design flows of different NR-IQA models, andmay point out promising ways to improve a model or to combine the best aspects of multiple models.
Third, when switching to the average ratio (Eq.
, we find that UNIQUE is the least stable modelconsistently for all FR-IQA methods, suggesting that the overparameterization of UNIQUE creates anabundant space for overfitting the current IQA datasets.
Although the observations have been obtainedfrom the expensive psychophysical experiment on only twelve images, we empirically find these tobe consistent across a wide range of images with different content and distortion complexities.
To compare the perturbations generated for different NR-IQA models in a more intuitive way, wevisualize the absolute residual images ( i .
e ., ( | x 0 − x ⋆ | ) in Fig.
3, where DISTS is used as the imagefidelity measure.
The primary observation is that the difference in perturbations provides usefulclues on how they extract quality-aware features.
Specifically, perturbations for BRISQUE [12]mainly emerge in smooth and textured regions, where manipulation of individual and product oflocally normalized luminances is much easier, as a way of cheating the built NSS model.
The learnedcodebook in CORNIA [21] contains many Dirac delta functions of different locations and edge filtersof different orientations, for which the selected texture and edge patches from a test image give themaximum response and thus are used for quality computation.
This may explain the perturbations in
Fig.
3 (c) are concentrated along the edges and in strong textures.
The blocking perturbations in Fig. 3(d) appear primarily on the objects ( e .
g ., the lighthouse and the fence) and along strong edges ( e .
g .,the image borders due to zero padding in DNNs).
We believe this arises from the spatial pyramidpooling [69] layer used in Ma19 [22].
When the pyramid pooling is replaced with global averagepooling, the perturbations by Ma19 resemble those in Fig.
3 (e) by UNIQUE.
In addition, compare to
BRISQUE and CORNIA that only accept grayscale images, the perturbations by Ma19 and UNIQUEoccur in all color channels (not shown).
Finally, nearly all pixel perturbations are less than 4 / 255 ,justifying the effectiveness of our psychophysical experiment to identify counterexamples that arebelow JNDs.
Conclusion
We have described a perceptual attack on NR-IQA models, based on which their perceptual robustnesshas been thoroughly evaluated.
The proposed attack has close connections to previous techniques.
Wefound that neither the conventional knowledge-driven NR-IQA models nor the modern DNN-basedmethods are inherently robust to perceptually imperceptible perturbations.
Moreover, the generatedadversarial examples by one NR-IQA model do not transfer in an efficient way to falsify other models,which contain valuable information to expose the design flows of respective models.
In the future, we plan to develop perceptually robust NR-IQA methods.
From the model constructionperspective, we will identify and combine robust building blocks of different NR-IQA methods.
From the optimization perspective, we may directly employ adversarial training as suggested by[37].
Meanwhile, we may draw inspiration from the connections between the proposed perceptualattack and previous analysis by synthesis techniques, and develop robust regularizers [70] to facilitatetraining robust NR-IQA methods.
Checklist
For all authors.
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Sec.
We focus on evaluation
of perceptual robustness on NR-IQA models
in this paper, leaving the development of
perceptually robust models to future work.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[No]
But we will
release all the source code and data upon acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See both Sec.
4.1 and the Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See the Appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] As we described in
Sec. 4.1, we use the publicly available LIVE IQA database [7].
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] LIVE IQA database is freely available online.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[Yes] See the descriptions of the details of the psychophysical experiment
in Sec. 4.1 and a screenshot of the user interface in the Appendix.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A AppendixA.1 Implementation Details
We evaluate all four FR-IQA models ( i .
e ., Minkowski distance, SSIM, LPIPS, and DISTS) for makingpredictions of JNDs for natural images.
Specifically, we collect four undistorted images from the
LIVE dataset [7], on which small perturbations are applied.
We present each pair of a perturbedimage and the associated source image along with the quality estimated by an FR-IQA model in
Fig. 6, from which we find that although Fig.
6 (b), (d), (f), and (h) are assessed as being of highquality by the FR-IQA models, all the added perturbations are above JND.
Therefore, we have toconduct well-controlled psychophysical experiments to verify that the perceptually imperceptiblecounterexamples are below JND.
The subjective experiment was carried out in an indoor officeenvironment with a normal lighting condition (with approximately 200 lux) and without reflectingceiling walls and floors.
The peak luminance of the displayed images is mapped to 200 cd/m 2 .
Theseare suggested in the BT.
500 recommendations.
To generate candidate perceptually imperceptible counterexamples, we use 32 different lambdavalues: [0.2, 0.4, 0.6, 0.8, 1, 3, 5, 7, 9, 20, 40, 60, 80, 100, 300, 500, 700, 900, 2,000, 4,000, 6,000,8,000, 10,000, 30,000, 50,000, 70,000, 90,000, 200,000, 600,000, 1,000,000, 5,000,000, 9,000,000].
All the experiments are conducted on a machine equipped with an Intel(R) Core(TM) i9-9900K CPU@ 3.60GHZ, 48G RAM, and an NVIDIA Geforce GTX 2080Ti 11G GPU.
A.2 One perturbation to fail multiple NR-IQA models
Under the current circumstance that the counterexamples are not transferable (see Table 2), it isworth exploring one perceptually imperceptible perturbation to simultaneously fail multiple NR-IQAmodels.
Conceptually, it is straightforward to extend the current computational method to look forsuch universal perturbation by replacing the second term in Eq.
2 with the average of discrepancyover N NR-IQA models:
[Equation]
We have made a preliminary experiment to search for such a perturbation on an initial image asshown in Fig.
8 (a).
By using the extended attack in Eq.
7 to simultaneously attack BRISQUE,
CORNIA, Ma19, and UNIQUE with DISTS as the image fidelity measure, we find an image (Fig. 8(b)) with perturbation below JNDs with human-in-the-loop.
We also visualize the perturbation withan amplified magnitude in Fig.
8 (c), from which we observe a perturbation with mixed patterns,indicating that the resulting perturbation originates from multiple different NR-IQA models.
Finally,we compare the quality predictions of Fig.
8 (a) and Fig.
8 (b) by the four NR-IQA models in Table 3,reaching an important observation that the generated single perturbation can simultaneously changethe quality predictions of the four NR-IQA models on the counterexample (note that we have limitedthe range of quality predictions of all four NR-IQA models within [0 , 10] by manually enforcingβ 1 = 10 and β 2 = 0 in Eq. 5).
[Table]
[Table] [Table]
[Table] [Table]
A.3 Defending against the perceptual attack
One of the key observations is that the counterexamples generated from one NR-IQA model arenot transferrable to falsify other models.
Intuitively, we may directly leverage model ensembletricks to mitigate the vulnerability of perceptual quality metrics, but it turns out not a viable solutionas indicated by the preliminary results in Sec.
A.2.
Following the common practice of using themethod of Madry et al.
[37], we have also conducted some preliminary experiments to adversarially training the UNIQUE model.
Counterexamples during training are generated under an ℓ ∞ -constraintwith a radius of 4255 , and the number of attacking iterations and the step size are set to 5 and 2255 ,respectively.
After training, we test the adversarially trained model using the proposed perceptualattack, and conduct an informal psychophysical experiment to verify the perceptual imperceptibilityof the found counterexamples (by two of the authors).
As shown in Table 4, vanilla adversarialtraining significantly improves the robustness of the UNIQUE model.
In the future, we will delvedeep into the design and training of perceptually robust NR-IQA models with improved effectivenessand efficiency.
[Table]
A.4 Visualization of perturbations
[Table]
To visualize what the perceptual attacks look like, we include perceptual counterexamples withamplified perturbations in Fig.
It deserves to note that the perturbations are imperceptible (verifiedby the psychophysical experiments) if we do not scale their intensities.
We also show pairs of initial images and their associated counterexamples without amplification ofthe perturbations in Fig.
It is clear that the perturbations are invisible, verifying the effectiveness ofthe psychophysical experiments to make sure that the perturbations are below the JNDs.
[Figure] [Figure]
[Figure] [Figure]