Original: Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples
Revised: Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples

Original: Abstract
Revised: Abstract

Original: Evaluating robustness of machine-learning models to adversarial examples is a challenging problem.
Revised: Evaluating robustness of machine-learning models to adversarial examples is a challenging problem.

Original: Many defenses have been shown to provide a false sense of robustness by causing gradient-based attacks to fail, and they have been broken under more rigorous evaluations.
Revised: Many defenses have been shown to provide a false sense of robustness by causing gradient-based attacks to fail, and they have been broken under more rigorous evaluations.

Original: Although guidelines and best practices have been suggested to improve current adversarial robustness evaluations, the lack of automatic testing and debugging tools makes it difficult to apply these recommendations in a systematic manner.
Revised: Although guidelines and best practices have been suggested to improve current adversarial robustness evaluations, the lack of automatic testing and debugging tools makes it difficult to apply these recommendations in a systematic manner.

Original: In this work, we overcome these limitations by: (i) categorizing attack failures based on how they affect the optimization of gradientbased attacks, while also unveiling two novel failures affecting many popular attack implementations and past evaluations; (ii) proposing six novel indicators of failure , to automatically detect the presence of such failures in the attack optimization process; and (iii) suggesting a systematic protocol to apply the corresponding fixes.
Revised: In this work, we overcome these limitations by: (i) categorizing attack failures based on how they affect the optimization of gradientbased attacks, while also unveiling two novel failures affecting many popular attack implementations and past evaluations; (ii) proposing six novel indicators of failure , to automatically detect the presence of such failures in the attack optimization process; and (iii) suggesting a systematic protocol to apply the corresponding fixes.

Original: Our extensive experimental analysis, involving more than 15 models in 3 distinct application domains, shows that our indicators of failure can be used to debug and improve current adversarial robustness evaluations, thereby providing a first concrete step towards automatizing and systematizing them.
Revised: Our extensive experimental analysis, involving more than 15 models in 3 distinct application domains, shows that our indicators of failure can be used to debug and improve current adversarial robustness evaluations, thereby providing a first concrete step towards automatizing and systematizing them.

Original: Our open-source code is available at: https://github.com/pralab/IndicatorsOfAttackFailure .
Revised: Our open-source code is available at: https://github.com/pralab/IndicatorsOfAttackFailure .

Original:  Introduction
Revised:  Introduction

Original: Despite their unprecedented success in many different applications, machine-learning models have been shown to be vulnerable to adversarial examples [40, 4], i.e., inputs intentionally crafted to mislead such models at test time.
Revised: Despite their unprecedented success in many different applications, machine-learning models have been shown to be vulnerable to adversarial examples [40, 4], i.e., inputs intentionally crafted to mislead such models at test time.

Original: While hundreds of defenses have been proposed to overcome this issue [46, 29, 43, 36], many of them turned out to be ineffective, as their robustness to adversarial inputs was significantly overestimated.
Revised: While hundreds of defenses have been proposed to overcome this issue [46, 29, 43, 36], many of them turned out to be ineffective, as their robustness to adversarial inputs was significantly overestimated.

Original: In particular, some defenses were evaluated by running existing attacks with inappropriate hyperparameters (e.g., using an insufficient number of iterations), while others were implicitly obfuscating gradients, thereby causing the optimization of gradient-based attacks to fail.
Revised: In particular, some defenses were evaluated by running existing attacks with inappropriate hyperparameters (e.g., using an insufficient number of iterations), while others were implicitly obfuscating gradients, thereby causing the optimization of gradient-based attacks to fail.

Original: Such defenses have been shown to yield much lower robustness under more rigorous robustness evaluations and using attacks carefully adapted to evade them [10, 2].
Revised: Such defenses have been shown to yield much lower robustness under more rigorous robustness evaluations and using attacks carefully adapted to evade them [10, 2].

Original: To prevent such evaluation mistakes, and help develop better defenses, evaluation guidelines and best practices have been described in recent work [11].
Revised: To prevent such evaluation mistakes, and help develop better defenses, evaluation guidelines and best practices have been described in recent work [11].

Original: However, they have been mostly neglected as their application is non-trivial; in fact, 13 defenses published after the release of these guidelines were found again to be wrongly evaluated, reporting overestimated adversarial robustness values [41].
Revised: However, they have been mostly neglected as their application is non-trivial; in fact, 13 defenses published after the release of these guidelines were found again to be wrongly evaluated, reporting overestimated adversarial robustness values [41].

Original: It has also been shown that, even when following such guidelines, robustness can still be overestimated [33].
Revised: It has also been shown that, even when following such guidelines, robustness can still be overestimated [33].

Original: Finally, while recently-proposed frameworks combining diverse attacks seem to provide more reliable adversarial robustness evaluations [13, 44], it is still unclear whether and to what extent the considered attacks can also be affected by subtle failures.
Revised: Finally, while recently-proposed frameworks combining diverse attacks seem to provide more reliable adversarial robustness evaluations [13, 44], it is still unclear whether and to what extent the considered attacks can also be affected by subtle failures.

Original: To address these limitations, in this work we propose the first testing approach aimed to debug and automatically detect misleading adversarial robustness evaluations.
Revised: To address these limitations, in this work we propose the first testing approach aimed to debug and automatically detect misleading adversarial robustness evaluations.

Original: The underlying idea, similar to traditional software testing, is to lower the entry barrier for researchers and practitioners towards performing more reliable robustness evaluations.
Revised: The underlying idea, similar to traditional software testing, is to lower the entry barrier for researchers and practitioners towards performing more reliable robustness evaluations.

Original: To this end, we provide the following contributions:
Revised: To this end, we provide the following contributions:

Original: (i) we categorize
Revised: (i) we categorize

Original: four
Revised: four

Original: known attack failures by
Revised: known attack failures by

Original: connecting
Revised: connecting

Original: them to the
Revised: them to the

Original: optimization
Revised: optimization

Original: process of gradient-
Revised: process of gradient-

Original: based attacks, which enabled us also to identify two additional, never-before-seen failures affecting many popular attack implementations and past evaluations (
Revised: based attacks, which enabled us also to identify two additional, never-before-seen failures affecting many popular attack implementations and past evaluations (

Original: Sect
Revised: Sect

Original: 2.1); (ii) we propose six indicators of attack failures (IoAF), i.e., quantitative metrics derived by analyzing the optimization process of gradient-based attacks, which can automatically detect the corresponding failures (Sect. 2.2); and (iii) we suggest a systematic, semi-automated protocol to apply the corresponding fixes (Sect. 2.3).
Revised: 2.1); (ii) we propose six indicators of attack failures (IoAF), i.e., quantitative metrics derived by analyzing the optimization process of gradient-based attacks, which can automatically detect the corresponding failures (Sect. 2.2); and (iii) we suggest a systematic, semi-automated protocol to apply the corresponding fixes (Sect. 2.3).

Original: We empirically validate our approach on three distinct application domains (images, audio, and malware), showing how recently-proposed, wrongly-evaluated defenses could have been evaluated correctly by monitoring the IoAF values and following our evaluation protocol (Sect. 3).
Revised: We empirically validate our approach on three distinct application domains (images, audio, and malware), showing how recently-proposed, wrongly-evaluated defenses could have been evaluated correctly by monitoring the IoAF values and following our evaluation protocol (Sect. 3).

Original: We also open source our code and data to enable reproducibility of our findings at https://github.com/pralab/
Revised: We also open source our code and data to enable reproducibility of our findings at https://github.com/pralab/

Original: IndicatorsOfAttackFailure .
Revised: IndicatorsOfAttackFailure .

Original: We conclude by discussing related work (Sect. 4), limitations, and future research directions (Sect. 5).
Revised: We conclude by discussing related work (Sect. 4), limitations, and future research directions (Sect. 5).

Original:  Debugging Adversarial Robustness Evaluations
Revised:  Debugging Adversarial Robustness Evaluations

Original: We introduce here our automated testing approach for adversarial robustness evaluations, based on the design of novel metrics, referred to as Indicators of Attack Failure (IoAF) , each aimed to detect a specific failure within the optimization process of gradient-based attacks.
Revised: We introduce here our automated testing approach for adversarial robustness evaluations, based on the design of novel metrics, referred to as Indicators of Attack Failure (IoAF) , each aimed to detect a specific failure within the optimization process of gradient-based attacks.

Original: 2.1 Optimizing Gradient-based Attacks
Revised: 2.1 Optimizing Gradient-based Attacks

Original: Before introducing the IoAF, we discuss how gradient-based attacks are optimized, and provide a generalized attack algorithm that will help us to identify the main attack failures.
Revised: Before introducing the IoAF, we discuss how gradient-based attacks are optimized, and provide a generalized attack algorithm that will help us to identify the main attack failures.

Original: In particular, we consider attacks that solve the following optimization problem:
Revised: In particular, we consider attacks that solve the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where x ∈ [0 , 1] d is the d -dimensional input sample, y t ∈ Y = { 1 , . .
Revised: where x ∈ [0 , 1] d is the d -dimensional input sample, y t ∈ Y = { 1 , . .

Original: . , c } is the target class label (chosen to be different from the true label y of the input sample), δ ∈ R d is the perturbation applied to the input sample during the optimization, and θ are the parameters of the model.
Revised: . , c } is the target class label (chosen to be different from the true label y of the input sample), δ ∈ R d is the perturbation applied to the input sample during the optimization, and θ are the parameters of the model.

Original: The loss function
Revised: The loss function

Original: L is defined such that minimizing it amounts to having the perturbed sample x + δ misclassified as y t .
Revised: L is defined such that minimizing it amounts to having the perturbed sample x + δ misclassified as y t .

Original: Typical examples include the Cross-Entropy (CE) loss, or the so-called logit loss [10], i.e.,
Revised: Typical examples include the Cross-Entropy (CE) loss, or the so-called logit loss [10], i.e.,

Original: L ( x + δ , y t , θ ) = max k ̸ =
Revised: L ( x + δ , y t , θ ) = max k ̸ =

Original: y t f k ( x + δ , θ ) − f y t ( x + δ , θ ) , being f i ( · , θ ) the model’s prediction ( logit ) for class i .
Revised: y t f k ( x + δ , θ ) − f y t ( x + δ , θ ) , being f i ( · , θ ) the model’s prediction ( logit ) for class i .

Original: 1 Let us finally discuss the constraints in Eq.
Revised: 1 Let us finally discuss the constraints in Eq.

Original: While the ℓ p -norm constraint ∥ δ ∥ p ≤ ϵ bounds the maximum perturbation size, the box constraint x + δ ∈ [0 , 1] d ensures that the perturbed sample stays withing the given (normalization) bounds.
Revised: While the ℓ p -norm constraint ∥ δ ∥ p ≤ ϵ bounds the maximum perturbation size, the box constraint x + δ ∈ [0 , 1] d ensures that the perturbed sample stays withing the given (normalization) bounds.

Original: Transfer Attacks.
Revised: Transfer Attacks.

Original: When the target model is either non-differentiable or not sufficiently smooth [2], a surrogate model ˆ θ can be used to provide a smoother approximation of the loss L , and facilitate the attack optimization.
Revised: When the target model is either non-differentiable or not sufficiently smooth [2], a surrogate model ˆ θ can be used to provide a smoother approximation of the loss L , and facilitate the attack optimization.

Original: The attack is thus optimized on the surrogate model, and then evaluated against the target model.
Revised: The attack is thus optimized on the surrogate model, and then evaluated against the target model.

Original: If the attack is successful, then it is said to correctly transfer to the target [30].
Revised: If the attack is successful, then it is said to correctly transfer to the target [30].

Original: ▷ point θ
Revised: ▷ point θ

Original: 3 δ 0 ←
Revised: 3 δ 0 ←

Original: 4 for i ∈
Revised: 4 for i ∈

Original: 1 , n ] do δ i ← δ i − 1 − α ∇ x L ( x + δ i − 1 , y t ; ˆ θ )
Revised: 1 , n ] do δ i ← δ i − 1 − α ∇ x L ( x + δ i − 1 , y t ; ˆ θ )

Original: Compute gradient update(
Revised: Compute gradient update(

Original: δ i ←
Revised: δ i ←

Original: Generalized Attack Algorithm.
Revised: Generalized Attack Algorithm.

Original: We provide here a generalized algorithm, given as Algorithm 1, which summarizes the main steps followed by gradient-based attacks to solve Problem (1)-(2).
Revised: We provide here a generalized algorithm, given as Algorithm 1, which summarizes the main steps followed by gradient-based attacks to solve Problem (1)-(2).

Original: The algorithm starts by defining an initialization point (line 1), which can be the input sample x , a randomly-perturbed version of it, or even a sample from the target class [5].
Revised: The algorithm starts by defining an initialization point (line 1), which can be the input sample x , a randomly-perturbed version of it, or even a sample from the target class [5].

Original: If the target model θ is either non-differentiable or not sufficiently smooth, a surrogate model ˆ θ can be used to approximate it, and perform a transfer attack (line 2).
Revised: If the target model θ is either non-differentiable or not sufficiently smooth, a surrogate model ˆ θ can be used to approximate it, and perform a transfer attack (line 2).

Original: The attack then iteratively updates the point to find an adversarial example (line 4), computing one (or more) gradient updates in each iteration (line 5), while the perturbation δ i +1 is projected onto the feasible domain (i.e., the intersection of the constraints in
Revised: The attack then iteratively updates the point to find an adversarial example (line 4), computing one (or more) gradient updates in each iteration (line 5), while the perturbation δ i +1 is projected onto the feasible domain (i.e., the intersection of the constraints in

Original: Eq. 2) via a projection operator Π (line 6).
Revised: Eq. 2) via a projection operator Π (line 6).

Original: The algorithm finally returns the best perturbation across the whole attack path , i.e., the perturbed sample that evades the (target) model with the lowest loss (line 7).
Revised: The algorithm finally returns the best perturbation across the whole attack path , i.e., the perturbed sample that evades the (target) model with the lowest loss (line 7).

Original: This generalized attack algorithm provides the basis for identifying known and novel attack failures, by connecting each failure to a specific step of the algorithm, as discussed in the next section.
Revised: This generalized attack algorithm provides the basis for identifying known and novel attack failures, by connecting each failure to a specific step of the algorithm, as discussed in the next section.

Original: Other Perturbation Models.
Revised: Other Perturbation Models.

Original: We conclude this section by remarking that, even if we consider additive ℓ p -norm perturbations in our formulation, the proposed approach can be easily extended to more general perturbation models (e.g., represented as x ′
Revised: We conclude this section by remarking that, even if we consider additive ℓ p -norm perturbations in our formulation, the proposed approach can be easily extended to more general perturbation models (e.g., represented as x ′

Original: =
Revised: =

Original: h ( x , δ ) , being x ′ the perturbed feature representation of a valid input sample, and h a manipulation function parameterized by δ , as in [17]).
Revised: h ( x , δ ) , being x ′ the perturbed feature representation of a valid input sample, and h a manipulation function parameterized by δ , as in [17]).

Original: In fact, our approach can be used to debug and evaluate any attack as long as it is optimized via gradient descent, regardless of the given perturbation model and constraints, as also demonstrated in our experiments (A.3,A.4, A.5).
Revised: In fact, our approach can be used to debug and evaluate any attack as long as it is optimized via gradient descent, regardless of the given perturbation model and constraints, as also demonstrated in our experiments (A.3,A.4, A.5).

Original: 2.2 Indicators of Attack Failure
Revised: 2.2 Indicators of Attack Failure

Original: We introduce here our approach, compactly represented in Fig.
Revised: We introduce here our approach, compactly represented in Fig.

Original: 1, by describing the identified attack failures, along with the corresponding indicators and mitigation strategies.
Revised: 1, by describing the identified attack failures, along with the corresponding indicators and mitigation strategies.

Original: We describe two main categories of failures, respectively referred to as loss-landscape and attack-optimization failures.
Revised: We describe two main categories of failures, respectively referred to as loss-landscape and attack-optimization failures.

Original: Loss-landscape Failures, Mitigations, and Indicators.
Revised: Loss-landscape Failures, Mitigations, and Indicators.

Original: The first category of failures depends on the choice of the loss function L and of the target model θ , regardless of the specific attack implementation.
Revised: The first category of failures depends on the choice of the loss function L and of the target model θ , regardless of the specific attack implementation.

Original: In fact, it has been shown that the objective function in Eq.
Revised: In fact, it has been shown that the objective function in Eq.

Original: (1) may exhibit obfuscated gradients [10, 2], which prevent gradient-based attacks to find adversarial examples even when they exist within the feasible domain.
Revised: (1) may exhibit obfuscated gradients [10, 2], which prevent gradient-based attacks to find adversarial examples even when they exist within the feasible domain.

Original: We report two known failures linked to this issue, referred as F 1 and F 2 below.
Revised: We report two known failures linked to this issue, referred as F 1 and F 2 below.

Original: F 1 : Shattered Gradients.
Revised: F 1 : Shattered Gradients.

Original: This failure is reported in
Revised: This failure is reported in

Original: It
Revised: It

Original: compromises the
Revised: compromises the

Original: computation of
Revised: computation of

Original: the
Revised: the

Original: input gradient
Revised: input gradient

Original: ∇ x L , and the whole execution of the
Revised: ∇ x L , and the whole execution of the

Original: attack
Revised: attack

Original: , when
Revised: , when

Original: at
Revised: at

Original: least one of the model’s components (e.g., a specific layer) is non-differentiable or causes numerical instabilities (
Revised: least one of the model’s components (e.g., a specific layer) is non-differentiable or causes numerical instabilities (

Original: F in Fig. 3)
Revised: F in Fig. 3)

Original: M 1 : Use BPDA.
Revised: M 1 : Use BPDA.

Original: This failure can be overcome using the Backward Pass Differentiable Approximation (BPDA) [3, 41], i.e., replacing the derivative of the problematic components with the identity matrix.
Revised: This failure can be overcome using the Backward Pass Differentiable Approximation (BPDA) [3, 41], i.e., replacing the derivative of the problematic components with the identity matrix.

Original: I 1 : Unavailable Gradients.
Revised: I 1 : Unavailable Gradients.

Original: Despite the failure and mitigation being known, no automated
Revised: Despite the failure and mitigation being known, no automated

Original: , systematic approach has ever been proposed to detect F 1 .
Revised: , systematic approach has ever been proposed to detect F 1 .

Original: This newly-proposed indicator is able to automatically detect the presence of non-differentiable components and numerical instabilities when computing gradients; in particular, for each given input x , if the input gradient ∇ x L ( x , y t , θ ) has zero norm, or if its computation returns an error, we set I 1 ( x ) = 1 (and 0 otherwise).
Revised: This newly-proposed indicator is able to automatically detect the presence of non-differentiable components and numerical instabilities when computing gradients; in particular, for each given input x , if the input gradient ∇ x L ( x , y t , θ ) has zero norm, or if its computation returns an error, we set I 1 ( x ) = 1 (and 0 otherwise).

Original: Attack Optimization
Revised: Attack Optimization

Original: F a il u r es
Revised: F a il u r es

Original: I nd i ca t
Revised: I nd i ca t

Original: o r s
Revised: o r s

Original: UnavailableGradients
Revised: UnavailableGradients

Original: Silent Success
Revised: Silent Success

Original: Incomplete Optimization
Revised: Incomplete Optimization

Original: Use BPDA
Revised: Use BPDA

Original: Use EoT
Revised: Use EoT

Original: Fix Attack Implementation
Revised: Fix Attack Implementation

Original: Change Loss ( Adaptive )
Revised: Change Loss ( Adaptive )

Original: Unstable Predictions
Revised: Unstable Predictions

Original: Tune Step Size and Iterations
Revised: Tune Step Size and Iterations

Original: M i t
Revised: M i t

Original: i g a t i on s
Revised: i g a t i on s

Original: Unconstrained Attack Failure
Revised: Unconstrained Attack Failure

Original: Change Loss ( Bad Local
Revised: Change Loss ( Bad Local

Original: Minimum )
Revised: Minimum )

Original: Loss Landscape (Obfuscated Gradients)
Revised: Loss Landscape (Obfuscated Gradients)

Original: TransferFailure
Revised: TransferFailure

Original: I I
Revised: I I

Original: I I
Revised: I I

Original: I I
Revised: I I

Original: Shattered Gradients
Revised: Shattered Gradients

Original: F
Revised: F

Original: Stochastic Gradients
Revised: Stochastic Gradients

Original: F
Revised: F

Original: Implementation Errors
Revised: Implementation Errors

Original: F
Revised: F

Original: Non-converging Attack
Revised: Non-converging Attack

Original: F
Revised: F

Original: Non-adaptive Attack
Revised: Non-adaptive Attack

Original: F
Revised: F

Original: Unreachable Misclassification
Revised: Unreachable Misclassification

Original: F
Revised: F

Original: M M
Revised: M M

Original: M M
Revised: M M

Original: M M
Revised: M M

Original: Loss/Model-specific fixes to ensure gradients are smooth
Revised: Loss/Model-specific fixes to ensure gradients are smooth

Original: Attack-specific fixes to ensure attack optimization runs correctly * * * * * * * * * *
Revised: Attack-specific fixes to ensure attack optimization runs correctly * * * * * * * * * *

Original: F 2 : Stochastic Gradients.
Revised: F 2 : Stochastic Gradients.

Original: This failure is reported in [2, 41].
Revised: This failure is reported in [2, 41].

Original: The input gradients can be computed, but their value is uninformative, as following the gradient direction does not correctly minimize the loss function L .
Revised: The input gradients can be computed, but their value is uninformative, as following the gradient direction does not correctly minimize the loss function L .

Original: Such issue arises when models introduce randomness in the inference process [22], or when their training induces noisy gradients [43] ( F 2 in Fig. 3).
Revised: Such issue arises when models introduce randomness in the inference process [22], or when their training induces noisy gradients [43] ( F 2 in Fig. 3).

Original: M 2 : Use EoT. Previous work mitigates this failure using Expectation over Transformation (EoT) [2, 41], i.e., by averaging the loss L over the same transformations performed by the randomized model, or over random perturbations of the input (sampled from a given distribution).
Revised: M 2 : Use EoT. Previous work mitigates this failure using Expectation over Transformation (EoT) [2, 41], i.e., by averaging the loss L over the same transformations performed by the randomized model, or over random perturbations of the input (sampled from a given distribution).

Original: I 2 : Unstable Predictions.
Revised: I 2 : Unstable Predictions.

Original: Despite the failure and mitigation being known, no automated, systematic approach has been considered to automatically detect F 2 .
Revised: Despite the failure and mitigation being known, no automated, systematic approach has been considered to automatically detect F 2 .

Original: This novel indicator measures the relative variability V ( x ) of the loss function L around the input samples.
Revised: This novel indicator measures the relative variability V ( x ) of the loss function L around the input samples.

Original: Given an input sample x , we draw s perturbed inputs uniformly from a small ℓ 2 ball centered on x , with radius r , and compute
Revised: Given an input sample x , we draw s perturbed inputs uniformly from a small ℓ 2 ball centered on x , with radius r , and compute

Original: V ( x ) = min( 1 s (cid:80) si =1 | ( L 0 − L i ) /L 

Revised: V ( x ) = min( 1 s (cid:80) si =1 | ( L 0 − L i ) /L

Original: | , 1) ∈ [0 , 1] , where L 0 is the loss obtained at x , L 1 , ..., L s are the loss values computed on the perturbed samples, and the min operator is used to upper bound
Revised: | , 1) ∈ [0 , 1] , where L 0 is the loss obtained at x , L 1 , ..., L s are the loss values computed on the perturbed samples, and the min operator is used to upper bound

Original: V ( x ) ≤ 1 .
Revised: V ( x ) ≤ 1 .

Original: We then set I 2 ( x )
Revised: We then set I 2 ( x )

Original: = 1 if V ( x ) ≥ τ (and 0 otherwise).
Revised: = 1 if V ( x ) ≥ τ (and 0 otherwise).

Original: In our experiments, we set the parameters of this indicator as s = 100 , r = 10 − 3 and τ = 10% , as explained in Sect. 3.
Revised: In our experiments, we set the parameters of this indicator as s = 100 , r = 10 − 3 and τ = 10% , as explained in Sect. 3.

Original: Attack-optimization Failures, Mitigations, and Indicators.
Revised: Attack-optimization Failures, Mitigations, and Indicators.

Original: The second category of failures is connected to problems encountered when running a gradient-based attack to solve Problem (1)-(2) with
Revised: The second category of failures is connected to problems encountered when running a gradient-based attack to solve Problem (1)-(2) with

Original: Algorithm 1.
Revised: Algorithm 1.

Original: This category encompasses failures F 3 - F 6 in Fig.
Revised: This category encompasses failures F 3 - F 6 in Fig.

Original: F 3 : Implementation Errors.
Revised: F 3 : Implementation Errors.

Original: We are the first to identify and characterize this failure, which affects many widely-used attack implementations and past evaluations.
Revised: We are the first to identify and characterize this failure, which affects many widely-used attack implementations and past evaluations.

Original: In particular, we find that many attacks return the last sample of the attack path (line 7 of Algorithm 2.1) even if it is not adversarial, discarding valid adversarial examples found earlier in the attack path ( F 3 in Fig. 3).
Revised: In particular, we find that many attacks return the last sample of the attack path (line 7 of Algorithm 2.1) even if it is not adversarial, discarding valid adversarial examples found earlier in the attack path ( F 3 in Fig. 3).

Original: This issue affects: (i) the PGD attack implementations [25] currently used in the four most-used Python libraries for crafting adversarial examples, i.e., Foolbox, CleverHans, ART, and Torchattacks (for further details, please refer to A.6); and also (ii) AutoAttack [13], as it returns the initial sample if no adversarial example is found, leading to flawed evaluations when used in transfer settings [15].
Revised: This issue affects: (i) the PGD attack implementations [25] currently used in the four most-used Python libraries for crafting adversarial examples, i.e., Foolbox, CleverHans, ART, and Torchattacks (for further details, please refer to A.6); and also (ii) AutoAttack [13], as it returns the initial sample if no adversarial example is found, leading to flawed evaluations when used in transfer settings [15].

Original: M 3 : Fix Attack Implementation.
Revised: M 3 : Fix Attack Implementation.

Original: To fix the issue, we propose to modify the optimization algorithm to return the best result.
Revised: To fix the issue, we propose to modify the optimization algorithm to return the best result.

Original: This mitigation can also be automated using the same mechanism used to automatically evaluate the IoAF, i.e., by wrapping the attack algorithm with a function that keeps track of the best sample found during the attack optimization and eventually returns it.
Revised: This mitigation can also be automated using the same mechanism used to automatically evaluate the IoAF, i.e., by wrapping the attack algorithm with a function that keeps track of the best sample found during the attack optimization and eventually returns it.

Original: I 3 : Silent Success.
Revised: I 3 : Silent Success.

Original: We design I 3 ( x ) as a binary indicator that is set to 1 when the final point in the path is not adversarial, but an adversarial example is found along the attack path (and 0 otherwise).
Revised: We design I 3 ( x ) as a binary indicator that is set to 1 when the final point in the path is not adversarial, but an adversarial example is found along the attack path (and 0 otherwise).

Original: F 4 : Non-converging Attack.
Revised: F 4 : Non-converging Attack.

Original: This failure is reported in [41], noting that the flawed evaluations performed by Buckman et al.
Revised: This failure is reported in [41], noting that the flawed evaluations performed by Buckman et al.

Original: [6] and Pang et al.
Revised: [6] and Pang et al.

Original: [28], respectively, used only 7 and 10 steps of
Revised: [28], respectively, used only 7 and 10 steps of

Original: PGD for testing their defenses.
Revised: PGD for testing their defenses.

Original: More generally, attacks may not reach convergence ( F 4 in Fig.
Revised: More generally, attacks may not reach convergence ( F 4 in Fig.

Original: 3) due to inappropriate choices of their hyperparameters, including not only an insufficient number of iterations n (line 4 of Algorithm 1), but also an inadequate step size α (line 5 of Algorithm 1).
Revised: 3) due to inappropriate choices of their hyperparameters, including not only an insufficient number of iterations n (line 4 of Algorithm 1), but also an inadequate step size α (line 5 of Algorithm 1).

Original: [Figure]
Revised: [Figure]

Original: v
Revised: v

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: F
Revised: F

Original: [Figure]
Revised: [Figure]

Original: F
Revised: F

Original: [Figure]
Revised: [Figure]

Original: Iterations
Revised: Iterations

Original: [Figure]
Revised: [Figure]

Original: M 4 : Tune Step Size and Iterations.
Revised: M 4 : Tune Step Size and Iterations.

Original: The failure can be patched by increasing either the step size α or the number of iterations n .
Revised: The failure can be patched by increasing either the step size α or the number of iterations n .

Original: I 4 : Incomplete Optimization.
Revised: I 4 : Incomplete Optimization.

Original: To automatically evaluate if the attack has not converged, we propose a novel indicator that builds a monotonically decreasing curve ˆ L for the loss, by keeping track of the best minimum found at each attack iteration (a.k.a.
Revised: To automatically evaluate if the attack has not converged, we propose a novel indicator that builds a monotonically decreasing curve ˆ L for the loss, by keeping track of the best minimum found at each attack iteration (a.k.a.

Original: cumulative minimum curve, shown in black in Fig. 2).
Revised: cumulative minimum curve, shown in black in Fig. 2).

Original: Then, we compute the relative loss decrement D ( x ) in the last k iterations (out of n ) as:
Revised: Then, we compute the relative loss decrement D ( x ) in the last k iterations (out of n ) as:

Original: D ( x ) =
Revised: D ( x ) =

Original: | ˆ L ( n ) − ˆ L ( n − k ) | / (max i ˆ L ( i ) − min
Revised: | ˆ L ( n ) − ˆ L ( n − k ) | / (max i ˆ L ( i ) − min

Original: i
Revised: i

Original: ˆ L ( i ) )
Revised: ˆ L ( i ) )

Original: ∈ [0 , 1] .
Revised: ∈ [0 , 1] .

Original: We set
Revised: We set

Original: I 4 ( x ) = 1 if D ( x ) ≥ µ (and 0 otherwise).
Revised: I 4 ( x ) = 1 if D ( x ) ≥ µ (and 0 otherwise).

Original: In our experiments, we set the parameters k
Revised: In our experiments, we set the parameters k

Original: = 10 and µ = 1% , as detailed in Sect.
Revised: = 10 and µ = 1% , as detailed in Sect.

Original: F 5 : Non-adaptive attack.
Revised: F 5 : Non-adaptive attack.

Original: This failure is only qualitatively discussed in [3, 11, 41], showing that many previously-published defenses can be broken by developing adaptive attacks that specifically target the given defense mechanism.
Revised: This failure is only qualitatively discussed in [3, 11, 41], showing that many previously-published defenses can be broken by developing adaptive attacks that specifically target the given defense mechanism.

Original: However, to date, it is still unclear how to evaluate if an attack is really adaptive or not (as it is mostly left to a subjective judgment).
Revised: However, to date, it is still unclear how to evaluate if an attack is really adaptive or not (as it is mostly left to a subjective judgment).

Original: We try to shed light on this issue below.
Revised: We try to shed light on this issue below.

Original: We first note that this issue arises when existing attacks are executed against defenses that may either have non-differentiable components [16] or use additional detection mechanisms [26].
Revised: We first note that this issue arises when existing attacks are executed against defenses that may either have non-differentiable components [16] or use additional detection mechanisms [26].

Original: In both cases, the attack is implicitly executed in a transfer setting against a surrogate model which retains the main structure of the target, but ignores the problematic components/detectors; e.g., Das et al.
Revised: In both cases, the attack is implicitly executed in a transfer setting against a surrogate model which retains the main structure of the target, but ignores the problematic components/detectors; e.g., Das et al.

Original: [16] simply removed the JPEG compression component from the model.
Revised: [16] simply removed the JPEG compression component from the model.

Original: However, it should be clear that optimizing the attack against such a bad surrogate ˆ θ (line 2 of Algorithm 1) is not guaranteed to also bypass the target model θ ( F 5 in Fig. 3).
Revised: However, it should be clear that optimizing the attack against such a bad surrogate ˆ θ (line 2 of Algorithm 1) is not guaranteed to also bypass the target model θ ( F 5 in Fig. 3).

Original: M 5 : Change Loss (Adaptive).
Revised: M 5 : Change Loss (Adaptive).

Original: Previous work [3, 11, 41] applied custom fixes by using better approximations of the target, i.e., loss functions and surrogate models that consider all the components of the defense in the attack optimization.
Revised: Previous work [3, 11, 41] applied custom fixes by using better approximations of the target, i.e., loss functions and surrogate models that consider all the components of the defense in the attack optimization.

Original: This is unfortunately a non-trivial step to automate.
Revised: This is unfortunately a non-trivial step to automate.

Original: I 5 : Transfer Failure.
Revised: I 5 : Transfer Failure.

Original: Despite the failure and some qualitative ideas to implement mitigations being known, it is unclear how to measure and evaluate if an attack is really adaptive.
Revised: Despite the failure and some qualitative ideas to implement mitigations being known, it is unclear how to measure and evaluate if an attack is really adaptive.

Original: To this end, we propose an indicator that detects F 5 by evaluating the misalignment between the loss L optimized by the attack on the surrogate ˆ θ and the same function evaluated on the target θ .
Revised: To this end, we propose an indicator that detects F 5 by evaluating the misalignment between the loss L optimized by the attack on the surrogate ˆ θ and the same function evaluated on the target θ .

Original: We thus set I 5 ( x ) =
Revised: We thus set I 5 ( x ) =

Original: if the attack sample bypasses the surrogate but not the target model (and 0 otherwise).
Revised: if the attack sample bypasses the surrogate but not the target model (and 0 otherwise).

Original: F 6 : Unreachable Misclassification.
Revised: F 6 : Unreachable Misclassification.

Original: We are the first to identify and characterize this failure.
Revised: We are the first to identify and characterize this failure.

Original: It assumes that gradient-based attacks can get stuck in bad local minima (e.g., characterized by flat regions), where no adversarial example is found ( F 6 in Fig. 3).
Revised: It assumes that gradient-based attacks can get stuck in bad local minima (e.g., characterized by flat regions), where no adversarial example is found ( F 6 in Fig. 3).

Original: When this failure is present, unconstrained attacks (i.e., attacks with ϵ → ∞ in Eq. 2) are also expected to fail, even if adversarial examples exist for sure in this setting (since the feasible domain includes samples from all classes).
Revised: When this failure is present, unconstrained attacks (i.e., attacks with ϵ → ∞ in Eq. 2) are also expected to fail, even if adversarial examples exist for sure in this setting (since the feasible domain includes samples from all classes).

Original: M 6 : Change Loss (Bad Local Minimum).
Revised: M 6 : Change Loss (Bad Local Minimum).

Original: We argue that this failure can be fixed by modifying the loss function optimized by the attack, to avoid attack paths that lead to bad local minima.
Revised: We argue that this failure can be fixed by modifying the loss function optimized by the attack, to avoid attack paths that lead to bad local minima.

Original: However, similarly to M 5 , this is not a trivial issue to fix, and it is definitely not easy to automate.
Revised: However, similarly to M 5 , this is not a trivial issue to fix, and it is definitely not easy to automate.

Original: I 6 : Unconstrained Attack Failure.
Revised: I 6 : Unconstrained Attack Failure.

Original: Despite being difficult to find a suitable mitigation for F 6 , detecting it is straightforward.
Revised: Despite being difficult to find a suitable mitigation for F 6 , detecting it is straightforward.

Original: To this end, we run an unconstrained attack on the given input x , and we set I 6 ( x ) = 

Revised: To this end, we run an unconstrained attack on the given input x , and we set I 6 ( x ) =

Original: if such an attack fails (and 0 otherwise).
Revised: if such an attack fails (and 0 otherwise).

Original: 2.3 How to Use Our Framework
Revised: 2.3 How to Use Our Framework

Original: We summarize here the required steps that developers and practitioners should follow to debug their robustness evaluations with our IoAF.
Revised: We summarize here the required steps that developers and practitioners should follow to debug their robustness evaluations with our IoAF.

Original: • Initialize Evaluation.
Revised: • Initialize Evaluation.

Original: The evaluation is initialized with a chosen attack, along with its hyperparameters, number of samples, and the model to evaluate.
Revised: The evaluation is initialized with a chosen attack, along with its hyperparameters, number of samples, and the model to evaluate.

Original: • Mitigate Loss-landscape Failures.
Revised: • Mitigate Loss-landscape Failures.

Original: Before computing the attack, the evaluation should get rid of loss-landscape failures, by computing I 

Revised: Before computing the attack, the evaluation should get rid of loss-landscape failures, by computing I

Original: and I 2 , and applying M 1 and M 2 accordingly.
Revised: and I 2 , and applying M 1 and M 2 accordingly.

Original: These indicators are computed on a small random subset of N samples, as the presence of either F 1 or F 2 would render useless the execution of the attack on all points.
Revised: These indicators are computed on a small random subset of N samples, as the presence of either F 1 or F 2 would render useless the execution of the attack on all points.

Original: Both F 1 and F 2 are reported if they are detected on at least one sample in the given subset.
Revised: Both F 1 and F 2 are reported if they are detected on at least one sample in the given subset.

Original: • Run the Attack.
Revised: • Run the Attack.

Original: If F 1 and F 2 are absent or have been fixed, the attack can be run on all samples.
Revised: If F 1 and F 2 are absent or have been fixed, the attack can be run on all samples.

Original: • Mitigate Attack-specific Failures.
Revised: • Mitigate Attack-specific Failures.

Original: Once the attack completes its execution, indicators I 3 , I 

Revised: Once the attack completes its execution, indicators I 3 , I

Original: and I 5 should be checked.
Revised: and I 5 should be checked.

Original: Depending on their output, M 3 , M 4 , and M 5 are applied, and the attack is repeated (on the affected samples).
Revised: Depending on their output, M 3 , M 4 , and M 5 are applied, and the attack is repeated (on the affected samples).

Original: If no failures are found, but the attack is still failing, the value of I 6 should be checked, and M 6 applied if needed.
Revised: If no failures are found, but the attack is still failing, the value of I 6 should be checked, and M 6 applied if needed.

Original: As I 1 and I 2 , I 6 is also computed on a subset of N samples to avoid unnecessary computations.
Revised: As I 1 and I 2 , I 6 is also computed on a subset of N samples to avoid unnecessary computations.

Original: The failure is reported if I 6 = 1 for at least one sample.
Revised: The failure is reported if I 6 = 1 for at least one sample.

Original: Let us finally remark that when no indicator triggers, one should not conclude that robustness may not be worsened by a different, more powerful attack.
Revised: Let us finally remark that when no indicator triggers, one should not conclude that robustness may not be worsened by a different, more powerful attack.

Original: This is indeed a problem of any empirical evaluation, including software testing – which does not ensure that software is bug-free, but only that some known issues get fixed.
Revised: This is indeed a problem of any empirical evaluation, including software testing – which does not ensure that software is bug-free, but only that some known issues get fixed.

Original: Our methodology, similarly, highlights the presence of known failures in adversarial robustness evaluations and suggests how to mitigate them with a systematic protocol, taking a first concrete step toward making robustness evaluations more systematic and reliable.
Revised: Our methodology, similarly, highlights the presence of known failures in adversarial robustness evaluations and suggests how to mitigate them with a systematic protocol, taking a first concrete step toward making robustness evaluations more systematic and reliable.

Original:  Experiments
Revised:  Experiments

Original: In this section, we demonstrate the effectiveness of our approach by re-evaluating the robustness ofdefenses, and using our IoAF framework to fix them.
Revised: In this section, we demonstrate the effectiveness of our approach by re-evaluating the robustness ofdefenses, and using our IoAF framework to fix them.

Original: Additionally, we evaluate 6 more robust image classification models from a widely-used benchmark, discovering that their robustness claims might be unreliable.
Revised: Additionally, we evaluate 6 more robust image classification models from a widely-used benchmark, discovering that their robustness claims might be unreliable.

Original: We also conduct the following additional experiments, reported in the appendix: (i) we describe how we set the thresholds τ for I 2 and µ for I 4 in A.2; and (ii) we offer evidence that
Revised: We also conduct the following additional experiments, reported in the appendix: (i) we describe how we set the thresholds τ for I 2 and µ for I 4 in A.2; and (ii) we offer evidence that

Original: IoAF can encompass perturbation models beyond ℓ p norms by analyzing the robustness evaluations of one Windows malware detector in A.3, one Android malware detector in A.4, and one audio keyword-spotting model in A.4.
Revised: IoAF can encompass perturbation models beyond ℓ p norms by analyzing the robustness evaluations of one Windows malware detector in A.3, one Android malware detector in A.4, and one audio keyword-spotting model in A.4.

Original: Experimental Setup.
Revised: Experimental Setup.

Original: We run our attacks on an Intel ® Xeon ® CPU E5-2670 v3, with 48 cores,
Revised: We run our attacks on an Intel ® Xeon ® CPU E5-2670 v3, with 48 cores,

Original: GB of RAM, equipped with a Nvidia Quadro M6000 with 24 GB of memory, and we leverage the
Revised: GB of RAM, equipped with a Nvidia Quadro M6000 with 24 GB of memory, and we leverage the

Original: SecML library [32] to implement our methodology.
Revised: SecML library [32] to implement our methodology.

Original: We show failures of gradient-based attacks onpreviously published defenses [16, 22, 29, 43, 28, 45, 38, 21, 47], by inspecting them with our IoAF, and most of them [16, 22, 29, 43, 28, 45] are characterized by wrong robustness evaluations.
Revised: We show failures of gradient-based attacks onpreviously published defenses [16, 22, 29, 43, 28, 45, 38, 21, 47], by inspecting them with our IoAF, and most of them [16, 22, 29, 43, 28, 45] are characterized by wrong robustness evaluations.

Original: We choose these defenses because it is known that their original evaluations are incorrect [2, 41]; our goal here is to show that IoAF would have identified and mitigated such errors.
Revised: We choose these defenses because it is known that their original evaluations are incorrect [2, 41]; our goal here is to show that IoAF would have identified and mitigated such errors.

Original: We include the evaluation of an adversarial detector [38] to highlight that IoAF is able to detect possible failures of this family of classifiers.
Revised: We include the evaluation of an adversarial detector [38] to highlight that IoAF is able to detect possible failures of this family of classifiers.

Original: We include the robustness evaluations of a Wide-ResNet [47] and an adversarially-trained model [21], to highlight that our indicators do not trigger when they inspect sound evaluations.
Revised: We include the robustness evaluations of a Wide-ResNet [47] and an adversarially-trained model [21], to highlight that our indicators do not trigger when they inspect sound evaluations.

Original: We defer details of the selected original evaluations to A.1.
Revised: We defer details of the selected original evaluations to A.1.

Original: Lastly, to demonstrate that automated attacks do not provide guarantees on performing correct evaluations (without human intervention), we apply the AutoPGD attack [13] with Cross-Entropy loss (APGD CE ) and the Difference of Logits Ratio
Revised: Lastly, to demonstrate that automated attacks do not provide guarantees on performing correct evaluations (without human intervention), we apply the AutoPGD attack [13] with Cross-Entropy loss (APGD CE ) and the Difference of Logits Ratio

Original: [Table]
Revised: [Table]

Original: (APGD DLR ).
Revised: (APGD DLR ).

Original: Robustness is computed with the robust accuracy (RA) metric, quantified as the ratio of samples classified correctly within a given perturbation bound ϵ .
Revised: Robustness is computed with the robust accuracy (RA) metric, quantified as the ratio of samples classified correctly within a given perturbation bound ϵ .

Original: For I 1 , I 2 , and I 6 we set N = 10 , and for I 4 we set k = 10 .
Revised: For I 1 , I 2 , and I 6 we set N = 10 , and for I 4 we set k = 10 .

Original: For I 2 , we set the number of sampled neighboors s = 100 , and the radius of the ℓ 2 ball r = 10 − 3 , to match the step size α of the evaluations.
Revised: For I 2 , we set the number of sampled neighboors s = 100 , and the radius of the ℓ 2 ball r = 10 − 3 , to match the step size α of the evaluations.

Original: The thresholds τ of I 2 and µ of I 4 are set to 10% and 1% respectively (details about their calibration in A.2).
Revised: The thresholds τ of I 2 and µ of I 4 are set to 10% and 1% respectively (details about their calibration in A.2).

Original: All the robustness evaluations are performed on 100 samples from the test dataset of the considered model, and for each attack we evaluate the robust accuracy with ϵ = 8 / 255 for CIFAR models, and ϵ = 0 .
Revised: All the robustness evaluations are performed on 100 samples from the test dataset of the considered model, and for each attack we evaluate the robust accuracy with ϵ = 8 / 255 for CIFAR models, and ϵ = 0 .

Original: 5 for the
Revised: 5 for the

Original: MNIST ones.
Revised: MNIST ones.

Original: The step size α is set to match the original evaluations (as detailed in A.1).
Revised: The step size α is set to match the original evaluations (as detailed in A.1).

Original: Identifying and Fixing Attack Failures.
Revised: Identifying and Fixing Attack Failures.

Original: We now delve into the description of the considered evaluations, which failures we detect and how we mitigate them, reporting the results in Table 1.
Revised: We now delve into the description of the considered evaluations, which failures we detect and how we mitigate them, reporting the results in Table 1.

Original: Correct Evaluations (ST, ADV-T).
Revised: Correct Evaluations (ST, ADV-T).

Original: We first evaluate the robustness analysis of the Wide-ResNet and the adversarially-trained model, by applying PGD with CE loss, with n = 100 (number of steps) and α = 0 .
Revised: We first evaluate the robustness analysis of the Wide-ResNet and the adversarially-trained model, by applying PGD with CE loss, with n = 100 (number of steps) and α = 0 .

Original: 03 (step size).
Revised: 03 (step size).

Original: Since no gradient obfuscation techniques have been used, the loss landscape indicators do not trigger.
Revised: Since no gradient obfuscation techniques have been used, the loss landscape indicators do not trigger.

Original: Also, since the attacks smoothly converge, these evaluations do not trigger any attack optimization indicator.
Revised: Also, since the attacks smoothly converge, these evaluations do not trigger any attack optimization indicator.

Original: Their robust accuracy is, respectively, 0% and 48%.
Revised: Their robust accuracy is, respectively, 0% and 48%.

Original: Defensive Distillation (DIST) .
Revised: Defensive Distillation (DIST) .

Original: Papernot et al.
Revised: Papernot et al.

Original: [29] use distillation to train a classifier to saturate the last layer of the network, making the computations of gradients numerically unstable and impossible to calculate, triggering the I 1 indicator, but also I 6 indicator as the optimizer can not explore the space.
Revised: [29] use distillation to train a classifier to saturate the last layer of the network, making the computations of gradients numerically unstable and impossible to calculate, triggering the I 1 indicator, but also I 6 indicator as the optimizer can not explore the space.

Original: To patch this evaluation, we apply M 1 to overcome the numerical instability caused by DIST, forcing the attack to leverage the logits of the model instead of computing the softmax.
Revised: To patch this evaluation, we apply M 1 to overcome the numerical instability caused by DIST, forcing the attack to leverage the logits of the model instead of computing the softmax.

Original: Such a fix reduces the robust accuracy from 95% to 1%.
Revised: Such a fix reduces the robust accuracy from 95% to 1%.

Original: In contrast, APGD DLR manages to decrease the robust accuracy to 0%, as it avoids the saturation issue by considering only the logits of the model, while APGD CE triggers the same issues of the original evaluation.
Revised: In contrast, APGD DLR manages to decrease the robust accuracy to 0%, as it avoids the saturation issue by considering only the logits of the model, while APGD CE triggers the same issues of the original evaluation.

Original: k-Winners Take All (k-WTA) .
Revised: k-Winners Take All (k-WTA) .

Original: Xiao et al.
Revised: Xiao et al.

Original: [43] develop a classifier with a very noisy loss landscape, that destroys the meaningfulness of the directions of computed gradients.
Revised: [43] develop a classifier with a very noisy loss landscape, that destroys the meaningfulness of the directions of computed gradients.

Original: The original evaluation triggers I 2 , since the loss landscape is characterized by frequent fluctuations, but also I 3 , as the applied
Revised: The original evaluation triggers I 2 , since the loss landscape is characterized by frequent fluctuations, but also I 3 , as the applied

Original: PGD [25] returns the last point of the attack discarding adversarial examples found within the path.
Revised: PGD [25] returns the last point of the attack discarding adversarial examples found within the path.

Original: Moreover, due to the noise, the attack is not always reaching convergence, as signaled by I 4 , also performing little exploration of the space, as flagged by I 6 .
Revised: Moreover, due to the noise, the attack is not always reaching convergence, as signaled by I 4 , also performing little exploration of the space, as flagged by I 6 .

Original: Hence, we apply M 2 , performing EoT on the PGD loss, sampling 2000 points from a Normal distribution N (0 , σ 2 I ) , with σ = 8 / 255 .
Revised: Hence, we apply M 2 , performing EoT on the PGD loss, sampling 2000 points from a Normal distribution N (0 , σ 2 I ) , with σ = 8 / 255 .

Original: We fix the implementation with M 3 , and increase the iterations with M 4 , thus reducing the robust accuracy from 67% to 9%.
Revised: We fix the implementation with M 3 , and increase the iterations with M 4 , thus reducing the robust accuracy from 67% to 9%.

Original: However, our evaluation still triggers I 

Revised: However, our evaluation still triggers I

Original: and I 6 , implying that for some points the result can be improved further by increasing the number of steps or the smoothing parameter γ .
Revised: and I 6 , implying that for some points the result can be improved further by increasing the number of steps or the smoothing parameter γ .

Original: Both
Revised: Both

Original: APGD CE and APGD DLR trigger
Revised: APGD CE and APGD DLR trigger

Original: I 2 , since they are not applying EoT, and they partially activate I 6 , similarly to the original evaluation.
Revised: I 2 , since they are not applying EoT, and they partially activate I 6 , similarly to the original evaluation.

Original: Interestingly, APGD DLR converges better than APGD CE , as the latter triggers
Revised: Interestingly, APGD DLR converges better than APGD CE , as the latter triggers

Original: I 4 .
Revised: I 4 .

Original: Both attacks decrease the robust accuracy to 35% and 28%, a worst estimate than the one computed with the patched attack that directly handles the presence of the gradient obfuscation.
Revised: Both attacks decrease the robust accuracy to 35% and 28%, a worst estimate than the one computed with the patched attack that directly handles the presence of the gradient obfuscation.

Original: Input Transformations (IT) .
Revised: Input Transformations (IT) .

Original: Guo et al.
Revised: Guo et al.

Original: [22] apply random affine input transformations to input images, producing a noisy loss landscape that varies at each prediction, thus its original robustness evaluation
Revised: [22] apply random affine input transformations to input images, producing a noisy loss landscape that varies at each prediction, thus its original robustness evaluation

Original: I 2 , I 3 , and I 4 indicators, for the same reasons of k-WTA.
Revised: I 2 , I 3 , and I 4 indicators, for the same reasons of k-WTA.

Original: Again, we apply M 2 , M 3 and M 4 , using EoT with γ = 200 and σ = 8 / 255 , decreasing the robust accuracy from 32% to 0%.
Revised: Again, we apply M 2 , M 3 and M 4 , using EoT with γ = 200 and σ = 8 / 255 , decreasing the robust accuracy from 32% to 0%.

Original: Still, for some points (even if adversarial)
Revised: Still, for some points (even if adversarial)

Original: the objective could still be improved, as I 4 is active for them.
Revised: the objective could still be improved, as I 4 is active for them.

Original: Both APGD CE and
Revised: Both APGD CE and

Original: APGD DLR are able to slightly decrease the robust accuracy to 12%, but they both trigger the I 2 and
Revised: APGD DLR are able to slightly decrease the robust accuracy to 12%, but they both trigger the I 2 and

Original: I 6 indicator since they are not addressing the high variability of the loss landscape.
Revised: I 6 indicator since they are not addressing the high variability of the loss landscape.

Original: Ensemble Diversity (EN-DV) .
Revised: Ensemble Diversity (EN-DV) .

Original: Pang et al.
Revised: Pang et al.

Original: [28] propose an ensemble model, evaluated with only 10 steps of PGD, thus triggers the I 5 indicator since the attack is not reaching convergence.
Revised: [28] propose an ensemble model, evaluated with only 10 steps of PGD, thus triggers the I 5 indicator since the attack is not reaching convergence.

Original: As a consequence, also the unbounded attack is failing, thus triggering I 6 .
Revised: As a consequence, also the unbounded attack is failing, thus triggering I 6 .

Original: We apply the M 4 mitigation and we set n = 100 , as already done by previous work [41].
Revised: We apply the M 4 mitigation and we set n = 100 , as already done by previous work [41].

Original: This fix changes the robust accuracy from 48% to 0%.
Revised: This fix changes the robust accuracy from 48% to 0%.

Original: Not surprisingly, both APGD CE and APGD DLR are able to bring the robust accuracy to 0%, since they both automatically modulate the step size.
Revised: Not surprisingly, both APGD CE and APGD DLR are able to bring the robust accuracy to 0%, since they both automatically modulate the step size.

Original: Turning a Weakness into a Strength (TWS) .
Revised: Turning a Weakness into a Strength (TWS) .

Original: Yu et al.
Revised: Yu et al.

Original: [45] propose an adversarial example defense, composed of different detectors.
Revised: [45] propose an adversarial example defense, composed of different detectors.

Original: For simplicity, we only consider one of these, and we repeat their original evaluation with PGD without the normalization operation on the computed gradients, and n = 50 .
Revised: For simplicity, we only consider one of these, and we repeat their original evaluation with PGD without the normalization operation on the computed gradients, and n = 50 .

Original: This attack does not converge, triggering I 4 , and if fails to navigate the loss landscape due to the very small gradients, triggering also I 6 .
Revised: This attack does not converge, triggering I 4 , and if fails to navigate the loss landscape due to the very small gradients, triggering also I 6 .

Original: We apply M 4 , by setting n = 100 , and we use the original implementation of PGD (with the normalization step), hence applying also M 6 , achieving a drop in the robust accuracy from 77% to 1%.
Revised: We apply M 4 , by setting n = 100 , and we use the original implementation of PGD (with the normalization step), hence applying also M 6 , achieving a drop in the robust accuracy from 77% to 1%.

Original: For only one point, our evaluation triggers I 4 , as better convergence could be achieved.
Revised: For only one point, our evaluation triggers I 4 , as better convergence could be achieved.

Original: Moving forward, APGD CE finds a robust accuracy of 3%, as it is modulating the step size along with using the normalization of the gradient.
Revised: Moving forward, APGD CE finds a robust accuracy of 3%, as it is modulating the step size along with using the normalization of the gradient.

Original: On the other hand,
Revised: On the other hand,

Original: APGD DLR triggers the I 6 indicator and the robust accuracy remains at 68%, as the attack is not able to avoid the rejection applied by the model.
Revised: APGD DLR triggers the I 6 indicator and the robust accuracy remains at 68%, as the attack is not able to avoid the rejection applied by the model.

Original: JPEG Compression (JPEG-C) .
Revised: JPEG Compression (JPEG-C) .

Original: Das et al.
Revised: Das et al.

Original: [16] pre-process all input samples using JPEG compression , before feeding the input to the undefended network.
Revised: [16] pre-process all input samples using JPEG compression , before feeding the input to the undefended network.

Original: We apply this defense on top of a model with a reverse sigmoid layer [23], as was done in [44].
Revised: We apply this defense on top of a model with a reverse sigmoid layer [23], as was done in [44].

Original: The authors firstly state that they can not directly attack the defense due to its non-differentiability (thus triggering I 1 ), then they apply the
Revised: The authors firstly state that they can not directly attack the defense due to its non-differentiability (thus triggering I 1 ), then they apply the

Original: DeepFool [27] attack against the undefended model.
Revised: DeepFool [27] attack against the undefended model.

Original: This attack has low transferability, as it is not maximizing the misclassification confidence [18].
Revised: This attack has low transferability, as it is not maximizing the misclassification confidence [18].

Original: The robustness evaluation triggers I 5 since the attack computed on the undefended model is not successful on the real defense.
Revised: The robustness evaluation triggers I 5 since the attack computed on the undefended model is not successful on the real defense.

Original: To patch this evaluation, we apply M 1 , approximating the back-propagation of the JPEG compression and the reverse sigmoid layers with BPDA [41], and we apply M 5 by using PGD to maximize the misclassification confidence of the model.
Revised: To patch this evaluation, we apply M 1 , approximating the back-propagation of the JPEG compression and the reverse sigmoid layers with BPDA [41], and we apply M 5 by using PGD to maximize the misclassification confidence of the model.

Original: This fix is sufficient to reduce its robust accuracy from
Revised: This fix is sufficient to reduce its robust accuracy from

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 74% to 0.01%.
Revised: 74% to 0.01%.

Original: Since APGD can not attack non-differentiable models
Revised: Since APGD can not attack non-differentiable models

Original: [Table]
Revised: [Table]

Original: we run the attack against the undefended target and we transfer the adversarial examples on the defense.
Revised: we run the attack against the undefended target and we transfer the adversarial examples on the defense.

Original: However, both of them trigger I 5 as transfer attacks are not effective
Revised: However, both of them trigger I 5 as transfer attacks are not effective

Original: [Table]
Revised: [Table]

Original: keeping the robust accuracy to 51% and 70%.
Revised: keeping the robust accuracy to 51% and 70%.

Original: Deep Neural Rejection (DNR) .
Revised: Deep Neural Rejection (DNR) .

Original: Sotgiu et al.
Revised: Sotgiu et al.

Original: [38] propose an adversarial detector encoding it as an additional class that captures the presence of attacks.
Revised: [38] propose an adversarial detector encoding it as an additional class that captures the presence of attacks.

Original: Similarly to TWS, it was evaluated with PGD with no normalization of the norms of gradients.
Revised: Similarly to TWS, it was evaluated with PGD with no normalization of the norms of gradients.

Original: The attack does not converge, as it triggers I 4 , but also sometimes gets trapped inside the rejection class, triggering I 6 .
Revised: The attack does not converge, as it triggers I 4 , but also sometimes gets trapped inside the rejection class, triggering I 6 .

Original: Hence, we apply both M 4 and
Revised: Hence, we apply both M 4 and

Original: M 6 , by increasing the number of iterations and considering an attack that avoids minimizing the score of the rejection class.
Revised: M 6 , by increasing the number of iterations and considering an attack that avoids minimizing the score of the rejection class.

Original: These fixes reduce the robust accuracy from 75% to 3%.
Revised: These fixes reduce the robust accuracy from 75% to 3%.

Original: Interestingly, only
Revised: Interestingly, only

Original: APGD CE is able to reduce the robust accuracy to 2%, also noting that it might be still beaten since it is triggering I 4 .
Revised: APGD CE is able to reduce the robust accuracy to 2%, also noting that it might be still beaten since it is triggering I 4 .

Original: On the contrary, APGD DLR is reducing the robust accuracy only to 20%, by also triggering I 6 .
Revised: On the contrary, APGD DLR is reducing the robust accuracy only to 20%, by also triggering I 6 .

Original: By manually inspecting the outputs of the model, we found that its scores tend to have all the same values, due to the SVM-RBF kernel that assigns low prediction outputs to samples that are outside the distributions of the training samples.
Revised: By manually inspecting the outputs of the model, we found that its scores tend to have all the same values, due to the SVM-RBF kernel that assigns low prediction outputs to samples that are outside the distributions of the training samples.

Original: This causes the DLR loss to saturate due to the denominator becoming extremely small, making this attack weak against the DNR defense.
Revised: This causes the DLR loss to saturate due to the denominator becoming extremely small, making this attack weak against the DNR defense.

Original: Evaluation of Robust Image Classification Models.
Revised: Evaluation of Robust Image Classification Models.

Original: We evaluate 6 defenses recently published on top-tier venues, available through RobustBench [12, 37, 42, 20, 35] or their official repository [39], by applying APGD CE and APGD DLR , using an ℓ ∞ perturbation bounded by ϵ = 8 / 255 .
Revised: We evaluate 6 defenses recently published on top-tier venues, available through RobustBench [12, 37, 42, 20, 35] or their official repository [39], by applying APGD CE and APGD DLR , using an ℓ ∞ perturbation bounded by ϵ = 8 / 255 .

Original: We report the results in Table 2.
Revised: We report the results in Table 2.

Original: Interestingly, we found that all these evaluations are unreliable, as they trigger the I 6 indicator.
Revised: Interestingly, we found that all these evaluations are unreliable, as they trigger the I 6 indicator.

Original: Hence, even without bounds on the perturbation model, the optimizer is unable to reach regions where adversarial examples are found.
Revised: Hence, even without bounds on the perturbation model, the optimizer is unable to reach regions where adversarial examples are found.

Original: This suggests that applying M 6 , i.e. making the attacks adaptive, can improve the trustworthiness of these robustness evaluations.
Revised: This suggests that applying M 6 , i.e. making the attacks adaptive, can improve the trustworthiness of these robustness evaluations.

Original:  Related Work
Revised:  Related Work

Original: Robustness Evaluations.
Revised: Robustness Evaluations.

Original: Prior work focused on re-evaluating already-published defenses [9, 2, 41, 11], showing that their adversarial robustness was significantly overestimated.
Revised: Prior work focused on re-evaluating already-published defenses [9, 2, 41, 11], showing that their adversarial robustness was significantly overestimated.

Original: The authors then suggested qualitative guidelines and best practices to avoid repeating the same evaluation mistakes in the future.
Revised: The authors then suggested qualitative guidelines and best practices to avoid repeating the same evaluation mistakes in the future.

Original: However, the application of such guidelines remained mostly neglected, due to the inherent difficulty of applying them in an automated and systematic manner.
Revised: However, the application of such guidelines remained mostly neglected, due to the inherent difficulty of applying them in an automated and systematic manner.

Original: To overcome this issue, in this work we have provided a systematic approach to debug adversarial robustness evaluations via the computation of the IoAF, which identify and characterize the main causes of failure of gradient-based attacks known to date.
Revised: To overcome this issue, in this work we have provided a systematic approach to debug adversarial robustness evaluations via the computation of the IoAF, which identify and characterize the main causes of failure of gradient-based attacks known to date.

Original: In addition, we have provided a semi-automated protocol that suggests how to apply
Revised: In addition, we have provided a semi-automated protocol that suggests how to apply

Original: Benchmarks.
Revised: Benchmarks.

Original: Instead of re-evaluating robustness of previously-proposed defenses with an ad-hoc approach, as the aforementioned papers do, some more systematic benchmarking approaches have been proposed.
Revised: Instead of re-evaluating robustness of previously-proposed defenses with an ad-hoc approach, as the aforementioned papers do, some more systematic benchmarking approaches have been proposed.

Original: Ling et al.
Revised: Ling et al.

Original: [24] have proposed DEEPSEC, a benchmark that tests several attacks against a wide range of defenses.
Revised: [24] have proposed DEEPSEC, a benchmark that tests several attacks against a wide range of defenses.

Original: However, this framework was shown to be flawed by several implementation issues and problems in the configuration of the attacks [7].
Revised: However, this framework was shown to be flawed by several implementation issues and problems in the configuration of the attacks [7].

Original: Croce et al.
Revised: Croce et al.

Original: [14] have proposed RobustBench, a benchmark that accepts state-of-the-art models as submissions, and tests them automatically with AutoAttack [13].
Revised: [14] have proposed RobustBench, a benchmark that accepts state-of-the-art models as submissions, and tests them automatically with AutoAttack [13].

Original: The same authors have also recently introduced some textual warnings when executing AutoAttack, which are displayed if the strategy encounters specific issues (e.g., randomness and zero gradients), taking inspiration from what we have done more systematically in this paper.
Revised: The same authors have also recently introduced some textual warnings when executing AutoAttack, which are displayed if the strategy encounters specific issues (e.g., randomness and zero gradients), taking inspiration from what we have done more systematically in this paper.

Original: Yao et al.
Revised: Yao et al.

Original: [44] have proposed an approach referred to as Adaptive
Revised: [44] have proposed an approach referred to as Adaptive

Original: AutoAttack , which automatically tests a variety of attacks, by applying different configurations of parameters and objective functions.
Revised: AutoAttack , which automatically tests a variety of attacks, by applying different configurations of parameters and objective functions.

Original: This approach has been shown to outperform AutoAttack by some margin, but at the expense of being much more computationally demanding.
Revised: This approach has been shown to outperform AutoAttack by some margin, but at the expense of being much more computationally demanding.

Original: While all these approaches are able to automatically evaluate adversarial robustness, they mostly blindly apply different attacks to show that the robust accuracy of some models can be decreased by a small fraction.
Revised: While all these approaches are able to automatically evaluate adversarial robustness, they mostly blindly apply different attacks to show that the robust accuracy of some models can be decreased by a small fraction.

Original: None of the proposed benchmarks has implemented any mechanisms to detect known failures of gradient-based attacks and use them to patch the evaluations, except for the aforementioned warnings present in AutoAttack, and a recent flag added to RobustBench.
Revised: None of the proposed benchmarks has implemented any mechanisms to detect known failures of gradient-based attacks and use them to patch the evaluations, except for the aforementioned warnings present in AutoAttack, and a recent flag added to RobustBench.

Original: This flag reports a potentially flawed evaluation when the black-box (square) attack finds lower robust accuracy values than the gradient-based attacks APGD CE , APGD DLR , and FAB.
Revised: This flag reports a potentially flawed evaluation when the black-box (square) attack finds lower robust accuracy values than the gradient-based attacks APGD CE , APGD DLR , and FAB.

Original: However, this is not guaranteed to correctly detect all flawed evaluations, as finding adversarial examples with black-box attacks is even more complicated and computationally demanding.
Revised: However, this is not guaranteed to correctly detect all flawed evaluations, as finding adversarial examples with black-box attacks is even more complicated and computationally demanding.

Original: Our approach finds the same issues without running any computationally-demanding black-box attack, but just inspecting the failures in the optimization process of gradient-based attacks.
Revised: Our approach finds the same issues without running any computationally-demanding black-box attack, but just inspecting the failures in the optimization process of gradient-based attacks.

Original: For these reasons, we firmly believe that our approach based on the IoAF will be extremely useful not only to debug in a more systematic and automated manner future robustness evaluations, but also to improve the current benchmarks.
Revised: For these reasons, we firmly believe that our approach based on the IoAF will be extremely useful not only to debug in a more systematic and automated manner future robustness evaluations, but also to improve the current benchmarks.

Original:  Conclusions and Future Work
Revised:  Conclusions and Future Work

Original: In this work, we have proposed the first automated testing approach that helps debug adversarial robustness evaluations, by characterizing and quantifying four known and two novel causes of failures of gradient-based attacks.
Revised: In this work, we have proposed the first automated testing approach that helps debug adversarial robustness evaluations, by characterizing and quantifying four known and two novel causes of failures of gradient-based attacks.

Original: To this end, we have developed six Indicators of Attack Failure (IoAF), and connected them with the corresponding (semi-automated) fixes.
Revised: To this end, we have developed six Indicators of Attack Failure (IoAF), and connected them with the corresponding (semi-automated) fixes.

Original: We have demonstrated the effectiveness of our methodology by analyzing more than 15 models on three different domains, showing that most of the reported evaluations were flawed, and how to fix them with the systematic protocol we have defined.
Revised: We have demonstrated the effectiveness of our methodology by analyzing more than 15 models on three different domains, showing that most of the reported evaluations were flawed, and how to fix them with the systematic protocol we have defined.

Original: An interesting direction for future work would be to define mitigations that can be applied in a fully-automated way, including how to make attacks adaptive.
Revised: An interesting direction for future work would be to define mitigations that can be applied in a fully-automated way, including how to make attacks adaptive.

Original: Still, our work provides a set of debugging tools and systematic procedures that aim to significantly reduce the number of manual interventions and the required skills to detect and fix flawed robustness evaluations.
Revised: Still, our work provides a set of debugging tools and systematic procedures that aim to significantly reduce the number of manual interventions and the required skills to detect and fix flawed robustness evaluations.

Original: To conclude, we firmly believe that integrating our work in current attack libraries and benchmarks will help avoid factual mistakes in adversarial robustness evaluations which have substantially hindered the development of effective defense mechanisms against adversarial examples thus far.
Revised: To conclude, we firmly believe that integrating our work in current attack libraries and benchmarks will help avoid factual mistakes in adversarial robustness evaluations which have substantially hindered the development of effective defense mechanisms against adversarial examples thus far.

Original: Acknowledgements
Revised: Acknowledgements

Original: This work has been partly supported by the PRIN 2017 project RexLearn (grant no. 2017TWNMH2), funded by the Italian Ministry of Education, University and Research; and by BMK, BMDW, and
Revised: This work has been partly supported by the PRIN 2017 project RexLearn (grant no. 2017TWNMH2), funded by the Italian Ministry of Education, University and Research; and by BMK, BMDW, and

Original: the
Revised: the

Original: Province of Upper Austria in the frame of the COMET programme managed by FFG in the COMET module S3AI; by the TESTABLE project, funded by the European Union’s Horizon 2020 research and innovation programme (grant no. 101019206); and by the ELSA project, funded by the EuropeanUnion’s Horizon Europe research and innovation programme (grant no. 101070617). A Appendix A.1 Additional Details on Models and Original Evaluations
Revised: Province of Upper Austria in the frame of the COMET programme managed by FFG in the COMET module S3AI; by the TESTABLE project, funded by the European Union’s Horizon 2020 research and innovation programme (grant no. 101019206); and by the ELSA project, funded by the European

Original: We collect here the implementation details of the models considered in our experimental analysis, along with their original evaluations used to determine their adversarial robustness.

Original: Distillation (DIST).

Original: Papernot et al.

Original: [29], develop a model to have zero gradients around the training points.

Original: However, this technique causes the loss function to saturate and produces zero gradients for the Cross-Entropy loss, but the defense was later proven ineffective when removing the Softmax layer [8].

Original: We re-implemented such defense, by training a distilled classifier on the MNIST dataset to mimic the original evaluation.

Original: Then, we apply the original evaluation, by using ℓ ∞ -PGD [25], with step size α = 0 .

Original: 01 , maximum perturbation ϵ = 0 .

Original: 3 for 50 iterations on 100 samples, resulting in a robust accuracy of 95%.

Original: k-Winners-Take-All (k-WTA).

Original: Xiao et al.

Original: [43] propose a defense that applies discontinuities on the loss function, by keeping only the top-k outputs from each layer.

Original: This addition causes the output of each layer to drastically change even for very close points inside the input space.

Original: Unfortunately, this method only causes gradient obfuscation that prevents the attack optimization to succeed, but it was proven to be ineffective by leveraging more sophisticated attacks [41].

Original: We leverage the implementation provided by Tramèr et al. .

Original: [41] trained on CIFAR10, and we replicate the original evaluation by attacking it with ℓ ∞ -PGD [25] with a step size of α = 0 .

Original: 003 , maximum perturbation ϵ = 0 .

Original: 031 and 50 iterations, scoring a robust accuracy of 67% on 100 samples.

Original: Input-transformation Defense (IT).

Original: Guo et al.

Original: [22] propose to preprocess input images with random affine transformations, and later feed them to the neural network.

Original: This preprocessing step is differentiable, thus training is not affected by these perturbations, and the network should be more resistant to adversarial manipulations.

Original: Similarly to k-WTA, this model is also affected by a highly-variable loss landscape, leading to gradient obfuscation [3], again proven to be ineffective as a defense [41].

Original: We replicate the original evaluation, by attacking this defense with PGD [25] with α = 0 .

Original: 003 , with 10 iterations and a maximum perturbation of 0 . 031 , scoring 32% of robust accuracy. Ensemble diversity (EN-DV). Pang et al.
Revised: Union’s Horizon Europe research and innovation programme (grant no. 101070617).

Original: [28] propose a defense that is composed of different neural networks, trained with a regularizer that encourages diversity.

Original: This defense was found to be evaluated with a number of iterations not sufficient for the attack to converge [41].

Original: We adopt the implementation provided by Tramèr et al.

Original: Then, following its original evaluation, we apply ℓ ∞ -PGD [25], with step size α = 0 .

Original: 003 , maximum perturbation ϵ = 0 .

Original: 031 for 10 iterations on 100 samples, resulting in a robust accuracy of 48%.

Original: Turning a Weakness into a Strength (TWS).

Original: Yu et al.

Original: [45] propose a defense that applies a mechanism for detecting the presence of adversarial examples on top of an undefended model, measuring how much the decision changes locally around a sample.

Original: The authors evaluated this model with a self-implemented version of PGD that does not apply the sign operator to the gradients and hinders the optimization performance as the magnitude of the gradients is too small for the attack to improve the objective [41].

Original: We apply this defense on a VGG model trained on CIFAR10, provided by PyTorch torchvision module 2 .

Original: Following the original evaluation, we attack this model with ℓ ∞ -PGD without the normalization of the gradients, with step size α = 0 .

Original: 01 , maximum perturbation ϵ = 0 .

Original: 031 for 50 iterations on 100 samples, and then we query the defended model with all the computed adversarial examples, scoring a robust accuracy of 77%.

Original: JPEG Compression (JPEG-C).

Original: Das et al.

Original: [16] propose a defense that applies the JPEG compression to input images, before feeding them to a convolutional neural network.

Original: We combine this defense with a defense from Lee et al.

Original: [23], originally proposed against black-box model-stealing attacks.

Original: This defense adds a reverse sigmoid layer after the model, causing the gradients to be misleading.

Original: This model was evaluated first directly, raising errors because of the non-differentiable transformation applied to the input, then re-evaluated with BPDA but with DeepFool, a minimum-distance attack, and finally found vulnerable to maximum-confidence attacks [3].

Original: We attack this model by replicating the original evaluation proposed by the author, and we first apply a standard PGD [25] attack that triggers the F 1 failures (as also mentioned by the author of the defense).

Original: Hence, we continue the original evaluation by applying the DeepFool attack against a model without the compression, with maximum perturbation ϵ = 0 .

Original: 031 , 100 iterations on 100 samples.

Original: These samples are transferred to the defended model, scoring a robust accuracy of 85%.

Original: Deep Neural Rejection (DNR).

Original: Sotgiu et al.

Original: [38] propose an adversarial detector built on top of a pretrained network, by adding a rejection class to the original output.

Original: The defense consists of several

Original: SVMs trained on top of the learned feature representation of selected internal layers of the original neural network, and they are combined to compute the rejection score.

Original: If this score is higher than a threshold, the rejection class is chosen as the output of the prediction.

Original: The defense was evaluated, similarly to TWS, with a PGD implementation not using the sign operator, leading the sample to reach regions where the gradients become unusable for improving the objective.

Original: This model was never found vulnerable by others, hence we are the first to show its evaluation issues.

Original: Following the original evaluation, we apply PGD without the normalization of gradients, with α = 0 . 3 , 50 iterations with maximum perturbation ϵ = 0 .

Original: 031 , and the scored robust accuracy is 75%.

Original: A.2 Thresholding Indicators I 2 and I

Original: We discuss here how to set the threshold values τ and µ , respectively used to compute I 2 and I 4 .

Original: For I 2 , we report the value of V ( x ) for each model, averaged over the considered N = 10 samples (along with its standard deviation) in Table 3a.

Original: As one may notice, models with unstable/noisy gradients exhibit values higher than 0 . 4 , whereas non-obfuscated models exhibit values that are very close to 0 (and the standard deviations are negligible).

Original: We set τ = 10% as the per-sample threshold in this case, but any other value between 0 . 1 and 0 .

Original: 3 would still detect the failure correctly on all samples.

Original: This threshold is thus not tight, but a conservative choice is preferred, given that missing a flawed evaluation would be far more problematic than detecting a non-flawed evaluation.

Original: For I 4 , we report the mean value (and standard deviation) of D ( x ) for each model and averaged over

Original: N = 10 samples, in Table 3b.

Original: Here, attacks that use few iterations (EN-DV), and attacks that use

Original: PGD without the step normalization (TWS, DNR) trigger the indicator.

Original: The evaluations that trigger already I 2 are not trusted as it is not worth checking convergence for models that present obfuscated gradients.

Original: Again, we used for I 4 a very conservative threshold to avoid missing the failure.

Original: [Table]

Original: A.3 Application on Windows Malware

Original: We replicate the evaluation of the MalConv neural network [34] for Windows malware detection done by Demetrio et al.

Original: We used the "Extend" attack, which manipulates the structure of Windows programs while preserving the intended functionality [17].

Original: We replicate the same setting described in the paper by executing the Extend attack on 100 samples (all initially flagged as malware by

Original: MalConv), and we report its performances and the values of our indicators in Table 4.

Original: As highlighted by the collected results, this evaluation can be improved by using BPDA instead of the sigmoid applied on the logits of the model ( M 1 ), and also by patching the implementation to return the best point in the path ( M 3 ).

Original: Since MalConv applies an embedding layer to impose distance metrics on byte values (that have no notion of norms and distance), we need to adapt I 2 to sample points inside such embedding space of the neural network.

Original: Hence, the sampled values must then be projected back to byte values by inverting the lookup function of the embedding layer [17].

Original: However, this change to

Original: I 2 is minimal, and all the other indicators are left untouched.

Original: [Table]

Original: A.4 Application on Android Malware Detection

Original: We replicate the evaluation of the Drebin malware detector by Arp et al.

Original: [1] from Demontis et al.

Original: The considered attack only injects new objects into Android applications to ensure that feature-space samples can be properly reconstructed in the problem space.

Original: We limit the budget of the attack to include only 5 and 25 new objects, and we report the collected results in Table 5.

Original: The attack is implemented using the PGD-LS [19] implementation from SecML [32].

Original: Note that, since the evaluated model is a linear SVM, the input gradient is constant and proportional to the feature weights of the model.

Original: Accordingly, the attack is successfully executed without failures.

Original: [Table]

Original: A.5 Application on Keyword Spotting

Original: To demonstrate the applicability of our indicators to different domains, we applied our procedure to the audio domain, using a reduced version of the Google Speech Commands Dataset 4 , including only the 4 keywords “ up ”, “ down ”, “ left ”, and “ right ”.

Original: We first convert the audio waveforms to linear spectrograms and then use these spectrograms to train a ConvNet ( AUDIO-ConvNet ) that achieves 99% accuracy on the test set.

Original: The spectrograms are then perturbed in the feature space using the

Original: PGD ℓ 2 attack (in the feature space) and transformed back to the input space using the Griffin-Lim transformation [31].

Original: The samples are transformed again and passed through the network to ensure the attack still works after the reconstruction of the perturbed waveform.

Original: We leverage the PGD ℓattack with n = 200 , α = 5 , and ϵ = 50 .

Original: Our indicators do not require any change to be applied to this domain.

Original: We report the values of our indicators and the robust accuracy in Table 6.

Original: [Table]

Original: A.6 Implementation Errors of Adversarial Machine Learning Libraries

Original: As discussed in Sect.

Original: 2, we found F 3 in the PGD implementations of the most widely-used adversarial robustness libraries, namely Cleverhans (PyTorch 6 , Tensorflow 7 , JAX 8 ), ART (NumPy 9 , PyTorch 10 , and Tensorflow 11 ), Foolbox (EagerPy 12 , which wraps the implementation of NumPy, PyTorch,

Original: Tensorflow, and JAX), Torchattacks (PyTorch 13 ).

Original: We detail further in Table 7 the specific versions and statistics.

Original: In particular, we report the GitHub stars ( ✰ ) to provide an estimate of the number of users potentially impacted by this issue.

Original: Let us also assume that there are a large number of defenses that have been evaluated with these implementations (or their previous versions, which most likely have the same problem).

Original: After this problem is fixed, all these defenses should be re-evaluated with the more powerful version of the attack, perhaps revealing faulty robustness performances.

Original: [Table]

Original: A.7 Pseudo-code of Indicators

Original: We report here the implementation of our proposed indicators.

Original: Since we write here a pseudo-code, we refer to the repository for the actual Python code.

Original: Algorithm 2: Pseudo-code for the Unavailable Gradients indicator I 1 ( x ) .

Original: 6 https://

Original: .com/

Original: -lab torch/ attacks/projected_gradient_descent.py 7 https://github

Original: Input : x , input sample; L ( · , y ; θ ) , the loss function of the attack with fixed label and model; r , noise radius; s , number of neighboring samples;

Original: τ threshold for triggering the indicator

Original: [-Output : The value of I 2 ( x ) 1 L (0) ← L ( x j , y ; θ ) 2 x (1) , ... x ( s ) ∼ U ( x j − r 
-]
Revised: [-Output : The value of I 2 ( x ) 1 L (0) ← L ( x j , y ; θ ) 2 x (1) , ... x ( s ) ∼ U ( x j − r 
-]

Original: [-I , x j + r 2 I ) 
-]
Revised: [-I , x j + r 2 I ) 
-]

Original: L (1) , ..., L ( s ) ← L ( x (1) , y ; θ ) , ..., L ( x ( s ) , y ; θ ) 4 V ( x ) ← min( 1 s (cid:80) mi =1 | ( L (0) − L ( i ) ) /L

Original: (0) | , 1) 5 return V

Original: > τ

Original: Input

Original: Output : The value of I 3 ( x ) return adversarial

Original: Input : x , the initial sample; δ 0 , ..., δ n , the attack path; L ( · , y ; θ ) , the loss function of the attack with fixed label and model; k , the length of the last part of the loss to retain; µ threshold for triggering the indicator

Original: [-Output : The value of I 4 ( x ) 1 L (0) , ..., L ( n ) ← L ( x + δ 0 , y ; θ ) , ..., L ( x + δ n , y ; θ ) 2 L (0) , ..., L ( n ) ← rescale ( L (0) , ..., L ( n ) , [0 , 1] ) 
-]
Revised: [-Output : The value of I 4 ( x ) 1 L (0) , ..., L ( n ) ← L ( x + δ 0 , y ; θ ) , ..., L ( x + δ n , y ; θ ) 2 L (0) , ..., L ( n ) ← rescale ( L (0) , ..., L ( n ) , [0 , 1] ) 
-]

Original: ˆ L (0) , ..., ˆ L ( n ) = cumulative-minimum ( L (0) , ..., L ( n ) ) 4 D ( x ) ← max(ˆ L ( n − k − 1) , ..., ˆ L ( n ) )

Original: − min(ˆ L ( n − k − 1) , ..., ˆ L ( n ) ) 5 return D ( x ) >