The Interplay between Distribution Parameters and the Accuracy-Robustness Tradeoff in Classiﬁcation
Abstract
Adversarial training tends to result in models that are less accurate on natural (unperturbed) examples compared to standard models.
This can be attributed to either an algorithmic shortcoming or a fundamental property of the training data distribution, which admits different solutions for optimal standard and adversarial classiﬁers.
In this work, we focus on the latter case under a binary Gaussian mixture classiﬁcation problem.
Unlike earlier work, we aim to derive the natural accuracy gap between the optimal Bayes and adversarial classiﬁers, and study the effect of different distributional parameters, namely separation between class centroids, class proportions, and the covariance matrix, on the derived gap.
We show that under certain conditions, the natural error of the optimal adversarial classiﬁer, as well as the gap, are locally minimized when classes are balanced, contradicting the performance of the
Bayes classiﬁer where perfect balance induces the worst accuracy.
Moreover, we show that with an (cid:96) ∞ bounded perturbation and an adversarial budget of (cid:15) , this gap is Θ( (cid:15) 2 ) for the worst-case parameters, which for suitably small (cid:15) indicates the theoretical possibility of achieving robust classiﬁers with near-perfect accuracy, which is rarely reﬂected in practical algorithms.
1. Introduction
It is well known that despite the huge success of deep learning in different tasks such as computer vision and natural language processing, deep models suffer from the existence of adversarial examples (Szegedy et al., 2014).
Adversarial training and its variants (Madry et al., 2018; Zhang et al., 2019; Wang et al., 2020; Carmon et al., 2019) have been one of the most successful defenses against adversarial attacks.
However, models obtained via these training methods demonstrate an undesirable drop in the natural accuracy.
There have been many attempts to justify this accuracy drop.
Tsipras et al.
(2019) showed that the objectives of standard and adversarial training can be fundamentally at odds with each other by proving the existence of plausible settings in which optimal standard and adversarial classiﬁers exploit inherently different features.
Built upon this idea, Zhang et al.
(2019) provided an algorithm to train models with a balance between robustness and accuracy.
Furthermore, Javanmard et al.
(2020) proved the existence of such an innate tradeoff even in the simple case of Gaussian linear regression.
Strikingly, Yang et al.
(2020) empirically revealed that different classes in benchmark image classiﬁcation problems are separated enough, which leads to the existence of classiﬁers that are both Bayes and adversarially optimal.
Thus in these cases, the tradeoff can not be attributed to distributional properties.
Another line of work points at the statistical hardness of adversarial training as the cause of the natural accuracy drop.
This hardness was ﬁrst proved by Schmidt et al.
(2018) in terms of a higher sample complexity for adversarial learning in a binary Gaussian mixture setting.
Bhattacharjee et al.
(2020) also showed a higher sample complexity for adversarial training in separated settings.
Rice et al.
(2020) demonstrated that virtually all adversarial training methods suffer from a huge generalization gap between training and test errors.
Empirically, it has been observed that adversarially trained models can enjoy better generalization through a larger training set (Carmon et al., 2019; Raghunathan et al., 2020).
Perhaps closest to this work are that of Dobriban et al.
(2020) and Javanmard & Soltanolkotabi (2020).
Javanmard & Soltanolkotabi (2020) study the output of the adversarial training algorithm on a similar binary Gaussian mixture problem in high-dimensional settings.
However, our work focuses on information-theoretically optimal classiﬁers, i.e. optimal classiﬁers when we are aware of the underlying data distribution and have access to its parameters.
Dobriban et al.
(2020) also study information-theoretically optimal standard and adversarial classiﬁers, and arrive at conditions that imply their difference and subsequently lead to robustness-accuracy tradeoff.
However, they do not quantify the exact drop in accuracy and how it can be affected by distributional parameters.
We indeed show that even though the two optimal classiﬁers tend to be different most of the time, with an (cid:96) ∞ adversarial budget of (cid:15) , their gap in accuracy is Θ( (cid:15) 2 ) at worst, which can be negligible under practical settings.
Besides, we prove the existence of non-trivial global maxima in the natural risk of the optimal adversarial classiﬁer, as well as its gap with the Bayes optimal classiﬁer, under class imbalance.
2. Preliminaries and Notation
The training set is deﬁned as
S n = { ( x (1) , y (1) ) , ..., ( x ( n ) , y ( n ) ) } , where ( x ( i ) , y ( i ) ) are i.i.d.
samples drawn from a distribution D , with x ∈ X ⊆ R d and y ∈ {− 1 , +1 } .
For a positive-deﬁnite matrix A , we deﬁne (cid:107) x (cid:107)
A = √ x T Ax .
Finally, we denote the CDF of the standard Gaussian distribution by Φ .
We deﬁne the adversarial risk and the empirical adversarial risk of a classiﬁer with the weight vector w with respect to the loss function (cid:96) over distribution D as
[Equation]
Furthermore, the optimal adversarial weights, when optimized on the entire distribution and the training set, can be deﬁned as
[Equation]
respectively, where W is the class of all classiﬁers (represented by weight vectors), which we search from.
We can also deﬁne the natural counterpart of the above deﬁnitions, denoted by nat superscript rather than adv , by setting (cid:15) = 0 .
One quantity that is of particular interest in this paper, is the gap between the natural risk of the optimal adversarial and the Bayes optimal classiﬁers.
Thus, we deﬁne
[Equation]
Although we consider one of the strongest threat models in the literature, the (cid:96) ∞ bounded adversary, our results are easily extensible to other attack norms.
3. Warm-up: Linear Loss Function
In this section, we brieﬂy discuss the case of linear classiﬁers with bounded weights when trained on a linear loss function, i.e. we are interested in the following setting
[Equation]
Considering the simple linear loss function will help us understand how a particular property of the distribution, namely µ , can inﬂuence the inherent existence of a tradeoff between natural and adversarial accuracy.
The analysis of this section is similar to Chen et al.
(2020), which also considered linear loss functions and analyzed the generalization gap between standard and adversarial classiﬁers.
Though they considered only the (cid:96) ∞ regularization case, which does not capture the entire truth, and they restrict their analysis to
Gaussian and Bernoulli distributions.
The next proposition, proved in Appendix A, will quantify the effect of µ .
Proposition 1.
Consider w nat and w adv that are obtained using W p and (cid:96) lin .
For any arbitrary distribution D , let µ = E x,y ∼D
[ yx ] .
Then, the following conditions are necessary and sufﬁcient to have w nat = w adv for different cases of p :
When p = 1 , we need (cid:107) µ (cid:107) ∞ ≥ (cid:15) or µ = 0 .
When p = ∞ , for each i ∈ [ d ] , we either need µ i =or | µ i | ≥ (cid:15) .
When 1 < p < ∞ , there must exist c ≥ (cid:15) such that for each i ∈
[ d ] , we either have µ i = 0 or | µ i | =
c .
While Proposition 1 generally stipulates a large enough distance between class centroids, in the case of 1 < p < ∞ the condition is much more exclusive, essentially requiring all non-zero elements of µ to be equal in magnitude.
As will be seen later, this condition reappears with the 0loss and Gaussian classiﬁcation, which hints at its possible importance beyond the settings considered here.
It is easy to provide a ﬁnite-sample analysis and observe that the same conditions of Proposition 1 control whether w natn and w advn converge given enough samples, or that their gap is bounded away from zero.
We will formally state and prove this result in
Appendix B. 4. Complete Analysis of the Tradeoff for Binary Gaussian Mixtures
In this section, we study optimal standard and adversarial classiﬁers, when samples are generated from a binary
Gaussian mixture model.
Note that in this section, we choose the 0-1 loss function, which for a linear classiﬁer with weight vector w and bias w 0 can be deﬁned as
[Equation]
Deﬁnition 2.
Let D be a distribution on R d × Y , where
Y = {− 1 , +1 } , such that
[Equation]
where p is a probability density function.
We call D a ( µ, Σ , π + ) -binary
Gaussian mixture.
This class of distributions has been used in many recent works on the theory of adversarial machine learning (Schmidt et al., 2018; Chen et al., 2020; Dobriban et al., 2020), although the form that we consider here is more general than prior works.
It is well known that the standard (Bayes) optimal classiﬁer for this distribution, i.e. the classiﬁer that minimizes the natural risk, is a linear classiﬁer with w = Σ − 1 µ and w
= ln
( π + /π − ).
Several prior work have established optimal adversarial classiﬁers for the binary Gaussian mixture (Bhagoji et al., 2019; Dan et al., 2020; Dobriban et al., 2020).
Here, we restate a formulation from Dan et al.
(2020), which we found useful for our analysis, with a slight extension to incorporate class imbalance.
The proof is deferred to Appendix C.
Proposition 3.
Let distribution D be a ( µ, Σ , π + ) -binary
Gaussian mixture according to Deﬁnition 2.
Then, the optimal adversarial classiﬁer, which is also linear, is given by
[Equation]
For the next sections, we implicitly assume (cid:15) < (cid:107) µ (cid:107) ∞ , omitting the trivial case of w adv =
according to Proposition 3.
4.1. Class Imbalance
In this section, we investigate the effect of class imbalance on the natural accuracy of the optimal adversarial classiﬁer and its gap with the accuracy of the Bayes classiﬁer.
Although Dobriban et al.
(2020) derived the optimal adversarial classiﬁer for binary and ternary Gaussian mixtures with respect to the (cid:96) 2 perturbation norm with imbalanced class priors, they did not analyze the impact of this imbalance on the natural accuracy of that classiﬁer.
It is well known that under constant Σ and µ , and varying π + , the Bayes optimal classiﬁer for the binary Gaussian mixture model of Deﬁnition 2 achieves better accuracies as π + moves away from 12 .
Perhaps surprisingly, we prove that under certain conditions, this is not the case for the natural risk of the optimal adversarial classiﬁer.
The behavior of this risk as a function of π + can have two distinct regimes. In the 0. + 0.
[Figure]
=1.
=1.
=2.
= imbalanced.
ﬁrst one, it behaves similarly to the natural risk of the Bayes classiﬁer, increasing monotonically before and decreasing monotonically after 12 .
We call this regime standard .
In the second regime however, this risk will have 3 extreme values, a local minimum at 12 and two maxima located symmetrically around 12 .
We call this regime surprising .
Moreover, this pattern can be seen in the gap between the accuracies of the Bayes and the optimal adversarial classiﬁer as well.
In other words, unlike standard classiﬁcation, when we are in the surprising regime, having perfectly balanced classes makes adversarial classiﬁcation more naturally accurate and closes its gap with natural classiﬁcation, compared to a slight class imbalance.
This behavior is illustrated in Figure 1.
The next theorem, proved in Appendix D, precisely characterizes the condition that separates the two regimes for both the natural risk of the adversarial classiﬁer and the gap.
Theorem 4.
Let distribution D be a ( µ, Σ , π + ) -binary
Gaussian mixture according to Deﬁnition 2, and let c := 2 (cid:104) µ, Σ − 1 ( µ − (cid:15)z ∗ ) (cid:105) and d := 2 (cid:107) µ − (cid:15)z ∗ (cid:107) Σ − 1 where z ∗ is the solution of Equation (10) .
Consider R nat D ,(cid:96) 0-1 ( w adv ) as a function of π + ∈ (0 , 1) under ﬁxed Σ and µ .
This function is surprising if and only if
[Equation]
and standard otherwise.
Furthermore, consider G D ,(cid:96) 0-1 , i.e. the gap, as a function of π + ∈ (0 , 1) under ﬁxed Σ and µ .
This function is surprising if and only if
[Equation]
and standard otherwise. Remark. As expected, the satisfaction of (13) also implies the satisfaction of (14). Although not proved formally, we believe that the transition to the surprising regime generally occurs with a non-trivially (less than (cid:107) µ (cid:107) ∞ ) large enough (cid:15) .
For instance, let Σ be diagonal and let all elements of µ have the same magnitude | µ | . Then, (cid:15) > | µ | 2 would sufﬁce to satisfy (13) and subsequently (14). Note that in this case, the latter is satisﬁed for all (cid:15) .
4.2. Gap Lower and Upper Bounds
In this section, we focus on the case where π + = π − .
We begin our discussion by specifying the conditions under which the adversarial and Bayes optimal classiﬁers are equivalent, i.e. w nat and w adv are parallel, in the following proposition which is proved in
Appendix E. This is indeed an extension to the results of Dobriban et al.
(2020), where they only consider isotropic covariance matrices.
Theorem 5.
Let distribution D be a ( µ, Σ , 12 ) -binary
Gaussian mixture according to Deﬁnition 2.
Then the optimal adversarial classiﬁer is also a Bayes optimal classiﬁer if and only if a constant c ≥ (cid:15) exists such that
[Equation]
Remark.
For a diagonal Σ , the condition in Theorem 5 will be the same as the condition for 1 < p < ∞ in Proposition 1, which demands a strong symmetry between all non-zero elements of µ that is unimaginable to hold in practice.
As a consequence of Theorem 5, it is reasonable to assume that in most natural settings, there is indeed a tradeoff between optimizing standard accuracy and robustness.
However, the natural question to ask is, when the conditions stated above are not met, how much accuracy drop should we expect?
The next theorem, proved in Appendix F, addresses this question.
Theorem 6.
Let distribution D be a ( µ, Σ , 12 ) -binary
Gaussian mixture according to Deﬁnition 2, and let (cid:15)
[Equation]
where
[Equation]
Then we have
[Equation]
Moreover
[Equation]
when Σ is diagonal and min i ∈ [ d ] | µ i | ≥ (cid:15) [Equation] we have
[Equation]
Remark.
One can easily verify C Σ ,µ ≥ 0 using the H¨older’s inequality.
In fact, when C Σ ,µ = 0 , there will be no tradeoff.
Equation (90) presents a tighter upper bound without the requirement of (cid:15) ≤ [Equation] µ
[Equation]
,µ , which is omitted here for simpler presentation.
There are two interesting observations to be made surrounding Theorem 6:
• As the optimal standard accuracy tends to one,
i
e. (cid:107) µ [Equation] 1 → ∞ , the gap vanishes exponentially quickly.
• In high dimensional cases (as d grows), an interesting setting occurs when
[Equation]
O [Equation] 1) , e g. when Σ is isotropic and µ has equal constant elements.
In this case, the gap remains negligible with the growth of d , especially if (cid:15) (cid:28)
[Equation]
In order to better understand the information-theoretical limit provided by Theorem 6, we provide further numerical experiments in Appendix G.
5. Conclusion
In this work, we studied the effect of the parameters of the underlying generative distribution on the natural accuracy gap between the Bayes optimal and the optimal adversarial classiﬁers.
In particular, when the underlying distribution is a binary Gaussian mixture, we proved an interesting characteristic of adversarial classiﬁers, that their natural error and its gap with the Bayes classifer are locally minimized when classes are balanced, contrary to the Bayes error rate that is maximized in this case.
Furthermore, we speciﬁed the exact conditions on µ and Σ when there is no tradeoff.
Although these conditions are highly unlikely to be met, we showed that the gap is roughly Θ( (cid:15) 2 ) in the worst-case scenario, which can be negligible in many applications with small enough (cid:15) .
These results can help as a guideline to ensure that newly developed robust learning algorithms are close to information-theoretical optimality.
While this work addressed the question of informationtheoretically optimal achievable gap, it is interesting to study the limits of ﬁnite-sample algorithms next, which can provide valuable insights for practitioners in designing robust deep learning algorithms.
A. Proof of Proposition
We begin by proving the following lemma, which will be useful in proving Theorem 8 as well
[Equation]
Lemma 7.
Let w adv , w nat and µ be deﬁned as in Proposition 1.
Then, for any arbitrary distribution D , we have
[Equation]
where β (cid:15)
[Equation]
η (cid:15),p , γ (cid:15) :
R d → R d are deﬁned as
[Equation]
Moreover, w nat can be obtained by setting (cid:15)
= 0 .
First [Equation] we simplify the adversarial loss as follows
[Equation]
where we used the fact that (cid:96) 1 and (cid:96) ∞ norms are dual norms for the last equality.
Thus, our problem is simpliﬁed to
[Equation]
We start with the case of p = 1 .
Using the Holder’s inequality we have
[Equation]
If (cid:15) > (cid:107) µ (cid:107) ∞ , then for any w (cid:54) = 0 , R adv D ,(cid:96) will be strictly positive, thus the optimal solution would be w adv = 0 .
Otherwise, with setting w adv =
Wβ (cid:15) ( µ ) where β (cid:15) is deﬁned as in Equation (20), R adv D ,(cid:96) will achieve the minimum of its lower bound and is therefore minimized.
Next, we move to the case where 1 < p < ∞ .
In this case, we will form the Lagrange function
[Equation]
Note that for minimizing the Lagrange function, we can decouple it and minimize the term corresponding to each w i individually.
Thus, we seek to minimize
[Equation]
Using the above equation’s subgradients for λ > 0 , we arrive at the following solution
[Equation]
Now, by choosing the appropriate λ such that (cid:107) w (cid:107) p
=
W we will arrive at the desired result.
Finally, we present the proof for the p = ∞ case.
In this case, Equation 4 can be written as
[Equation]
Therefore, we can optimize each w j separately.
Note that when | µ j | < (cid:15) the term corresponding to w j is non-negative and will be minimized by choosing w j = 0 .
When | µ j | = (cid:15) choosing any w j such that w j µ j ≥ 0 will be a minimizer, and we choose w j = W sign( µ j ) .
Otherwise, choosing w j = W sign( µ j ) will minimize the corresponding term, which will ﬁnalize the proof.
From here, one can easily verify the truth of Proposition 1 using Equations (20), (21), and (22), by comparing the cases of (cid:15) > 0 and (cid:15)
=
(which gives the standard classiﬁer, i.e. w nat ).
B. Finite-Sample Analysis of Training with the Linear Loss Function
First, it is worthwhile to note that unlike the rest of the paper, the weights obtained in this section are stochastic as they are learned using the training set S n , without having knowledge about the distribution.
As a brief description, the theorem presented below states that when we perform training over S n using W p and (cid:96) lin as in (7) and (8), then:
• When p = 1 , if (cid:107) µ (cid:107) ∞ > (cid:15) then we have w advn =
w natn w.h.p.
, and if 0 < (cid:107) µ (cid:107) ∞ < (cid:15) we have G ( n ) D ,(cid:96) lin
= W (cid:107) µ (cid:107) ∞ w .h.p.
• When p = ∞ , if min i ∈
[ d ] | µ i | ≥ (cid:15) , then w advn =
w natn w.h.p.
, and otherwise, i.e. if there exists an index j such that 0 < | µ j | < (cid:15) , then G ( n ) D ,(cid:96) lin ≥ W | µ j | w.h.p.
Note that these are the same conditions in order to have w nat = w adv in Proposition 1, indicating that the conditions not only characterize when the information-theoretically optimal classiﬁers are identical, but also when the result of the two training algorithms are probably the same.
Theorem 8.
Consider w natn and w advn obtained via training on S n using W p and (cid:96) lin as in Equations (7) and (8) .
Let X be such that sup x ∈X (cid:107) x (cid:107) ∞ ≤ A < ∞ , and let µ = E ( x,y )
∼D
[ yx ] .
Deﬁne r := (cid:107) µ (cid:107) ∞ − max { k ∈ [ d ]: | µ k | < (cid:107) µ (cid:107) ∞ } | µ k | and let r = 0 when all elements of µ are equal.
Deﬁne τ := min j ∈ [ d ] | µ j | − (cid:15) .
Then 1.
When p = 1 , if (cid:107) µ (cid:107) ∞ ≥ (cid:15) then we have
[Equation]
Furthermore, if 0 < (cid:107) µ (cid:107) ∞ < (cid:15) , then we have
P [ R nat D ,(cid:96) ( w advn ) − R nat D ,(cid:96) ( w natn ) = W (cid:107) µ (cid:107) ∞
[Equation]
When p = ∞ , if min i ∈ [ d ] | µ i | ≥ (cid:15) we have
[Equation]
Conversely, if there exists an index j such that 0 < | µ j | < (cid:15) , then we have
[Equation]
Remark.
w advn =
w natn can be ambiguous when the optimal classiﬁers are not unique.
By this notation, we mean that there is at least some classiﬁer that is both naturally and adversarially optimal.
On the other hand, when talking about
R nat D ,(cid:96) ( w advn ) − R nat D ,(cid:96) ( w natn ) , the statement holds for all possible solutions w natn and w advn .
Proof.
Let ˆ µ := 1 n (cid:80) ni
=
y ( i ) x ( i ) and note that one can simply substitute µ with ˆ µ in Lemma 7 to obtain closed-form expressions for w advn and w natn .
First, consider the case where p = 1 and (cid:107) µ (cid:107) ∞ > (cid:15) .
Choose a ﬁxed i such that | µ i | = (cid:107) µ (cid:107) ∞ .
In order to have w advn =
w natn , it sufﬁces to have (cid:107) ˆ µ (cid:107) ∞ ≥ (cid:15) , and consequently it sufﬁces that | ˆ µ i | ≥ (cid:15) .
Using the Hoeffding inequality, we have
[Equation]
proving Equation (23).
Next, in order to prove (24), we show that whenever 0 < (cid:107) µ (cid:107) ∞ < (cid:15) , we have R nat D ,(cid:96) ( w advn )
=
and R nat D ,(cid:96) ( w natn ) =
− W (cid:107) µ (cid:107) ∞ w.h.p.
The ﬁrst condition will be met whenever (cid:107) ˆ µ (cid:107) ∞ < (cid:15) , which implies w advn = 0 .
For the probability of this event, we have
[Equation]
The second condition can be achieved whenever for a ﬁxed i with | µ i | = (cid:107) µ (cid:107) ∞ and for every k such that | µ i | > | µ k | we have | ˆ µ i | >
| ˆ µ k | .
Using the triangle inequality, it sufﬁces to have | ˆ µ i − µ i | < | µ i |−| µ k |and | ˆ µ k − µ k | < | µ i |−| µ k |.
P
[Equation]
Combining (27) and (28) completes the proof of (24).
Next, we consider the p = ∞ and min i ∈
[ d ] | µ i | ≥ (cid:107) µ (cid:107) ∞ case.
For w advn and w natn to be different we need | ˆ µ j | < (cid:15) to hold for at least one j ∈ [ d ] .
Thus, using the union bound we have
[Equation]
which proves (25).
Finally, for the p = ∞ and 0 < | ˆ µ j | < (cid:15) case, we need to show that 0 < | ˆ µ j | < (cid:15) w.h.p.
As a result, the corresponding element in w natn will be nonzero, while the corresponding element in w advn will be zero, which will cause the difference in natural risks.
For 0 < | ˆ µ j | < (cid:15) it is sufﬁcient to have | ˆ µ j − µ j | < min( (cid:15) − | µ j | , | µ j ) .
Thus
[Equation]
With this, the proof of (26) is completed and all cases have been covered.
C. Proof of Proposition
Several proofs of this statement already exist in the literature (Dan et al., 2020; Bhagoji et al., 2019; Dobriban et al., 2020). Here, we adopt a proof due to Dan et al. (2020) with a slight modiﬁcation to incorporate class imbalance.
We begin by ﬁnding the optimal linear adversarial classiﬁer, and then show that no other (non-linear) classiﬁer can achieve better adversarial accuracy.
First, we derive a closed form expression for the adversarial risk of a linear classiﬁer with weight vector w and bias w 0 .
For a sample ( x, y ) , the adversary will introduce a noise δ such that y (cid:104) w, x + δ (cid:105) =
min (cid:107) v (cid:107) ∞ ≤ (cid:15) y (cid:104) w, x + v (cid:105) =
y (cid:104) w, x (cid:105) − (cid:15) (cid:107) w (cid:107) 1 .
Therefore
R adv D ,
[Equation]
w
[Equation]
w
[Equation]
=
[Equation]
[Equation]
[Equation] [Equation] (cid:107) w
[Equation]
Since multiplying w and w 0 by any positive scalar does not change the risk, there exist optimal solutions for which (cid:107) w [Equation] Σ = 1 .
Applying this restriction, we end up with
[Equation]
It is easy to see that optimizing w is independent from w 0 , and we should minimize −(cid:104) w, µ (cid:105) + (cid:15) (cid:107) w (cid:107) 1 subject to (cid:107) w (cid:107) Σ = 1 .
We can convexify the problem by relaxing the condition to (cid:107) w (cid:107) Σ ≤ 1 and verifying that the solution meets (cid:107) w (cid:107) Σ = 1 at the end.
Now, we can restate the problem of min (cid:107) w (cid:107) Σ ≤ 1 [-, i.e. if w i (cid:54) = 
 then z i = sign(-]−(cid:104) w, µ (cid:105) + (cid:15) (cid:107) w (cid:107) 1 as min (cid:107) w (cid:107) [-
-]Σ ≤ 1 max (cid:107) z (cid:107) ∞ ≤ 1 −(cid:104) w, µ − (cid:15)z (cid:105) .
Since the objective is concave-convex and the constraints are convex sets, we can change the min-max order using von Neumann’s
Minimax theorem, which results in max (cid:107) z (cid:107) ∞ ≤ 1 min (cid:107) w (cid:107) Σ ≤ 1 −(cid:104) w, µ − (cid:15)z (cid:105) .
It is straightforward to see that the solution to the inner maximization is given by w adv =
Σ 1 ( µ − (cid:15)z ) (cid:107) µ − (cid:15) (cid:107) Σ − 1 , and we are left with the outer maximization problem which is max [Equation] ≤ [Equation]
µ − [Equation]
(cid:107) Σ − 1 . We further show that we must have z ∗ i = sign( w advi ) whenever w advi (cid:54) = 0 , which will be used later in the proof.
Returning to the problem of z ∗ = arg min (cid:107) z [Equation] ∞ ≤ 1 (cid:107) µ − (cid:15)z [Equation]
2Σ − 1 and using the method of Lagrange multipliers with dual variables u 1 , ..., u d , we have [Equation]
[Equation]
At the optimal point, the subgradient satisﬁes ∂ z L ( z ∗ , u ∗ ) = 0 .
Therefore,
[Equation]
where ∂ z i | z ∗ i | = sign( z ∗ i ) whenever z ∗ i (cid:54) = 0 and any real number in [ − 1 , 1] otherwise.
One can see that when w advi (cid:54) = 0 , we must have u ∗ i > 0 and z ∗ i = sign( w advi ) due to the complementary slackness condition.
It remains to ﬁnd the optimal value for w 0 .
Let K := (cid:104) w adv , µ (cid:105) − (cid:15) (cid:107) w adv (cid:107) 1 .
Then
[Equation] [Equation]
Setting ∂R adv {+D ,(cid:96) ( w adv ,w 0 ) ∂w= 0 leads to w ∗ 
 = ln( π + /π − ) 2 K , and one can observe that ∂R adv D ,(cid:96) ( w adv+}
[Equation]
w[-adv ,w 0 ) ∂w= 0 leads to w ∗ 
 = ln( π + /π − ) 2 K , and one can observe that ∂R adv-]
[Equation]
∂w>
if and only if w 0 > w ∗ 0 , thus w ∗ 0 is achieving the minimum value of R adv D ,(cid:96) ( w adv , w 0 ) .
Furthermore, by plugging w adv in K , we have
K = [Equation] µ [Equation]
Σ
[Equation]
Since multiplying both w adv and w ∗ 0 by a positive scalar will not change the classiﬁer, we multiply both by K , thus arriving at w adv =
Σ − 1 ( µ − (cid:15)z ∗ ) and w adv
[Equation]
, as the theorem claims.
It remains to show that this linear classiﬁer is indeed optimal among all (potentially non-linear) classiﬁers.
We ﬁrst present a slightly extended version of Lemma 6.4 from Dan et al.
(2020), and apply it in a similar fashion to their work.
Lemma 9.
Let distribution D and D (cid:48) be a ( µ, Σ , π + ) and a ( µ − (cid:15)z ∗ , Σ , π + ) -binary
Gaussian mixture respectively.
For any classiﬁer f : R d → {− 1 , +1 } we have
[Equation]
Proof.
First, note that we have
[Equation]
Moreover, by deﬁnition we have (cid:107) (cid:15)z ∗ (cid:107) ∞ ≤ (cid:15) , which implies that
E x ∼N ( µ, Σ)
[Equation]
and a similar lower bound for the negative case.
Thus we have
[Equation]
which completes the proof of the ﬁrst inequality.
For the second inequality, note that the Bayes optimal classiﬁer for D (cid:48) is indeed a linear classiﬁer with w adv =
Σ − 1 ( µ − (cid:15)z ∗ ) and w adv [Equation] , and any other classiﬁer has a higher natural risk compared to the stated Bayes risk.
Now, all we need to show is that the adversarial risk of ( w adv , w adv 0 ) is equal to the lower bound stated in Lemma 9.
For this, note that whenever w advi (cid:54) =
we have z ∗ i
= sign( w advi ) .
As a result, we have (cid:104) z ∗ , w adv (cid:105) =
(cid:107) w adv (cid:107) 1 and thus (cid:104) w adv , µ (cid:105) − (cid:15) (cid:107) w adv (cid:107)
=
(cid:104)
w adv , µ − (cid:15)z ∗ (cid:105) .
Plugging this result into Equation (30) implies that the lower bound in Lemma 9 is equal to R adv D ,(cid:96) 0-1 ( w adv , w adv 0 ) which demonstrates that for any classiﬁer f we have R adv D ,(cid:96) 0-1 ( f ) ≥ R adv D ,(cid:96) 0-1 ( w adv , w adv 0 ) .
With that, the proof of Proposition 3 is complete.
D. Proof of Theorem
Both conditions in Theorem 4 have a similar proof.
First, for the sake of simplicity, we deﬁne
[Equation]
[Equation]
[Equation]
and π := π + .
The proof is based on analyzing the second derivative of our desired function with respect to π .
Deﬁne the following function which simpliﬁes the proof
[Equation]
From Proposition 3 and Equation (30) by setting (cid:15) = 0 , it is obvious that
[Equation]
[Equation]
The next lemmas are essential for proving both theorems.
Lemma 10.
Consider c and d deﬁned by Equations (39) and (40) .
We have
[Equation]
Proof.
Replace c,d by their values.
we have
[Equation]
Recall from Appendix C that we have z ∗ i =
sign(Σ − 1 ( µ − (cid:15)z ∗ )) i whenever (Σ − 1 ( µ − (cid:15)z ∗ ))
i (cid:54) = 0 , hence the last inequality holds.
Note that if (cid:15) = 0 , we will have cd 2 = 12 .
Lemma 11.
Consider the function F deﬁned by Equation (41) .
The second derivative of this function is
[Equation]
where φ ( x ) = ddx Φ( x ) = 1 √ 2 π e − x 2 2 .
Proof.
First, we have
[Equation]
Now, differentiating F (cid:48) ( π ) results in
[Equation]
Lemma 12.
The second derivative (with respect to π while other parameters are constant) of the natural risk of a classiﬁer with parameters ( c, d ) deﬁned according to Equations (39) and (40) is given by
[Equation]
where C is a positive constant
[Equation]
Proof.
Using Lemma [Equation] we have
[Equation]
Lemma 13.
Consider the following function
[Equation]
where A and β are positive real numbers and α ≥ 0 .
There exists x 0 ∈ R + ∪ { 0 } such that the domain of S(x) is ( −∞ , − x 0 ) ∪ ( x 0 , ∞ ) .
Furthermore, there exists x 1 ∈ R + such that S ( x ) is monotonically decreasing on ( x 0 , x 1 ) and monotonically increasing on ( x 1 , ∞ ) .
Proof.
Since S ( x ) is an even function, we only consider the non-negative part of its domain ( x ≥ 0 ) as everything is symmetric about zero.
We have two distinct cases for A .
A ≤ 0 .
5 implies that the domain of S is R .
On the other hand,
A > 0 .
5 imposes the following condition on x ≥ 0 to ensure that the logarithmic term remains valid
[Equation]
The RHS is strictly decreasing on (0 , ∞ ) and tends to 0 as x → ∞ .
However, the LHS, which is tanh( Ax ) , is strictly increasing on (0 , ∞ ) .
Hence, there exists x 0 such that the above inequality holds for all x ∈ ( x 0 , ∞ ) .
Also, The domain of
S is ( −∞ , − x 0 ) ∪ ( x 0 , ∞ ) .
The proof of ﬁrst statement is complete.
We break down the proof of the second statement to the following three cases.
If α = 0 , we have
[Equation]
[Equation]
The denominator is always positive on the domain of S ( x ) .
When A ≤ 0 .
5 , it is clear that the numerator is also positive.
If A ≥ 0 .
5 , we have
[Equation]
Again, the ﬁrst inequality is correct on the domain of S , and the second and third inequalities follow by multiplying by
A and adding a positive value respectively.
Thus, S (cid:48)
( x ) is negative on ( x 0 , ∞ ) , and x 1 = ∞ .
Therefore, S is strictly decreasing on ( x 0 , ∞ ) , and doesn’t have an increasing stage.
The proof for this case is complete.
α > 0 and A ≥ 0 .
5 : Another form of S(x) is
[Equation]
[Equation]
We will use this alternative form of S in the following arguments.
The ﬁrst derivative of S is
[Equation]
and after simpliﬁcation, we have
[Equation]
Furthermore, (50) implies e Ax − e −
Ax e Ax + e − Ax ≥ β ( A − 0 . 5) x ⇒ x ≥ β ( A − 0 . 5) .
Consequently, ( β ( A − 0 . 5)) 2 ≤ x 2 holds and we can ﬁnally conclude that S (cid:48)(cid:48)
( x ) > 0 .
Therefore, S ( x ) is strictly convex on ( x 0 , ∞ ) , which guarantees the existence of x 1 (potentially inﬁnite) as mentioned in the lemma.
Assume that A ≤ 0 .
5 and α > 0 .
We want to show that S (cid:48)
( x ) has at most one root in x > 0 , and S (cid:48) ( x ) > 0 as x → ∞ .
Recall that when A ≤ 0 .
5 , the domain of S is R .
Note that S (cid:48) (0) is 0 .
Let
[Equation]
[Equation]
Then we have S (cid:48)
( x ) =
L ( x ) − T ( x ) .
Assume for now that there exists a such that T ( x ) is strictly concave on (0 , a ) and strictly decreasing on ( a, ∞ ) .
Then, since L is a linear function with a positive slope, L ( x ) =
T ( x ) can have at most one solution on (0 , a ] .
If such solution exists, then T ( a ) ≤ L ( a ) and hence, no more solutions exist beyond a .
Otherwise, we will either have T ( a ) > L ( a ) , and exactly one solution will exist beyond a , or have T ( a ) <
L ( a ) and no solution exists on x > 0 .
In conclusion, there can be at most one solution on (0 , ∞ ) , and clearly, T ( x ) <
L ( x ) as x → ∞ .
All that is left to prove is the existence of a .
We will show that T (cid:48)
( x ) is strictly decreasing until a , and is negative beyond a .
Similar to the derivative of S ( x ) , we have
[Equation]
Note that
[Equation]
While the LHS in the above inequality is constant, the RHS is a strictly increasing function of x , meaning that there is exactly one a such that T (cid:48)
( x ) < 0 for all x > a , meaning that T is strictly decreasing beyond a .
Take this a .
When x < a , the nominator is positive and strictly decreasing in x , while the denominator is strictly increasing in x , since
A ≤ 0 .
As a result, for x < a , T (cid:48) ( x ) is strictly decreasing in x , meaning that T is strictly concave in this area.
We have proved the existence of the promised a , and the proof of this case, as well as the entire lemma, is complete.
Lemma 14.
There exists π ∗ such that R c,d ( π ) is monotonically decreasing on ( π ∗ , 1) , or equivalently, R (cid:48)
c,d ( π ) is negative on this interval.
Proof.
we have
[Equation]
[Equation]
By deﬁnition of F and Equation (45), we have
[Equation]
Hence,
[Equation]
And it shows that there exists a π ∗ such that R (cid:48)
c,d ( π ) is negative on ( π ∗ , 1) .
Lemma 15.
Let G ( π ) , the natural accuracy gap between the optimal adversarial and Bayes optimal classiﬁers, be deﬁned as
[Equation]
where w adv and w nat are the weight of the optimal adversarial and Bayes classiﬁers, respectively.
There exists a π ∗ such that G ( π ) is monotonically decreasing on ( π ∗ , 1) .
Proof.
We show that there exists a π ∗ such that G (cid:48)
( π ) is negative on ( π ∗ , 1) .
Let c (cid:48) and d (cid:48) be deﬁned using Equations (39) and (40) when (cid:15)
= 0 .
Then, we have
[Equation]
By deﬁnitions of c (cid:48) and d (cid:48) and Equation (45) [Equation] we have
[Equation]
Hence
[Equation]
and
[Equation]
Using deﬁnitions of d and d (cid:48) , it is obvious that d < d (cid:48) .
Therefore, there exists π ∗ such that for all π ∈ ( π ∗ , 1) , we have the following inequalities,
[Equation]
Since Φ is concave on R + , for any a.b > 0 we have Φ( b ) − Φ( a ) ≤ φ ( a )( b − a ) .
Setting a = − c +ln π 1 − π d and b = − c (cid:48)
+ln π 1 − π d (cid:48) would result in,
[Equation]
Combining above inequalities, we have
[Equation]
where we used the fact that for large enough π we have Φ (cid:16) − c − ln [Equation] − [Equation]
d
[Equation]
(cid:17) < 0 , and φ ≥ 0 .
It sufﬁces to show that,
[Equation]
[Equation]
When π → 1 , the LHS in the last inequality tends to 1 d , while the RHS tends to ∞ (since d < d (cid:48) ), and is monotonically increasing.
Consequently, we set π ∗ large enough such that the last inequality holds, which completes the proof.
D.1. Proof of Equation (13)
We begin by proving the following claim: if R (cid:48)(cid:48) c,d has no roots in (0 . 5 , 1) , then R c,d is monotonically decreasing on (0 . 5 , 1) , and if R (cid:48)(cid:48) c,d has exactly one root in (0 . 5 , 1) and R (cid:48)(cid:48) c,d (0 . 5) > 0 , then R c,d is increasing at ﬁrst and then decreasing, yielding exactly one global maximum in (0 . 5 , 1) .
The ﬁrst part of the claim is true since R (cid:48) c,d (0 . 5)
=
and R c,d is eventually decreasing, meaning that it must be concave, and subsequently monotonically decreasing on (0 . 5 , 1) .
For the second part,
R c,d will be convex and increasing at ﬁrst, and concave after that.
Since R c,d is eventually decreasing as well, it will also achieve its global maximum during its convex phase.
Assume π is a root of R (cid:48)(cid:48) c,d .
By using Lemma 12, we have
[Equation]
Let x =
ln
π
1 − π , A = cd 2 − 12 , and β = d 2 .
We should have
[Equation]
Similar to proof of ﬁrst part of Lemma 13, when A > 0 .
5 the above equation has only one solution in x > 0 and when
A ≤ 0 .
5 , cannot have any roots in x > 0 .
Moreover, one can verify that when A > 0 we have R (cid:48)(cid:48) c,d (0 . 5) > 0 as well.
Thus,
A > 0 .
5 and A ≤ 0 .
5 exactly differentiate between the two regimes.
Finally, note that
[Equation]
D.2. Proof of Equation (14)
The proof is similar to proof of Equation (13).
We want to analyze the second derivation of G ( π ) deﬁned by Equation (64).
Recall that we only consider π ≥ 12 .
Assume that π is root of G (cid:48)(cid:48)
( π ) .
Then we have
[Equation]
Recall that for the Bayes optimal classiﬁer, we have c (cid:48)
d (cid:48) 2 = 12 from Lemma 10.
Recall from Lemma 12 we have
[Equation]
Setting A = cd 2 − 12 and x = ln π 1 − π implies
[Equation]
Therefore
[Equation] in order to have G [Equation] ( π ) =
we must have
[Equation]
⇐⇒
ln
[Equation] − c [Equation] Let β = d 2 and α = 12 ( 1 d 2 − 1 d (cid:48) 2 ) .
It is clear that α, β > 0 .
As a result, we have
[Equation]
where T is a constant.
From Lemma 13 we know that S ( x ) in (0 , ∞ ) has two stages, decreasing at ﬁrst, and then increasing.
Similarly, T + S ( x ) has these two stage in x > 0 .
We have two cases 1.
If G (cid:48)(cid:48) ( 12 ) ≤ 0 or equivalently, using the deﬁnition of G (cid:48)(cid:48) ( π [Equation]
[Equation]
then, one can observe from Equation (79) that A ≤ 0 .
5 and per Lemma 13 the domain of S ( x ) is (0 , ∞ ) .
Furthermore,
S (0) + T ≤ 0 holds according to G (cid:48)(cid:48) ( 12 ) ≤ 0 .
From Lemma 13, we know that there exists x 1 such that S ( x ) is decreasing on (0 , x 1 ) and increasing on ( x 1 , ∞ ) .
Combined with the fact that S (0) + T ≤ 0 , we can conclude that
S ( x ) + T has at most one root in (0 , ∞ ) .
Similarly, G (cid:48)(cid:48) has at most a root in (0 . 5 , 1) .
Furthermore, Lemma 15 ensures that G is eventually decreasing.
Following the ﬁrst claim of the proof of Equation (13), we know that G (cid:48)(cid:48) (0) ≤ 0 and G (cid:48)(cid:48) having no roots in (0 . 5 , 1) imply that G is monotonically decreasing on (0 . 5 , 1) , which completes the proof in this case.
If G (cid:48)(cid:48) ( 12 ) > 0 , or equivalently
[Equation]
then Lemma 13 shows that S ( x ) + T has at most two roots in (0 , ∞ ) , which implies that G (cid:48)(cid:48) has at most two roots in (0 . 5 , 1) , since it is impossible for G (cid:48)(cid:48) to have roots corresponding to x out of the domain of S ( x ) .
Moreover, 12 is a local minimal of G since G (cid:48) ( 12 ) = 0 and G (cid:48)(cid:48) ( 12 ) > 0 .
As G is eventually decreasing, G cannot remain convex on the entirety of (0 . 5 , 1) , meaning that G (cid:48)(cid:48) has either exactly one or two roots.
When G (cid:48)(cid:48) has one root, similar to the proof of
Equation (13), G is increasing at ﬁrst and then decreasing on (0 . 5 , 1) , with exactly one global maximum.
When G (cid:48)(cid:48) has two roots, G is convex and increasing at ﬁrst, then concave, and then convex again.
Note that since G is eventually decreasing, it must be decreasing on the entirety of its ﬁnal convex phase, which means that it has already become decreasing on its concave phase, and has reached its maximum value in this phase as well.
Once again, we have shown that G is increasing at ﬁrst and then decreasing on (0 . 5 , 1) , and the proof of this case, as well as the entire theorem, is complete.
E. Proof of Theorem
Based on Proposition 3, we can write w adv =
Σ − 1 ( µ − (cid:15)z ∗ ) , where z ∗ = arg min (cid:107)
z (cid:107) ∞ ≤ 1 (cid:107) µ − (cid:15)z (cid:107) 2Σ − 1 .
In order for the two classiﬁers to be equivalent, we need w adv = λw nat for some λ > 0 .
Since w nat = Σ − 1 µ , this would imply z ∗ = µ (1 − λ ) (cid:15) .
Moreover, recall from Appendix C that {+whenever w advi (cid:54) = 
+}
we have z ∗ i
= sign( w advi ) , which here it would imply z ∗ i =
sign((Σ − 1 µ ) i ) whenever (Σ − 1 µ ) i (cid:54)
= 0 .
Besides, based on λ > 1 or λ < 1 , we must either have sign( z ∗ i ) =
sign( µ i ) or sign( z ∗ i ) =
− sign( µ i ) for all i .
Since Σ is positive deﬁnite, we must have (cid:104) µ, Σ − 1 µ (cid:105) ≥ 0 , therefore we always have sign( z ∗ i ) =
sign( µ i ) , i.e. λ < 1 .
In conclusion, we have
[Equation]
[Equation] where c = (cid:15) 1 − λ > (cid:15) .
Conversely, note that whenever the above conditions hold, one can easily verify that w adv = Σ − 1 µ is a valid solution, completing the proof.
F. Proof of Theorem
Recall that w adv = Σ − 1 ( µ − (cid:15)z ∗ ) , where z ∗ = arg min (cid:107) z (cid:107) ∞ ≤ 1 (cid:107) µ − (cid:15)z (cid:107) 2Σ − 1 .
Also, when π + = π − = 12 we have
[Equation]
Furthermore, using the H¨older’s inequality we have (cid:104) z ∗ , Σ − 1 µ
[Equation]
≤
[Equation]
z ∗ [Equation]
∞ [Equation] Σ − 1 µ (cid:107) 1 ≤ (cid:107) Σ − 1 µ (cid:107) 1 , and by deﬁnition of z ∗ we have (cid:107) µ − (cid:15) sign(Σ − 1 µ ) (cid:107) Σ − 1 ≥ (cid:107) µ − (cid:15)z ∗ (cid:107) Σ − 1 .
Taking (cid:15) <
(cid:107) µ (cid:107) 2Σ − 1 2 (cid:107)
Σ − 1 µ (cid:107) 1 into account, we have
[Equation]
For simplicity, we deﬁne A := [Equation] [Equation] 2Σ
[Equation] 1 and B := (cid:107) sign( [Equation] 1 µ
[Equation]
2Σ
[Equation]
1 (cid:107) µ (cid:107) 2Σ −.
Then [Equation] [Equation] where the second and third inequalities follow from the facts that (cid:15) ≤ 12 A and 1 √ 1+ x ≥ 1 − x 2 for all x > − 1 .
Recall that
C Σ ,µ = (cid:107) sign(Σ − 1 µ ) (cid:107) 2Σ − 1 − (cid:107) Σ − 1 µ (cid:107) 21 (cid:107) µ (cid:107) 2Σ − 1 .
Thus, it is straightforward to see
[Equation]
[Equation]
Besides, Φ is convex on R − .
Therefore, for any a, b ≤ 0 we have Φ( b ) − Φ( a ) ≤ e − b 2 2 √ 2 π ( b − a ) .
Consequently, we have
[Equation]
Moreover, with (cid:15) ≤ (cid:107) µ (cid:107) Σ − 1 2 √ C Σ ,µ we have −(cid:107) µ (cid:107) Σ − 1 + 2 C Σ ,µ (cid:15) 2 (cid:107) µ (cid:107) Σ − 1 ≤ −(cid:107) µ (cid:107) Σ − 1and g ≤ 2 e
[Equation]
{+− 
+}
C Σ ,µ
[Equation]
√ 2 π (cid:107) µ (cid:107) Σ −.
In order to prove the lower bound, we ﬁrst redeﬁne A and B as A :
= (cid:104) z ∗ , Σ − 1 µ (cid:105) (cid:107) µ (cid:107) 2Σ − 1 and B :=
(cid:107) z ∗ (cid:107) 2Σ − 1 (cid:107) µ (cid:107) 2Σ − 1 . Now we have R nat
[Equation]
( w
[Equation]
(cid:18) −(cid:107)
µ [Equation]
Σ
[Equation]
[Equation]
[Equation] (cid:19) [Equation] Furthermore, by the deﬁnition of z ∗ , when Σ is diagonal and min i ∈ [ d ] | µ i | ≥ (cid:15) , it is easy to observe that z ∗ = sign( µ ) =
sign(Σ − 1 µ ) .
As a result, we have B − A 2 = C Σ ,µ (cid:107) µ (cid:107) 2Σ − 1 .
In addition, the conditions of the theorem stipulate that C Σ ,µ (cid:15) 2 (cid:107) µ (cid:107) 2Σ
[Equation]
1 ≤ 14 , and for x ≤ 12 we have
[Equation]
x 3 .
Combined with (92), we have
[Equation]
Again, the convexity of Φ on R − yields that for any a, b ≤ 0 we have Φ( b ) − Φ( a ) ≥
e − a 2 2 √ 2 π ( b − a ) .
Thus, we have
[Equation]
G. Numerical Results
In the section, we present numerical experiments that validate Theorems 4 and 6.
Furthermore, we discover a new trend in the gap as the adversarial budget increases.
A theoretical analysis of this trend can be an interesting line of future work.
G.1. Class Imbalance
In Section 4.1 we analyzed the impact of class imbalance on the natural risk of the optimal adversarial classiﬁer and the natural risk gap between the two classiﬁers.
Theorem 4 proposes the necessary and sufﬁcient condition for transitioning between two different regimes, one with a single global maximum and another with two global maxima.
Here, we validate this theorem in two cases.
In the ﬁrst case, we set µ = (1 . 5 , 2 , 4) and Σ = 3 I 3 .
In Figure 2, we can observe the occurrence of two distinct regimes for both the natural risk of the optimal adversarial classiﬁer and the gap, which demonstrates our desired result.
For example, (cid:15) = 1 .
5 does not satisfy the condition of Equation (13).
Hence, the natural risk of adversarial classiﬁer has a concave shape with a single maximum at 12 .
On the other hand, (cid:15) = 2 .
5 satisﬁes this condition.
As a result, we have the promised two maxima located symmetrically around 12 and a local minimum at 12 .
Similarly, for a non-diagonal covariance matrix, we consider µ = (2 , 1 , 3) and
[Equation]
Figure 3 shows the result of this case, which conforms with our theoretical results.
Note that the natural risk of the Bayes classiﬁer does not depend on the adversarial budget ( (cid:15) ), and as expected, we only have a single maximum at 12 for this risk.
In this section and the next ones, in order to obtain exact risks for the optimal adversarial classiﬁer and the gap, we need to numerically solve Equation (10) and obtain z ∗ .
We used CVXPY (Diamond & Boyd, 2016) to this end.
G.2. Gap Lower and Upper Bounds
In this experiment, we validate our bounds for the natural risk gap between the two classiﬁers.
Theorem 6 provides a general upper bound for all balanced binary Gaussian mixture settings, as well as a lower bound when the covariance matrix is diagonal.
For the upper bound, we consider Equation (90) as it provides a tighter bound.
Firstly, we evaluate both the upper and lower bounds for a diagonal covariance matrix.
With µ = (1 . 5 , 2 , 4) and Σ = 3 I 3 , Figure 4a depicts the exact value of the gap as a function of (cid:15) along with our proposed lower and upper bounds.
Since for the non-diagonal setting we only have an upper bound, it is the only bound plotted in Figure 4b, when µ = (1 , 1 , 1 . 5) and
[Equation]
While our bounds are tight enough for the cases studied here, we show that they can be improved in the next section. [Figure] u r a l [Figure]
i
[Figure]
l
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure]  0 .1 0 .
[Figure]

 .
We only have an upper bound in this case [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure]
G.3. The Gap and Adversarial Budget
In this experiment, we aim to delve deeper into the behavior of the natural risk gap as a function of the adversarial budget, (cid:15)
[Figure]
We plot the gap as a function of (cid:15) with π + = 12 and µ = (1 , 2 , 3 , 3 . 4) and Σ =  
3 1 11 3 01 0 30 0 1  .
As shown in Figure
[Figure]
the curve has 3 break points, i.e. points where the function is non-differentiable.
The ﬁrst is betweenand 0 .
5 , the second is 2 and the last occurs between 2 . 5 and 3 .
The existence of these points is due to the (cid:96) ∞ norm constraint of the optimization problem in Equation (10).
The piecewise behavior of the gap encourages locating the break points and providing a tighter gap between each two break points in the future.
G.4. The Gap and Imbalanced Priors
In this experiment, we compare the trend of the natural risk gap as a function of (cid:15) , obtained with different π + , when we have ﬁxed µ = (1 , 2 , 3) and Σ =   3 11 31 0  .
We have two observations from Figure 6, which their theoretical study is left as future work.
In contrast to the case of π + = 12 where the gap remains constant beyond (cid:15) = 2 , when π + (cid:54) = 12 the gap still increases after this point.
This observation shows that the convergence point of the gap, i e. the (cid:15) beyond which the gap is constant, varies with π + . 2.
The worst priors for the gap vary with (cid:15) .
For instance, when (cid:15) ∈ (1 . 5 , 2) , π + = 0 .
8 achieves a higher gap compared to π + = 0 . 7 , which reverses beyond (cid:15) > 2 . 5 .