Maximizing the robust margin provably overﬁts on noiseless data
Abstract
Numerous recent works show that overparameterization implicitly reduces variance, suggesting vanishing beneﬁts for explicit regularization in high dimensions.
However, this narrative has been challenged by empirical observations indicating that adversarially trained deep neural networks suffer from robust overﬁtting.
While existing explanations attribute this phenomenon to noise or problematic samples in the training data set, we prove that even on entirely noiseless data, achieving a vanishing adversarial logistic training loss is suboptimal compared to regularized counterparts.
1. Introduction
A modern narrative in machine learning suggests that regularization is superﬂuous for good performance of large overparameterized models.
This perspective is motivated by theoretical and experimental ﬁndings that analyze the population risk in a setup where the training and test data are drawn from the same distribution (see Hastie et al.
Bartlett et al.
; Yang et al.
; Nakkiran et al.
(2020) and references therein).
However, the successful adoption of machine learning models in real-world applications crucially hinges on the models’ robustness to adversarial attacks or distribution shifts.
A popular method to achieve low robust risk is to minimize a robust training loss, for example using adversarial training (Goodfellow et al., 2015).
Mounting empirical evidence suggests that when minimizing the robust loss, the narrative that regularization is superﬂuous is incorrect: additional regularization such as early-stopped adversarial training often leads to more robust generalization (Rice et al., 2020; Sagawa et al., 2020a;b).
For example, Rice et al.
(2020) show that overparameterized deep neural networks that are adversarially trained with (cid:96) ∞ perturbations beneﬁt from early stopping on image data
< konstantin.donhauser@ethz.ch > , Alexandru T , ifrea < tifreaa@inf.ethz.ch > .
Accepted by the ICML 2021 workshop on A Blessing in Disguise: The Prospects and Perils of Adversarial Machine Learning.
Copyright 2021 by the author(s).
Without noise
With noise      d/n 0 .0 .0 .0 .0 .
R i s k
( λ → 0 ) -
R i s k
( λ o p t ) sets.
This phenomenon is sometimes referred to as robust overﬁtting .
For noisy settings, previous works have proposed the following explanation for robust overﬁtting: since the robust risk ampliﬁes estimation errors, its variance is larger, and regularization can hence be beneﬁcial for robust generalization (Sanyal et al., 2021).
However, Figure 4 in Appendix B shows that even when estimating mostly noiseless signals, robust overﬁtting persists!
This observation does not only apply to image data but also to simple linear models: Figure 1 shows the gap in performance between (cid:96) ∞ -robust logistic regression regularized by a ridge penalty with optimal parameter λ opt and the robust max-margin solution.
Existing theoretical results neither predict nor explain those observations.
In particular, it is perhaps unexpected that for noiseless data, strictly sacriﬁcing robust data ﬁt by increasing the weight of the ridge penalty term leads to better generalization.
In this paper, we show for robust logistic regression and a sparse ground truth that in the high-dimensional regime, where d > n , maximizing the
robust margin leads to suboptimal robust generalization.
Speciﬁcally, • We prove that a strictly positive ridge parameter leads to a systematic improvement in generalization compared to the robust max-margin solution.
Our results provide the ﬁrst rigorous proof for robust overﬁtting even in the absence of noise.
• We experimentally show a perhaps unexpected effect of noise in the training procedure: noisy data might not be robustly linearly separable, rendering the robust max-margin estimator infeasible and hence inducing spurious regularization effects.
2. Risk minimization framework
In this section, we describe the setup for our theoretical analysis of adversarial training with the logistic loss.
We deﬁne the data generating process, the standard and robust population risks, and formally introduce the estimators that we analyze.
2.1. Problem setting
We consider a discriminative data-generating model for classiﬁcation with covariates (or features) x ∈ R d drawn from an isotropic Gaussian distribution x ∼ N (0 , I d ) .
In this paper we focus primarily on training with noiseless data, namely we observe deterministic labels given by y = sgn (cid:104) θ (cid:63) , x (cid:105) ∈ {− 1 , +1 } for all covariates x ∈ R d , where the ﬁxed vector θ (cid:63) denotes the ground truth.
We consider the scenario where the ground truth has inherent low-dimensional structure in the form of sparsity and pick θ (cid:63) =
, 0) (cid:62) .
As we discuss later in this section, (cid:96) ∞ robustness and sparsity via its convex surrogate, the (cid:96)norm, are related objectives.
The results in this paper are of asymptotic nature and hold when d/n → γ as both the dimensionality d and the number of samples n tend to inﬁnity.
This high-dimensional regime is widely studied in the literature (B ¨ uhlmann & Van
De Geer, 2011; Wainwright, 2019) as it yields precise predictions for many real-world problems where both the input dimension and the data set size are large.
It is also the predominant setting considered in previous theoretical papers that discuss overparameterized linear models (Dobriban &
Wager, 2018; Hastie et al., 2019; Ali et al., 2020; Deng et al., 2021; Javanmard et al., 2020; Javanmard & Soltanolkotabi, 2020; Sur & Cand ` es, 2019).
2.2. Robust risk for evaluation
The broad application of ML models in real-world decisionmaking processes increases requirements on their robustness.
For example, for the image domain, robust classiﬁers should yield the same prediction when an image is attacked via an additive imperceptible (cid:96) ∞ -perturbation that does not change the ground truth label.
In this case, the estimator which has zero standard population risk also achieves zero robust population risk.
Transferred to linear classiﬁcation, we require such additive consistent perturbations to be orthogonal to θ (cid:63) , that is δ ∈ U c ( (cid:15) ) :
= { δ ∈ R d : (cid:107) δ (cid:107) ∞ ≤ (cid:15) and (cid:104)
θ (cid:63) , δ (cid:105) = 0 } .
We hence evaluate the adversarially robust risk of a parameter θ with respect to consistent (cid:96) ∞ - perturbations, deﬁned as follows:
[Equation]
(1) where the expectation is taken over the marginal feature distribution P and 1 is the indicator function.
For (cid:15) =
we obtain the population 0-1 risk, also called standard risk, denoted by R ( θ ) .
For the 1 -sparse ground truth θ (cid:63) , the risks have a closed-form expression given by
Lemma A.1 in Appendix A.1.
2.3. Interpolating and regularized estimator
We say that the data is robustly separable when the robust max-margin solution exists.
Its direction is given by ˆ
[Equation]
that for all
[Equation]
Robust separability is guaranteed for noiseless data and for consistent perturbations U ( (cid:15) )
=
U c ( (cid:15) ) that are orthogonal to the ground truth θ (cid:63) .
Unfortunately, minimization with respect to consistent perturbations in Equation (2) requires full knowledge of the ground truth during training, thus leaking information about θ (cid:63) .
Instead, many papers to date (e.g. Javanmard et al.
; Javanmard & Soltanolkotabi (2020)) consider the unrestricted (and hence inconsistent) (cid:96) ∞ -perturbation set U
ic
[Equation]
However, when training with inconsistent perturbations, it is possible that the perturbed data crosses the true decision boundary.
Thus, a minimizing solution might ﬁt wrong labels even if the actual training samples are noiseless.
We want to study the purely noiseless setting and hence focus primarily on adversarial training with consistent perturbations.
According to classical wisdom, this setting is also the one in which we expect regularization to help the least.
We revisit inconsistent adversarial training in Section 4.2 and show that its effects can at least spuriously improve the robust risk.
We compare the robust max-margin interpolator with the ridge-regularized robust logistic regression.
We can write the minimizer of the regularized (cid:96) ∞ robust logistic loss as follows:
[Equation]
Notice that for robustly separable data, the direction of the robust max-margin solution aligns with that of the minimizer ˆ θ λ for λ → 0 , i.e., ˆ θ 0 = lim λ
→ 0 ˆ θ λ .
This fact follows from the results in (Rosset et al., 2004) when using the closed form expression for the consistent robust risk (1).
Since the robust risk is independent of the estimator norm, we refer
[Figure] [Figure]
d/n 0 .0 .0 .0 .
R i s k (a
Standard risk Robust risk
Normalized robust margin
[Figure]
Iteration
[Figure]
to the robust max-margin solution as the normalized vector ˆ θ 0 / (cid:107) ˆ θ 0 (cid:107) 2 and for brevity simply call it the max-margin solution.
Finally, we remark that for standard logistic regression, gradient descent converges to the same direction as the maxmargin solution (Ji & Telgarsky, 2019; Soudry et al., 2018), and, as argued in (Javanmard & Soltanolkotabi, 2020), this result can also be extended to adversarial training (3).
3. Regularization for robustly separable data
In this section, we prove that the robust max-margin solution (2) overﬁts since it generalizes worse than the regularized estimator (3) with optimally chosen λ > 0 for entirely noiseless training data.
We derive precise asymptotic predictions that show a systematic beneﬁt of ridge regularization for robust logistic regression as d, n → ∞ and d/n → γ .
We further show that early-stopped gradient descent on the unregularized robust logistic loss in Equation (3) yields much smaller population risks than the robust max-margin solution at convergence.
3.1. Beneﬁts of ridge regularization
We now present an informal statement describing the asymptotic standard and robust risk for linear classiﬁcation with the logistic loss as d, n → ∞ .
We refer to Appendix D for the precise statement and the proof, which is inspired by the works of Javanmard & Soltanolkotabi (2020); Salehi et al.
and uses the Convex Gaussian Minimax Theorem (CGMT)
(Gordon, 1988; Thrampoulidis et al., 2015).
Theorem 3.1 (Informal) .
Assume that (cid:15) =
(cid:15) 0 / √ d for some constant (cid:15) 0 independent of n, d and θ (cid:63) =
Then, the robust and standard risks of the estimator ˆ θ λ from
Equation (2) , (3) with respect to consistent attacks and λ ≥converge in probability as d, n
[Equation] [Equation]
with erf the error function,
I (
[Equation] [Equation]
[Equation]
and where ν (cid:63) =
(cid:113) ( ν (cid:63) ⊥ )
+ ( ν (cid:63) (cid:107) ) 2 , and ν (cid:63) ⊥ , ν (cid:63) (cid:107) , δ (cid:63) are the unique solution of a scalar optimization problem speciﬁed in Appendix D that depends on θ (cid:63) , γ, (cid:15) 0 and λ .
We illustrate the theorem’s claim in Figure 2a where we show the standard and robust risks obtained by the asymptotic theoretical predictions for d, n → ∞ and the risks obtained from simulations for n = 1000 and (cid:15) = 0 .
The theoretical curves are obtained by solving the scalar optimization problem speciﬁed in Appendix D; the empirical setup is described in Appendix C. We observe that the precise asymptotics indeed predict beneﬁts of regularization for robust logistic regression on noiseless training data.
This trend is further supported by our simulation results closely following the theoretical curves.
Furthermore, in Figure 2b, we observe for a ﬁxed overparameterization ratio d/n =how optimizing the robust logistic loss well beyond 100% robust training accuracy (i.e. the robust margin becomes positive) substantially hurts generalization.
[Figure]
Label noise
[Figure]
3.2. Beneﬁts of early stopping
In addition to varying the ridge coefﬁcient λ , we observe the same trends as for 1 /λ → ∞ on the optimization path when training with gradient descent as the iterations t → ∞ .
Figure 2c indicates that early stopped gradient descent can improve the robust risk for logistic regression, similar to ridge regularization.
Both early stopping and ridge regularization avoid the max-margin solution t → ∞ and λ → ∞ respectively, and yield an estimator with signiﬁcantly lower standard and robust risk.
4. The beneﬁts of avoiding robust separability
In the previous section we focused on noiseless settings and studied the generalization performance of regularized estimators that do not maximize the robust margin.
In this section, we show how, surprisingly, as a consequence of avoiding robust separability, adding noisy labels in the training loss also leads to an estimator with better robust generalization than the max-margin solution of the corresponding noiseless problem.
However, we note that explicit regularization is still the preferred choice of avoiding the maxmargin solution as it yields a signiﬁcantly lower robust risk.
4.1. Avoiding the max-margin solution via label noise
We ﬁrst consider the case where we introduce label noise to the training data.
Figure 3a shows the robust and standard risk together with the training loss of estimator ˆ θ λ (3) for λ → 0 with varying percentages of ﬂipped labels.
For low noise levels, the data is robustly separable and the training loss vanishes, yielding the max-margin solution (2).
For high enough noise levels, the constraints in (2) become infeasible and the training loss of the resulting estimator starts to increase.
This effectively induces implicit regularization as the robust risk at this point starts to decrease again.
While it is well known that covariate noise can act as an implicit regularization (Bishop, 1995), in contrast to common intuition, we show that the performance can also improve when introducing wrong labels in the training loss as it prevents the estimator from converging to the max-margin solution as λ → 0 .
While this spuriously implies that label noise beneﬁts robustness, the estimator with optimal ridge parameter λ opt always outperforms the unregularized estimator, even if the data is not robustly-separable.
4.2. Avoiding the max-margin solution via inconsistent training
For large (cid:15) and inconsistent perturbations U ic ( (cid:15) ) , even noiseless training data might not be robustly separable, rendering the max-margin solution infeasible for λ → 0 .
Consequently, this leads to a similar effect as adding label noise as discussed in Section 4.1.
Indeed, Figure 3b reveals that the resulting estimator trained with inconsistent adversarial perturbations achieves a better performance than the corresponding max-margin estimator (2).
5. Conclusions
In this paper, we prove that on high-dimensional input data, the ridge-regularized robust logistic loss yields a solution that is more robust than the unregularized robust max-margin estimator, even on entirely noiseless data.
We further show experimentally that early-stopped gradient descent yields similar beneﬁts and discuss other phenomena that unexpectedly improve robust generalization by avoiding the max-margin solution.
A. Setting
In Section A.1 we derive closed-form expressions of the standard and robust risks (1).
Furthermore, Section A.2 discusses that the robust risk (1) upper-bounds the worst case risk under distributional mean shifts.
A.1. Closed-form of robust risk for classiﬁcation
Similarly to linear regression, we can express the robust and standard risk for the linear classiﬁcation model in Section 3 as stated in the following lemma.
Lemma A.1.
Assume that P X is the isotropic Gaussian distribution and θ (cid:63) =
(1 , 0 , · · · , 0) (cid:62) .
Then, 1.
For any non-decreasing loss (cid:96) : R → R we have
[Equation]
For the 0 - 1 loss the robust risk (1) with respect to (cid:96) ∞ perturbations is given by
[Equation]
(cid:105) (cid:107)
[Equation]
(cid:107)
[Equation]
(cid:18) (cid:15) (cid:107) Π ⊥ θ (cid:107)
[Equation]
(cid:107)
[Equation]
(cid:107)
[Equation]
(cid:19)
[Equation]
(cid:18) (cid:15) (cid:107) Π ⊥ θ (cid:107)(cid:107) θ (cid:107), (cid:104)
θ (cid:63) , θ (cid:105) (cid:107)
θ (cid:107)(cid:19) , (5) with
[Equation]
Proof.
We ﬁrst prove Eq (4).
Because (cid:96) is non-increasing, we have
[Equation]
While minimization over δ has no closed-form solution in general, we note that for θ (cid:63) =
(1 , 0 , · · · , 0) , we get min (cid:107) δ
i
[Equation]
≤
[Equation]
−
[Equation]
Π ⊥ θ (cid:107) 1 and Equation (4).
Let 1 { E } be the indicator function, 1 if the event E holds.
Since (cid:96) ( · ) = 1 ·≤ 0 is non-increasing we can use (4) and write
[Equation]
(cid:107)
[Equation]
Let (cid:98) Π (cid:107) :
= 1 (cid:107) θ (cid:107) 22 θθ (cid:62) be the projection onto the subspace spanned by θ and (cid:98)
Π ⊥ :
= I d − (cid:98) Π (cid:107) the projection onto the orthogonal subspace.
Since X is a vector with i.i.d.
standard normal distributed entries, we equivalently have
[Equation]
[Equation]
with Z 1 , Z 2 two independent standard normal random variables.
For brevity of notation, deﬁne ν
=
(cid:15) (cid:107) Π ⊥ θ (cid:107) 1 (cid:107)
θ (cid:107)and b ( Z 1 , Z 2 ) =
sgn (cid:16) Z 1 (cid:107) (cid:98) Π (cid:107) θ (cid:63) (cid:107)
+ Z 2 (cid:107) (cid:98)
Π ⊥ θ (cid:63) (cid:107) 2 (cid:17)
=:
sgn( β (cid:62) Z ) with β (cid:62) =
( (cid:107) (cid:98) Π (cid:107) θ (cid:63) (cid:107) 2 , (cid:107) (cid:98) Π ⊥ θ (cid:63) (cid:107) 2 ) .
Deﬁne the event
A = { sgn (cid:16) Z 1 (cid:107) (cid:98) Π (cid:107) θ
(cid:63)
[Equation]
≤ 0 } .
Because Z 2 is symmetric, the distribution of
Z 1 b ( Z 1 , Z 2 ) is symmetric, hence we can rewrite the risk
[Equation]
and derive expression for T 1 , T 2 separately.
Step 1: Proof for T
Note that due to (cid:107) θ (cid:63) (cid:107) 2 = 1 we have (cid:107) β (cid:107) 2 =
and recall that T 1 = P ( β (cid:62) Z ≤
| Z 1 ≥ 0) .
Using the fact that both Z 1 and Z 2 are independent standard normal distributed random variables, a simple geometric argument then yields that T
= απ with α = arc cos
[Equation]
=
[Equation]
Noting that β 1 = (cid:107) (cid:98) Π (cid:107) θ (cid:63) (cid:107) 2 = (cid:104) θ (cid:63)
[Equation]
θ (cid:105) (cid:107) θ (cid:107) 2 then yields
T
[Equation]
=
[Equation]
Step 2:
Proof for T
First, assume that (cid:104) θ (cid:63) , θ (cid:105) ≥ 0 .
We separate the event V
= { Z 1 ≤ ν, b ( Z 1 , Z 2 ) ≥ 0 } into two events
V =
V 1 ∪ V
V 1 = { Z 1 ≤ ν, Z 2 ≥ 0 } and V 2 = { Z 1 ≤ ν, b ( Z 1 , Z 2 ) ≥ 0 , Z 2 ≤ 0 } .
The conditional probability of the ﬁrst event is directly given
[Equation]
Hence it only remains to ﬁnd an expression for P ( V 2 | Z 1 ≥ 0) .
Letting µ denote the standard normal distribution, we can write
[Equation]
[Table]
Together with Step 1, Equation (5) follows by noting that β 21 + β 22 = 1 .
Finally, the proof for the case where (cid:104) θ (cid:63) , θ (cid:105) ≤follows exactly from the same argument and the proof is complete.
A.2. Distribution shift robustness and consistent adversarial robustness
In this section we rigorously introduce distribution shift robustness and show the relation to consistent (cid:96) p adversarial robustness for certain types of distribution shifts.
When learned models are deployed in the wild, the i.i.d.
assumption does not always hold.
That is, the test loss might be evaluated on samples from a slightly different distribution.
Shifts in the mean of the covariate distribution is a standard intervention studied in the invariant causal prediction literature (B ¨ uhlmann et al., 2020; Chen & B ¨ uhlmann, 2020).
For mean shifts in the null space of the ground truth θ (cid:63) we deﬁne an alternative evaluation metric that we refer to as the distributionally robust risk deﬁned as follows:
[Equation]
0 } where V p is the neighborhood of mean shifted probability distributions.
A duality between distribution shift robustness and adversarial robustness has been established in earlier work such as (Sinha et al., 2018) for general convex, continuous losses (cid:96) test .
For our setting, the following lemma holds.
Lemma A.2.
For any (cid:15) ≥ 0 and θ , we have ˜ R (cid:15) ( θ ) ≤ R (cid:15) ( θ ) .
Proof.
The proof follows directly from the deﬁnition and consistency of the perturbations U p ( (cid:15) ) and orthogonality of the mean shifts for the neighborhood V p .
By deﬁning the random variable w = x − µ P for x ∼ P we have the distributional equivalence
[Equation]
for x (cid:48) ∼ Q and x ∼ P with µ Q − µ P = δ and hence
[Equation]
where the ﬁrst line follows from orthogonality of the mean-shift to θ (cid:63) .
B. Neural networks on sanitized binary MNIST
[Figure]
a cc u r a c y g a i n
Figure 4 shows that robust overﬁtting in the overparameterized regime also occurs for single hidden layer neural networks on an image classiﬁcation problem that we chose to be arguably devoid of noise.
We consider binary classiﬁcation on MNIST classes 1 vs 3 and further reduce variance by removing “difﬁcult” samples: We train networks of width p ∈ { 10 , 100 , 10 k } on the full MNIST training dataset and discard all images that take at least one of the models more than 100 epochs of training to learn.
While some recent work argues that such sanitation procedures can effectively mitigate robust overﬁtting (Dong et al., 2021), we still observe a signiﬁcant increase from the ﬁnal test robust accuracies to the best (early stopped) ones in
Figure 4.
We train all networks on a subset of n = 2000 training samples using vanilla mini-batch stochastic gradient descent with learning rate ν p = (cid:112) 0 .
1 /p that we multiply by 0 .
1 after 300 epochs.
This learning rate schedule minimizes the training loss efﬁciently; we did not perform tuning using test or validation data.
For the robust test error, we approximate worst-case (cid:96) ∞ perturbations using 10-step SGD attacks on each test sample.
C. Experimental details
In this section we provide additional details on our experiments.
All our code including instructions and hyperparameters can be found here: https://github.com/michaelaerni/interpolation_robustness .
C.1. Empirical predictions
If not mentioned otherwise, all empirical simulations use noiseless i.i.d.
samples from our synthetic data model as described in Section 2.1.
Whenever we simulate label noise, we ﬂip a ﬁxed percentage of all training labels chosen uniformly at random.
However, we calculate all risks in closed-form without noise and, in the robust case, with consistent perturbations.
We approximate the integral for the robust 0-1 risk in Theorem 3.1 using a numerical integral solver since we cannot obtain a solution analytically.
We ﬁt all logistic regression models except in Figure 2c by minimizing the (regularized) robust logistic loss (3) using the
CVXPY library in combination with the Mosek convex programming solver.
To obtain the max-margin solution whenever feasible for λ → 0 , we optimize the constrained problem (2) directly since (3) with λ = 0 has many optimal solutions.
In Figure 1, we use (cid:15) = 0 .
1 for the robust risk and consistent adversarial training on n = 1000 samples.
In the noisy case, we ﬂip 2% of all training labels.
For Figure 2c, we run zero-initialized gradient descent on the unregularized loss ( λ = 0 ) for 500 k iterations.
We start with a small initial step size of 0 .
01 that we double every 30 k steps until iteration 300 k .
C.2. Theoretical predictions
In order to obtain the asymptotic theoretical predictions for logistic regression in Figure 2a corresponding to the empirical simulations with n = 1000 and (cid:15) = 0 .
05 , we obtain the solution of the optimization problems in Theorem D.1,D.2 with (cid:15)
= 0 .
√ 1000 γ by solving the system of equations in Corollary D.4,D.5 using MATLAB ’s optimization toolbox where we approximate expectations via numerical integration.
D. Details on Theorem 3.
In this section we give a formal statement for Theorem 3.1.
The results are based on the Convex Gaussian Minimax Theorem (CGMT)(Gordon, 1988; Thrampoulidis et al., 2015).
The results presented in this section have similarities with the ones in (Javanmard & Soltanolkotabi, 2020), however, we study the discriminative data model with features drawn from a single
Gaussian and 1 -sparse ground truth.
In contrast, in the paper (Javanmard & Soltanolkotabi, 2020), the authors study a generative data model with features drawn from two Gaussians.
Furthermore, several papers study logistic regression for isotropic Gaussian features in high dimensions (Salehi et al., 2019; Sur & Cand`es, 2019), but the analysis is focused on the standard risk and the authors do not consider adversarial robustness.
An immediate consequence of the proof of Lemma A.1 is that the adversarial loss from Equation (3) for the 1 -sparse ground truth writes as
[Equation]
Let M f ( x, t ) =
min y 12 t ( x − y ) 2 + f ( y ) be the Moreau envelope and let Z (cid:107) , Z ⊥ be two independent standard normal distributed random variables.
We can now state Theorem D.1 which describes the asymptotic risk of ˆ θ λ , for λ > 0 , and for the asymptotic regime where d, n → ∞ .
The proof of the theorem can be found in Appendix D.3.
Theorem D.1.
Assume that the we have i.i.d.
random features x i drawn from an isotropic Gaussian with noiseless observations
y i =
sgn( (cid:104) x
i , θ (cid:63) (cid:105) ) and ground truth θ (cid:63) =
, 0) (cid:62) .
Further, assume that λ > 0 and (cid:15) =
(cid:15) 0 / √ d , where (cid:15) 0 is a numerical constant.
Let ( ν (cid:63) ⊥ , ν (cid:63)
[Equation]
, r
[Equation]
, τ
[Equation]
) be the unique solution of
[Equation]
Then, for λ > 0 the estimator ˆ θ λ from Equation (3) with logistic loss and consistent (cid:96) ∞ attacks satisﬁes asymptotically as d, n → ∞ , d/
[Equation]
The convergences hold true in probability.
For λ > 0 , the Equation (9) has a unique minimizer.
In contrast, for λ = 0 the minimizer of Equation (9) is not unique since the data is robustly seperable.
As discussed in Section 3, for λ =
we study instead the penalized max-margin solution from Equation (2).
The asymptotic behavior of the corresponding solution is characterized in Theorem D.2 with proof in
Appendix D.4.
Theorem D.2.
Assume that the we have i.i.d.
random features x i drawn from an isotropic Gaussian with noiseless observations
y i =
sgn( (cid:104) x
i , θ (cid:63) (cid:105) ) and θ (cid:63) =
(1 , 0 , · · · , 0) (cid:62) as described in Section 3.
Further, assume that (cid:15) =
(cid:15) 0 / √ d .
Let ( ν (cid:63) ⊥ , ν (cid:63) (cid:107) , r (cid:63) , δ (cid:63) , ζ (cid:63) , κ (cid:63) ) be the unique solution of
[Equation]
Then, the estimator ˆ θ 0 ( (cid:15) ) from Equation (2) with logistic loss and consistent (cid:96) ∞ attacks satisﬁes asymptotically as d, n → ∞ , d/n → γ
[Equation]
The convergences hold true in probability.
Remark D.3.
Theorem 3.1 is obtained from Theorem D.1,D.2 when inserting the expression from Equation (11) , (13) into the expression of the risk in Lemma A.1.
D.1. Solving the optimization problems
To obtain concrete predictions from Theorems D.1 and D.2 we need to solve the corresponding optimization problems.
Using the fact that the objective is concave in r, µ respectively r, ζ and convex in all other variables, we ﬁnd the unique solution by solving the systems of equations stated in Corollary D.4,D.5 presented below — assuming that the solution is not obtained on the boundary.
For numerical predictions, we solve the systems of equations using a least squares solver in
MATLAB where we approximated the expectations using a standard numerical integrator.
Regularized estimate ( λ > 0 ): We make use of the following well known facts on the derivative of the Moreau envelope (e.g. see (Jourani et al., 2014)):
[Equation]
where Prox f ( x, µ ) is the proximal operator
Prox
[Equation]
Let
M = E Z (cid:107) ,Z ⊥ (cid:2) M (cid:96) ( Z (cid:107) ν
(cid:107) + Z ⊥ ν ⊥ − (cid:15)δ, τr ) (cid:3) and denote with
[Equation]
the derivative of the expected Moreau envelope with respect to ν (cid:107) and similar for the other variables.
Further, deﬁne
[Equation]
and denote with g µ
respectively g
r the corresponding partial derivatives.
This allows us to express the optimization objective from Theorem D.1 as
[Equation]
Since the problem is concave in r, µ and convex in the other variables, we can equivalently solve the system of equations that results from setting the ﬁrst order conditions, i.e. ∇ C = 0 , which gives us the following corollary.
Corollary D.4.
Assume that the solution of Equation (10) from Theorem D.1 is not obtained on the boundary.
Then, the solution satisﬁes the following system of Equations.
[Equation]
Robust max- (cid:96) 2 -margin interpolator ( λ = 0 ):
Similarly, we can also reformulate the optimization problem in Theorem
D.2.
For brevity of notation, let T =
E Z (cid:107) ,Z ⊥ (cid:104) max (cid:0) 0 , 1 + (cid:15)
δ − Z (cid:107) ν (cid:107) + Z ⊥ ν ⊥ (cid:1) 2 (cid:105) and
[Equation]
Further, denote with T ν ⊥ the derivative of T with respect to ν ⊥ and similarly for the other variables.
We use the same notation for the partial derivatives of H .
Finally, we can write the optimization objective from Theorem D.2 as
[Equation]
Since, once again, the solution of the optimization problem must satisfy ∇ C =
if it does not lie on the boundary, we analogously obtain the following corollary.
Corollary D.5.
Assume that the solution of Equation (12) from Theorem D.2 is not obtained on the boundary.
Then, the solution satisﬁes the following system of Equations.
[Equation]
D.2. Label noise
While our results assume noiseless observations y i = sgn( (cid:104)
x i , θ (cid:63) (cid:105) ) , Theorem D.1,D.2 can be extended to the case where additional label noise is added to the observations.
That is, it can be extended to the case
where y i = sgn
( (cid:104) x i , θ (cid:63) (cid:105) )
ξ i with ξ i i.i.d.
distributed and P ( ξ i = 1) = 1 − σ P ( ξ i = − 1) =
σ with label noise σ .
Note that as discussed in Section 4.2, the robust max-margin solution (2) might not exists for noisy observations.
In this case, the estimate (3) has a unique solution also for λ = 0 .
In fact, asymptotically we can ﬁnd a threshold γ (cid:63) such that for any γ < γ (cid:63) the robust max-margin solution does not exists and for any γ ≥ γ (cid:63) the robust max-margin solution exists.
The threshold can be found using the CGMT when following the same argument as in Theorem 6.1 (Javanmard & Soltanolkotabi, 2020).
Finally, we remark that when λ > 0 or λ = 0 and γ < γ (cid:63) we can apply Theorem D.1 when replacing | Z (cid:107) | with ξ | Z (cid:107) | .
Similarly, for λ = 0 and γ ≥ γ (cid:63) we can apply Theorem D.2 when replacing | Z (cid:107) | with ξ | Z (cid:107) | where ξ is drawn form the same distribution as ξ i deﬁned above.
D.3. Proof of Theorem D.
The proof is similar to the proof of Theorem 6.4 (Javanmard & Soltanolkotabi, 2020).
Denote with X ∈ R n × d the input data matrix and with y ∈ R n the vector containing the observations.
Recall that the estimator ˆ θ is given by
[Equation]
such that v = D y X θ,
[Equation]
14) where (cid:96) ( x ) =
log(
+ exp( − x )) is the logistic loss, X ∈ R n × d is the data matrix and D y the diagonal matrix with entries ( D y )
i,i = y i .
We can then introduce the Lagrange multipliers u ∈ R n to obtain:
[Equation]
Furthermore, we can separate X = XΠ ⊥ + XΠ (cid:107) , which gives us:
[Equation]
Convex Gaussian Minimax Theorem We can now make use of the CGMT, which states that
[Equation]
with ψ convex in θ and concave in u has asymptotically pointwise the same solution when d, n → ∞ , d/n → γ as
[Equation]
[Equation]
with g ∈ R d , h ∈ R n random vectors with i.i.d.
standard normal distributed entries and compact sets U θ
and U u .
As commonly referred to in the literature, we call Equation (24) the primal optimization problem and Equation (17) the auxiliary optimization problem.
We omit the precise statement and refer the reader to (Thrampoulidis et al., 2015).
The CGMT has already been used in several works studying high dimensional asymptotic logistic regression (Salehi et al., 2019) and also when training with adversarial attacks (Javanmard & Soltanolkotabi, 2020).
Essentially, the application of the CGMT is based on the following facts:
The objective (15) is concave in u and convex in v, θ .
We can restrict u, v, θ to compact sets without changing the solution.
For θ we note that this is a consquence of λ >and for u we note that the stationary condition requires u i = (cid:96) (cid:48) ( v i − (cid:15) (cid:107) Π ⊥ θ (cid:107) 1 ) .
XΠ ⊥ is independent of the observations y and of XΠ (cid:107) . As a result, the CGMT states that the solution of the primal optimizatin problem (15) asymptotically concentrates around the same value the solution of the following auxiliary optimization problem
[Equation]
where g ∈ R d and h ∈ R n are vectors with i.i.d.
standard normal distributed entries.
Scalarization of the optimization problem We now aim to simplify the optimization problem.
In a ﬁrst step, we maximize over u .
For this, deﬁne r = (cid:107) u (cid:107) 2 / √ n , which allows us to equivalently write:
[Equation]
where we have used the fact that (cid:107) u (cid:62) D y (cid:107)
=
(cid:107) u (cid:107) 2 .
In order to proceed, we want to separate Π ⊥ θ form the loss (cid:96) ( v, Π ⊥ θ ) :
= 1 n (cid:80) ni =1 (cid:96) ( v i − [Equation] Π
[Equation] Denoting by ˜ (cid:96) the conjugate of (cid:96) we can write (cid:96) ( v, Π ⊥ θ ) in terms of its conjugate with respect to Π ⊥ θ :
[Equation]
where for the second identity we use the derivation for the conjugate of (cid:96) from Lemma A.2 in the paper (Javanmard &
Soltanolkotabi, 2020).
Next, we note that because the problem is convex in r and concave in θ, v , we can swap maximization with minimization.
Interchanging the order of maximization over w and minimization over δ , we get
[Equation]
where we used again the fact that we can interchange the order maximization and minimization.
Next, we simplify the optimization over θ .
Write Π (cid:107) θ
=
Π (cid:107) 1 ν (cid:107) where ν (cid:107) ∈ R and let ν ⊥ = (cid:107) Π ⊥ θ (cid:107) 2 .
We can simplify
[Equation]
In order to obtain a low dimensional scalar optimization problem, we still need to scalarize the optimization over w and v .
For this, we replace the term (cid:107) D y XΠ [Equation] 1 ν (cid:107) + D y hν ⊥ − v (cid:107) 2 with its square, which is achieved by using the following identity min
[Equation]
Hence,
[Equation]
interchanging again the order of maximization and minimization, we can now separately solve the following two inner optimization problems:
[Equation]
Equation (20) Let ST t ( x ) =
(cid:26) | x | ≤ t sgn( x )( | x | − t ) else be the soft threshold function.
We have
[Equation]
where we used in the third line that the ground truth θ (cid:63) is 1 -sparse and in the last line that the expectation exists for
Z ∼ N (0 , 1) .
Finally, we can further simplify
[Equation]
Hence
[Equation]
we can conclude the ﬁrst term
[Equation]
Equation (21)
For the second term we also aim to apply the law of large numbers.
We have
[Equation]
where in the last line we used that ( D y XΠ
[Equation]
Π ⊥
[Equation]
=
[Equation]
σ
[Equation]
x
[Equation]
x
[Equation]
i
[Equation]
has the same distribution as | Z (cid:107) | with Z (cid:107) ∼ N (0 , 1) .
Further, to apply the law of large numbers, we need to show that
the
Moreau envelope exists.
Similarly to Theorem 1 (Salehi et al., 2019), this follows immediately when noting that
M (cid:96) ( x, µ ) ≤ (cid:96) ( x ) = log(1 + exp( − x )) ≤ log(2)
+ | x | .
Finally, we obtain the desired optimization problem in Equation (10) when combining these results.
[-This follows when using similar arguments as in Lemma A.
-]
Convergences Finally, note that the optimum δ (cid:63) in Equation (18) satisﬁes δ (cid:63) = 1 √ d (cid:107) Π ⊥ θ (cid:107) , and similarly the optima ν (cid:63) ⊥ and ν (cid:63) (cid:107) in Equation (19) satisfy ν (cid:63)
⊥ = (cid:107) Π
⊥ θ (cid:107) 2 and ν (cid:63) (cid:107) =
(cid:104) θ, θ (cid:63) (cid:105) .
Hence we can conclude the proof as the solutions (10), (14) concentrate asymptotically as d, n → ∞ around the same optima.
D.4. Proof of Theorem D.
The proof is similar to the proof of Theorem 6.3 (Javanmard & Soltanolkotabi, 2020).
Recall the robust max-margin solution from Equation (2): min
[Equation]
[-(cid:107) 
-]22 such that (cid:104) θ,
[Equation]
for all i
[Equation] After introducing the Lagrange multipliers ζ and u we can write equivalently:
[Equation]
23) and separating again X = XΠ ⊥ + XΠ (cid:107) , we get
[Equation]
Convex Gaussian Minimax Theorem Since the adversarial attacks are consistent and the observations are noiseless, we know the solution in Equation (22) exists for all d, n .
Yet, in order to apply the CGMT, we have to show that we can restrict u and θ to compact sets.
This follows from a simple trick as explained in Section D.3.1 in (Javanmard & Soltanolkotabi, 2020).
Hence, the primal optimization problem from Equation (24) can be asymptotically replaced with the following auxiliary optimization:
[Equation]
Scalarization of the optimization problem The goal is again to scalarize the optimization problem.
As a ﬁrst step, we can solve the optimization over u when deﬁning r = (cid:107)
[Equation]
n :
[Equation]
where max is taken elementwise over the vector.
We now want to separate (cid:107) Π ⊥ θ (cid:107) 2 from the term in max( . ) .
This is achieved by introducing the variable ν ⊥ ≥ 0 and the Lagrange multiplier κ .
Further [Equation] we set ν (cid:107) =
(cid:104)
θ (cid:63) , Π (cid:107) θ (cid:105) , which allows us to equivalently write
[Equation]
We further note that
[Equation]
Next
[Equation]
by completion of the squares we get
[Equation]
with (cid:107) g/ √ d (cid:107) 22 → 1 .
Hence
[Equation]
it only remains to solve the inner optimization over Π ⊥ θ .
We can write:
[Equation]
where (cid:96) H is the Huber loss, given by (cid:96) H ( x, y ) =
[Equation] [Equation]
y
[Equation]
| x | − 0 .
5 y ) .
Finally, we can conclude the proof from
[Equation]
Convergence We note again that the optimum δ (cid:63) in Equation (23) satisﬁes δ (cid:63) = 1 √ d (cid:107) Π ⊥ θ (cid:107) , and similarly the optima ν (cid:63) ⊥ and ν (cid:63) (cid:107) in Equation (26) satisfy ν (cid:63)
⊥ = (cid:107) Π
⊥ θ (cid:107) 2 and ν (cid:63) (cid:107) =
(cid:104) θ, θ (cid:63) (cid:105) .
Hence we can conclude the proof as the solutions (24), (25) concentrate asymptotically as d, n → ∞ around the same optima.