Tikhonov Regularization is Optimal Transport Robust under Martingale Constraints
Anonymous Author(s)
AffiliationAddress email
Abstract
Distributionally robust optimization has been shown to offer a principled way toregularize learning models.
In this paper, we find that Tikhonov regularization isdistributionally robust in an optimal transport sense (i.e., if an adversary choosesdistributions in a suitable optimal transport neighborhood of the empirical measure),provided that suitable martingale constraints are also imposed.
Further, we introduce a relaxation of the martingale constraints which not only provides a unifiedviewpoint to a class of existing robust methods but also leads to new regularizationtools.
To realize these novel tools, tractable computational algorithms are proposed.
As a byproduct, the strong duality theorem proved in this paper can be potentiallyapplied to other problems of independent interest
Introduction
Regularization is an important tool in machine learning which is used in, for instance, reducingoverfitting [23].
Recently, ideas from distributionally robust optimization (DRO) have led to a freshviewpoint on regularization precisely in connection to overfitting; see, e.g., [7, 2, 20, 10, 29, 24, 25, 5]and the references therein.
In these references it is shown that many standard regularization-based estimators arise as the solutionof a min-max game in which one wishes to minimize a loss over a class of parameters againstan adversary that maximizes the out-of-distribution impact of any given parameter choice, thatis, the adversary perturbs the empirical distribution in a certain way.
The choice of adversarialdistributions or perturbations in DRO is often non-parametric thus providing reassurance that thedecision is reasonably robust to a wide range of out-of-distribution perturbations.
For example,one such non-parametric choice is given by employing optimal transport costs [31] to construct aso-called distributional uncertainty set (e.g. a Wasserstein ball around the empirical distribution) forthe adversary to choose.
[27] shows that optimal transport-based DRO (OT-DRO) is closely relatedto adversarial robustness in the sense of steepest gradient loss contamination.
This can be furtherexplained by OT-DRO’s hidden connection with generalized Lipschitz regularization [7].
Thus,understanding if a well-known regularization technique is actually distributionally robust and in whatsense, allows us to understand its out-of-distribution benefits and potentially introduce improvements.
In this paper, we introduce a novel set of regularization techniques which incorporate martingaleconstraints into the OT-DRO framework.
Our starting point is the conventional OT-DRO formulation.
The conventional OT-DRO formulation can generally be interpreted as perturbing each data pointin such a way that the average size perturbation is less than a given budget.
In addition to thisconventional formulation, we will impose a martingale constraint in the joint distribution of theempirical data and the resulting adversarially perturbed data.
Why do we believe that the martingale constraint makes sense as a regularization technique?
It turnsout that two random variables X
and ¯ X form a martingale in the sense that E [ ¯ X | X ] =
X
if and only if the distribution of ¯ X dominates X in convex order [30].
In this sense, the adversary ¯ X will havehigher dispersion in non-parametric sense than the observed data X but in a suitably constrainedway so that the average locations are preserved.
This novel OT-DRO constrained regularization, webelieve, is helpful to potentially combat conservative solutions, see [16].
Moreover, by allowing asmall amount of violation in the martingale property, we can control the regularization properties ofthis constraint, thus obtaining a natural interpolation towards the conventional OT-DRO formulationand potentially improved regularization performance.
We point out that related optimal transportproblems with martingale constraints have been studied in robust mathematical finance [1, 8].
Consider, for example, the linear regression setting with the exact martingale constraints, whichmeans that for any given observed data point, the conditional expectation of the additive perturbationunder the worst-case joint distribution equals zero.
Surprisingly, we show that the resulting martingale
DRO model is exactly equivalent to the ridge regression [18] with the Tikhonov regularization.
Tothe best of our knowledge, this paper is the first work to interpret the Tikhonov regularization froma DRO perspective showing that it is distributionally robust in a precise non-parametric sense.
Instark contrast, it is well-known that the conventional OT-based DRO model ( without the martingaleconstraint) is identical to the regularized square-root regression problem [2].
Therefore, introducingan additional power in norm regularization (i.e., converting square-root regression to Tikhonovregularization) can be translated into adding martingale constraints in the adversarial perturbationsthus reducing the adversarial power.
A natural question that arises here is whether we can interpolatebetween the conventional DRO model and the Tikhonov regularization, and further improve them.
We will provide a comprehensive and positive answer to the above question in this paper.
The key ideahere is to relax the equality constraint on the conditional expectation of the adversarial violation
andthus allow a small perturbation of the martingale property to gain more flexibility of the uncertaintyset.
This idea leads to another novel model, termed the perturbed martingale DRO in the sequel.
Intuitively, if the relaxation is sufficiently loose, the perturbed martingale DRO model will reduce tothe conventional DRO, which is formally equivalent to setting an infinite amount of possible violationsfor the martingale constraint.
By contrast, if no violation is allowed, the perturbed martingale DROwill automatically reduce to the exact counterpart — Tikhonov regularization.
As a result, we areable to introduce a new class of regularizers via the interpolation between the conventional DROmodel and the Tikhonov regularization.
Furthermore, such insightful interpolation also works for a broad class of nonlinear learning models.
Inspired by our extensive exploration of linear regression, the developed martingale DRO model canalso provide a new principled adversarial training procedure for deep neural networks.
Extensiveexperiments are conducted to demonstrate the effectiveness of the proposed perturbed martingale
DRO model for both linear regression and deep neural network training under the adversarial setting.
We summarize our main contributions as below:
• We reveal a new hidden connection in this paper, that is, Tikhonov regularization is optimaltransport robust when exact martingale constraints (
i
e., convex order between adversary andempirical data) are imposed.• Upon this surprising finding, we develop a new perturbed martingale DRO model, which not onlyprovides a unified viewpoint of existing regularization techniques, but also leads to a new class ofrobust regularizers.•
We introduce an easy-to-implement computational approach to realize the model benefits inpractice, for both linear regression and neural network training under the adversarial setting.•
As a byproduct, the strong duality theorem proved in this paper, acting as the main technical tool,can be potentially applied to other problems of independent interest.
Preliminaries
Let us introduce some basic definitions and concepts preparing for the later analysis.
Definition 2.1 (Optimal transport costs and the Wasserstein distance [21, 31]) .
Suppose that c ( · , · ) : d × R d → [0, ∞ ] is a lower semi-continuous cost function such that c ( X , X ) = 0 for every X ∈ R d .
The optimal transport cost between two distributions Q and P supported on R d is defined as
[Equation]
Here, P ( X × X ) is the set of joint probability distribution π of ( ¯ X , X ) supported on X × X while
P 1 π and P 2 π respectively refer to the marginals of ¯ X and X under the joint distribution π .
If c ( X , ¯ X ) =
∥
¯ X − X ∥ is any given norm on R d , then D recovers the Wasserstein distance [31].
Inthis paper, we are interested in a flexible family of functions for the computational tractability, socalled the Mahalanobis cost functions in the form of c ( ¯ X , X ) =
( X − ¯ X )
⊤ M
( X − ¯ X ), where Mis a d -by- d positive definite matrix.
Next, we consider the conventional OT-DRO problem:
[Equation]
, where L β ( (cid:98) P , ρ ) ≜
[Equation]
where (cid:98) P ≜ 1 N (cid:80) Ni
=
δ X i is the empirical distribution.
Using Definition 2.1, we have L β ( (cid:98) P , ρ ) =
max Q : D ( Q , (cid:98) P ) ≤ ρ E Q [ ℓ ( f β ( ¯ X ))] , which is the worst-case expected loss under all possible distributionsaround the empirical measure (cid:98) P at most ρ with respect to the OT distance.
It is well-known thatunder appropriate assumptions, the DRO problem (2.1) is equivalent to the regularized square-rootregression problem.
Proposition 2.2 ([2, Proposition 2.]) .
Suppose that (i) the loss function ℓ ( · ) is a convex quadraticfunction, i.e., ∇ 2 ℓ ( · ) = γ > 0 , where γ is a constant, (ii) the feature mapping f β ( ¯ X ) =
β
⊤ ¯ X islinear, and (iii) the ground cost c is the squared Euclidean norm on X = R d .
Then
[Equation]
We also present the strong duality result for a general class of optimal transport based DRO modelswith martingale constraints.
This result serves as our main technical tool for reformulating the DROmodels, and it can also be applied to other semi-infinity structured DRO models, which could be ofindependent interests.
We consider the primal problem
[Equation]
and its associated dual form
[Equation]
Here, f : X → R is upper semi-continuous and L 1 -integrable.
The next theorem states the strongduality result linking these two problems.
Theorem 2.3 (Strong duality) .
Let (cid:98) P ≜ 1 N (cid:80) i ∈
[ N ] δ X i be the reference measure.
Suppose that (i)every sample point is in the interior of the cone generated by X
, i.e., X i ∈ int ( cone ( X )) ∀ i ∈
[ N ] ,and (ii) the ambiguity radius ρ > 0 .
Then the strong duality holds, i.e., Val(Primal) = Val(Dual) .
Remark 2.4.
At the heart of our analysis tools is the abstract semi-infinite duality theory for coniclinear program [26, Proposition 3.4].
Notably, it will be tricky and subtle to reformulate our probleminto the standard form and further carefully check the general Slater condition.
Moreover, we indeedfill the technical gap in [19, Theorem 4.2].
Notation.
We use S d ++ to denote the set of d -by- d positive definite matrices and ∥ X ∥ M ≜ √ X ⊤ MXfor any X ∈ R d , M ∈ S d ++ ; if M is the identity matrix, then we omit M and write ∥ X ∥ ≜ 118 √
X ⊤ X .
We use δ X to denote the Dirac measure at X and let (cid:98) P ≜ 1 N (cid:80) Ni =
δ X i be the empiricalmeasure constructed from sample { X 1 , . . .
, X N } .
We use E P to denote the integration over P : 120 E
P [ f ( X )]
= (cid:82)
X
f
( X )d P .
Specifically, for ( ¯ X , X ) following joint distribution π , E π
[ c ( ¯ X , X )] =
(cid:82) X×X c ( ¯ X , X )d π , E π
[ f ( ¯ X )] = (cid:82) X×X f
( ¯ X )d π .
We use ℓ ( · ) to denote the loss function applied tothe parametrized feature mapping f β .
Let ∇ ℓ ( · ), ∇ 2 ℓ ( · ) be the first and second order derivative ofℓ ( · ) respectively.
Notably, we use L β ( (cid:98) P , ρ ) to denote the objective function of the conventional DROmodel (2.1); we use L β ( (cid:98) P , ρ ) to refer to the exact martingale DRO model (3.1); we use L β ( (cid:98) P , ρ , ϵ )to refer to the perturbed martingale DRO model (3.2).
Tractable Reformulations
In this section, we introduce an optimal transport-based DRO model with the exact martingaleconstraint at first.
That is, on top of the vanilla DRO model [3], we add an additional martingaleequality constraint on its coupling.
It is surprisingly interesting to find out that the resulting DROapproach is equivalent to empirical risk minimization with Tikhonov regularization.
Naturally, wecan relax the equality constraint and thus allow a small violation of the martingale property to enrichthe uncertainty set.
Formally, by sending violation size to infinity in the martingale constraint, ourrelaxation allows to interpolate between the conventional DRO formulation (i.e. with no martingaleconstraints) and Tikhonov regularization (which involves exact martingale constraints).
Therefore,this relaxation further leads to a new class of regularizers in a principled way, which improves upon
Tikhonov regularization as we show in our experiments.
Assumption 3.1.
The following assumptions hold throughout.
(i)
The ground cost c ( · , · ) is the Mahalanobis cost with the weighting matrix
M ∈ S d ++ ,
(ii)
The domain X is unconstrained, i.e., X = R d .
3.1 Optimal Transport-based DRO
with Martingale Constraints
To start with, we investigate the exact martingale DRO problem:min β L β ( (cid:98) P , ρ ), where L β ( (cid:98) P , ρ ) ≜   sup π
E π [ ℓ ( f β ( ¯ X ))] s .
t .
π ∈ P ( X × X )
E π (cid:2) c ( ¯ X , X ) (cid:3) ≤ ρ , P 2 π = (cid:98) P
E π
[ ¯ X
[Equation] [Equation]
(cid:98) P -a.s.
, (3.1) and ρ ≥ 0 is the radius of uncertainty set centered at (cid:98) P .
Note that because (cid:98) P is the empirical measure,the martingale constraint implies that the conditional expected value of the perturbation obtainedby modifying each observed data point equals the observed data point itself.
The quantity L β ( (cid:98) P , ρ )is referred to as the worst-case expected loss of the model parameter β under the martingale DROmodel.
It is easy to see that L β ( (cid:98) P , ρ ) ≤ L β ( (cid:98) P , ρ ) , where L β is defined as in (2.1).
This is becausethe adversary in (3.1) has a smaller feasible set, and thus is less powerful than the adversary in (2.1).
Hence, the martingale DRO solution for problem (3.1) is considered to be less conservative than theconventional DRO solution for problem (2.1).
The next result asserts that the martingale DRO problem coincides with the Tikhonov regularizationproblem under similar conditions of Proposition 2.2.
Proposition 3.2 (Tikhonov equivalence) .
Suppose that (i) the loss function ℓ ( · ) is a convex quadraticfunction, i.e., ∇ 2 ℓ ( · ) = γ > 0 , and (ii) the feature mapping f β ( ¯ X ) =
β
⊤ ¯ X is linear.
Then we have
[Equation]
If the Mahalanobis matrix M is the identity matrix, the martingale DRO model (3.1) recovers the
Tikhonov regularization problem.
Proof of Proposition 3.2.
By a change of the variable, let ∆ =
¯ X − X
and we have
[Equation] [Equation]
The last equality follows from the general Hölder’s inequality.
To achieve the equality, we can, forexample, take a normally distributed random variable C with mean 0 and variance ρ and which isindependent of X , and then let ∆ = CM − 1 β .
Example 3.3 (Linear regression) .
Let X ⊤ ≜
( Y , Z ⊤ ) ∈ R d and β ⊤ ≜ (1, − b ⊤ ) ∈ R d , we haveβ ⊤ X
= Y − b ⊤ Z .
For any Q ∈ S d − 1 ++ , we take M = diag (+ ∞ , Q ) , which implies that we do notallow transport of the response Y , then the problem (3.1) with γ = 2 becomes
[Equation]
In Appendix C.1, we give another instructive proof based on the strong duality result in Section 2.
For general convex loss functions, we have the following certificate of robustness that provides upperand lower bound for the worst-case DRO loss in (3.1).
Corollary 3.4 (General convex loss functions) .
Suppose that (i) the loss function ℓ ( · ) is µ -stronglyconvex and C -smooth, that is, ℓ ( θ ) ≥ ℓ ( θ ′ )
+ ∇ ℓ ( θ ′ )
( θ ′ − θ ) + µ 2 ( θ ′ − θ ) 2 and ℓ ( θ ) ≤ ℓ ( θ ′ )
+∇ ℓ ( θ ′ )
( θ ′ − θ )
+ C 2 ( θ ′ − θ ) 2 hold for all θ , θ ′ ∈ R , and (ii) the feature mapping f β ( ¯ X ) =
β ⊤ ¯ X islinear.
Then we have for any ρ ≥ 0 ,
[Equation]
Example 3.5 (Logistic regression) .
Let X
=
Y Z ∈ R d , where Z ∈ R d , Y ∈ {± 1 } , and ℓ ( t )
=
log(1 + exp( − t )) , where ℓ ( · ) satisfies Assumption (i) in Corollary 3.4 with C
= 14 .
Then we have
[Equation]
3.2 Optimal Transport-based DRO with Perturbed Martingale Constraints
Now we turn to the relaxation of the martingale constraint to improve upon Tikhonov regularizationand gain more flexibility.
We consider the perturbed martingale coupling based DRO model (perturbedmartingale DRO):
[Equation]
where L β ( (cid:98) P , ρ , ϵ ) ≜
[Equation]
The parameter ϵ controls the allowed violations of the martingale constraint for the adversary.
Itis trivial that if we set ϵ = 0 , then we obtain L β ( (cid:98) P , ρ , 0) =
L β ( (cid:98) P , ρ ) and we recover the exactmartingale DRO model (3.1).
If we set ϵ = + ∞ then the martingale constraint becomes ineffective,thus we have L β ( (cid:98) P , ρ , + ∞ ) =
L β ( (cid:98) P , ρ ) and model (3.2) collapses to the DRO formulation (2.1).
We thus can think of ϵ as an interpolating parameter connecting two extremes: the conventional DROmodel (2.1) (at ϵ = + ∞ ) and the exact martingale DRO model (3.1) (at ϵ = 0 ).
However, the resulting optimization problem (3.2) constitutes an infinite-dimensional optimizationproblem over probability distributions and thus appears to be computationally intractable.
Toovercome this issue, we leverage Theorem 2.3 and prove that the problem (3.2) is actually equivalentto a finite-dimensional problem.
To begin with, we present one crucial proposition that can be appliedto more general settings.
Proposition 3.6 (General loss functions and feature mappings) .
Suppose that the loss function ℓ ( · ) 188 and the feature mapping f β ( · ) are upper semi-continuous.
Then, for any ρ > 0 and ϵ > 0 , the 189 perturbed martingale DRO model (3.2) admits
[Equation]
Sketch of proof.
The key step is to decouple (3.2) as a two-layer optimization problem:
[Equation]
Then we invoke Theorem 2.3 for the inner maximization problem over π and apply the Sion’sminimax theorem [28] for the outer maximization over η .
The desired result is obtained.
Due to the specific structure of the quadratic cost and the linear feature mapping, the dual variableλ admits a closed-form representation and α ∈ R N × d in (3.3) can be reduced to N parallel onedimensional optimization problems.
Upon this observation, we thus conduct an instructive andintuitive reformulation in Theorem 3.7 for linear regression with proof detailed in Appendix C.2.
Theorem 3.7 (Linear regression) .
Suppose that (i) the loss function ℓ ( · ) is a convex quadraticfunction, i.e., ∇ 2 ℓ ( · ) = γ > 0 and (ii) the feature mapping f β ( ¯ X ) =
β
⊤ ¯ X is linear.
Then, theperturbed martingale DRO model (3.2) admits:
[Equation]
where the additional regularizer is defined as
[Equation]
and G β = ( ∇ ℓ
[Equation]
Obviously, compared with the exact martingale constraint investigated in Proposition 3.2, the perturbed constraint we consider here involves an additional term R ( β ) .
Fortunately, armed with Lemma
D.3, we are able to shed light on its intuitive interpretation based on the quantitative relationshipbetween modeling parameters ϵ and ρ .
First, we discuss two extreme cases — there are interestinghidden connections between (3.2) and other existing regularization techniques.
Remark 3.8.
Intuitively, if ϵ is relatively small, the optimal s ⋆ is zero which means that all perturbedmartingale constraints will be active.
Precisely, if ϵ 2 ≤ ρ , the additional regularization term satisfies
R ( β ) =
ϵ E (cid:98) P [ ∥∇ X ℓ ( f β ( X ))
∥ M − 1 ] , which is so-called Jacobian or input gradient regularizer in theliterature when M is an identity matrix.
Recently, it has received much intention owing to its abilityto improve adversarial robustness [32, 9].
Conversely, if ρ is relatively small, the optimal s ⋆ is equalto G β , implying that none of perturbed martingale constraints is active.
In fact, if ϵ 2 ≥ Nρ , wehave R ( β ) =
(cid:113) ρ E (cid:98) P [ ∥∇ X ℓ ( f β ( X ))
∥ 2 M − 1 ] .
Then, we can easily figure out that L β ( (cid:98) P , ρ , ϵ ) can bereduced to the conventional OT-based DRO model in Proposition 2.2.
Then for the middle case, it is natural to infer that only part of constraints will be active.
In Lemma
D.3, we justify this conjecture rigorously.
We refer the reader to Appendix D for the details.
As such,the proposed martingale DRO model takes the first step bridging the input gradient regularizationand regularized square-root regression problem in a unified framework.
On the other hand, it alsoopens up an exciting brand new avenue of robustified regularizers.
In the next section, we validate itseffectiveness for the adversarial training task.
Optimization Algorithms
To take advantage of the proposed perturbed martingale DRO model, a natural question here iswhether we can address problem (3.2) in a tractable manner.
In this section, we answer the abovequestion in the affirmative by developing two different computational paradigms for linear regressionand deep neural network respectively.
4.1 Subgradient Method for Linear Regression
Unfortunately, the resulting formulation we conducted in Theorem 3.7 (i.e., (3.5)) for linear regressionis potentially intractable for optimizing ( β , s ) jointly.
One obvious computational challenge here isthat the overall problem is not necessarily convex in ( β , s ) , even if the original problem is convex overβ .
This is because the minima of convex functions is not convex.
However, under a mild assumption,it is possible to give a reliable computational routine for solving the resulting problem over β directly.
To start with, a key observation is L β ( (cid:98) P , ρ , ϵ ) is convex over β .
This is essentially from the factthat the pointwise supremum of a class of convex functions is still convex.
To check the details,
[Figure]
L β ( (cid:98) P , ρ , ϵ ) is originally defined in (3.2) and E π [ ℓ ( f β ( ¯ X ))] is convex over β for all π .
Thus, a naturalyet simple algorithm is subgradient method.
The main difficulty is to obtain the correct subgradientoracle.
Since the first two terms are smooth and strongly convex, we know that R ( β ) is weaklyconvex and thus subdifferentially regular.
All the subdifferential concepts are coincide.
As such,we may simply use the Clarke subdifferential [6] in the sequel.
Moreover, the sum rule is hold forcomputing the subgradient of L β ( (cid:98) P , ρ , ϵ ) due to the weakly convexity.
We have∂
[Equation]
The remaining question is how to compute the Clarke subdifferential of R ( β ) .
When ϵ 2 ≤ ρ andϵ 2 ≥
Nρ , R ( β ) will enjoy the convex composite structure.
Thus, we can get the correct subgradientby invoking the chain rule developed in [22, Theorem 10.6] directly.
The more subtle and tricky caseis the middle one — ρ < ϵ 2 <
Nρ .
Without of loss generality, we assume that M = I for simplicity.
Based on [6, Theorem 2.3.9], we have
[Equation]

[Equation]
where S ( β ) is the optimal solution set and Conv {·} denotes the convex hull.
If we assume theinclusion here is tight, then the vanilla subgradient method will converge to the optimal solution withthe rate O (1 / √ K ) [4].
Empirically, we find out that the resulting subgradient method works welland the violated case will never happen.
4.2 A New Principled Adversarial Training Procedure for Deep Learning
In this subsection, we develop inexact stochastic gradient-type methods for (3.3) and thus are able torealized the benefits of the proposed DRO model in adversarial learning tasks.
As the nonconvexityof f β ( · ) , the inner maximization problem over ∆ i will be no longer tractable.
Therefore, we leveragethe methodology proposed in [27] to gain the computational efficiency, that is, regarding the dualvariable as a modeling parameter.
To proceed, we make the same smoothness assumption in [27,
Assumption B] (i.e., see Assumption E.1 in appendix for details).
Lemma 4.1 (Convex-concave minimax theorem without compactness) .
Suppose that f : R n → R isconvex and level bounded and g : R m → R is strongly convex.
Then we have
[Equation]
Leveraging Lemma 4.1 and the smoothness assumption, (3.3) leads to a simple and instructive form:
[Equation]
In contrast to the vanilla DRO model studied in [27], (4.1) further constrains the perturbation into a
Euclidean ball with the correlation information M .
Moreover, we can observe that the magnitude of ϵdecides how many martingale constraints will be active, which also perfectly matches our theoreticalresults and interpretations established for linear regression, see Theorem 3.7 and Remark 3.8. [Figure]
martingale DRO leaves bigger margins from than other methods.
From a computational viewpoint, if λ is large enough (see Lemma E.2 for details), the inner maximization problem is strongly concave and thus the outer minimization problem over β will besmooth.
This motivates Algorithm 1, an inexact stochastic gradient method for Problem (4.1).
Theconvergence guarantee is provided in [27, Theorem 2].
It is worthwhile mentioning that the resultingnew principled adversarial training is extremely easy to implement by only adding three lines of
Pytorch
Algorithm 1: Martingale Distributionally Robust Optimization with Adversarial Training
Input : Sampling distribution (cid:98) P , stepsize sequence { t k } K − 1 k =0 ; for k = 0, 1, 2, · · · , K − 1 do
Sample X k ∼ (cid:98) P and find an η -approximate maximizer ˆ∆
k satisfying ∥ ˆ∆
k − ∆ ⋆k ∥ ≤ η , where ∆ ⋆k = arg max ∥ ∆ ∥ M ≤ ϵ (cid:8) ℓ ( f β k ( X k + ∆)) − λ ∥
∆ ∥ 2 M (cid:9) .
Set β k +1 ← β k − t k ∇ β ℓ ( f β k ( X k + ˆ∆ k )).
Numerical Results
In this section, we validate the effectiveness of our methods (referred to as martingale DRO ) onboth linear regression and deep neural networks under the adversarial setting.
All simulations areimplemented using Python 3.8 on: (1) a computer running Windows 10 with a 2.80GHz, Intel(R)
Core(TM) i7-1165G7 processor and 16 GB of RAM, and (2) Google Colab with NVIDIA Tesla
P100 GPU and 16 GB of RAM.
As for the adversarial setting, we consider three types of attack, thedetailed definitions of which are collected in Appendix F.
5.1 Linear Regression
To start with, we demonstrate the effectiveness of the proposed martingale DRO model (3.5) withthe quadratic loss function and linear feature mapping, i.e., ℓ ( f β ( X ))
=
12 ( Y − b ⊤ Z ) 2 with X ⊤ ≜( Y , Z ⊤ ) and β ⊤ ≜ (1, − b ⊤ ) , where Z is the feature vector and Y is the target variable.
In thisexperiment, we test our method on three LIBSVM regression real world datasets 1 .
More specifically,we randomly select 60% of the data to train the models and the rest as our test data.
To showcasethe effectiveness of martingale DRO model under adversarial setting, we apply one-step projectedgradient method (PGM) attack [17] on test data and report the performance in terms of the theroot-mean-square error (RMSE) on adversarial test data, where RMSE ≜ (cid:113) 1 N (cid:80) i ( ˆ β ⊤ x ( i ) adv − y ( i ) adv )
2and ˆ β is the estimator of β .
All numerical results with different step sizes for PGM attack are collected in Figure 1.
As we mentioned in Proposition 3.2, since the exact martinagle DRO model(i.e. , ϵ = 0 ) is equivalent to Tikhonov regularization, we choose the same hyperparameter ρ = 0.08for ridge regression and martingale DRO mode for fair comparison.
We can easily observe that themartingale DRO can beat other two benchmarks over three real world dataset consistently when thestep size for PGD attack is relatively large.
This result also corroborates our theoretical intuition —the additional regularization R ( β ) can further improve the adversarial robustness.
5.2 Deep Neural Network for Adversarial Training
To begin with, we generate the synthetic training data { ( Y i , Z i ) } i ∈I with a wide margin as follows:generate i.i.d.
Z j ∼ N (0, I 2 ) , where Z ∈ R 2 , I 2 is the identity matrix in R 2 ; set I
= {
j : ∥
Z j ∥ 2 / ∈( √ 2 /η , η √ 2) } , where η = 1.6 ; let Y i
= sign( ∥ Z i ∥ 2 − √ 2), ∀ i ∈ I .
We train a neural network with3 hidden layers of size 4, 3 and 2 and ELU activations between layers.
We compare our approach (cf.martingale DRO) with ERM and the conventional DRO approach developed in [27].
More detailsabout the experiment setup are collected in Appendix F.
Experimental results on synthetic data are shown in Figure 2.
Test data are shown in darkgray andgray with different shapes, which are generated by the previous procedure with a smaller margin( η = 1.2 ).
Classification boundaries are shown in blue, green, and orange for ERM, DRO, andmartingale DRO respectively, as well as with the true class boundaries of the test data.
Intuitively, theboundary generated by ERM is too close to the true inner boundary since the majority of points areof darkgray class, while the DRO approach pushes the classification boundary outwards.
However,as illustrated in Figure 2(a), the DRO approach suffers from over-conservativeness and becomesentangled with the boundary of the outer gray class.
In contrast, our martingale DRO boundarylands in between the two extremes and it leaves bigger margin.
Figure 2(b) explicitly shows thequalitative difference between these two methods in perturbation, where the Martingale perturbationis constrained below ϵ while the DRO perturbation is unconstrained.
Moreover, as shown in theprevious sections, decreasing non-zero ϵ ’s push the perturbed martingale constraints towards theexact martingale constraints and force the classification boundary increasingly inward.
More resultsare in Appendix F.
Then, we validate our method on the MNIST dataset [15].
For the classifier, we train a neural networkequipped with 8 × 8 , 6 × 6 , 5 × 5 convolutional filter layers and ELU activations followed by a fullyconnected layer and softmax output.
To show the robustness of our method, we test the performanceof four methods (ERM, DRO, Jacobian regularization [13] and martingale DRO) under the PGD and
FGSM attacks (Definition F.1) with test error defined to be: 1 − classification accuracy .
In Figures 3a and 3b, our martingale DRO model outperforms the other methods and still providerobustness under the ∞ -norm FGSM attacks.
In Figure 3c, we show the performance of our modelwith different ϵ ’s to show the sensitivity to parameter ϵ .
As expected, when ϵ is relatively small, themodel is not flexible enough and shows large test error.
Alternatively, as ϵ becomes large, our modelwill behave like the original DRO model since the ϵ -constraint is almost invalid in this case.
Figure 4 visualizes the different levels of robustness for the four methods.
For each test data point,we perturb the image using the DRO attacks (Definition F.2) with decaying level of perturbation andrespectively record the first perturbed images that each model correctly classifies.
In Figure 4, theoriginal label is 6 and all methods output the correct prediction, whereas in the adversarial examplethat the DRO model predicts 6 , the correct classification seems unreasonable to human eyes (see
Appendix F for more examples).
This observation shows an insight that the original DRO model istoo conservative in predicting and our model puts more constraints on the perturbation when trainingthus providing a model that is more consistent to human eyes.
Closing Remarks
In this paper, we find that the OT-based DRO model is equivalent to Tikhonov regularization whenexact martingale constraints are imposed.
Upon this interesting hidden connection, we introduce anew model called the perturbed martingale DRO model, which not only provides a unified viewpointto several common robust methods but also leads to new regularization tools.
Empirically, we validatethe effectiveness of our model in addressing the conservativeness issue for the conventional DROmodel.
From the statistical perspective, how to optimally select the size of uncertainty regions andthe perturbation size of the martingale constraint, is a natural problem to be further explored. [Figure]
Predict Predict
Predict Predict
Predict Predict
Predict Predict[-
-]
Predict Predict
Predict[-
-] Predict
[Figure]
Predict Predict
Predict Predict
Predict Predict
Predict Predict
Original
ERM
Jacobian Regularization
DRO
Martingale DRO
Perturbations on a test datapoint [Figure] Predict Predict
Predict
{+Predict 
+}
Predict
Predict
Predict
Predict
Predict
Predict
Predict
Predict [Figure] Predict Predict
Predict Predict
Predict[-
 Predict Predict Predict-]
Predict[-
 Predict Predict Predict-]
[-Predict 
-]
Predict
Predict
[-Predict Predict Predict 
 Predict-]Original
[-Predict Predict Predict 
 Predict-]ERM
[-Predict 
-]
Jacobian Regularization
DRO
Martingale DRO
Perturbations on a test datapoint (b) ERM
Predict[-
 Predict Predict Predict-]
Predict[-
 Predict Predict Predict-]
Predict[-
 Predict Predict Predict-]
Predict {+
+}
Predict Predict Predict Predict
Predict Predict Predict Predict
Predict Predict Predict Predict
Predict
Predict
Predict
Predict
Predict
Predict
Predict
Predict
Predict
Predict
Predict
Predict Original ERM Jacobian Regularization
DRO
Martingale DRO
Perturbations on a test datapoint (c) Jacobian Regularization
Predict
Predict
Predict
{+Predict 
+}
Predict Predict Predict Predict
Predict Predict [-
-]Predict
Predict Predict Predict
Predict Predict [-
-]Predict
Predict Predict Predict
Predict Predict [-
-]Predict
Predict Predict Predict
Predict Predict
Original ERM Jacobian Regularization DRO
Martingale DRO Perturbations on a test datapoint (d) DRO
Predict Predict
Predict {+Predict 
 Predict Predict+}
Predict Predict Predict Predict
Predict Predict Predict Predict
Predict Predict Predict Predict
Predict Predict Predict Predict
Predict Predict Predict Predict
Predict Predict
Original
ERM
Jacobian Regularization
DRO
Martingale DRO
Perturbations on a test datapoint
[Figure]
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See section 3(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A](d)
Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]