Constrained Stochastic Nonconvex Optimization with State-dependent Markov Data
Abstract
We study stochastic optimization algorithms for constrained nonconvex stochastic optimization problems with Markovian data.
In particular, we focus on the case when the transition kernel of the Markov chain is state-dependent.
Such stochastic optimization problems arise in various machine learning problems including strategic classification and reinforcement learning.
For this problem, we study both projection-based and projection-free algorithms.
In both cases, we establish that the number of calls to the stochastic first-order oracle to obtain an appropriately defined ϵ -stationary point is of the order
O (1 /ϵ 2 . 5 ) .
In the projection-free setting we additionally establish that the number of calls to the linear minimization oracle is of order O
(1 /ϵ 5 . 5 ) .
We also empirically demonstrate the performance of our algorithm on the problem of strategic classification with neural networks
Introduction
We consider the following stochastic optimization problem
[Equation]
where (i) the expectation is taken over the stationary distribution, π θ , of the random vector x , (ii) F (and hence f ) is a potentially non-convex function in θ , and (iii) Θ is a compact and convex constraint set.
Stochastic approximation algorithms for solving problem (1), given an independent and identically distributed ( iid ) data stream { x k } k drawn from π , are well-studied.
Such iid assumptions are commonly made in various machine learning and statistical problems including empirical risk minimization [SSBD14], sparse recovery [BJMO12] and compressed sensing [FR13, Lan20].
We refer to [MB11, ABRW12, RSS12, GL13, SZ13, LZ16, ACD + 19] for a partial list of non-asymptotic upper and lower bounds on the oracle complexity of widely-used stochastic approximation algorithms like the Stochastic Gradient Descent (SGD) and the Stochastic Conditional Gradient Algorithm.
Our focus in this work is on the case when the data sequence { x k } k is drawn from a Markov chain with a state-dependent transition kernel P θ .
Such a setting arises in several machine learning applications including but not limited to strategic classification [HMPW16, CDP15, MDPZH20,
LW22] and reinforcement learning [Bar92, GSK13, ZJM21, KMMW19, QW20].
Despite their prevalence in practice, a deeper understanding of the non-asymptotic oracle complexity of stochastic approximation for Markovian data is only now starting to emerge.
We establish non-asymptotic oracle complexity results for the stochastic conditional gradient algorithm for non-convex constrained stochastic optimization with Markovian data.
To establish our results, from a methodological point-of-view, we leverage the moving-average stochastic gradient estimation technique recently used in [ZSM + 20, GRW20, XBG22] in the context of constrained optimization with iid data.
This technique avoids having to a use a mini-batch of samples in each iteration, which turns out to be crucial in the non- iid setup we consider.
From a theoretical point-of-view, we assume the so-called drift conditions, a classical assumption in Markov Chain literature [AMP05].
This ensures the existence of a solution to the Poisson equation associated with the underlying Markov chain [DMPS18] which enables one to decompose the noise present in the stochastic gradient into three components: a martingale difference sequence, a time-decaying sequence, and a telescopic sum type sequence.
The key idea of our paper is to use this decomposition to construct an auxiliary sequence of iterates with a time-decaying noise-variance and show that these sequence of iterates are close to the iterates of the original sequence produced by our algorithm.
This novel technique in then used in combination with a merit-function based analysis to establish the oracle complexity results.
1.1 Motivating Example
Problems of the form in (1) arise in various important applications, e.g., strategic classification, and reinforcement learning as mentioned above.
Below we illustrate the motivation of this work through the example of strategic classification with adapted best response [LW22].
In strategic classification, there is a learner whose task is to classify a given dataset which is collected from a set of agents .
Given the knowledge of the classifier, the agents can distort some of their personal features, in order to get classified in a predetermined target class.
This scenario arises in various applications, e.g., spam email filtering, and credit score classification.
Optimizing the classifier to classify such strategically modified data where the agents modify the data iteratively can be formulated as problem (1).
Formally, let the classifier be h ( x, θ ) where x ∈ R
d is the feature and θ is the parameter to be optimized.
h ( x ; · ) :
Θ → R is potentially nonconvex.
Let the loss function be logistic loss which for a sample ( x, y ) , where y ∈ {− 1 , 1 } denotes the corresponding class, is given by,
[Equation]
We use x S , and x − S to denote the subset of feature x which are respectively strategically modifiable, and non-modifiable by the agents.
Then the modified feature (the best response) x ′ S reported by the agent is the solution to the following optimization problem:
[Equation]
where c ( x, x ′ ) is the cost of modifying x S to x ′ S .
Let the agents iteratively learn x ′ S similar to [LW22].
Note that unlike [LW22], where the authors deploy a logistic regression classifier and the closed form solution of the best response is readily known to the agents, it may not be the case in general.
In that case the agents have to possibly learn the best response x ′ S using some iterative optimization algorithm.
For example, if the agents use Gradient Ascent then, at every iteration k , a set I k of n 1 ≤ M randomly chosen agents out of M agents modify their features as:
[Equation]
where α is the stepsize.
With a little abuse of notation, we use ∇ h ( x k − 1 S,i ; θ ) in (4) to denote the fact that the gradient is with respect to x k − 1 S,i while x − S,i remains unchanged.
This introduces the state-dependent Markov chain dynamics in the training data.
The objective function, analogous to f ( θ ) in (1), is
[Equation]
where π θ is the stationary joint distribution of ( x, y ) , and Θ is a convex and compact set, e.g., sparsity inducing constraint ∥ θ ∥ 1 ≤ R from some R > 0 .
The loss evaluated at a single data point ( x, y ) ,
L ( θ ; x, y ) , is analogous to F ( θ ; x ) in (1).
[DX20], and [LW22] study this problem theoretically and empirically respectively in an unconstrained strongly convex setting.
Our results takes a step towards analyzing this problem in constrained nonconvex setting.
We empirically show the performance of the stochastic conditional gradient algorithm on a strategic classification problem in Section 4.1.
1.2 Preliminaries and Main Contributions
Before we present our main contributions, we introduce our convergence criterion.
In constrained optimization literature, most commonly used convergence criteria are: (i) Gradient Mapping (GM), and (ii) Frank-Wolfe Gap (FW-gap).
The Gradient Mapping at a point ¯ θ ∈ Θ is defined as
[Equation]
where Π Θ ( x ) denotes the orthogonal projection of the vector x onto the set Θ , i.e.,
[Equation]
We will use Π Θ (¯ θ, ∇ f
(¯ θ ) , β ) to denote Π Θ (cid:0) ¯ θ − ∇ f
(¯ θ ) /β (cid:1) when there is no confusion.
Note that when Θ ≡ R d we have G Θ (¯ θ, ∇ f
(¯ θ ) , β ) = ∇ f
(¯ θ ) .
In other words, for constrained optimization gradient mapping plays an analogous role of the gradient for unconstrained optimization.
The gradient mapping is a frequently used measure in the literature as a convergence criterion for nonconvex constrained optimization [Nes18].
We should emphasize here that although the gradient mapping cannot be computed in the stochastic setting, one can still use it as a convergence measure.
[BG22] shows that the above notion of convergence criterion is closely related to the so-called
Frank-Wolfe Gap .
The FW-gap is defined as
[Equation]
The following proposition from [BG22] establishes the relation between the gradient mapping criterion and the Frank-Wolfe gap:
Proposition 1.1 [BG22] Let g Θ ( · ) be the Frank-Wolfe gap defined in (6) and G Θ ( · ) be the gradient mapping defined in (5) .
Then, we have
[Equation]
Moreover, under standard regularity assumption in smooth optimization (specifically, Assumption 2.1, and 2.2), we have
[Equation]
In this work we use a suboptimality measure, closely related to both GM and the FW-gap.
At point ¯ θ ∈ Θ , we define the suboptimality measure V (¯ θ, z ) :
R d × R d → R as [GRW20]
[Equation]
where z , formally defined in Algorithm 1, is the moving-average estimate of ∇ f (¯ θ ) .
We show the relation among V ( θ, z ) , and GM G Θ ( θ, z, β ) in the following proposition.
Proposition 1.2 Let { z k } be the sequence generated in Algorithm 1.
Then, for k = 1 , 2 , · · · , N , we have ∥G Θ ( θ k , z k , β ) ∥ 22 ≤ max(2 , 2 β 2 ) V ( θ k , z k ) .
The proof is provided in Appendix A.
The main objective of this work is to find an ϵ -stationary solution to (1), where an ϵ -stationary solution is defined as follows:
Definition 1 A point ¯ θ is said to be an ϵ -stationary solution to (1) , if E (cid:2) V (¯ θ, z ) (cid:3) ≤ ϵ , where the expectation is taken over all the randomness involved in the problem.
For stochastic Frank-Wolfe-type algorithms, the oracle complexity is measured in terms of number of calls to the Stochastic First-order Oracle (SFO) and the Linear Minimization Oracle (LMO) used to the solve the sub-problems of the algorithm which involves minimizing a linear function over the convex constraint set.
Formally, we have the following definition.
Definition
For a given point θ ∈ Θ , SFO returns the stochastic gradient ∇ F ( θ, x ) .
Given a vector z , LMO returns a vector v :=
argmin y ∈ Θ ⟨ z, y ⟩ .
[Table]
Hence, in this work, the oracle complexity is measured in terms of the number of calls to SFO and
LMO required by the proposed algorithm to obtain an ϵ -stationary solution as in Definition 1.
With the above preliminaries, we now list our main contributions :
• In Theorem 3.1, we show that the number of calls to the SFO and LMO required by the stochastic conditional gradient-type method in Algorithm 1, with state-dependent Markovian data, is of order O ( ϵ − 2 . 5 ) and O ( ϵ − 5 . 5 ) respectively.
To the best of our knowledge,
these
are the first oracle complexity results for projection-free one-sample stochastic optimization algorithm for constrained nonconvex optimization in the Markovian setting.
• In Theorem 3.2, for the sake of completion, we also show that the number of calls to the SFO and LMO required for the case of state-independent Markovian data is of the order ˜ O
( ϵ − 2 ) and ˜ O
( ϵ − 3 ) respectively.
In particular, this turns out to be of the same order as that of iid data ignoring the
logarithmic factors.
A summary of the our contributions is provided in Table 1.
We also empirically evaluate our algorithm on a strategic classification problem with 2-layer neural network classifier and show that the proposed method obtains encouraging results.
We provide an experiment on single-index model regression with sparsity-inducing nuclear-norm ball constraint in Appendix 4.2.
1.3 Related Work
Stochastic Optimization with Dependent Data.
Understanding stochastic approximation algorithms like SGD with dependent data in the asymptotic setting has been well-explored in the optimization literature.
We refer to [KY03, Bor09, BMP12] for a text-book introduction to such classical results.
A few recent results include [AMP05, TD17].
In the unconstrained non-asymptotic setting, [DAJJ12] studies convex optimization with ergodic data sequence.
[DL22] uses multi-level gradient estimator and analyze AdaGrad for nonconvex optimization with Markovian Data.
Block coordinate descent with homogeneous Markov chain has been analyzed in [SSXY20] for nonconvex unconstrained optimization.
[DX20] studies stochastic optimization with decision-dependent data distribution for strongly convex functions in the context of strategic classification.
Sample-average approximation algorithms for constrained convex optimization with ϕ -mixing data was considered in [WPT + 21].
[SSY18], and [AL22] analyze projected SGD for constrained nonconvex optimization with time-homogeneous Markov chain.
None of these works consider statedependent data distribution except [DX20].
But unlike [DX20], we consider constrained nonconvex optimization.
There also exists work in the reinforcement learning literature on understanding stochastic optimization with Markovian data; see, for example [XXLZ20, BRS18, DNPR20].
However, such works are invariably focused on specific objective functions arising in the reinforcement learning setup, while our focus is on obtaining results for a general class of functions.
Conditional Gradient-Type Method.
There has been significant recent advancements in the conditional gradient algorithm literature although it was developed long back [FW56, LP66]; see [Mig94, Jag13, LJJ15, LJJ15, HJN15, GKS21, BS17], for a non-exhaustive list of recent works.
[HK12, HL16] provided expected oracle complexity results for stochastic conditional gradient algorithm in the stochastic convex setup.
Better rates were provided by a sliding procedure in [LZ16].
In the non-convex setting, [RSPS16, YSC19, HL16] considered variance reduced stochastic conditional gradient algorithms, and provided expected oracle complexities.
[QLX18] analyzed the sliding algorithm in the non-convex setting and provided results for the gradient mapping criterion.
All of the above works use increasing orders of mini-batch based gradient-estimate.
To avoid mini-batches, a moving-average gradient estimator based on only one-sample in each iteration for a stochastic conditional gradient-type algorithm was proposed in [MHK20] and [ZSM + 20] for the convex and non-convex setting.
However, several restrictive assumptions have been made in
[MHK20] and [ZSM + 20].
Specifically, [ZSM + 20] requires that the stochastic gradient G 1 ( x, ξ 1 ) has uniformly bounded function value, gradient-norm, and Hessian spectral-norm, and the distribution of the random vector ξ 1 has an absolutely continuous density p such that the norm of the gradient of log p and spectral norm of the Hessian of log p has finite fourth and second-moments respectively.
In contrasts, we do not require such stringent assumptions.
Assumptions
We now introduce the precise assumptions we make in this work.
Let F k be the filtration generated by { θ 0 , · · · , θ k , z 0 , · · · , z k , x 1 , · · · , x k } .
For any mapping g : R d → R d define the following norm with respect to a function V : R d → [1 , ∞ ) :
∥ g ∥ V = sup x ∈ X ( ∥ g ( x ) ∥ 2 / V ( x )) , and let L V = { g :
R d → R d , sup x ∈ X ∥ g ∥ V < ∞} .
Assumption 2.1 (Constraint set)
The set Θ ⊂ R d is convex and closed with max x,y ∈ Θ
∥ x − y ∥ 2 ≤ D Θ , form some D Θ > 0 .
Assumption 2.2 Let f be a continuously differentiable function.
Assumption 2.3 Let ξ k +1 ( θ k , x k +1 ) :
=
∇ F ( θ k , x k +1 )
− ∇ f ( θ k ) .
Then,
[Equation]
Assumption 2.4 Let { x k } k be a Markov chain with transition kernel P θ .
For any θ ∈ Θ , P θ is irreducible and aperiodic.
Additionally, there exists a function V : R d → [1 , ∞ ) and a constant α ≥ 2 such that for any compact set Θ ′ ⊂ Θ :
(a) There exist a set C ⊂ R d , an integer I , constants 0 < λ < 1 , b , κ , δ > 0 , and a probability measure ν such that,
[Equation]
where B R d is the Borel σ -algebra over R d .
(b) There exists a constant c > 0 , such that, for all x ∈ R d and for all θ, θ ′ ∈ Θ ′ ,
[Equation]
(c) There exists a constant c > 0 , such that, for all ( θ, θ ′ ) ∈
[Equation]
Some comments regarding the assumptions are in order.
Assumption 2.1, and Assumption 2.2 are common for constrained optimization [GRW20, XBG22, AL22, ZSM + 20].
Assumption 2.1, and
Assumption 2.2 together imply the Lipschitz continuity of f ( · ) , i.e., there is a constant L > 0 such that for any θ 1 , θ 2 ∈ Θ , we have | f ( θ 1 ) − f
( θ 2 ) | ≤ L ∥
θ 1 − θ 2 ∥ 2 .
Assumption 2.3 is common in stochastic optimization literature.
Assumption 2.4(a) is a frequently used assumption in Markov chain literature.
It implies that for every θ ∈ Θ , there exists a stationary distribution π θ ( x ) , and the chain is V α -uniformly ergodic [AMP05].
Assumption 2.4(c) provides smoothness guarantee on the function f ( · ) .
More formally, we have the following proposition.
Proposition 2.1 (Lipschitz continuous gradient [AMP05]) Let Assumption 2.4 be true.
Then f ( · ) has Lipschitz continuous gradient, i.e., there is a constant L G > 0 such that for any θ 1 , θ 2 ∈ Θ :
[Equation]
Finally, the most important implication of Assumption 2.4 is that it ensures the existence and regularity of a solution u ( θ, x ) to Poisson equation of the transition kernel P θ given by u ( θ, x ) − P θ u
( θ, x ) =
∇ F ( θ, x ) − ∇ f ( θ ) .
Solution of Poisson equation has been crucial in analyzing additive functionals of Markov chain (see [AMP05] for details).
In this work, the Poisson equation solution facilitates a decomposition of the noise as presented in Lemma 3.1 which is a key component of our analysis.
Main Result
In this section we present our main result on the oracle complexity to establish a bound on E [ V ( θ k , z k )] .
In order to do so we use Algorithm 1, and 2 similar to [XBG22].
If an exact
Input: z 0 , θ 0 ∈ R d , η k = ( N + k ) − a , 1 / 2 < a < 1 , β . for k
= 1 , 2 , · · · , N do y k =
  min y ∈ Θ (cid:110) ⟨ z
[Equation]
[Equation]
− θ
[Equation]
(Projection)
[Equation]
No Projection)
[Equation]
end for
Output: θ R where P ( R = i ) = η i (cid:80) Nj =1 η j for i = 1 , 2 , · · · , N .
Input: z, θ, β, t, ω .
Set w
= θ for i = 1 , 2 , · · · , t − 1 do
Find v i such that
[Equation]
D
[Equation]
w for
Output minimizer of the following subproblem, which is the projection of θ k − z k /β on to Θ , is available, then Algorithm 1 is same as ASA algorithm introduced in [GRW20].
[Equation]
When a projection operator is unavailable or computationally costly, we use Algorithm 2 instead to solve (17).
At iteration k , Algorithm 2 finds an approximate solution to (17) based on the conditional gradient algorithm.
Algorithm 2 needs access to LMO which is often much cheaper and simpler to compute than projection operator.
We should emphasize that our results are not limited to ICG method but are valid for any method which can solve (17) within an error of the order of { η k } .
Theorem 3.1 Let Assumption 2.1-2.4 be true.
Then, for Algorithm 1,
(a) when a projection operator is available, choosing
[Equation]
for k = 1 , 2 , · · · , N we have
[Equation]
(b) when Algorithm 2 is used to solve (17) , choosing
[Equation]
for k = 1 , 2 , · · · , N we have
[Equation]
where the expectations are taken with respect to all the randomness of the algorithm, and an independent integer random variable R ∈ { 1 , 2 , · · · , N } with probability mass function,
[Equation]
Remark 1 Note that total number of LMO calls are (cid:80)
Nk =
1 t k = (cid:80) Nk =1 t k = (cid:80) Nk
=1 ( N + k ) 2 a =
O ( N 11 / 5 ) .
In other words, to achieve ∥G Θ ( θ R , ∇ f ( θ R ) , β ) ∥ 22 ≤ E
[ V ( θ R , z R )] ≤ ϵ , SFO and LMO complexities are respectively ϵ − 2 .
5 , and ϵ − 5 .
Note that the SFO complexity will be ϵ − 2 .
5 as long as one has an approximation of the projection operator with approximation error O ( η k ) .
Remark
In Theorem 3.1, one obtains sublinear rate max( N a − 1 , N 2 − 4 a ) with η k = ( N + k ) − a for 1 / 2 < a < 1 .
Choosing a = 3 / 5 provides the fastest rate of convergence.
Before sketching the outline of the proof, we present the following lemma which provides a decomposition of the noise ξ k ( θ k − 1 , x k ) – one of the key result used in the proof of the main theorem.
The lemma and its proof are almost same as Lemma A.5 in [Lia10] with the only difference that unlike [Lia10], where the iterates are of SGD, we need to prove it for the iterates of Algorithm 1.
We provide the proof in Appendix A.
Lemma 3.1 Let Assumption 2.1-2.4 be true.
Then the following decomposition takes place:
[Equation]
where, { e k } is martingale difference sequence, E [ ∥ ν k ∥ 2 ] ≤ η k , and ζ k =
(˜ ζ k − ˜ ζ k +1 ) /η
k , where
E
[Equation]
≤
[Equation]
k .
Outline of the proof of Theorem 3.1:
A key step in the analysis of Algorithminvolves controlling the expectation of interaction with noise of the form ⟨∇ f ( θ k ) − ∇ f ( θ k − 1 ) , ξ k +1 ( θ k , x k +1 ) ⟩ .
For iid or martingale difference data it is easy to control because E [ ⟨∇ f ( θ k ) − ∇ f
( θ k − 1 ) , ξ k +1 ( θ k , x k +1 ) ⟩|F k ] = 0 .
But this is no longer true for Markov chain data.
To resolve the issue, first notice that under our assumptions, the noise sequence ξ k can be decomposed into the sum of a martingale difference sequence { e k } and some residual terms { ν k } , and { ζ k } as shown in Lemma 3.1.
Then the key step is to introduce a different sequence of hypothetical iterates (˜ θ k , ˜ y k , ˜
z k ) for which the noise is small enough so that we can bound E (cid:104) V (˜ θ k , ˜ z k ) (cid:105) , and then show that these hypothetical iterates and the original sequence generated by Algorithm 1 are close enough so that E [ V ( θ k , z k )] is of the same order
as E (cid:104) V
(˜ θ k , ˜ z k ) (cid:105) .
This step is the main novelty of the proof.
Specifically, consider the following sequence:
[Equation]
[Equation] [Equation]
This also means,
[Equation] [Equation]
where, ˜ ϵ
k = e k + ν k + ˜ ζ k .
Note that by Lemma 3.1, E [ e k ] = 0 , and E (cid:104)(cid:13)(cid:13)(cid:13)
ν
k + ˜ ζ k (cid:13)(cid:13)(cid:13) 2 (cid:105) ≤ η k .
First we show that by choosing η k = ( N + k ) − a , 1 / 2 < a < 1 , and t k = 1 /η 2 k one has E (cid:104) ∥
˜ θ k − θ k ∥ 22 (cid:105) =
O (cid:0) N 2 − 4 a (cid:1) , and E [ V ( θ k , z k )] ≤ 2 E (cid:104) V
(˜ θ k , ˜ z k ) (cid:105)
+ O (cid:0) N 2 − 4 a (cid:1) .
Then we establish the bound on
V (˜ θ k , ˜ z k ) .
Combining the above two facts proves Theorem 3.1.
We defer the detailed proof to
Appendix A.1.
3.1 State-independent Markov Chain
While our main goal in this work is to analyze Algorithm 1 for constrained nonconvex optimization with state-dependent Markov chain data, we provide the following result on the complexity of
Algorithm 1 for Markov chain data with state-independent transition kernel for the sake of completion.
Here we use P to denote the transition kernel (as opposed to P θ for state-dependent kernel).
Note that under Assumption 2.4(a), for each θ , the chain is V -uniformly ergodic, and hence, exponentially mixing [MT12] in the following sense:
Definition
A Markov chain is said to be exponentially mixing, if there exists C, r > 0 such that, for any initial state x ,
[Equation]
where P n ( x, · ) is the distribution of X n with initial state X
0 = x .
Now we present our result on the complexity of Algorithm 1 to find an ϵ -stationary solution to (1) for exponentially-mixing Markov chain data with state-independent transition kernel.
Theorem 3.2 Let Assumption 2.1-2.3 be true.
Let Assumption 2.4(a)-(b) be true with P θ replaced by
P .
Then, for Algorithm 1,
(a) when the projection operator is available, choosing
[Equation]
for k = 1 , 2 , · · · , N we have
[Equation]
(b) when Algorithm 2 is used, choosing
[Equation]
for k = 1 , 2 , · · · , N we have
[Equation]
where the expectation is taken with respect to all the randomness of the algorithm, and an independent integer random variable R ∈ { 1 , 2 , · · · , N } whose probability mass function is given by,
[Equation]
We defer the proof to the Appendix.
Remark 3 To find an ϵ -stationary point, the total number of calls to SFO and LMO are ˜ O (cid:0) ϵ − 2 (cid:1) , and ˜ O (cid:0) ϵ − 3 (cid:1) , where ˜ O ( · ) denotes the order ignoring logarithmic factors.
Remark 4 The authors of [AL22] obtain the same rate as in Theorem 3.2 for constrained (but projection-based) nonconvex optimization with state-independent exponentially mixing data.
In the state-dependent case, since the transition kernel of the Markov chain is controlled by θ k , and the transition kernel is assumed to be only Lipschitz smooth in θ (15) , the chain does not necessarily exponentially mix.
In the state-independent case, since the chain mixes exponentially we obtain the same rate as well.
While their results are for projection-based algorithms, we analyze a projectionfree LMO-based algorithm since LMO is often computationally cheaper than projection.
Experimental Evaluation 4.1 Strategic Classification
In this section we illustrate our algorithm on the strategic classification problem as described in
Section 1.1 with the GiveMeSomeCredit 4 dataset.
The main task is a credit score classification
10^ −        20 k l og V ( θ k , z k )
ProjectionICG 10^ 0.
20 k
T e s t
A cc u r a cy
ProjectionICG problem where the bank (learner) has to decide whether a loan should be granted to a client.
Given the knowledge of the classifier the clients (agents) can distort some of their personal traits in order to get approved for a loan.
Here we use a 2-layer neural network with width m as the classifier, given by
[Equation]
where υ ( · ) is the activation function, W i ∈ R d , W
=
[ W 1 , W 2 , · · · , W d ] ⊤ ∈ R m × d , A
= ( A 1 , A 2 , · · · , A m ) ∈ R m , B
=
( B 1 , B 2 , · · · , B m ) ∈ R m .
We will use θ to collectively denote ( W , A , B ) .
We impose the constraint of sparsity on the classifier given by ∥ θ ∥ 1 ≤ R for some R > 0 .
As loss function we consider logistic loss as shown in (2).
We consider a quadratic cost given by c ( x, x ′ ) =
∥ x S − x ′ S ∥
22 / (2 λ ) where λ is the sensitivity of the underlying distribution on θ .
We assume that the agents iteratively learn x ′ S similar to [LW22].
Note that unlike [LW22], the closed form of best response is not known here.
So we assume that the agents use Gradient Ascent (GA) to learn the best response.
For ∥ θ ∥ 1 ≤ R constraint, the LMO in Algorithm 2 at iteration k is given by − R sign ( q i ) , where i = argmax j =1 , ··· ,d | q j | , q = z + β ( w k − θ ) , and q j is the j -th coordinate of q .
We select a subset of randomly chosen M = 2000 samples (agents) such that the dataset is balanced.
Each agent has 10 features.
Note that since Algorithm 1 computes the gradient on one sample at every iterate, the computation time is independent of the total number of agents.
We assume that the agents can modify Revolving Utilization, Number of Open Credit Lines, and Number of Real
Estate Loans or Lines.
In this experiment we set n 1 = 200 .
Similar to [LW22], we set α = 0 .
5 λ , and λ = 0 .
For the classifier, the activation function is chosen as sigmoidal , and m = 400 .
We set N = 20000 , and R = 4000 .
All the parameters of Algorithm 1 are chosen as described in (19).
Figure 4.1 shows that Algorithm 1 finds an ϵ -stationary point of the strategic classification problem.
We show that Algorithm 1 performs comparably with Averaged Stochastic Approximation with the projection operator.
Each curve in Figure 4.1 is an average of 50 repetitions.
4.2 Single Index Model with Trace-norm Ball Constraint
In this section we illustrate our algorithm on a synthetic example of single-index model regression with a nuclear-norm constraint on the model parameter.
Let ∥·∥ ∗ denote the nuclear norm.
The features { x k } k ∈ R d 1 × d 2 are a matrix-valued time-series given by,
[Equation]
where A ∈ R d 1 × d 1 matrix with spectral radius less than 1 , E k ∈ R d 1 × d 2 is the noise matrix with each entry of E k is iid N (0 , 1) random variable, W k is a Bernoulli (0 . 5) random variable, and υ ∈ R .
For a fixed θ k
= θ , { x k } k has a stationary distribution as shown in Proposition 1 of [CXY21].
{ E k } k , and { W k } k are iid sequence.
This Markov chain follows conditions (b) and (c) of Assumption 2.4 since the evolution of x k only involves linear terms in θ k .
The responses { y k } k are generated according to the following single index model,
[Equation]
where { ˜ E k } k is an iid sequence of standard normal random variables, θ ∗ ∈ R
d
1 × d 2 is a matrix with ∥ θ ∗ ∥ ∗ ≤ 1 , and g ( · ) :
R → R is the link function.
For this experiment we choose g ( x ) =
3 x +
[Figure]
5 sin( x ) .
Since y k only depends on x k , and g is a Lipschitz continuous function of θ , Assumption 2.
holds for ( x k , y k ) .
It is easy to see that Assumptions 2.1 - 2.3 holds for this example.
The constraint set is given by ∥ θ
∥ ∗ ≤ 1 , i.e., we assume that θ ∗ has a low-rank structure.
The goal is to minimize the expected squared loss with the constraint ∥ θ ∥ ∗ ≤ 1 , i.e.,
[Figure]
[Equation]
[Figure]
The advantages of conditional-gradient based method for nuclear-norm ball constrained problems have been studied extensively [JS10, Jag13, HJN15].
The main advantage of ICG-based method is that calculating the LMO in this case requires computation of the leading singular vector of gradient matrix whereas to calculate the projection on the trace-norm ball one needs to compute the complete singular value decomposition.
Let u 1 , v 1 are the leading left and right singular vectors of the noisy gradient matrix evaluated at ( θ ; x, y ) , − 2( y − g ( x ⊤ θ ))
g ′ ( x ⊤ θ ) x .
Then the LMO is given by − u 1 v ⊤ 1 .
For this experiment we choose d 1 = 10 , d 2 = 20 , υ = 0 .
1 , and N = 2000 .
Rest of the parameters of
Algorithm 1 are chosen according to Theorem 3.1.
In Figure 2, we compare the projection-based and
ICG based version of Algorithm 1 with respect to V ( θ k , z k ) , test Mean Squared Error (MSE), and ∥ θ k − θ ∗ ∥ F where ∥·∥ F is the Fröbenius norm.
Figure 2 shows that the performance of projectionbased and the ICG-based versions of Algorithm 1 are almost same.
Each plot in Figure 2 is the average of 50 repetitions.
Discussion
In this work we provide oracle complexity results for the stochastic conditional gradient algorithm to find an ϵ -stationary point of a constrained nonconvex optimization problem with state-dependent
Markovian data.
In Theorem 3.1, we show that the number of calls to the SFO and LMO required by the stochastic conditional gradient-type method in Algorithm 1, with state-dependent Markovian data, is O ( ϵ − 2 . 5 ) and O ( ϵ − 5 . 5 ) respectively.
To the best of our knowledge, these are the first oracle complexity results in this setting.
In Theorem 3.2, we show that SFO and LMO complexity in the case of state-independent Markovian data is ˜ O
( ϵ − 2 ) and ˜ O
( ϵ − 3 ) respectively, which matches the corresponding results in the iid setting.
There are various avenues for further extensions.
Establishing lower bounds on the oracle complexity of projection-free algorithms in the Markovian data setting is extremely interesting.
It is also intriguing to establish upper and lower bounds on the oracle complexity for more general types of dependent data sequences arising in applications, including ϕ and α mixing sequences.
Yet another exciting direction is that of designing algorithms adaptive to the dependency in the data that achieve potentially better oracle complexity bounds.