A Stochastic Prox-Linear Method for CVaR Minimization
Abstract
We develop an instance of the stochastic prox-linear method for minimizing the Conditional Valueat-Risk (CVaR) objective.
CVaR is a risk measure focused on minimizing worst-case performance, deﬁned as the average of the top quantile of the losses.
In machine learning, such a risk measure is useful to train more robust models.
Although the stochastic subgradient method (SGM) is a natural choice for minimizing CVaR objective, we show that the prox-linear algorithm can be used to better exploit the structure of the objective, while still providing a convenient closed form update.
We then specialize a general convergence theorem for the prox-linear method to our setting, and show that it allows for a wider selection of step sizes compared to SGM.
We support this theoretical ﬁnding experimentally, by showing that the performance of stochastic prox-linear is more robust to the choice of step size compared to SGM.
1. Introduction
The most common approach to ﬁt a model parametrized by θ ∈ R n to data, is to minimize the expected loss over the data distribution, that is
[Equation]
But in many cases, the expected loss may not be the suitable objective to minimize.
When robustness or safety of the model are concerned, the emphasis should rather be on the extreme values of the distribution rather than the average value.
For instance, in distributionally robust optimization, the goal is to optimize the model for the worst case distribution around some ﬁxed distribution [14].
In extreme risk-averse settings, such as when safety is the top priority, one would minimize the maximum loss within a training set [30].
These applications can all be formulated as minimizing the expectation of the losses that are above some cutoff value,
[Equation]
where α β ( θ ) is the upper β –quantile of the losses.
For example, for β = 0 . 9 , the problem in Equation 2 is to minimize the expectation of the worst 10% of the losses.
In this work, we investigate the use of the stochastic prox-linear (SPL) method introduced by Duchi and Ruan [15] for solving Equation 2.
The possibility of applying SPL to CVaR minimization was mentioned in Davis and Drusvyatskiy [13], but not explored.
We ﬁrst derive a closed-form update for
SPL, and show why it is particularly well suited for minimizing CVaR.
We give its convergence rates for convex and Lipschitz losses by specializing existing results from Davis and Drusvyatskiy [13].
Through several experiments comparing the stochastic prox-linear method to stochastic subgradient we show that the prox-linear algorithm is more robust to the choice of step size.
We conclude with a discussion on several future applications for minimizing CVaR in machine learning.
1.1. Background
The CVaR objective was ﬁrst introduced in ﬁnance as an alternative measure of risk, also known as the expected shortfall [2, 16].
Many applications in ﬁnance can be formulated as CVaR minimization problems, such as portfolio optimization [21, 25], insurance [17] and credit risk management [1].
The seminal work of Rockafellar and Uryasev [27] proposed a variational formulation of the CVaR objective that is amenable to standard optimization methods.
This formulation has since inspired considerable research in applications spanning machine learning and adjacent ﬁelds, such as ν -SVM [18, 32], robust decision making and MDPs [7, 9–11, 29], inﬂuence maximization and submodular optimization [24, 26, 33], fairness [34], and federated learning [23].
Though it ﬁnds many applications, the CVaR objective is typically difﬁcult to minimize.
It is nonsmooth even when the individual losses (cid:96) ( · ; z ) are continuously differentiable.
Indeed, if P does not admit a density — which is the case for all empirical distributions over training data — the variational objective is not everywhere differentiable.
To address this, Laguel et al.
[22] developed subdifferential calculus for a number of equivalent CVaR formulations and proposed minimizing a smoothed version of the dual objective.
On the other hand, several works [20, 31] apply the stochastic subgradient method directly to the variational formulation proposed by Rockafellar and Uryasev [27], which is well-deﬁned regardless of the distribution P .
However, as we elaborate in Section 3, this approach is oblivious to the special structure of the variational form of the CVaR objective.
2. Problem setup
Let (cid:96) ( θ ; z ) be the loss associated with the model parameters θ ∈ R d and a measurable random variable z ( ω ) on some background probability space (Ω , F , P ) .
When z follows a distribution P with density p ( z ) , the cumulative distribution function on the loss for a ﬁxed θ is given by P [ (cid:96) ( θ ; z ) ≤ α ] = (cid:82) (cid:96) ( θ ; z ) ≤ α p ( z ) dz , which we assume is everywhere continuous with respect to α .
Let β be a conﬁdence level, for instance β = 0 .
The
Value-at-Risk (VaR) of the model is the lowest α such that with probability β , the loss will not exceed α .
Formally,
VaR β ( θ ) :
= min { α ∈ R : P [ (cid:96) ( θ ; z ) ≤ α ] ≥ β } , (3)
The Conditional Value-at-Risk (CVaR) is the expectation of the upper tail starting at VaR β , illustrated in Figure 1: CVaR [Equation] VaR
[Equation]
[Figure]
Clearly, the CVaR always upper bounds the VaR.
Our goal is to minimize CVaR β over θ ∈ R d , but directly minimizing Equation 4 is not straightforward.
Fortunately, Rockafellar and Uryasev [27] introduced a variational formulation where the solution to
[Equation]
is such that θ ∗ is the solution to Equation 4, and we obtain α ∗ = VaR β ( θ ) as a byproduct.
3. The Stochastic Subgradient Method
A natural choice for minimizing Equation 5 is the stochastic subgradient method (SGM).
Letting ∂f denote the convex subdifferential of f , at each step t we sample z ∼ P uniformly and compute a subgradient g t from the subdifferential
[Equation]
Given some step size sequence { λ t } > 0 , the SGM then takes the step x t +1 = x
t − λ t g t , where g t ∈ ∂F β ( θ t , α t ; z ) , and x = ( θ, α ) T .
For reference, the complete SGM algorithm is given in Algorithm 2.
SGM is very sensitive to the step size choice and may diverge if not carefully tuned.
This issue can be explained from a modeling perspective [13].
Indeed, SGM can be written as a model-based method where at each iteration t , it uses the following linearization of the sampled F β ( x ; z ) at the current point x t :
[Equation]
This provides an approximate, stochastic model of the objective F β ( x ) .
The SGM update is then a proximal step on this model, that is
[Equation]
The issue with this model m t ( x ; z ) is that it uses a linearization to approximate the max {· , 0 } function.
The linearization, which can take negative values, is a poor approximation of the nonnegative max {· , 0 } operation.
The main insight to the SPL method is to leverage the structure of
F β ( x ) as a truncated function. This structure allows for a more accurate model that still has an easily computable proximal operator.
4. Stochastic prox-linear method for CVaR minimization
Here we introduce an alternative model for our objective that only linearizes inside the max {· , 0 } , which is a strictly more accurate model when the objective is convex [3].
In particular, we use m t ( x ; z
[Equation]
for some v t ∈ ∂(cid:96) ( θ t ; z ) .
[Figure]
Note that because the subgradient of the objective is 0 in θ , the SGM model is constant in θ .
The algorithm resulting from Equation 8 using this model is known as the stochastic prox-linear (SPL) method [15].
Figure 2 illustrates the difference between the two models.
We derive the closed-form updates of SPL, which can be found in Algorithm 1.
Furthermore, the cost of computing each iteration of SPL is of the same order of computing an iteration of SGM.
In addition, we instantiate the convergence analyses from Davis and Drusvyatskiy [13] in the case of CVaR minimization, and compare the rates for SGM and SPL for losses satisfying the following Assumption.
Assumption 1 (Convex, subdifferentiable, and Lipschitz)
There exist square integrable random variables M : Ω → R such that for a.e. z ∈ Ω and all θ ∈ R d , the sample losses (cid:96) ( θ ; z ) are convex, subdifferentiable 1 , and M ( z ) -Lipschitz.
Theorem 1 (Convergence rates of SGM and SPL under convexity) Suppose Assumption 1 holds.
Let x ∗ =
( θ ∗ , α ∗ )
T be a minimizer of F β ( θ, α ) , and ∆ =
(cid:107) x 0 − x ∗ (cid:107) for an arbitrary initialization x 0 .
Consider the iterates ( x t ) T +1 t =1 given by SGM in Algorithm 2 or by SPL in Algorithm 1.
There exists L > 0 such that by using a constant step size λ t = λ √ T +1 with λ = ∆ L √ 2 , we have
[Equation]
where ¯ x T = 1 T (cid:80) T +1 t =1 x t is the averaged iterate.
In particular, the Lipschitz constants are given by
[Equation]
This result follows from Theorem 4.4 in Davis and Drusvyatskiy [13], and we verify the assumptions necessary in Appendix B. Note that λ > 0 can be chosen independent of L and ∆ at the cost of a worse bound in Equation 10. Since L 2 is smaller for SPL, it allows for larger step sizes and improved constants in the convergence rate of Equation 10, though both methods enjoy the same asymptotic rate of O (1 / √ T ) .
In practice, the exact value of L is typically unavailable and λ is tuned heuristically via a grid search.
Since SPL has theoretical guarantees under a wider range of valid step sizes, it should be easier to tune in practice.
Our numerical experiments corroborate the theory, at least when the loss functions (cid:96) ( · ; z ) are smooth.
When (cid:96) is itself nonsmooth we ﬁnd the beneﬁts to be negligible, though the performance of SPL remains at least as good as that of SGM.
5. Experiments
We now compare the empirical performance of SGM and SPL for minimizing the CVaR objective (Equation 5) using synthetic data.
Similar to the setup of Holland and Haress [20], described in detail in Appendix C, we ﬁx β = 0 .
95 and experiment with various combinations of loss functions (cid:96) ( · ; z ) and data distributions controlled by noise ζ .
We employ a decreasing step size λ t = λ / √ t
+
and study the sensitivity of both methods to the initial step λ 0 = λ , varied over a logarithmically-spaced grid.
Since the expectation in the objective is difﬁcult to compute in closed form, we evaluate the suboptimality gaps using an empirical average over N = 10 6 data points sampled i.i.d.
from the corresponding distribution under a single ﬁxed seed.
We denote this approximation by ˜ F β ( θ, α ) .
Figure 3 shows the ﬁnal suboptimality achieved by SGM and SPL for different λ .
For smooth losses (squared and logistic)
we see that SPL is signiﬁcantly more robust and admits a much larger range of λ for which it does not diverge.
Interestingly, for the absolute loss there is barely a difference.
The same stability to larger step size can be observed by instead looking at the minimum number of iterations required to achieve (cid:15) ﬁnal suboptimality ˜ F ( θ, α ) − ˜ F ∗ ≤ (cid:15) (see Figure 5). Finally, we also include the same set of experiments on two real datasets.
Details and results can be found in Appendix C.1.
6. Conclusion and future work
Our numerical evidence suggests that for the CVaR minimization problem, while both SGM and SPL can be tuned to achieve similar performance, SPL is strictly more tolerant to misspeciﬁed step sizes.
To further speed up SPL and make it more competitive over SGM, some natural heuristics include the use of non-uniform sampling to bias towards training examples with higher losses (as in Curi et al.
[12], Sagawa et al.
[28]), as well as updating the parameters θ t and α t using different step sizes.
Efﬁcient CVaR minimization with a stochastic algorithm opens up the possibility for new applications in machine learning.
For instance, we could now consider models that trade-off between low average risk and heavy tails by adding the CVaR objective as a regularizer:
[Equation]
where ρ > 0 is a parameter that captures this trade-off.
Controlling this trade-off is important as machine learning models are increasingly deployed in safety-critical applications that call for
[Figure]
control over the likelihood of failure. As future work, we also see applications in training neural networks, where CVaR can be used to disincentivize the activations from being saturated too often, and thus help in speeding up training.
This would offer an alternative to normalization layers, such as batchnorm or layernorm, which are used to bring the activations within a suitable range of the activations and thus avoid saturation.
Appendix A. SPL derivation for CVaR minimization
Before deriving the SPL updates, we ﬁrst introduce the following lemma based on the truncated model from Asi and Duchi [4].
Lemma 1 (Truncated model)
Consider the problem
[Equation]
for some scalar c and vector a ∈ R n .
The solution can be written in closed-form as
[Equation]
Proof Note that x t +1 is the proximal point of the function
[Equation]
, with h ( z ) = max { z, 0 } , b = c − (cid:104) a, x t (cid:105) .
centered at x ≡ x t .
Using Beck [5, Theorem 6.15], we have
[Equation] [Equation] [Equation]
In turn, the max function is the support function of the interval [0 , 1] .
By Beck [5, Theorem 6.46], it follows that
[Equation]
Plugging Equation 14 into Equation [Equation] we obtain
[Equation]
Writing proj [0 , 1] ( v )
=
min { max { v, 0 } , 1 } yields the result.
We now derive the the SPL updates.
Recall that for the CVaR objective, using the model in Equation 9, the stochastic model-based approach solves the following problem at each iteration:
[Equation]
for some v t ∈ ∂(cid:96) ( θ t ; z ) .
We can apply Lemma 1 by transforming f t into the truncated model form.
First, we combine the α with its regularization term,
[Equation]
Using a change of variable ˆ α = α and ˆ α
[Equation]
α
[Equation]
λ, (15) the quadratic regularization plus α is simpliﬁed to
[Equation]
Adding and subtracting ˆ α t , the linearization inside the max {· , 0 } then becomes
[Equation]
Letting x =
( θ, ˆ α ) T and x t
= ( θ t , ˆ α t ) T , we arrive at the form in Lemma 1 (up to constants) with
[Equation]
) and a = [Equation] t
[Equation]
The step size from Lemma 1 is
[Equation]
which, combined with the deﬁnition of ˆ α in Equation
[Equation]
translates to the following updates:
If c < 0 = ⇒ (cid:96) ( θ t ; z ) <
α t − λ , then η =
[Equation]
and
[Equation]
If c > λ (cid:107) a (cid:107)( > 0) , which implies checking for the case
[Equation]
Then the step size taken is η = λ , and the updates are
[Equation]
Otherwise,
0 <
c (cid:107) a (cid:107)
< λ
, then η = c
[Equation]
a (cid:107) 2 , and the updates are
[Equation]
Together, these give us the closed-form updates of the SPL method, collected in Algorithm 1.
1: initialize: θ 0 ∈ R d , α 0 ∈ R , hyperparameter: λ >2: for t = 0 , 1 , 2 , . .
, T − 1 do 3:
Sample data point z ∼ P , compute (cid:96) ( θ t ; z ) and v t ∈ ∂(cid:96) ( θ t ; z ) 4: λ t ← λ/ √ t +5: if α t > (cid:96) ( θ t ; z ) + λ t then (cid:46) α t too big 6: θ t +1 ← θ t 7: α t +1 ← α t − λ t 8: else if α t < (cid:96) ( θ t ; z ) − λ t 1 − β (cid:16) (cid:107) v t (cid:107) 2 + β (cid:17) then (cid:46) α t too small 9: θ t +1 ← θ t − λ t 1 − β v t 10: α t +1 ← α t + λ t 1 − β β 11: else (cid:46) α t in middle range 12: ν ← (cid:96) ( θ t )+ λ t − α t λ t ( (cid:107) v t (cid:107) 2 +1 ) 13: θ t +1 ← θ t − λ t ν ∇ (cid:96) ( θ t ; z ) 14: α t +1 ← α t − λ t + λ t ν 15: end if 16: end for 17: return ¯ x T =
[Equation]
We also include the closed-form updates for the stochastic subgradient method applied to CVaR minimization in Algorithm 2.
Appendix B. Proof of Theorem
Proof To apply Theorem 4.4 in Davis and Drusvyatskiy [13], we must ﬁrst verify their assumptions (B1)-(B4) hold.
We will enumerate these under their following general setup: writing the CVaR
1: initialize: θ 0 ∈ R d , α 0 ∈ R , hyperparameter: λ >2: for t = 0 , 1 , 2 , . .
, T − 1 do 3:
Sample data point z ∼ P , compute (cid:96) ( θ t ; z ) and v t ∈ ∂(cid:96) ( θ t ; z ) 4: λ t ← λ/ √ t +5: if α t ≥ (cid:96) ( θ t ; z ) then (cid:46) α t too big 6: θ t +1 ← θ t 7: α t +1 ← α t − λ t 8: else (cid:46) α t too small 9: θ t +1 ← θ t − λ t 1 − β v t 10: α t +1 ← α t + λ t 1 − β β 11: end if 12: end for 13: return ¯ x T = 1 T +1 (cid:80) T +1 t =1 ( θ t , α t ) T objective as F β ( x ) = f [Equation] +
[Equation] [Equation] ) , (17) and interpret r ( x ) = 0 for SGM while r ( x ) = α for SPL.
In the SPL case, we further write f ( x ) =
E z [ h ( c ( x ; z ))] where h ( · ) = 1 1 − β max {· , 0 } and c ( x ; z ) =
(cid:96) ( θ ; z ) − α .
Recall that the stochastic one-sided models used are
SGM
[Equation]
where g t ∈ ∂F β ( x t ; z ) (18)
SPL
[Equation]
where u t ∈ ∂c ( x t ; z )
[Equation]
19) and the update in Equation 8 is equivalent to
[Equation]
The assumptions we need to verify are the following, adapted from Davis and Drusvyatskiy [13]:
(B1) ( Sampling )
It is possible to generate i i
.d. realizations z 1 , z 2 , · · · ∼ P . (B2) ( One-sided accuracy ) There is an open set U containing dom r and a measurable function ( x, y ; z ) (cid:55)→ g x ( y ; z ) , deﬁned on U × U × Ω , satisfying
[Equation]
and
[Equation]
Assumptions (B1) follows trivially from i.i.d. sampling, while (B2) follows from convexity of (cid:96) ( · ; z ) , which results in
τ = 0 .
(B3) ( Weak-convexity ) The function f t ( x ; z ) + r ( x ) is η -weakly convex for all x ∈ U , a.e. z ∈ Ω . Since r ( x ) is also convex in both methods and both models are convex, (B3) holds with η = 0 .
(B4) ( Lipschitz property )
There exists a measurable function L : Ω → R + satisfying (cid:112)
E z
[Equation]
[Equation]
≤ L and such that [Equation] This can be easily proved for SGM using the Lipschitz assumption on (cid:96) ( · ; z ) and that max {· , 0 } is 1 -Lipschitz:
[Equation]
which gives us L 2 =
E z (cid:20)(cid:16)
+ 1 1 − β (cid:112) M ( z )
+ 1 (cid:17) 2 (cid:21) .
For SPL, (1 − β )( f t ( x t ; z ) − f t ( y ; z )) = max { (cid:96) ( θ t ; z ) − α t , 0 } − max { (cid:96) ( θ t ; z ) − (cid:104) v t , θ − θ t (cid:105) − α, 0 } ≤ [Equation]
t
[Equation]
0 } = max (cid:26)(cid:28)(cid:18)
[Equation]
[Equation]
(cid:27) =
⇒ f t ( x t ; z ) − f t
( y ; z ) ≤
[Equation]
This gives us the slightly improved constant for SPL:
[Equation]
All assumptions for Theorem 4.4 in Davis and Drusvyatskiy [13] are satisﬁed, and so under the step size λ t = λ √ T +1 , with λ =
∆ L √ 2 and an averaged ﬁnal iterate gives us the rate
[Equation]
Appendix C. Experiment details and additional results
For all problems we set the dimension to be d = 10 .
For regression problems, θ gen ∼ U ([0 , 1] d ) , and for classiﬁcation (logistic regression) we use θ gen ∼ U ([0 , 10] d ) to increase linear separability.
The loss functions and target generation schemes are listed in Table 1.
Each target of the corresponding problem contains an error (cid:15) from one of the distributions Table 2, which is intended to control the difﬁculty level of the problem.
For each error distribution and loss function combination, we draw N independent samples and use the following discretization as an approximation to Equation [Equation]
[Equation]
We use σ ( denote sigmoid function, and all
’s are Task Loss (cid:96) ( θ ; x, y )
Target
Regression 12 ( x T θ − y )
y
=
x T θ gen + ζ
Regression | x T θ − y | y
= x T θ gen + ζ
Classiﬁcation log (1 + exp ( − yx T θ ))
y
= 1 w.p. σ ( x T θ gen + ζ ) and − 1 otherwise.
We set β = 0 .
95 for all experiments, and thus have omitted β from all plot descriptions.
We run full-batch subgradient method with the adaptive Polyak step size from Hazan and Kakade [19] for a total of 2000 iterations divided into K = 100 iterations of the inner loop, using the lower bound 0 for the initial estimate of the minimum objective value.
We record the ﬁnal θ ∗ , α ∗ [-) . At initialization we let α 
-], and F
=[-
-]
F β ( θ ∗ , α
{+For the SPL against SGM comparison, we set α 0 = 
+}
and θ 0 ∼ N (0 , I d ) at initialization.
Both algorithms are run for T = 100 , 000 iterations using 5 different seeds that control the randomness of initialization and sampling during the course of optimization.
In the sensitivity plots (Figuresand 5), solid lines show the median values, while the shaded regions indicate the range over the random seeds. All objective evaluations are on ˜
F β (¯ θ t , ¯ α t ) using the averaged iterates.
Distribution of ζ Parameters
Normal( µ, σ 2 ) µ = 0 , σ = Gumbel( µ, β ) µ = 0 , β = LogNormal( µ, σ 2 ) µ = 2 , σ =
[Figure]
C.1. Additional experiment results
Figure 5 shows a similar sensitivity analysis to Figure 3 in the main text.
Instead of the sensitivity of ﬁnal suboptimality, here we show the sensitivity of the minimum number of iterations to reach (cid:15)
-suboptimality ˜ F
( θ, α ) − ˜ F ∗ ≤ (cid:15) .
[Figure]
i t e
[Figure]
a t i o n
[Figure]
Finally, we present the same experiment on two real datasets, YearPredictionMSD and (binary)
Covertype from the LIBSVM repository [8].
The objective now is the empirical CVaR
F β ( θ, α )
=
α + 1 − β n n (cid:88)
i = max { (cid:96) ( θ ; z i ) − α, 0 } where n is the number of examples in the training split.
The loss function (cid:96) ( · ; z i ) is the squared loss for
YearPredictionMSD , and logistic loss for Covertype .
Similar to the synthetic experiments, we set β = 0 .
95 and run full-batch subgradient method with the adaptive Polyak step size from
Hazan and Kakade [19] to compute θ ∗ and α ∗ , but for a total of 10000 iterations divided into K =iterations of the inner loop, as these datasets are large and more difﬁcult than the simulated ones.
For the comparison between SPL and SGM, we run both methods for 100 n iterations.
The results are in
Figure 6 and Figure 7, from which we draw similar conclusions as in the synthetic experiments.
[Figure]
Squared loss on YearPredictionMSD
[Figure]
e s t s e t
[Figure]
[Figure]
Logistic loss on Covertype
[Figure]
Logistic
[Figure]
on Covertype
[Figure]