Exploring the Algorithm-Dependent Generalization of AUPRC Optimization with List Stability
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Stochastic optimization of the Area Under the Precision-Recall Curve (AUPRC) isa crucial problem for machine learning.
Although various algorithms have beenextensively studied for AUPRC optimization, the generalization is only guaranteedin the multi-query case.
In this work, we present the ﬁrst trial in the single-querygeneralization of stochastic AUPRC optimization.
For sharper generalizationbounds, we focus on algorithm-dependent generalization.
There are both algorithmic and theoretical obstacles to our destination.
From an algorithmic perspective,we notice that the majority of existing stochastic estimators are unbiased only whenthe sampling strategy is unbiased, and is leave-one-out unstable due to the nondecomposability.
To address these issues, we propose a sampling-rate-invariantunbiased stochastic estimator with superior stability.
On top of this, the AUPRCoptimization is formulated as a composition optimization problem, and a stochastic algorithm is proposed to solve this problem.
From a theoretical perspective,standard techniques of the algorithm-dependent generalization analysis cannotbe directly applied to such a listwise compositional optimization problem.
Toﬁll this gap, we extend the model stability from instancewise losses to listwiselosses and bridge the corresponding generalization and stability.
Additionally, weconstruct state transition matrices to describe the recurrence of the stability, andsimplify calculations by matrix spectrum.
Practically, experimental results on threereal-world datasets speak to the effectiveness and soundness of our framework.
Introduction
Area Under the Precision-Recall Curve (AUPRC) is a widely used metric in the machine learningcommunity, especially in learning to rank, which effectively measures the trade-off between precisionand recall of a ranking model.
Compared with threshold-speciﬁed metrics like accuracy and recall@k,
AUPRC reﬂects a more comprehensive performance by capturing all possible thresholds.
In addition,literature has shown that AUPRC is insensitive toward data distributions [20], making it adaptableto largely skewed data.
Beneﬁting from these appealing properties, AUPRC has become one of thestandard metrics in various applications, e.g. , retrieval [54, 57, 22, 40], object detection [44, 48, 15],medical diagnosis [49, 35], and recommendation systems [16, 71, 1, 63, 2].
Over the past decades, the importance of AUPRC has prompted extensive researches on direct
AUPRC optimization.
Early work focuses on full-batch optimization [44, 43, 26].
However, in theera of deep learning, the rapidly growing scale of models and data makes these full-batch algorithmsinfeasible.
Therefore, in recent years, it has raised an increasing favor of the stochastic AUPRCoptimization [9, 12, 31, 45, 53, 68].
See Appendix A for more on related work.
Despite the promoting performance of these methods in various scenarios, the generalization of
AUPRC optimization algorithms is still an open problem.
Some studies [17, 62] provide provable generalization for AUPRC optimization in information retrieval.
In this scene, a dataset consists ofmultiple queries, where each query corresponds to a set of positive and negative samples.
However,these results require sufﬁcient queries to ensure small generalization errors, but leave the single-querycase alone, i.e. , whether the generalization error tends to zero with the length of a single-queryincreasing is still unclear .
This limits the adaptation scope of these methods.
To ﬁll this gap,in this paper we aim to design a stochastic optimization framework for AUPRC with a provablealgorithm-dependent generalization performance in the single-query case.
The target is challenging in three aspects: (a) Most AUPRC stochastic estimators are biased witha biased sampling rate.
Moreover, due to the non-decomposability, outputs of existing algorithmsmight change a lot with slight changes in the training data, which is called leave-one-out unstablein this paper.
Such an unstability is harmful to the generalization.
(b) The standard framework toanalyze the algorithm-dependent generalization requires the objective function to be expressed as asum of instancewise terms, while AUPRC involves a listwise loss.
(c)
The stochastic optimization of
AUPRC is a two-level compositional optimization problem, which is typically solved by alternateupdates.
This brings more complicated stability calculations.
In search of a solution to (a) , we propose a sampling-rate-invariant asymptotically unbiased stochasticestimator based on a reformulation of AUPRC.
Notably, to ensure the stability of the estimator, theobjective is formulated as a two-level compositional problem.
To solve this problem, we proposean algorithm that combines stochastic gradient descent (SGD), linear interpolation and exponentialmoving average.
Error analysis further supports the feasibility of our method, and inspires us to add asemi-variance regularization term.
Facing challenge (b) , we extend instancewise model stability to listwise model stability, and correspondingly put forward the generalization via stability of listwise problems.
On top of this, we bridgethe generalization of AUPRC and the stability of the proposed optimization algorithm.
As for challenge (c) , the key is to ﬁnd an upper bound on the variation of model parameters withslight jitter in the dataset.
Since the variables to be optimized are typically updated alternately inthe compositional optimization problem, we propose state transition matrices of these variables, andsimplify the calculations of the stability with matrix spectrum.
We also provide the convergenceanalysis of the proposed method.
Last but not least, empirical studies on three real-world datasets further validate the effectiveness andthe soundness of the proposed framework.
In a nutshell, the main contributions of this paper are summarized as follows:
• Algorithmically, a stochastic learning algorithm is proposed for AUPRC optimization.
Thecore of the proposed algorithm is a stochastic estimator which is sampling-rate-invariantasymptotically unbiased.•
Theoreticallly, we present the ﬁrst trial on the algorithm-dependent generalization of stochas-
tic AUPRC optimization.
To the best of our knowledge, it is also the ﬁrst work to analyze
the stability of stochastic compositional optimization problems.
• Technically, we extend the concept of the stability and generalization guarantee to listwisenon-convex losses.
Then we simplify the stability analysis of compositional objective by
matrix spectrum.
These techniques might be instructive for other complicated metrics.
Problem Formulation
2.1 Preliminaries on AUPRC
Notations.
Consider a set of N examples S
= { ( x i , y i ) } Ni =1 independently drawn from a samplespace D = X
× Y , where X is the input space and Y = {− 1 , 1 } is the label space.
For sake of thepresentation, denote the set of positive examples of S as S + = { x + i } N + i =1 , and similarly the set ofnegative examples is denoted as S − = { x − i } N − i =1 , where N + = |S + | , N − =
|S − | .
With a slightabuse of notation, we also denote S = S + ∪ S − if there is no ambiguity.
Generally, we assumethat the dataset is sufﬁciently large, such that N + / ( N + + N − ) =
P ( y = 1) :
= π .
Our target is tolearn a score function h w : X (cid:55)→ R with parameters w ∈ Ω ⊆ R d , such that the scores of positive examples are higher than negative examples.
Furthermore, when appling the score function to adataset S ∈ X N , we denote h w : X N (cid:55)→ R N , where the k -th element of h w ( S ) has the top- kvalues of { h w ( x ) | x ∈ S} .
Denote the asymptotic upper bound on complexity as O , and denoteasymptotically equivalent as (cid:16) .
In this work, our main interest is to optimize a score function in the view of AUPRC:
[Equation]
where ( x , y ) ∼ D , c refers to a threshold, and TPR ( c ) =
P ( h w ( x ) ≥ c | y = 1) , FPR ( c ) = 92 P ( h w ( x ) ≥ c | y = 0) .
For a ﬁnite set S , AUPRC is typically approximated by replacing thedistribution function P ( h w ( x ) ≥ c | y = 1) with its empirical cumulative distribution function [8, 19]: (cid:92) AUPRC
[Equation]
w ;
[Equation]
where (cid:91) TPR
[Equation]
c
[Equation]
= ˆ E
[Equation] [Equation]
( x ))] , (cid:96) 0 , 1 ( x ) = 1 ifx ≤ 0 or (cid:96) 0 , 1 ( x ) = 0 otherwise.
It has been shown that (cid:92) AUPRC is an unbiased estimator when
N + / ( N + + N − )
→ π and N → ∞ [8].
With the above estimation, we have the followingoptimization objective:
[Equation]
where σ ( x ) = x/ (1 + x ) is concave and monotonically increasing.
To make it smooth, surrogatelosses (cid:96) 1 , (cid:96) 2 are used to replace (cid:96) 0 , 1 in (cid:91) FPR and (cid:91) TPR respectively, yielding the following surrogateobjective:
[Equation]
where (cid:91) TPR ( c ; (cid:96) 2 )
=
ˆ E x ∼S + [ (cid:96) 2 ( c − h w ( x ))] , (cid:91) FPR ( c ; (cid:96) 1 ) =
ˆ E x ∼S − [ (cid:96) 1 ( c − h w ( x ))] .
Speciﬁcally, when N + / ( N + + N − )
=
π , it is equivalent to another commonly used formulation Average
Precision (AP) Loss :
[Equation]
2.2 Stochastic Learning of AUPRC
Under the stochastic learning framework for instancewise losses, the empirical risk F ( w ; S ) isexpressed as a sum of instancewise losses: F ( w ; S )
= 1 N (cid:80) x
∼S ˆ f ( w ; x ) , where ˆ f ( w ; x ) is thestochastic estimator of F ( w ; S ) .
Different from instancewise losses, listwise losses like AUPRCrequire a batch of samples to calculate the stochastic estimator.
Speciﬁcally, at each step, a subset of
S :
z = z + ∪ z − is randomly drawn, where z + consists of n + positive examples and z − consistsof n − negative examples.
Then a stochastic estimator of the loss function, denoted as ˆ f ( w ; z ) , iscomputed with z .
Similar to the instancewise case, we consider a variant of the empirical/population
AUPRC risks as approximations, which is a sum of stochastic losses w.r.t.
all posible z :
[Equation]
6) where M is the number of all posible z .
Unfortunately, due to the non-decomposability of theempirical AUPRC risk f ( w ; S ) , it is tackle to determine the approximation errors between F ( w ; S )and f ( w ; S ) in general.
Nonetheless, in Sec.
3.3 we argue that by selecting
Asymptotically Unbiased Stochastic AUPRC Optimization
In this section, we will present our SGD-style stochastic optimization algorithm of AUPRC.
In
Sec. 3.1, we propose surrogate losses to make the objective function differentiable.
In Sec. 3.2, wepresent details of the proposed stochastic estimator and the corresponding optimization algorithm.
Analyses on approximation errors are provided in Sec. 3.3.
3.1 Differentiable Surrogate Losses
Since (cid:96) 0 , 1 appears in both the numerator and denominator of Eq.
(4), simply implementing (cid:96) 1 , (cid:96) 2 witha single function [55, 9, 53] will bring difﬁculty to analyze the relationship between (cid:92) AUPRC ↓ ( w ; S )and f ( w ; S ) .
This motivates us to choose (cid:96) 1 ≥ (cid:96) 0 , 1 , (cid:96) 2 ≤ (cid:96) 0 , 1 , such that (cid:92) AUPRC ↓ ( w ; S ) ≤f ( w ; S ) , thus the original empirical risk could be optimized by minimizing its upper bound f ( w ; S ) .
Concretely, (cid:96) 1 and (cid:96) 2 are deﬁned as the one-side Huber loss and the one-side sigmoid loss:(cid:96) 1 ( x ) =
  − 2 x/τ 1 , x < 0 , (1 − x/τ 1 ) 2 , 0 ≤ x < τ 1 , 0 , x ≥ τ 1 .
(cid:96) 2 ( x ) =
  exp( − x/τ 2 ) −exp( − x/τ 2 )
+ 1 , x < 0 , 0 , x ≥ 0 .
Here τ 1 , τ 2 > 0 are hyperparameters.
(cid:96) 1 is convex and decreasing, which ensures the gap betweenpositive-negative pairs is effectively optimized.
Additionally, compared with the square loss and theexponential loss, (cid:96) 1 is more robust to noises.
(cid:96) 2 is Lipschitz continuous, and (cid:96) 2 → (cid:96) 0 , 1 with τ 2 → 0 .
3.2 Stochastic Estimator of AUPRC
The key to a stochastic learning framework is the design of the stochastic estimator (or the corresponding gradients), i.e. , ˆ f ( w ; z ) .
Existing methods [9, 72, 12] implement it with (cid:99) AP ↓ ( w ; z ) (Eq.
(5)),which might suffer from two problems:
(P1)
Comparing Eq.
and Eq
(5), it
can
be
seen that
only when n +
( n + + n − )
→ π , (cid:99) AP ↓
is an asymptotically unbiased estimator
However, it is hardly satisﬁed since the samplingstrategy is usually biased in practice
(P2)
Each term in the summation of (cid:99) AP ↓ is related to all instances of a batch, leading to weak
leave-one-out stability, i
e. , changing one instance might result in a relatively large ﬂuctuation
in the stochastic gradient, especially when changing a positive example.
To tackle the above problems, we ﬁrst substitute (cid:91) FPR ( h w ( x + ); (cid:96) 1 ) with ˆ E x ∼ z − [ (cid:96) 1 ( h w ( x + ) −h w ( x ))] , and then introduce an auxiliary vector v ∈ R N + to estimate (cid:91) TPR .
Formally, we proposethe following batch-based estimator:
[Equation]
] (cid:33)(cid:35) .
Such an estimator enjoys two advantages: in terms of P1 , it is asymptotically unbiased regardless ofthe sampling rate (see Sec.
3.3 for detailed discussions); as for P2 , we use v to substitute h w ( S + ) ,such that each positive example in a mini-batch only appears in one term.
Ideally, it can be consideredas using all positive examples in the dataset to estimate (cid:91) TPR instead of that from a mini-batch.
Withthe fact that n − (cid:29) n + , this makes the corresponding algorithm more stable.
Moreover, based on themodel stability, generalization bounds are available (see Sec. 4).
3.3 Analyses on Approximation Errors
In this subsection, we analyze errors from two approximations in the above algorithm: 1) the gapbetween F ( w ; S ) and the true AUPRC loss; 2) the gap between the interpolated scores φ ( h w ( z + ))and the true scores h w ( S + ) .
Proofs are provided in Appendix B.1.
−   4 x
D
en s i t y
Negative
Positive (
[Figure]
) Density functions of scores.
10000 Batch Size
N o r m a li z ed A U P
RC
Ours
AP ...
10000 Batch Size
N o r m a li z ed A U P
RC
Ours
AP ...
[Figure]
−
[Figure] [Figure] [Figure]
V a l ue
Interp.
Results
Target Values
[Figure]
Denote π = N + / ( N + + N − ) and π 0 = n + / ( n + + n − ) .
We would like to show that for allw ∈ Ω ,
E z
[ ˆ f ( w ; z )] is an unbiased estimator when n → ∞ , no matter how π 0 is chosen, while for
E z [ (cid:99) AP ↓ ( w ; z )] , it holds only when π 0 = π .
Since only one model w is considered, we let w t
=
win the update rule of v (Eq.
and we have the following proposition:
Proposition 1.
Consider updating v with Eq.
(10) for T steps, then we have
[Equation]
Remark 1.
Two conclusions could be drawn from the above proposition:
ﬁrst, if the linear interpolation is asymptotically unbiased (see next subsection), by choosing a large T or settingv 1 = E [ φ ( h w ( z + ))] , we have E
[ v ] ≈ h w ( S + ) ; second, by choosing a smaller β , v is more likelyto concentrate on h w ( S + ) .
Proposition 2.
Assume the linear interpolation is asymptotically unbiased.
Let κ=ˆ E c ∼ h w ( z + )
[ V ar x ∼S − [ (cid:96) 1 ( c − h w ( x ))]] , κ
=
ˆ E c ∼ h w ( z + )
[ V ar v ∼ v [ (cid:96) 2 ( c − v ))]] .
When κ 21 /n − →0 , κ 22 /n + → 0 , then there exists a positive scale H , such that
[Equation]
where P → refers to convergence in probability, and z ⊆ S refers to subsets described in Sec.
Remark 2.
The above proposition suggests that the proposed batch-based estimator is samplingrate-invariant asymptotically unbiased, while (cid:99) AP ↓ tends to be larger when the sampling rate of thepositive class is greater than the prior, and vice versa.
We also provide a non-asymptotic result in
Appendix B.2.
Simulation experiments are conducted as complementary to the theory.
Following previous work[8], the scores are drawn from three types of distributions, including binormal, bibeta and offsetuniform.
The results of binormal distribution are visualized in Fig.
1, and detailed descriptions andmore results are available in Appendix B.2.
These results are consistent with the above remark.
Next we further study the interpolation error.
For the sake of presentation, denote p : [0 , 1] (cid:55)→ R to bean increasing score function describing h w ( S + ) , where p ( x ) is the score in the bottom x -quantile ofh w ( S + ) .
Similarly, let ˆ p to be the interpolation results of E A [ h w ( z + )] .
Assume that E A [ h w ( z + )]are located in the ( i/n + ) -quantiles of p , where i ∈ [ n + ] , such that p ( i/n + )
=
ˆ p ( i/n + ) and allinterpolation intervals are with length 1 /n
+ .
The following proposition provides an upper bound ofthe approximation error (see [60] for proof):
Proposition 3 ( Linear Interpolation Error ) .
Let p, ˆ p be deﬁned as above.
Then we have
[Equation]
Similar to the last subsection, simulation results are shown in Fig.
1(c), which shows the expectederrors of linear interpolation are ignorable.
3.4 Optimization Algorithm
In the rest of this section, we focus on how to optimize F ( w ; S ) .
The main challenge is to designupdate rules for v , such that it could efﬁciently and effectively approximate h w ( S + ) without fullbatch scanning.
To overcome the challenge, we propose an algorithm called Stochastic Optimization of AUPRC (SOPRC) , which jointly updates model parameters w and the auxiliary vector v .
Asummary of the detailed process is shown as Alg. 1.
At step t , a batch of data is sampled from thetraining set, and then compute the corresponding scores.
Afterward, scores of positive examples aremapped into a N + -dimension vector with linear interpolation φ as shown in Alg. 2.
v t +1 are updatedwith the interpolated scores in a moving average manner.
Practically, n + , n − are ﬁnite, causing inevitable estimation errors in f ( w ; z i t , v t +1 ) .
Notice thatanother factor inﬂuencing the stochastic estimation errors, i.e. , κ 21 and κ 22 .
To reduce them, it isexpected that the variance of positive (negative) scores are small, which motivates us to add a varianceregularization term.
However, it might force to reduce positive scores that higher than the mean value,which is contrary to our target.
Therefore, we propose a semi-variance regularization term [4]:
[Equation]
where µ + = 1 n + (cid:80) x ∼ z + h w ( x ) , µ − = 1 n − (cid:80) x ∼ z − h w ( x ) , λ 1 , λ 2 are hyperparameters.
Finally, wecompute the gradients of f ( w ; z i t , v t +1 )
+ L var , and update parameters w with gradient descent.
Compute h w t ( z + i t ) and map the results into φ ( h w t ( z + i t )) with Alg. 2.
Update v with
[Equation]
[Equation]
Compute L var with Eq.
Update the model parameter:
[Equation]
8: end for
Algorithm
Input: A real value vector u ∈ R n where n < N + , range of target values [ b, B ] .
Output: Interpolated vector m = φ ( u ) .
1: Sort u in descending order.
2: Initialize m as 0 N + , let u 0 = max (2 u 1 − u 2 , b ) ,
u n +1 = min (2 u n − 2 u n − 1 , B ) .
3: for i = 1 to n do 4: for j = (cid:100) N
+ ( i − 1) n (cid:101) to (cid:104) N + · i n (cid:105) do 5: m j + = (cid:2)
( i − jn/N + ) u i −+(1 + jn/N + − i )
u i (cid:3) /6: end for 7: for j = (cid:104) N + · i n (cid:105) to (cid:98) N + · ( i +1) n (cid:99) do 8: m j + = (cid:2)
( i + 1 − jn/N + ) u i −+( jn/N + − i ) u i (cid:3) /9: end for 10: end for
Generalization of SOPRC via Stability
In this section, we turn to study the excess generalization error of the proposed algorithm.
Formally,following standard settings [5], we consider the test error of the model
A ( S ) trained on the trainingset S .
Our target is to seek an upper bound of the excess error E A, S [ F ( A ( S ))
− F ( w ∗ )] , wherew ∗ ∈ arg min w ∈ Ω E A, S [ F ( w ∗ )] .
It can be decomposed as:
[Equation]
The estimation error sources from the gap of minimizing the empirical risk instead of the expectedrisk.
In Sec. 4.1, we provide detailed discussion on the estimation error.
The optimization errormeasures the gap between the minimum empirical risk and the results obtained by the optimizationalgorithm, which will be studied in Sec. 4.2.
Detailed proofs of this section are available in Appendix
C. Before the formal presentation, we show the main assumptions:
Assumption 1 ( Bounded Scores & Gradient ) .
| ˆ f ( w ; · ) | ≤ B, (cid:107)∇ ˆ f ( w ; · ) (cid:107) 2 ≤ G for all w ∈ Ω .
Assumption 2 ( L-Smooth Loss ) .
(cid:107)∇ ˆ f ( w ; · ) − ∇ ˆ f ( ˜ w ; · ) (cid:107) 2 ≤ L (cid:107) w − ˜ w (cid:107) 2 for all w , ˜ w ∈ Ω .
Assumption 3 ( Lipschitz Continuous Functions ) .
| (cid:96) 1 ( x ) − (cid:96) 1 (˜ x ) | ≤ L 1 | x − ˜ x | , | (cid:96) 2 ( x ) − (cid:96) 2 (˜ x ) | ≤
L 2 | x − ˜ x | for all x, ˜ x ∈ [ − 2 B, 2 B ] .
(cid:107)
[Table]
4.1 Generalization of AUPRC via Model Stability
The generalization of SGD-style algorithms for instancewise loss has been widely studied withstability measure [38, 21, 28].
However, these results could not be directly applied to listwise losseslike AUPRC.
The main reason is that the estimation of each stochastic gradient requires a list ofexamples, and the estimation is usually biased.
Nonetheless, to bridge the optimization algorithm andthe generalization of AUPRC, we propose a listwise variant of on-average model stability [38] asfollows:
Deﬁnition 1 ( Listwise On-average Model Stability ) .
Let S = { ( x i , y i ) } Ni =1 and (cid:101) S
={ ( (cid:101) x i , y i ) }
Ni =1 be two sets of examples whose features are drawn independently from X .
For anyi = 1 , · · · , N , denote S ( i ) = { ( x 1 , y 1 ) , · · · , ( x i − 1 , y i − 1 ) , ( (cid:101) x
i , y i ) , ( x i +1 , y i +1 ) , · · · , ( x n , y n ) } .
Astochastic algorithm
A is listwise on-average model ( (cid:15) + , (cid:15) − ) -stable
if the following condition holds:
[Equation]
S
[Equation]
The following theorem shows that the estimation error is bounded by the above-deﬁned stability:
Theorem 1 ( Generalization via Model Stability ) .
Let a stochastic algorithm
A be listwise onaverage model ( (cid:15) + , (cid:15) − ) -stable
and
Asmp.
1 holds.
Then we have
[Equation]
With the above theorem, now we only need to focus on the model stability of the proposed algorithm.
Notice that in Alg. 1, both w t and v t are updated at each step, thus we have to consider the stabilityof both simultaneously.
The following lemma provides a recurrence for the stability w t and v t .
Lemma 1.
Let S , (cid:101) S , S
( i ) be constructed as Def. 1 and Asmp. 1, 2, 3 hold.
Let { w t } t and { w ( i ) t } tbe produced by Alg. 1 with S and S ( i ) , respectively.
Denote L = max { L w , L v /n
+ , C φ B, G/ 2 , B (cid:48) (cid:96) } ,
[Equation]
( i
[Equation]
1 N + (cid:80) y i = − 1 E S ,A (cid:104) m
( i ) t (cid:105) .
Then for all t ∈ [ T ] , by setting β t ≤ 2 C φ B/n + , we have
[Equation]
where I 3 is the 3 × 3 identity matrix and
[Equation]
Finally, we utilize the matrix spectrum of R + t and R − t to show that the model stability w.r.t.
Alg.
1decreases as the number of training examples increases (see Appendix C.2 for details):
Theorem 2.
Let λ =
LC η
(1 + (cid:112) 1 − β 2 + β ) , and assumptions in Lem.
1 hold.
By setting η t ≤ C η t ,β t = β (cid:16) 1 /n + and T ≤ N + , Alg. 1 is list on-average model stable with
[Equation]
4.2 Convergence of AUPRC Stochastic Optimization
Following previous work [24, 34], we study the optimization error of the proposed algorithm underthe Polyak-Łojasiewicz (PL) condition.
It has been shown that the PL condition holds for severalwidely used models including some classes of neural networks [13, 41].
Assumption 4 ( Polyak-Łojasiewicz Condition [34, 37]) .
Denote w ∗ = arg min w ∈ Ω F ( w ) .
Assume F satisfy the expectation version of PL condition with parameter µ > 0 , i.e.,
[Equation]
[Table]
The main difference to the existing convergence analysis on non-convex optimization is that thegradient estimation is biased.
Nonetheless, we show that the bias terms from Alg.
1 tend to 0 withsufﬁcient training data and training time (see Appendix C.3), leading to the following convergence:
Theorem 3.
Let Asmp.
1, 3, 4 hold.
By setting η t = 2 t +1 µ ( t +1) 2 and β t = β (cid:16) 1 /n
+ , we have
[Equation]
Theorem 4.
Let assumptions in Thm.
2 and 3 hold.
By setting T (cid:16) ( N + ) λ +1 2 λ +
( n + ) − 1 2 λ
+1 , we have
[Equation]
Remark 3.
Recall that λ =
LC η
(1 + (cid:112) 1 − β 2 + β ) and C η = 4 /µ , when β is small,we have λ ≈ 4 L/µ .
Here L/µ is a condition number determined by the model andsurrogate losses.
Notice that n + (cid:28) N + , n − (cid:28) N − , if λ = 1 , the generalizationbound is O (cid:0) ( N + ) − 2 / 3 · ( n + ) 4 / 3 + ( N − ) − 2 / 3 · ( n − ) 4 / 3 (cid:1) .
As λ increases, it increases to
O (cid:0) ( N + ) − 1 / 2 · ( n + ) 3 / 2 + ( N − ) − 1 / 2 · ( n − ) 3 / 2 (cid:1) .
Experiments
To validate the effectiveness of the proposed method, we conduct empirical studies on the imageretrieval task, in which data distributions are largely skewed and AUPRC is commonly used as anevaluation metric.
Detailed experimental settings are available in Appendix D.1.
5.1 Datasets
We evaluate the proposed method on three image retrieval benchmarks with various domains andscales, including Stanford Online Products (SOP) 1 [47], PKU VehicleID 2 [42] and iNaturalist 3[67].
We follow the ofﬁcial setting to split a test set from each dataset, and then further split the restinto a training set and a validation set by a ratio of 9 : 1 .
5.2 Main Results
We evaluate all methods with mean AUPRC (mAUPRC) and Recall@k .
mAUPRC measures themean value of the AUPRC over all queries, a.k.a.
mean average precision (mAP).
The performancecomparisons on test sets are shown in Tab. 1.
Consequently, we have the following observations: 1)
In all datasets, the proposed method surpasses all competitors in the view of mAUPRC, especially inthe large-scale long-tailed dataset iNaturalist.
This validates the advantages of our method in boosting
[Figure]
[Table]
the AUPRC of models.
Compared to pairwise losses, the AUPRC/AP optimization methods enjoybetter performance generally.
The main reason is that pairwise losses could only optimize modelsindirectly by constraining relative scores between positive and negative example pairs, while ignoringthe overall ranking. 3)
Although some pairwise methods like XBM have a satisfying performance on
Recall@1, their mAUPRC is relatively low.
It is caused by the limitation of Recall@1, i.e. , it focuseson the top-1 score while ignoring the ranking of other examples.
What’s more, this phenomenonshows the inconsistency of Recall@k and AUPRC, revealing the necessity of studying AUPRCoptimization.
More results are available in Appendix D.2.
To qualitatively demonstrate the effect ofthe proposed method, we also show the mean PR curves and convergence curves in Fig.
5.3 Ablation Studies
We further investigate the effect of different components of the proposed method.
Results are shownin Tab. 2, and more detailed statements and analyses are as follows.
Effect of Unbiased Estimator.
To show the performance drop caused by the biased estimator, wereplace the prior π in Eq.
(8) with n + / ( n + + n − ) .
Comparing line 1 and line 2, using the unbiasedestimator increases the mAUPRC by 1.3%, which is consistent with our theoretical results in Sec. 3.3.
Notably, the unbiased estimator is the main source of improvements in terms of mAUPRC.
Effect of v t .
To show the effect of introducing v t to estimate φ ( S + ) , we directly use φ ( z + ) insteadin the ﬁrst two lines.
Comparing line 2 and line 3, using v t could bring consistent improvements dueto the better generalization ability.
Effect of L var .
We show that shrinking variances could reduce the batch-based estimation errors.
Comparing line 3 and line 4, it can be seen that L var further boosts the proposed method.
Effect of Optimizer.
Comparing line 4 and line 5, it can be seen that the choice of optimizer onlyhas a slight inﬂuence.
Conclusion & Future Work
In this paper, we present a stochastic learning framework for AUPRC optimization.
To begin with, wepropose a stochastic AUPRC optimization algorithm based on an asymptotically unbiased stochasticestimator.
By introducing an auxiliary vector to approximate the scores of positive examples, theproposed algorithm is more stable.
On top of this, we study algorithm-dependent generalization.
First,we propose list model stability to handle listwise losses like AUPRC, and bridge the generalizationand the stability.
Afterward, we show that the proposed algorithm is stable, leading to an upper boundof the generalization error.
Experiments on three benchmarks validate the advantages of the proposedframework.
One limitation is the convergence rate is controlled by the scale of the dataset.
In thefurther, we will consider techniques like variance reduction to improve the convergence rate, andjointly consider the corresponding algorithm-dependent generalization.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Please see Sec.
(c) Did you discuss any potential negative societal impacts of your work?
[No] We haven’t
found any negative societal impact of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[No] Because it would be too computationally expensive.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
(b) Did you mention the license of the assets?
[Yes]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[No] Assets we used are open source.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]