Exploring the Algorithm-Dependent Generalization of AUPRC Optimization with List Stability
Abstract
Stochastic optimization of the Area Under the Precision-Recall Curve (AUPRC) is a crucial problem for machine learning.
Although various algorithms have been extensively studied for AUPRC optimization, the generalization is only guaranteed in the multi-query case.
In this work, we present the ﬁrst trial in the singlequery generalization of stochastic AUPRC optimization.
For sharper generalization bounds, we focus on algorithm-dependent generalization.
There are both algorithmic and theoretical obstacles to our destination.
From an algorithmic perspective, we notice that the majority of existing stochastic estimators are biased when the sampling strategy is biased, and is leave-one-out unstable due to the nondecomposability.
To address these issues, we propose a sampling-rate-invariant unbiased stochastic estimator with superior stability.
On top of this, the AUPRC optimization is formulated as a composition optimization problem, and a stochastic algorithm is proposed to solve this problem.
From a theoretical perspective, standard techniques of the algorithm-dependent generalization analysis cannot be directly applied to such a listwise compositional optimization problem.
To ﬁll this gap, we extend the model stability from instancewise losses to listwise losses and bridge the corresponding generalization and stability.
Additionally, we construct state transition matrices to describe the recurrence of the stability, and simplify calculations by matrix spectrum.
Practically, experimental results on three image retrieval datasets on speak to the effectiveness and soundness of our framework
Introduction
Area Under the Precision-Recall Curve (AUPRC) is a widely used metric in the machine learning community, especially in learning to rank, which effectively measures the trade-off between precision and recall of a ranking model.
Compared with threshold-speciﬁed metrics like accuracy and recall@k, AUPRC reﬂects a more comprehensive performance by capturing all possible thresholds.
In addition, literature has shown that AUPRC is insensitive toward data distributions [20], making it adaptable to largely skewed data.
Beneﬁting from these appealing properties, AUPRC has become one of the standard metrics in various applications, e.g. , retrieval [55, 58, 22, 41], object detection [45, 49, 15], medical diagnosis [50, 35], and recommendation systems [16, 73, 1, 64, 2].
Over the past decades, the importance of AUPRC has prompted extensive researches on direct
AUPRC optimization.
Early work focuses on full-batch optimization [45, 44, 26].
However, in (cid:3) Corresponding authors.
the era of deep learning, the rapidly growing scale of models and data makes these full-batch algorithms infeasible.
Therefore, in recent years, it has raised an increasing favor of the stochastic AUPRC optimization [9, 12, 31, 46].
Since AUPRC optimization is a stochastic dependent compositional optimization problem, general convergence rates are infeasible for AUPRC optimization.
To ﬁll this gap, [54, 70, 69] provide AUPRC optimization algorithms with provable convergence.
See
Appendix A for more on related work.
Despite the promoting performance of these methods in various scenarios, the generalization of
AUPRC optimization algorithms is still an open problem.
Some studies [17, 63] provide provable generalization for AUPRC optimization in information retrieval.
In this scene, a dataset consists of multiple queries, where each query corresponds to a set of positive and negative samples.
However, these results require sufﬁcient queries to ensure small generalization errors, but leave the singlequery case alone, i.e. , whether the generalization error tends to zero with the length of a singlequery increasing is still unclear .
This limits the adaptation scope of these methods.
To ﬁll this gap, in this paper we aim to design a stochastic optimization framework for AUPRC with a provable algorithm-dependent generalization performance in the single-query case.
The target is challenging in three aspects: (a) Most AUPRC stochastic estimators are biased with a biased sampling rate.
Moreover, due to the non-decomposability, outputs of existing algorithms might change a lot with slight changes in the training data, which is called leave-one-out unstable in this paper.
Such an unstability is harmful to the generalization.
(b) The standard framework to analyze the algorithm-dependent generalization requires the objective function to be expressed as a sum of instancewise terms, while AUPRC involves a listwise loss.
(c)
The stochastic optimization of
AUPRC is a two-level compositional optimization problem, which brings more complicated proofs of the stability.
In search of a solution to (a) , we propose a sampling-rate-invariant asymptotically unbiased stochastic estimator based on a reformulation of AUPRC.
Notably, to ensure the stability [28, 39, 37, 38] of the estimator, the objective is formulated as a two-level compositional problem by introducing an auxiliary vector for the ranking estimation.
Error analysis further supports the feasibility of our method, and inspires us to add a semi-variance regularization term.
To solve this problem, we propose an algorithm with provable convergence that combines stochastic gradient descent (SGD), linear interpolation and exponential moving average.
Facing challenge (b) , we extend instancewise model stability to listwise model stability, and correspondingly put forward the generalization via stability of listwise problems.
On top of this, we bridge the generalization of AUPRC and the stability of the proposed optimization algorithm.
As for challenge (c) , since the variables to be optimized are typically updated alternately in the compositional optimization problem, we propose state transition matrices of these variables, and simplify the calculations of the stability with matrix spectrum.
In a nutshell, the main contributions of this paper are summarized as follows:
(cid:15) Algorithmically, a stochastic learning algorithm is proposed for AUPRC optimization.
The core of the proposed algorithm is a stochastic estimator which is sampling-rate-invariant asymptotically unbiased.
(cid:15) Theoreticallly, we present the ﬁrst trial on the algorithm-dependent generalization of stochastic AUPRC optimization.
To the best of our knowledge, it is also the ﬁrst work to analyze the stability of stochastic compositional optimization problems.
(cid:15) Technically, we extend the concept of the stability and generalization guarantee to listwise non-convex losses.
Then we simplify the stability analysis of compositional objective by matrix spectrum.
These techniques might be instructive for other complicated metrics.
Problem Formulation 2.1 Preliminaries on AUPRC
Notations.
Consider a set of N examples S = f
( x i ; y i ) g Ni =1 independently drawn from a sample space D = X (cid:2) Y , where X is the input space and Y = f(cid:0) 1 ; 1 g is the label space.
For sake of the presentation, denote the set of positive examples of S as S + = f x + i g N + i =1 , and similarly the set of negative examples is denoted as S (cid:0) =
f x (cid:0) i g N (cid:0) i =1 , where N + = jS + j ; N (cid:0) = jS (cid:0) j .
With a slight abuse of notation, we also denote S = S + [ S (cid:0) if there is no ambiguity.
Generally, we assume that the dataset is sufﬁciently large, such that N + = ( N + + N (cid:0) ) =
P ( y = 1) :
= (cid:25) . Our target is to learn a score function h w : X 7! R with parameters w 2 Ω (cid:18) R d , such that the scores of positive examples are higher than negative examples.
Furthermore, when appling the score function to a dataset S 2 X N , we denote h w : X N 7! R N , where the k -th element of h w ( S ) has the top- k values of f h w ( x ) j x 2 Sg .
Denote the asymptotic upper bound on complexity as O , and denote asymptotically equivalent as ≍ .
In this work, our main interest is to optimize a score function in the view of AUPRC:
[Equation]
where ( x ; y ) (cid:24) D , c refers to a threshold, and TPR ( c ) =
P ( h w ( x ) (cid:21) c j y = 1) ; FPR ( c ) =
P ( h w ( x ) (cid:21) c j y = 0) .
For a ﬁnite set S , AUPRC is typically approximated by replacing the distribution function P ( h w ( x ) (cid:21) c j y = 1) with its empirical cumulative distribution function [8, 19]:
[Equation]
where [ TPR ( c ) =
^ E x (cid:24)S + [ ℓ 0 ; 1 ( c (cid:0) h w ( x ))] ; [ FPR ( c ) =
^ E x (cid:24)S (cid:0) [ ℓ 0 ; 1 ( c (cid:0) h w ( x ))] , ℓ 0 ; 1 ( x ) =[-
-]
if x (cid:20) 0 or ℓ 0 ; 1 ( x ) = 0 otherwise.
It has been shown that \ AUPRC is an unbiased estimator when
N + = ( N + + N (cid:0) ) !
(cid:25) and N ! 1 [8].
With the above estimation, we have the following optimization objective:
[Equation]
where (cid:27) ( x ) = x= (1 + x ) is concave and monotonically increasing.
To make it smooth, surrogate losses ℓ 1 ; ℓ 2 are used to replace ℓ 0 ; 1 in [ FPR and [ TPR respectively, yielding the following surrogate objective:
[Equation]
where [ TPR ( c ; ℓ 2 ) =
^ E x (cid:24)S + [ ℓ 2 ( c (cid:0) h w ( x ))] ; [ FPR ( c ; ℓ 1 ) =
^ E x (cid:24)S (cid:0) [ ℓ 1 ( c (cid:0) h w ( x ))] .
Speciﬁcally, when N + = ( N + + N (cid:0) ) = (cid:25) , it is equivalent to another commonly used formulation Average
Precision (AP) Loss :
[Equation]
2.2 Stochastic Learning of AUPRC
Under the stochastic learning framework for instancewise losses, the empirical risk F ( w ; S ) is expressed as a sum of instancewise losses: F ( w ; S )
=
1 N ∑ x (cid:24)S ^ f ( w ; x ) , where ^ f ( w ; x ) is the stochastic estimator of F ( w ; S ) .
Different from instancewise losses, listwise losses like AUPRC require a batch of samples to calculate the stochastic estimator.
Speciﬁcally, at each step, a subset of S : z = z + [ z (cid:0) is randomly drawn, where z + consists of n + positive examples and z (cid:0) consists of n (cid:0) negative examples.
Then a stochastic estimator of the loss function, denoted as ^ f ( w ; z ) , is computed with z .
Similar to the instancewise case, we consider a variant of the empirical/population
AUPRC risks as approximations, which is a sum of stochastic losses w.r.t.
all posible z :
[Equation]
where M is the number of all posible z .
Unfortunately, due to the non-decomposability of the empirical AUPRC risk f ( w ; S ) , it is tackle to determine the approximation errors between F ( w ; S ) and f ( w ; S ) in general.
Nonetheless, in Sec.
3.3 we argue that by selecting proper ^ f ( w ; z ) , F ( w ; S ) can be asymptotically unbiased estimator of f ( w ; S ) , which naturally makes F ( w ) an asymptotically unbiased estimator of 1 (cid:0) AUPRC.
In this case, ^ f is said to be an asymptotically unbiased stochastic estimator.
Moreover, if the unbiasedness holds under biased sampling rate, it is said to be sampling-rate-invariant asymptotically unbiased.
Asymptotically Unbiased Stochastic AUPRC Optimization
In this section, we will present our SGD-style stochastic optimization algorithm of AUPRC.
In
Sec. 3.1, we propose surrogate losses to make the objective function differentiable.
In Sec. 3.2, we present details of the proposed stochastic estimator and the corresponding optimization algorithm.
Analyses on approximation errors are provided in Sec. 3.3.
3.1 Differentiable Surrogate Losses
Since ℓ 0 ; 1 appears in both the numerator and denominator of Eq.
(4), simply implementing ℓ 1 ; ℓ 2 with a single function [56, 9, 54] will bring difﬁculty to analyze the relationship between \ AUPRC # ( w ; S ) and f ( w ; S ) .
This motivates us to choose ℓ 1 (cid:21) ℓ 0 ; 1 ; ℓ 2 (cid:20) ℓ 0 ; 1 , such that \ AUPRC # ( w ; S ) (cid:20) f ( w ; S ) , thus the original empirical risk could be optimized by minimizing its upper bound f ( w ; S ) .
Concretely, ℓ 1 and ℓ 2 are deﬁned as the one-side Huber loss and the one-side sigmoid loss:
[Equation]
Here (cid:28) 1 ; (cid:28) 2 > 0 are hyperparameters.
ℓ 1 is convex and decreasing, which ensures the gap between positive-negative pairs is effectively optimized.
Additionally, compared with the square loss and the exponential loss, ℓ 1 is more robust to noises.
ℓ 2 is Lipschitz continuous, and ℓ 2 ! ℓ 0 ; 1 with (cid:28) 2 ! 0 .
3.2 Stochastic Estimator of AUPRC
The key to a stochastic learning framework is the design of the stochastic estimator (or the corresponding gradients), i.e. , ^ f ( w ; z ) .
Existing methods [9, 74, 12] implement it with c AP # ( w ; z ) (Eq.
(5)), which might suffer from two problems:
(P1)
Comparing Eq.
(4) and Eq.
(5), it can be seen that only when n + = ( n + + n (cid:0) ) ! (cid:25) , c AP # is an asymptotically unbiased estimator.
However, it is hardly satisﬁed since the sampling strategy is usually biased in practice.
(P2)
Each term in the summation of c AP # is related to all instances of a batch, leading to weak leave-one-out stability,
i
e. , changing one instance might result in a relatively large ﬂuctuation in the stochastic gradient, especially when changing a positive example.
To tackle the above problems, we ﬁrst substitute [ FPR ( h w ( x + ); ℓ 1 ) with ^ E x (cid:24) z (cid:0) [ ℓ 1 ( h w ( x + ) (cid:0) h w ( x ))] , and then introduce an auxiliary vector v 2 R N + to estimate [ TPR .
Formally, we propose the following batch-based estimator:
[Equation]
Such an estimator enjoys two advantages: in terms of P1 , it is asymptotically unbiased regardless of the sampling rate (see Sec.
3.3 for detailed discussions); as for P2 , we use v to substitute h w ( S + ) , such that each positive example in a mini-batch only appears in one term.
Ideally, it can be considered as using all positive examples in the dataset to estimate [ TPR instead of that from a mini-batch.
With the fact that n (cid:0) ≫ n + , this makes the corresponding algorithm more stable.
Moreover, based on the model stability, generalization bounds are available (see Sec. 4).
3.3 Analyses on Approximation Errors
In this subsection, we analyze errors from two approximations in the above algorithm: 1) the gap between F ( w ; S ) and the true AUPRC loss; 2) the gap between the interpolated scores ϕ ( h w ( z + )) and the true scores h w ( S + ) .
Proofs are provided in Appendix B.1.
Denote (cid:25) = N + = ( N + + N (cid:0) ) and {+(cid:25) 
+} = n + = ( n + + n (cid:0) ) .
We would like to show that for all w 2 Ω , E z [ ^ f ( w ; z )] is an unbiased estimator when n ! 1 , no matter how (cid:25) 0 is chosen, while for
E z [ c AP # ( w ; z )] , it holds only when {+(cid:25) 
+} = (cid:25) .
Since only one model w is considered, we let w t =
w in the update rule of v (Eq.
and we have the following proposition:
Proposition 1.
Consider updating v with Eq.
(10) for T steps, then we have
[Equation]
Remark 1.
Two conclusions could be drawn from the above proposition:
ﬁrst, if the linear interpolation is asymptotically unbiased (see next subsection), by choosing a large T or setting v 1 = E
[ ϕ ( h w ( z + ))] , we have E [ v ] (cid:25) h w ( S + ) ; second, by choosing a smaller (cid:12) , v is more likely to concentrate on h w ( S + ) .
Proposition 2.
Assume the linear interpolation is asymptotically unbiased.
Let (cid:20)= ^ E c (cid:24)
{+h w ( z + ) [ V ar x (cid:24)S (cid:0) [ ℓ 1 ( c (cid:0) h w ( x ))]] , (cid:20) 
 = ^ E c (cid:24) h w ( z + )+}
[ V ar v (cid:24) v [ ℓ 2 ( c (cid:0) v ))]] .
When (cid:20) 21 =n (cid:0) ! 0 , (cid:20) 22 =n + ! 0 , then there exists a positive scale H , such that
[Equation]
where P ! refers to convergence in probability, and z (cid:18) S refers to subsets described in Sec. 2.2.
Remark 2.
The above proposition suggests that the proposed batch-based estimator is samplingrate-invariant asymptotically unbiased, while c AP # tends to be larger when the sampling rate of the positive class is greater than the prior, and vice versa.
We also provide a non-asymptotic result in
Appendix B.2.
Simulation experiments are conducted as complementary to the theory.
Following previous work [8], the scores are drawn from three types of distributions, including binormal, bibeta and offset uniform.
The results of binormal distribution are visualized in Fig.
1, and detailed descriptions and more results are available in Appendix B.2.
These results are consistent with the above remark.
Next we further study the interpolation error.
For the sake of presentation, denote p : [0 ; 1] 7! R to be an increasing score function describing h w ( S + ) , where p ( x ) is the score in the bottom xquantile of h w ( S + ) .
Similarly, let ^ p to be the interpolation results of E A [ h w ( z + )] .
Assume that E A [ h w ( z + )] are located in the ( i=n + ) -quantiles of p , where i 2 [ n + ] , such that p ( i=n + )
=
^ p ( i=n + ) and all interpolation intervals are with length 1 =n + .
The following proposition provides an upper bound of the approximation error (see [61] for proof):
Proposition 3 ( Linear Interpolation Error ) .
Let p; ^ p be deﬁned as above.
Then we have
[Equation]
Similar to the last subsection, simulation results are shown in Fig.
1(c), which shows the expected errors of linear interpolation are ignorable.
3.4 Optimization Algorithm
In the rest of this section, we focus on how to optimize F ( w ; S ) .
The main challenge is to design update rules for v , such that it could efﬁciently and effectively approximate h w ( S + ) without full-batch scanning.
To overcome the challenge, we propose an algorithm called Stochastic Optimization of AUPRC (SOPRC) , which jointly updates model parameters w and the auxiliary vector v .
A summary of the detailed process is shown as Alg. 1.
At step t , a batch of data is sampled from the training set, and then compute the corresponding scores.
Afterward, scores of positive examples are mapped into a N + -dimension vector with linear interpolation ϕ as shown in Alg.
v t +1 are updated with the interpolated scores in a moving average manner.
−   4 x
D
en s i t y
Negative
Positive (a) Density functions of scores.
10000 Batch Size
N o r m a li z ed A U P
RC
Ours
AP ...
10000 Batch Size
N o r m a li z ed A U P
RC
Ours
AP ...
[Figure]
V a
[Figure]
Interp.
Results
Target Values
[Figure]
Practically, n + ; n (cid:0) are ﬁnite, causing inevitable estimation errors in f ( w ; z i t ; v t +1 ) .
Notice that another factor inﬂuencing the stochastic estimation errors, i.e. , (cid:20) 21 and (cid:20) 22 .
To reduce them, it is expected that the variance of positive (negative) scores are small, which motivates us to add a variance regularization term.
However, it might force to reduce positive scores that higher than the mean value, which is contrary to our target.
Therefore, we propose a semi-variance regularization term [4]:
[Equation]
( h w ( x ) (cid:0) (cid:22) (cid:0) ) 2 ; (9) where (cid:22) + = 1 n + ∑ x (cid:24) z
+ h w ( x ) , (cid:22) (cid:0) =
1 n (cid:0) ∑ x (cid:24) z (cid:0) h w ( x ) , (cid:21) 1 ; (cid:21) 2 are hyperparameters.
Finally, we compute the gradients of f ( w ; z i t ; v t +1 )
+ L var , and update parameters w with gradient descent.
t t ) v + (cid:12) t ϕ ( h w t ( z + i t ))
w t +
= w t (cid:0) (cid:17)
(cid:1)
(cid:0) (cid:17) t (cid:1) ∇ f ( w t ; z i t ; v t +1 )
u 2 ; b ) ; u n +1 = min (2 u n (cid:0) 2 u n (cid:0) 1 ; B ) .
3: for i = 1 to n do 4: for j = ⌈ N + ( i (cid:0) 1) n ⌉ to [ N + (cid:1) i n ] do 5: m j + =
[ ( i (cid:0) jn
=N + ) u
i (cid:0)+(1 + jn=N + (cid:0) i ) u i ] =6: end for 7: for j = [ N + (cid:1) i n ] to ⌊ N + (cid:1) ( i +1) n ⌋ do 8: m j + = [ ( i + 1 (cid:0) jn
=N + ) u
i (cid:0)+( jn=N + (cid:0) i ) u i ] =9: end for 10: end for
Generalization of SOPRC via Stability
In this section, we turn to study the excess generalization error of the proposed algorithm.
Formally, following standard settings [5], we consider the test error of the model
A ( S ) trained on the training set S .
Our target is to seek an upper bound of the excess error E A; S [ F ( A ( S )) (cid:0) F ( w (cid:3) )] , where w (cid:3) 2 arg min w 2 Ω E A; S [ F ( w (cid:3) )] .
It can be decomposed as:
E
[Equation]
S )
[Equation]
[Equation]
S
[Equation]
] | {z }
Estimation Error +
[Equation]
] | {z }
Optimization Error :
The estimation error sources from the gap of minimizing the empirical risk instead of the expected risk.
In Sec. 4.1, we provide detailed discussion on the estimation error.
The optimization error measures the gap between the minimum empirical risk and the results obtained by the optimization algorithm, which will be studied in Sec.
Detailed proofs of this section are available in Appendix
C. Before the formal presentation, we show the main assumptions:
Assumption 1 ( Bounded Scores & Gradient ) .
j ^ f ( w ; (cid:1) ) j (cid:20) B; ∥∇ ^ f ( w ; (cid:1) ) ∥ 2 (cid:20) G for all w 2 Ω .
Assumption 2 ( L-Smooth Loss ) .
∥∇ ^ f ( w ; (cid:1) ) (cid:0) ∇ ^ f ( ~ w ; (cid:1) ) ∥ 2 (cid:20) L ∥ w (cid:0) ~ w ∥ 2 for all w ; ~ w 2 Ω .
Assumption 3 ( Lipschitz Continuous Functions ) .
j ℓ 1 ( x ) (cid:0) ℓ 1 (~ x ) j (cid:20) L 1 j x (cid:0) ~ x j , j ℓ 2 ( x ) (cid:0) ℓ 2 (~ x ) j (cid:20)
L 2 j x (cid:0) ~ x j for all x; ~ x 2 [ (cid:0) 2 B; 2 B ] .
∥ ϕ ( x ) (cid:0) ϕ
(~ x ) ∥ 2 (cid:20) C ϕ ∥ x (cid:0) ~ x ∥ 2 for all x ; ~ x 2 R N + .
4.1 Generalization of AUPRC via Model Stability
The generalization of SGD-style algorithms for instancewise loss has been widely studied with stability measure [39, 21, 28].
However, these results could not be directly applied to listwise losses like AUPRC.
The main reason is that the estimation of each stochastic gradient requires a list of examples, and the estimation is usually biased.
Nonetheless, to bridge the optimization algorithm and the generalization of AUPRC, we propose a listwise variant of on-average model stability [39] as follows:
Deﬁnition 1 ( Listwise On-average Model Stability ) .
Let S = f ( x i ; y i ) g Ni =1 and e S =
f ( e x i ; y i )
g Ni =1 be two sets of examples whose features are drawn independently from X .
For any i = 1 ; (cid:1) (cid:1) (cid:1) ; N , denote S ( i ) = f ( x 1 ; y 1 ) ; (cid:1) (cid:1) (cid:1) ; ( x i (cid:0) 1 ; y i (cid:0) 1 ) ; ( e x i ; y i ) ; ( x i +1 ; y i +1 ) ; (cid:1) (cid:1) (cid:1) ; ( x n ; y n ) g .
A stochastic algorithm A is listwise on-average model ( ϵ + ; ϵ (cid:0) )
-stable if the following condition holds: [Equation] S [Equation]
e S
[Equation]
A [Equation] The following theorem shows that the estimation error is bounded by the above-deﬁned stability:
Theorem 1 ( Generalization via Model Stability ) .
Let a stochastic algorithm
A be listwise onaverage model ( ϵ + ; ϵ (cid:0) ) -stable
and
Asmp.
1 holds.
Then we have
[Equation]
With the above theorem, now we only need to focus on the model stability of the proposed algorithm.
Notice that in Alg. 1, both w t and v t are updated at each step, thus we have to consider the stability of both simultaneously.
The following lemma provides a recurrence for the stability w t and v t .
Lemma 1.
Let S ; e S ; S ( i ) be constructed as Def. 1 and Asmp. 1, 2, 3 hold.
Let f w t g t and f w ( i ) t g t be produced by Alg. 1 with S and S ( i ) , respectively.
Denote L = max f L w ; L v =n + ; C ϕ B; G= 2 ; B ′ ℓ g , m
( i ) t
[Equation]
[Equation]
[ m
[Equation]
Then for all t 2 [ T ] , by setting (cid:12) t (cid:20) 2 C ϕ B=n + , we have
[Equation]
where I 3 is the 3 (cid:2) 3 identity matrix and R
[Equation]
[Equation] Finally, we utilize the matrix spectrum of R + t and R (cid:0) t to show that the model stability w.r.t.
Alg.decreases as the number of training examples increases (see Appendix C.2 for details):
Theorem 2.
Let (cid:21) =
LC (cid:17) (1+ √ 1 (cid:0) (cid:12) 2 + (cid:12) ) , and assumptions in Lem.
1 hold.
By setting (cid:17) t (cid:20) C (cid:17) t , (cid:12) t = (cid:12) ≍ 1 =n + and T (cid:20) N + , Alg. 1 is list on-average model stable with
[Equation]
[Table]
4.2 Convergence of AUPRC Stochastic Optimization
Following previous work [24, 34]
[Table]
we study the optimization error of the proposed algorithm under the Polyak-Łojasiewicz (PL) condition.
It has been shown that the PL condition holds for several widely used models including some classes of neural networks [13, 42].
Assumption 4 ( Polyak-Łojasiewicz Condition [34, 37]) .
Denote w (cid:3) =
arg min w 2 Ω F ( w ) .
Assume F satisfy the expectation version of PL condition with parameter (cid:22) > 0 , i.e.,
[Equation]
The main difference to the existing convergence analysis on non-convex optimization is that the gradient estimation is biased.
Nonetheless, we show that the bias terms from Alg.
1 tend to 0 with sufﬁcient training data and training time (see Appendix C.3), leading to the following convergence:
Theorem 3.
Let Asmp.
1, 3, 4 hold.
By setting (cid:17) t = 2 t +1 (cid:22) ( t +1) 2 and (cid:12) t = (cid:12) ≍
[Equation]
[-By setting T (cid:16) ( N + ) λ +1 2 λ +
-]
n + , we have
[Equation]
Theorem 4.
Let assumptions in Thm.
2 and 3 hold.
[-[ V ar x ∼S − [ (cid:96) 1 ( c − h w ( x ))]] , κ 
-]
By setting T ≍ (
[Equation]
n
[Equation]
(cid:0)
[Equation]
[-variables, thus according to Hoeffding’s inequality, for any (cid:15) > 
-]
1 , we have
E
[Equation]
[-Alg. 
-]
+1 (cid:1) ( n (cid:0) ) 3 (cid:21)
+1 2 (cid:21) + ) :
Remark 3.
[-[Equation] Lemma 2. Let z , z (cid:
 ) ∈-]Recall that (cid:21) =
LC (cid:17) (1 + √ 1 (cid:0) (cid:12) 2 + (cid:12) ) and C (cid:17) = 4 =(cid:22) , when (cid:12) is small, we have (cid:21) (cid:25) 4 L=(cid:22) .
Here L=(cid:22) is a condition number determined by the model and surrogate losses.
Notice that n + ≪
N + ; n (cid:0) ≪
N (cid:0) , if (cid:21) = 1 , the generalization bound is {+O ( ( N + ) (cid:0) 2 = 3 (cid:1) ( n + ) 4 = 
+}
+ ( N (cid:0) )
(cid:0) 2 = 3 (cid:1) ( n (cid:0) ) 4 = 3 ) .
[-Lη t Lβ 1 /N +     = T (cid:89) t = t(cid:34) 
-]As (cid:21) increases, it increases to
O ( ( N + ) (cid:0) 1 = 2 (cid:1) ( n + ) 3 = 2 + ( N (cid:0) ) (cid:0) 1 = 2 (cid:1) ( n (cid:0) )
3 = 2 ) .
Experiments
To validate the effectiveness of the proposed method, we conduct empirical studies on the image retrieval task, in which data distributions are largely skewed and AUPRC is commonly used as an evaluation metric.
More detailed experimental settings are provided in Appendix D.1.
The source code is available in https://github.com/KID-7391/SOPRC.git .
5.1 Implementation Details
Datasets.
We evaluate the proposed method on three image retrieval benchmarks with various domains and scales, including Stanford Online Products (SOP) 2 [48], PKU VehicleID 3 [43] and 2 https://github.com/rksltnl/Deep-Metric-Learning-CVPR16 .
Licensed MIT.https:// iNaturalist 4 [68].
We follow the ofﬁcial setting to split a test set from each dataset, and then further split the rest into a training set and a validation set by a ratio of 9 : 1 .
Network Architecture.
The feature extractor is implemented with ResNet-50 [29] pretrained on
ImageNet [60].
Following previous work [12, 9], the batch normalization layers are ﬁxed during training, and the output embeddings are mapped to 512-d with a linear projection.
Given L 2 normalized embeddings of a query image e q and a gallery list f e i g i , the scores are represented by the cosine similarity e ⊤ q e i for all i .
Optimization Strategy.
In the training phase, the input images are resized such that the sizes of the shorter sides are 256.
Afterward, we applied standard data augmentations including random cropping ( 224 (cid:2) 224 ) and random ﬂipping ( 50% ).
The model parameters are optimized in an endto-end manner as shown in Alg.
1, where (cid:12) = 0 : 001 and the weight decay is set to 4 (cid:2) 10 (cid:0) 4 .
The default batch size is set to 224, where each mini-batch is randomly sampled such that there are exactly 4 positive examples per category.
The learning rates are tuned according to performance on validation sets: for SOP, the learning rate are initialized as 0 : 01 and decays by 0 : 1 at the 15 k and 30 k iterations, T = 50 k ; for VehicleID, the learning rate are initialized as 0 : 001 and decays by 0 :at the 40 k and 80 k iterations, T = 100 k ; for iNaturalist, the learning rate are initialized as 0 :and decays by 0 : 1 at the 80 k and 110 k iterations, T = 130 k .
Competitors. We compare two types of competitors: 1) Pairwise Losses , including Contrastive
Loss [27], Triplet Loss [32], Multi-Similarity (MS) Loss [71], Cross-Batch Memory (XBM) [72].
These methods construct loss functions with image pairs or triplets.
2) Ranking-Based Losses , including SmoothAP [9], FastAP [12], DIR [58], BlackBox [33], and Area Under the ROC Curve Loss (AUROC) [76].
These methods directly optimize the ranking-based metrics.
Evaluation Metrics. In all experiments, we adopt evaluation metrics: mean AUPRC (mAUPRC) and Recall@k .
mAUPRC is also called mean average precision (mAP) in literature, which takes the mean value over the AUPRC of all queries. Recall@k measures the probability that at least one positive example is ranked in the top-k list.
5.2 Main Results
We evaluate all methods with mean AUPRC (mAUPRC) and Recall@k .
mAUPRC measures the mean value of the AUPRC over all queries, a.k.a.
mean average precision (mAP). The performance comparisons on test sets are shown in Tab. 1. Consequently, we have the following observations: 1) In all datasets, the proposed method surpasses all competitors in the view of mAUPRC, especially in the large-scale long-tailed dataset iNaturalist.
This validates the advantages of our method in boosting the AUPRC of models. Compared to pairwise losses, the AUPRC/AP optimization methods enjoy better performance generally. The main reason is that pairwise losses could only optimize models indirectly by constraining relative scores between positive and negative example pairs, while ignoring the overall ranking. Although some pairwise methods like XBM have a satisfying performance on Recall@1, their mAUPRC is relatively low.
It is caused by the limitation of Recall@1, i.e. , it focuses on the top-1 score while ignoring the ranking of other examples. What’s more, this phenomenon shows the inconsistency of Recall@k and AUPRC, revealing the necessity of studying AUPRC optimization.
More results are available in Appendix D.2.
To qualitatively demonstrate the effect of the proposed method, we also show the mean PR curves and convergence curves in Fig. The left two subﬁgures demonstrate that the proposed method achieves can effectively improve AUPRC. The right subﬁgure shows the affect of batch size, from which can be seen that a large batch size leads to better performance. One of the reasons is that a small batch size will amplify the AUPRC stochastic estimation error.
Such a problem have been addressed by maintaining inner gradient estimations [54, 70, 75].
Unfortunately, when applied to image retrieval problems, it needs to maintain intermediate variables for each positive pair, bringing high complexity in time and space, thus we leave this problem as further work.
We further investigate the effect of different components of the proposed method. Results are shown in Tab. 2, and more detailed statements and analyses are as follows. [Figure]
M ean
P r e c i s on
TripletBlackBoxFastAPSmoothAPOurs 0.
[Figure]
[Figure]
[Figure]
RC
[Figure]
Triplet
[Figure]
RC
[Figure] [Table] Effect of Unbiased Estimator. To show the performance drop caused by the biased estimator
[Table]
we replace the prior (cid:25) in Eq.
(8) with n + = ( n + + n (cid:0) ) .
Comparing line 1 and line 2, using the unbiased estimator increases the mAUPRC by 1.3%, which is consistent with our theoretical results in Sec. 3.3.
Notably, the unbiased estimator is the main source of improvements in terms of mAUPRC.
Effect of v t . To show the effect of introducing v t to estimate ϕ ( S + ) , we directly use ϕ ( z + ) instead in the ﬁrst two lines. Comparing line 2 and line 3, using v t could bring consistent improvements due to the better generalization ability. Effect of L var .
We show that shrinking variances could reduce the batch-based estimation errors. Comparing line 3 and line 4, it can be seen that L var further boosts the proposed method. Effect of Optimizer. Comparing line 4 and line 5, it can be seen that the choice of optimizer only has a slight inﬂuence.
Conclusion & Future Work
In this paper, we present a stochastic learning framework for AUPRC optimization. To begin with, we propose a stochastic AUPRC optimization algorithm based on an asymptotically unbiased stochastic estimator. By introducing an auxiliary vector to approximate the scores of positive examples, the proposed algorithm is more stable. On top of this, we study algorithm-dependent generalization.
First, we propose list model stability to handle listwise losses like AUPRC, and bridge the generalization and the stability.
Afterward, we show that the proposed algorithm is stable, leading to an upper bound of the generalization error.
Experiments on three benchmarks validate the advantages of the proposed framework.
One limitation is the convergence rate is controlled by the scale of the dataset.
In the further, we will consider techniques like variance reduction to improve the convergence rate, and jointly consider the corresponding algorithm-dependent generalization. Acknowledgments This work was supported in part by the National Key R&D Program of China under Grant 2018AAA0102000, in part by National Natural Science Foundation of China: U21B2038, 61931008, 6212200758 and 61976202, in part by the Fundamental Research Funds for the Central Universities, in part by Youth Innovation Promotion Association CAS, in part by the Strategic Priority Research Program of Chinese Academy of Sciences, Grant No. XDB28000000, in part by