Exploring the Algorithm-Dependent Generalization of AUPRC Optimization with List Stability
Abstract
Stochastic optimization of the Area Under the Precision-Recall Curve (AUPRC) is a crucial problem for machine learning.
Although various algorithms have been extensively studied for AUPRC optimization, the generalization is only guaranteed in the multi-query case.
In this work, we present the ﬁrst trial in the single-query generalization of stochastic AUPRC optimization.
For sharper generalization bounds, we focus on algorithm-dependent generalization.
There are both algorithmic and theoretical obstacles to our destination.
From an algorithmic perspective, we notice that the majority of existing stochastic estimators are biased only when the sampling strategy is biased, and is leave-one-out unstable due to the non-decomposability.
To address these issues, we propose a sampling-rate-invariant unbiased stochastic estimator with superior stability.
On top of this, the AUPRC optimization is formulated as a composition optimization problem, and a stochastic algorithm is proposed to solve this problem.
From a theoretical perspective, standard techniques of the algorithm-dependent generalization analysis cannot be directly applied to such a listwise compositional optimization problem.
To ﬁll this gap, we extend the model stability from instancewise losses to listwise losses and bridge the corresponding generalization and stability.
Additionally, we construct state transition matrices to describe the recurrence of the stability, and simplify calculations by matrix spectrum.
Practically, experimental results on three image retrieval datasets on speak to the effectiveness and soundness of our framework.
Introduction
Area Under the Precision-Recall Curve (AUPRC) is a widely used metric in the machine learning community, especially in learning to rank, which effectively measures the trade-off between precision and recall of a ranking model.
Compared with threshold-speciﬁed metrics like accuracy and recall@k,
AUPRC reﬂects a more comprehensive performance by capturing all possible thresholds.
In addition, literature has shown that AUPRC is insensitive toward data distributions [20], making it adaptable to largely skewed data.
Beneﬁting from these appealing properties, AUPRC has become one of the standard metrics in various applications, e.g. , retrieval [55, 58, 22, 41], object detection [45, 49, 15], medical diagnosis [50, 35], and recommendation systems [16, 73, 1, 64, 2].
Over the past decades, the importance of AUPRC has prompted extensive researches on direct
AUPRC optimization.
Early work focuses on full-batch optimization [45, 44, 26].
However, in the era of deep learning, the rapidly growing scale of models and data makes these full-batch algorithms infeasible.
Therefore, in recent years, it has raised an increasing favor of the stochastic AUPRC optimization [9, 12, 31, 46]. Since AUPRC optimization is a stochastic dependent compositional optimization problem, general convergence rates are infeasible for AUPRC optimization. To ﬁll this gap, [54, 70, 69] provide AUPRC optimization algorithms with provable convergence.
See Appendix A for more on related work.
Despite the promoting performance of these methods in various scenarios, the generalization of
AUPRC optimization algorithms is still an open problem.
Some studies [17, 63] provide provable generalization for AUPRC optimization in information retrieval.
In this scene, a dataset consists of multiple queries, where each query corresponds to a set of positive and negative samples.
However, these results require sufﬁcient queries to ensure small generalization errors, but leave the single-query case alone, i.e. , whether the generalization error tends to zero with the length of a single-query increasing is still unclear .
This limits the adaptation scope of these methods.
To ﬁll this gap, in this paper we aim to design a stochastic optimization framework for AUPRC with a provable algorithm-dependent generalization performance in the single-query case.
The target is challenging in three aspects: (a) Most AUPRC stochastic estimators are biased with a biased sampling rate.
Moreover, due to the non-decomposability, outputs of existing algorithms might change a lot with slight changes in the training data, which is called leave-one-out unstable in this paper.
Such an unstability is harmful to the generalization.
(b) The standard framework to analyze the algorithm-dependent generalization requires the objective function to be expressed as a sum of instancewise terms, while AUPRC involves a listwise loss.
(c)
The stochastic optimization of
AUPRC is a two-level compositional optimization problem, which brings more complicated proofs of the stability.
In search of a solution to (a) , we propose a sampling-rate-invariant asymptotically unbiased stochastic estimator based on a reformulation of AUPRC.
Notably, to ensure the stability [28, 39, 37, 38] of the estimator, the objective is formulated as a two-level compositional problem by introducing an auxiliary vector for the ranking estimation.
Error analysis further supports the feasibility of our method, and inspires us to add a semi-variance regularization term. To solve this problem, we propose an algorithm with provable convergence that combines stochastic gradient descent (SGD), linear interpolation and exponential moving average.
Facing challenge (b) , we extend instancewise model stability to listwise model stability, and correspondingly put forward the generalization via stability of listwise problems.
On top of this, we bridge the generalization of AUPRC and the stability of the proposed optimization algorithm.
As for challenge (c) , since the variables to be optimized are typically updated alternately in the compositional optimization problem, we propose state transition matrices of these variables, and simplify the calculations of the stability with matrix spectrum.
In a nutshell, the main contributions of this paper are summarized as follows:
• Algorithmically, a stochastic learning algorithm is proposed for AUPRC optimization.
The core of the proposed algorithm is a stochastic estimator which is sampling-rate-invariant asymptotically unbiased.
• Theoreticallly, we present the ﬁrst trial on the algorithm-dependent generalization of stochastic AUPRC optimization.
To the best of our knowledge, it is also the ﬁrst work to analyze the stability of stochastic compositional optimization problems.
• Technically, we extend the concept of the stability and generalization guarantee to listwise non-convex losses.
Then we simplify the stability analysis of compositional objective by matrix spectrum.
These techniques might be instructive for other complicated metrics.
Problem Formulation 2.1 Preliminaries on AUPRC
Notations.
Consider a set of N examples S
= { ( x i , y i ) } Ni =1 independently drawn from a sample space D
= X × Y , where X is the input space and Y = {− 1 , 1 } is the label space.
For sake of the presentation, denote the set of positive examples of S as S + = { x + i } N + i =1 , and similarly the set of negative examples is denoted as S − = { x − i } N − i =1 , where N + = |S + | , N − =
|S − | .
With a slight abuse of notation, we also denote S = S + ∪ S − if there is no ambiguity.
Generally, we assume that the dataset is sufﬁciently large, such that N + / ( N + + N − ) =
P ( y = 1) :
= π .
Our target is to learn a score function h w : X (cid:55)→ R with parameters w ∈ Ω ⊆ R d , such that the scores of positive examples are higher than negative examples.
Furthermore, when appling the score function to a dataset S ∈ X N , we denote h w : X N (cid:55)→ R N , where the k -th element of h w ( S ) has the top- k values of { h w ( x ) | x ∈ S} .
Denote the asymptotic upper bound on complexity as O , and denote asymptotically equivalent as (cid:16) .
In this work, our main interest is to optimize a score function in the view of AUPRC:
[Equation]
where ( x , y ) ∼ D , c refers to a threshold, and TPR ( c ) =
P ( h w ( x ) ≥ c | y = 1) , FPR ( c ) = P ( h w ( x ) ≥ c | y = 0) .
For a ﬁnite set S , AUPRC is typically approximated by replacing the distribution function P ( h w ( x ) ≥ c | y = 1) with its empirical cumulative distribution function [8, 19]:
[Equation]
[Equation]
where (cid:91) TPR ( c )
=
ˆ E x ∼S + [ (cid:96) 0 , 1 ( c − h w ( x ))] , (cid:91) FPR ( c )
=
ˆ E x ∼S − [ (cid:96) 0 , 1 ( c − h w ( x ))] , (cid:96) 0 , 1 ( x ) = {+
 if x+} ≤ 0 or (cid:96) 0 , 1 ( x ) = 0 otherwise.
It has been shown that (cid:92) AUPRC is an unbiased estimator when
N + / ( N + + N − )
→ π and N → ∞ [8].
With the above estimation, we have the following optimization objective:
[Equation]
where σ ( x ) = x/ (1 + x ) is concave and monotonically increasing.
To make it smooth, surrogate losses (cid:96) 1 , (cid:96) 2 are used to replace (cid:96) 0 , 1 in (cid:91) FPR and (cid:91) TPR respectively, yielding the following surrogate objective:
[Equation]
, (4) where (cid:91) TPR [Equation] c ; [Equation]
=
[Equation] E x ∼S + [ [Equation] 2 ( c − h w ( x ))] , (cid:91) FPR ( c ; (cid:96) 1 ) =
ˆ E x ∼S − [ (cid:96) 1 ( c − h w ( x ))] .
Speciﬁcally, when N + / ( N + + N − )
=
π , it is equivalent to another commonly used formulation Average
Precision (AP) Loss :
[Equation]
2.2 Stochastic Learning of AUPRC
Under the stochastic learning framework for instancewise losses, the empirical risk F ( w ; S ) is expressed as a sum of instancewise losses: F ( w ; S )
= 1 N (cid:80) x
∼S ˆ f ( w ; x ) , where ˆ f ( w ; x ) is the stochastic estimator of F ( w ; S ) .
Different from instancewise losses, listwise losses like AUPRC require a batch of samples to calculate the stochastic estimator.
Speciﬁcally, at each step, a subset of
S :
z = z + ∪ z − is randomly drawn, where z + consists of n + positive examples and z − consists of n − negative examples.
Then a stochastic estimator of the loss function, denoted as ˆ f ( w ; z ) , is computed with z .
Similar to the instancewise case, we consider a variant of the empirical/population
AUPRC risks as approximations, which is a sum of stochastic losses w.r.t.
all posible z :
[Equation]
where M is the number of all posible z .
Unfortunately, due to the non-decomposability of the empirical AUPRC risk f ( w ; S ) , it is tackle to determine the approximation errors between F ( w ; S ) and f ( w ; S ) in general.
Nonetheless, in Sec.
3.3 we argue that by selecting proper ˆ f ( w ; z ) , F ( w ; S ) can be asymptotically unbiased estimator of f ( w ; S ) , which naturally makes F ( w ) an asymptotically unbiased estimator of 1 − AUPRC. In this case, ˆ f is said to be an asymptotically unbiased stochastic estimator. Moreover, if the unbiasedness holds under biased sampling rate, it is said to be sampling-rate-invariant asymptotically unbiased.
Asymptotically Unbiased Stochastic AUPRC Optimization
In this section, we will present our SGD-style stochastic optimization algorithm of AUPRC.
In
Sec. 3.1, we propose surrogate losses to make the objective function differentiable.
In Sec. 3.2, we present details of the proposed stochastic estimator and the corresponding optimization algorithm.
Analyses on approximation errors are provided in Sec. 3.3.
3.1 Differentiable Surrogate Losses
Since (cid:96) 0 , 1 appears in both the numerator and denominator of Eq.
(4), simply implementing (cid:96) 1 , (cid:96) 2 with a single function [56, 9, 54] will bring difﬁculty to analyze the relationship between (cid:92) AUPRC ↓ ( w ; S ) and f ( w ; S ) .
This motivates us to choose (cid:96) 1 ≥ (cid:96) 0 , 1 , (cid:96) 2 ≤ (cid:96) 0 , 1 , such that (cid:92) AUPRC ↓ ( w ; S ) ≤ f ( w ; S ) , thus the original empirical risk could be optimized by minimizing its upper bound f ( w ; S ) .
Concretely, (cid:96) 1 and (cid:96) 2 are deﬁned as the one-side Huber loss and the one-side sigmoid loss: (cid:96) 1 ( x ) =
  − 2 x/τ 1 , x < 0 , (1 − x/τ 1 ) 2 , 0 ≤ x < τ 1 , 0 , x ≥ τ 1 .
(cid:96) 2 ( x ) =
  exp( − x/τ 2 ) −exp( − x/τ 2 )
+ 1 , x < 0 , 0 , x ≥ 0 .
Here τ 1 , τ 2 > 0 are hyperparameters.
(cid:96) 1 is convex and decreasing, which ensures the gap between positive-negative pairs is effectively optimized.
Additionally, compared with the square loss and the exponential loss, (cid:96) 1 is more robust to noises.
(cid:96) 2 is Lipschitz continuous, and (cid:96) 2 → (cid:96) 0 , 1 with τ 2 → 0 .
3.2 Stochastic Estimator of AUPRC
The key to a stochastic learning framework is the design of the stochastic estimator (or the corresponding gradients), i.e. , ˆ f ( w ; z ) .
Existing methods [9, 74, 12] implement it with (cid:99) AP ↓ ( w ; z ) (Eq.
(5)), which might suffer from two problems:
(P1)
Comparing Eq.
(4) and Eq.
(5), it can be seen that only when n + / ( n + + n − ) → π , (cid:99) AP ↓ is an asymptotically unbiased estimator.
However, it is hardly satisﬁed since the sampling strategy is usually biased in practice.
(P2)
Each term in the summation of (cid:99) AP ↓ is related to all instances of a batch, leading to weak leave-one-out stability,
i
e. , changing one instance might result in a relatively large ﬂuctuation in the stochastic gradient, especially when changing a positive example.
To tackle the above problems, we ﬁrst substitute (cid:91) FPR ( h w ( x + ); (cid:96) 1 ) with ˆ E x ∼ z − [ (cid:96) 1 ( h w ( x + ) − h w ( x ))] , and then introduce an auxiliary vector v ∈ R N + to estimate (cid:91) TPR .
Formally, we propose the following batch-based estimator:
[Equation]
Such an estimator enjoys two advantages: in terms of P1 , it is asymptotically unbiased regardless of the sampling rate (see Sec.
3.3 for detailed discussions); as for P2 , we use v to substitute h w ( S + ) , such that each positive example in a mini-batch only appears in one term.
Ideally, it can be considered as using all positive examples in the dataset to estimate (cid:91) TPR instead of that from a mini-batch.
With the fact that n − (cid:29) n + , this makes the corresponding algorithm more stable.
Moreover, based on the model stability, generalization bounds are available (see Sec. 4).
3.3 Analyses on Approximation Errors
In this subsection, we analyze errors from two approximations in the above algorithm: 1) the gap between F ( w ; S ) and the true AUPRC loss; 2) the gap between the interpolated scores φ ( h w ( z + )) and the true scores h w ( S + ) . Proofs are provided in Appendix B.1. Denote π = N + / ( N + + N − ) and π 0 = n + / ( n + + n − ) . We would like to show that for all w ∈ Ω ,
E z [ ˆ f ( w ; z )] is an unbiased estimator when n → ∞ , no matter how π 0 is chosen, while for E z [ (cid:99) AP ↓ ( w ; z )] , it holds only when π 0 = π .
Since only one model w is considered, we let w t = w in the update rule of v (Eq. and we have the following proposition: Proposition 1.
Consider updating v with Eq.
(10) for T steps, then we have
[Equation]
Remark 1.
Two conclusions could be drawn from the above proposition:
ﬁrst, if the linear interpolation is asymptotically unbiased (see next subsection), by choosing a large T or setting v 1 =
E [ φ ( h w ( z + ))] , we have E [ v ]
≈ h w ( S + ) ; second, by choosing a smaller β , v is more likely to concentrate on h w ( S + ) .
Proposition 2.
Assume the linear interpolation is asymptotically unbiased.
Let κ=
[Equation]
+ ) [ V ar v ∼ v [ (cid:96) 2 ( c − v ))]] .
When κ 21 /n − → 0 , κ 22 /n + → 0 , then there exists a positive scale H , such that
[Equation]
where P → refers to convergence in probability, and z ⊆ S refers to subsets described in Sec.
Remark 2.
The above proposition suggests that the proposed batch-based estimator is samplingrate-invariant asymptotically unbiased, while (cid:99) AP ↓ tends to be larger when the sampling rate of the positive class is greater than the prior, and vice versa.
We also provide a non-asymptotic result in
Appendix B.2.
Simulation experiments are conducted as complementary to the theory.
Following previous work [8], the scores are drawn from three types of distributions, including binormal, bibeta and offset uniform.
The results of binormal distribution are visualized in Fig.
1, and detailed descriptions and more results are available in Appendix B.2.
These results are consistent with the above remark.
Next we further study the interpolation error.
For the sake of presentation, denote p : [0 , 1] (cid:55)→ R to be an increasing score function describing h w ( S + ) , where p ( x ) is the score in the bottom x -quantile of h w ( S + ) .
Similarly, let ˆ p to be the interpolation results of E A [ h w ( z + )] .
Assume that E A [ h w ( z + )] are located in the ( i/n + ) -quantiles of p , where i ∈ [ n + ] , such that p ( i/n + )
=
ˆ p ( i/n + ) and all interpolation intervals are with length 1 /n
+ .
The following proposition provides an upper bound of the approximation error (see [61] for proof):
Proposition 3 ( Linear Interpolation Error ) .
Let p, ˆ p be deﬁned as above.
Then we have
[Equation]
Similar to the last subsection, simulation results are shown in Fig.
1(c), which shows the expected errors of linear interpolation are ignorable.
3.4 Optimization Algorithm
In the rest of this section, we focus on how to optimize F ( w ; S ) .
The main challenge is to design update rules for v , such that it could efﬁciently and effectively approximate h w ( S + ) without fullbatch scanning.
To overcome the challenge, we propose an algorithm called Stochastic Optimization of AUPRC (SOPRC) , which jointly updates model parameters w and the auxiliary vector v .
A summary of the detailed process is shown as Alg. 1.
At step t , a batch of data is sampled from the training set, and then compute the corresponding scores.
Afterward, scores of positive examples are mapped into a N + -dimension vector with linear interpolation φ as shown in Alg.
v t +1 are updated with the interpolated scores in a moving average manner.
−   4 x
D
en s i t y
Negative
Positive (a) Density functions of scores.
10000 Batch Size
N o r m a li z ed A U P
RC
Ours
AP ...
10000 Batch Size
N o r m a li z ed A U P
RC
Ours
AP ...
−2.
[Figure]
V a l ue
Interp.
Results
Target Values
[Figure]
Practically, n + , n − are ﬁnite, causing inevitable estimation errors in f ( w ; z i t , v t +1 ) .
Notice that another factor inﬂuencing the stochastic estimation errors, i.e. , κ 21 and κ 22 .
To reduce them, it is expected that the variance of positive (negative) scores are small, which motivates us to add a variance regularization term.
However, it might force to reduce positive scores that higher than the mean value, which is contrary to our target.
Therefore, we propose a semi-variance regularization term [4]: [Equation] where µ + = 1 n [Equation]
x
[Equation] [Equation] 2 are hyperparameters. Finally, we compute the gradients of f ( w ; z i t , v t +1 )
+ L var , and update parameters w with gradient descent.
Compute h w t ( z + i t ) and map the results into φ ( h w t ( z + i t ))
Alg
Update v with
[Equation]
Compute var with Eq
Update the parameter:
[Equation]
[Equation]
8: end for
N u
2 u 1 − u 2 , b ) , u n +1 = min (2 u n − 2 u n − 1 , B ) .
n do 4: for j = (cid:100) N + ( i − 1) n (cid:101) to (cid:104) N + · i n (cid:105) do 5: m j
[Equation]
= (cid:2) ( i
[Equation]
/N
[Equation]
) u
[Equation]
+ [Equation] [Equation] (cid:3) /6: end for 7: for j = (cid:104) N + · i n (cid:105)
Generalization of SOPRC via Stability
In this section, we turn to study the excess generalization error of the proposed algorithm.
Formally, following standard settings [5], we consider the test error of the model
A ( S ) trained on the training set S .
Our target is to seek an upper bound of the excess error E A, S [ F ( A ( S ))
− F ( w ∗ )] , where w ∗ ∈ arg min w ∈ Ω E A, S [ F ( w ∗ )] . It can be decomposed as: [Equation] The estimation error sources from the gap of minimizing the empirical risk instead of the expected risk.
In Sec. 4.1, we provide detailed discussion on the estimation error.
The optimization error measures the gap between the minimum empirical risk and the results obtained by the optimization algorithm, which will be studied in Sec.
Detailed proofs of this section are available in Appendix
C. Before the formal presentation, we show the main assumptions:
Assumption 1 ( Bounded Scores & Gradient ) .
| ˆ f ( w ; · ) | ≤ B, (cid:107)∇ ˆ f ( w ; · ) (cid:107) 2 ≤ G for all w ∈ Ω .
Assumption 2 ( L-Smooth Loss ) .
(cid:107)∇ ˆ f ( w ; · ) − ∇ ˆ f ( ˜ w ; · ) (cid:107) 2 ≤ L (cid:107) w − ˜ w (cid:107) 2 for all w , ˜ w ∈ Ω .
Assumption 3 ( Lipschitz Continuous Functions ) .
| (cid:96) 1 ( x ) − (cid:96) 1 (˜ x ) | ≤ L 1 | x − ˜ x | , | (cid:96) 2 ( x ) − (cid:96) 2 (˜ x ) | ≤ L 2 | x − ˜ x | for all x, ˜ x ∈ [ − 2 B, 2 B ] . (cid:107) φ ( x ) − φ ˜ x ) (cid:107) 2 ≤ C φ (cid:107) x − ˜ x (cid:107) 2 for all x , ˜ x ∈ R N + .
4.1 Generalization of AUPRC via Model Stability
The generalization of SGD-style algorithms for instancewise loss has been widely studied with stability measure [39, 21, 28].
However, these results could not be directly applied to listwise losses like AUPRC.
The main reason is that the estimation of each stochastic gradient requires a list of examples, and the estimation is usually biased.
Nonetheless, to bridge the optimization algorithm and the generalization of AUPRC, we propose a listwise variant of on-average model stability [39] as follows:
Deﬁnition 1 ( Listwise On-average Model Stability ) .
Let S = { ( x i , y i ) } Ni =1 and (cid:101) S
= { ( (cid:101) x i , y i ) }
Ni =1 be two sets of examples whose features are drawn independently from X .
For any i = 1 , · · · , N , denote S ( i ) = { ( x 1 , y 1 ) , · · · , ( x i − 1 , y i − 1 ) , ( (cid:101) x i , y i ) , ( x i +1 ,
y i +1 ) , · · · , ( x n , y n ) } .
A stochastic algorithm A is listwise on-average model ( (cid:15) + , (cid:15) − ) -stable
if the following condition holds:
[Equation]
The following theorem shows that the estimation error is bounded by the above-deﬁned stability:
Theorem 1 ( Generalization via Model Stability ) .
Let a stochastic algorithm
A be listwise onaverage model ( (cid:15) + , (cid:15) − ) -stable
and
Asmp.
1 holds.
Then we have
[Equation]
[Equation]
With the above theorem, now we only need to focus on the model stability of the proposed algorithm.
Notice that in Alg. 1, both w t and v t are updated at each step, thus we have to consider the stability of both simultaneously.
The following lemma provides a recurrence for the stability w t and v t .
Lemma 1.
Let S , (cid:101) S , S
( i ) be constructed as Def. 1 and Asmp. 1, 2, 3 hold.
Let { w t } t and { w ( i ) t } t be produced by Alg. 1 with S and S ( i ) , respectively.
Denote L = max { L w , L v /n
+ , C φ B, G/ 2 , B (cid:48) (cid:96) } , m
( i ) t
[Equation]
( i ) t (cid:105) .
Then for all t ∈ [ T ] , by setting β t ≤ 2 C φ B/n + , we have
[Equation]
where I 3 is the 3 × 3 identity matrix and
[Equation]
Finally, we utilize the matrix spectrum of R + t and R − t to show that the model stability w.r.t.
Alg.decreases as the number of training examples increases (see Appendix C.2 for details):
Theorem 2.
Let λ =
LC η
(1 + (cid:112) 1 − β 2 + β ) , and assumptions in Lem.
1 hold. By setting η t ≤ C η t , β t = β (cid:16) 1 /n + and T ≤ N + , Alg. 1 is list on-average model stable with [Equation] [Table]
4.2 Convergence of AUPRC Stochastic Optimization
Following previous work [24, 34]
[Table]
we study the optimization error of the proposed algorithm under the Polyak-Łojasiewicz (PL) condition.
It has been shown that the PL condition holds for several widely used models including some classes of neural networks [13, 42].
Assumption 4 ( Polyak-Łojasiewicz Condition [34, 37]) . Denote w ∗ = arg min w ∈ Ω F ( w ) . Assume F satisfy the expectation version of PL condition with parameter µ > 0 , i.e., [Equation]
The main difference to the existing convergence analysis on non-convex optimization is that the gradient estimation is biased.
Nonetheless, we show that the bias terms from Alg.
1 tend to 0 with sufﬁcient training data and training time (see Appendix C.3), leading to the following convergence:
Theorem 3.
Let Asmp.
1, 3, 4 hold.
By setting η t = 2 t +1 µ ( t +1) 2 and β t = β (cid:16) 1 /n + , we have [Equation] Theorem 4.
Let assumptions in Thm.
2 and 3 hold.
{+By setting T (cid:16) ( N + ) λ +1 2 λ +
 ( n + ) − 1 2 λ +1 ,+} we have E S ,
[Equation]
[Equation]
Remark 3.
Recall that λ =
LC η
(1 + (cid:112) 1 − β 2 + β ) and C η = 4 /µ , when β is small, we have λ ≈ 4 L/µ .
Here L/µ is a condition number determined by the model and surrogate losses.
Notice that n + (cid:28) N + , n − (cid:28) N − , if λ = 1 , the generalization bound is O (cid:0) ( N + ) − 2 / 3 · ( n + ) 4 / 3 + ( N − ) − 2 / 3 · ( n − ) 4 / 3 (cid:1) .
As λ increases, it increases to
O (cid:0) ( N + ) − 1 / 2 · ( n + ) 3 / 2 + ( N − ) − 1 / 2 · ( n − ) 3 / 2 (cid:1) .
Experiments
To validate the effectiveness of the proposed method, we conduct empirical studies on the image retrieval task, in which data distributions are largely skewed and AUPRC is commonly used as an evaluation metric.
More detailed experimental settings are provided in Appendix D.1.
The source code is available in https://github.com/KID-7391/SOPRC.git .
5.1 Implementation Details
Datasets.
We evaluate the proposed method on three image retrieval benchmarks with various domains and scales, including Stanford Online Products (SOP) 2 [48], PKU VehicleID 3 [43] and iNaturalist 4 [68].
We follow the ofﬁcial setting to split a test set from each dataset, and then further split the rest into a training set and a validation set by a ratio of 9 : 1 .
Network Architecture.
The feature extractor is implemented with ResNet-50 [29] pretrained on
ImageNet [60].
Following previous work [12, 9], the batch normalization layers are ﬁxed during training, and the output embeddings are mapped to 512-d with a linear projection.
Given Lnormalized embeddings of a query image e q and a gallery list { e i } i , the scores are represented by the cosine similarity e (cid:62)
q e i for all i .
Optimization Strategy.
In the training phase, the input images are resized such that the sizes of the shorter sides are 256.
Afterward, we applied standard data augmentations including random cropping ( 224 × 224 ) and random ﬂipping ( 50% ).
The model parameters are optimized in an end-to-end manner as shown in Alg.
1, where β
= 0 .
001 and the weight decay is set to 4 × 10 − 4 .
The default batch size is set to 224, where each mini-batch is randomly sampled such that there are exactlypositive examples per category.
The learning rates are tuned according to performance on validation sets: for SOP, the learning rate are initialized as 0 .
01 and decays by 0 .
1 at the 15 k and 30 k iterations,
T = 50 k ; for VehicleID, the learning rate are initialized as 0 .
001 and decays by 0 .
1 at the 40 k and 80 k iterations, T = 100 k ; for iNaturalist, the learning rate are initialized as 0 .
001 and decays by 0 .at
the 80 k and 110 k iterations, T = 130 k .
Competitors.
We compare two types of competitors: 1) Pairwise Losses , including Contrastive
Loss [27], Triplet Loss [32], Multi-Similarity (MS) Loss [71], Cross-Batch Memory (XBM) [72].
These methods construct loss functions with image pairs or triplets.
2) Ranking-Based Losses , including SmoothAP [9], FastAP [12], DIR [58], BlackBox [33], and Area Under the ROC Curve
Loss (AUROC) [76].
These methods directly optimize the ranking-based metrics.
Evaluation Metrics.
In all experiments, we adopt evaluation metrics: mean AUPRC (mAUPRC) and
Recall@k .
mAUPRC is also called mean average precision (mAP) in literature, which takes the mean value over the AUPRC of all queries.
Recall@k measures the probability that at least one positive example is ranked in the top-k list.
5.2 Main Results
We evaluate all methods with mean AUPRC (mAUPRC) and Recall@k .
mAUPRC measures the mean value of the AUPRC over all queries, a.k.a.
mean average precision (mAP).
The performance comparisons on test sets are shown in Tab. 1.
Consequently, we have the following observations: 1)
In all datasets, the proposed method surpasses all competitors in the view of mAUPRC, especially in the large-scale long-tailed dataset iNaturalist.
This validates the advantages of our method in boosting the AUPRC of models.
Compared to pairwise losses, the AUPRC/AP optimization methods enjoy better performance generally.
The main reason is that pairwise losses could only optimize models indirectly by constraining relative scores between positive and negative example pairs, while ignoring the overall ranking.
Although some pairwise methods like XBM have a satisfying performance on
Recall@1, their mAUPRC is relatively low. It is caused by the limitation of Recall@1, i.e. , it focuses on the top-1 score while ignoring the ranking of other examples. What’s more, this phenomenon shows the inconsistency of Recall@k and AUPRC, revealing the necessity of studying AUPRC optimization. More results are available in Appendix D.2.
To qualitatively demonstrate the effect of the proposed method, we also show the mean PR curves and convergence curves in Fig.
The left two subﬁgures demonstrate that the proposed method achieves can effectively improve AUPRC.
The right subﬁgure shows the affect of batch size, from which can be seen that a large batch size leads to better performance.
One of the reasons is that a small batch size will amplify the AUPRC stochastic estimation error.
Such a problem have been addressed by maintaining inner gradient estimations in a moving average manner [54, 70, 75].
Unfortunately, when applied to image retrieval problems, it needs to maintain intermediate variables for each sample pair, which will bring high complexity in time and space, thus we leave this problem as further work.
5.3 Ablation Studies
We further investigate the effect of different components of the proposed method.
Results are shown in Tab. 2, and more detailed statements and analyses are as follows.
[Figure]
i
[Figure]
TripletBlackBoxFastAPSmoothAPOurs
[Figure]
[Table]
Effect of Unbiased Estimator.
To show the performance drop caused by the biased estimator
[Table]
we replace the prior π in Eq.
(8) with n + / ( n + + n − ) .
Comparing line 1 and line 2, using the unbiased estimator increases the mAUPRC by 1.3%, which is consistent with our theoretical results in Sec. 3.3.
Notably, the unbiased estimator is the main source of improvements in terms of mAUPRC.
Effect of v t .
To show the effect of introducing v t to estimate φ ( S + ) , we directly use φ ( z + ) instead in the ﬁrst two lines.
Comparing line 2 and line 3, using v t could bring consistent improvements due to the better generalization ability.
Effect of L var .
We show that shrinking variances could reduce the batch-based estimation errors.
Comparing line 3 and line 4, it can be seen that L var further boosts the proposed method.
Effect of Optimizer.
Comparing line 4 and line 5, it can be seen that the choice of optimizer only has a slight inﬂuence.
Conclusion & Future Work
In this paper, we present a stochastic learning framework for AUPRC optimization.
To begin with, we propose a stochastic AUPRC optimization algorithm based on an asymptotically unbiased stochastic estimator.
By introducing an auxiliary vector to approximate the scores of positive examples, the proposed algorithm is more stable.
On top of this, we study algorithm-dependent generalization.
First, we propose list model stability to handle listwise losses like AUPRC, and bridge the generalization and the stability.
Afterward, we show that the proposed algorithm is stable, leading to an upper bound of the generalization error.
Experiments on three benchmarks validate the advantages of the proposed framework.
One limitation is the convergence rate is controlled by the scale of the dataset.
In the further, we will consider techniques like variance reduction to improve the convergence rate, and jointly consider the corresponding algorithm-dependent generalization.
Acknowledgments
This work was supported in part by the National Key R&D Program of China under Grant 2018AAA0102000, in part by National Natural Science Foundation of China: U21B2038, 61931008, 6212200758 and 61976202, in part by the Fundamental Research Funds for the Central Universities, in part by Youth Innovation Promotion Association CAS, in part by the Strategic Priority Research
Program of Chinese Academy of Sciences, Grant No. XDB28000000, in part by the China National Postdoctoral Program for Innovative Talents under Grant BX2021298, and in part by China
Postdoctoral Science Foundation under Grant 2022M713101.
B Details on Error Analysis
B.1 Proofs on the Error Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.2 Details of Simulation Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . .
C
Proofs on
Generalization via Stability
C.1 Generalization by List Model Stability . .
C.2 List Model Stability of SOPRC . .
C.3 Convergence Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D Experiments
D.1 Implementation Details . .
D.2 More Results . .
A Related Work A.1 AUPRC Optimization
To measure model performances in largely skewed datasets, Raghavan et al.
[57] use Precision-Recall (PR) curves to describe the trade-off between precision and recall, leading to a metric named Area
Under the PR Curve (AUPRC).
Beneﬁting from its insensitivity to data distribution, AUPRC has been widely used in imbalanced scenarios, such as information retrieval [17, 56, 44], recommendation systems [64, 16] and computer visions [9, 15, 12].
The important application value of AUPRC and the inconsistency with other metrics [20] have raised a wave of research on direct optimization of
AUPRC.
Early work can be roughly divided into discrete methods and continuous methods.
The ﬁrst technical route utilizes discrete optimization methods to directly optimize the non-differentiable objective, e.g. , Markov random ﬁeld model [44], randomized search [26], dynamic programming [77], and error driven method [10].
The second technical route seeks for continuous surrogate objective, like convex upper bound on AUPRC for support vector machines (SVMs) [45].
Unfortunately, limited by the high computational complexity, these methods are not suitable for deep learning.
With the increasing application of deep learning in ranking problems, stochastic optimization for
AUPRC has attracted the attention of researchers.
It is challenging since AUPRC is neither differentiable nor decomposable.
Therefore, the mainstream approaches tackle stochastic optimization of
AUPRC from two aspects: surrogate loss functions and batch-based estimators.
In the ﬁrst aspect, some methods replace the non-differentiable 0-1 loss with surrogate functions like exponential loss [56], sigmoid loss [9], and linear interpolation function [33].
Contrary to approximating the objective with differentiable functions, another route tackle this problem in an error driven style.
More speciﬁcally, Burges et al.
[10, 11] propose to decompose the gradients with chain rule, and then replace the differential w.r.t.
prediction scores with the difference.
Since the differential of scores w.r.t.
model parameters are available, the gradients can be obtained in this way.
This idea has been extended to solve the imbalance problem in object detection [15, 14, 49].
However, these methods cannot guarantee the relationship between the surrogate optimization objective and the original AUPRC.
In contrast, we propose a differentiable upper bound of AUPRC as a surrogate objective function.
As for the batch-based estimators, the mainstream approaches use a batch of examples to calculate average precision [9, 54, 55], or approximate precision and recall [58, 12, 30] with the histogram binning technique [65].
However, due to the biased sampling, these estimators are not asymptotically unbiased, leading to biased stochastic gradient estimations.
Moreover, since the number of positive examples sampled in a batch is typically limited, these algorithms are less stable, which is not conducive to the generalization.
Limitations of existing work motivate us to design a more appropriate estimator that (asymptotically) unbiased and stable.
In this work, we propose an unbiased estimator and further enhance the stability with an auxiliary set assisting the TPR estimation.
A.2 Generalization via Stability of Stochastic Optimization
The algorithmic stability [59, 7, 21] is a standard framework for generalization analysis.
Besides the original uniform stability [7], various types of stability are studied, e.g. , expected stability [62], hypothesis set stability [23], and on-average model stability [39].
Unlike another standard framework built on the Rademacher complexity [3, 53, 66], generalization via stability takes the optimization algorithm into account.
This is equivalent to constraining the hypothesis set to the possible optimization outcomes, which are usually in the neighborhood of the (local) optimum.
On the other hand, this framework can naturally consider two factors simultaneously, i.e. , optimization errors and estimation errors, allowing jointly consideration of generalization and convergence trade-offs [18].
These advantages enable stability to be applied in a wide variety of conditions [40, 47, 13].
However, existing techniques focus on instancewise or pairwise [37, 36] loss functions, while AUPRC is a listwise metric.
In addition, each term in the AUPRC are related to all instances, thus the stability is limited by the capacity of a batch.
Nonetheless, we propose a listwise variant of on-average model stability [39], and further develop generalization guarantees of our proposed stochastic optimization algorithm for AUPRC.
B Details on Error Analysis B.1 Proofs on the Error Bounds
Proposition 1. Consider updating v with Eq.
(10) for T steps, then we have
[Equation]
Proof.
With the update rule we have
[Equation] thus we have the expectation of v [Equation] and the variance
[Equation] Proposition 2.
Assume the linear interpolation is asymptotically unbiased.
Let κ= ˆ E c ∼ h w ( z + )
[ V ar x ∼S − [ (cid:96) 1 ( c − h w ( x ))]] , κ
=
ˆ E c ∼ h w ( z + )
[ V ar v ∼ v [ (cid:96) 2 ( c − v ))]] .
When κ 21 /n − → 0 , κ 22 /n + → 0 , then there exists a positive scale H , such that
[Equation]
where P → refers to convergence in probability, and z ⊆ S refers to subsets described in Sec.
Proof.
With sufﬁcient large N + and N − , we consider S as the population.
Given a threshold c ∈ R , we consider (cid:96) 1 ( c − h w ( x )) as i.i.d.
variables controlled by x with mean µ c, 1 and variance κ 2 c, 1 .
In this way, the surrogate (cid:91) FPR can be viewed as an average of these variables:
[Equation]
According to the central limit theorem, when n − → ∞ we have
[Equation]
(22) where (cid:32) refers to convergence in law. Similarly, consider (cid:96) 2 ( c − v )) as variables with mean µ c,and standard deviation κ 2 c, 2 , and denote
[Equation] = [Equation]
∼
[Equation]
[ (cid:96)
[Equation]
c
[Equation]
According to the portmanteau lemma [67] and Prop. 3, when n + → ∞ , we have
[Equation] [Equation]
Decompose the difference between ˆ f ( w ; z ) and (cid:92) AUPRC ↓ ( w ; S ) as
[Equation] [Equation]
Notice that z + is randomly sampled from S + , thus we have ( d ) = 0 , and we only need to focus on the ﬁrst three terms. To solve (a), for any threshold c , we consider a continuous function s (; c ) :
[Equation]
With the delta method [67], when n − → ∞ we have
[Equation]
Notice that the above result holds for all z and c . Consider a random variable X n − ∈ R Mn + whose elements are X cn − w.r.t.
all c , and a function Avg : R Mn
+ (cid:55)→ R outputs the average of the input. Obviously, the gradient of Avg at any point is a scaled unit vector. Therefore, according to the delta method, we have the following convergence:
[Equation] [Equation] where Σ = V ar [ X n − ] .
Denote
[Equation] then we have [Equation] According to the mean value theorem
[Equation] there exists a positive constant ζ a such that [Equation] [Equation]
and (27) can be rewritten as
[Equation]
Similarly, there exists ζ b , ζ c >
[Equation]
such that
[Equation]
To sum up, when ˆ E
[Equation]
∼
[Equation]
[Equation]
0 and ˆ E
[Equation]
0 , we have
[Equation]
where P → refers to convergence in probability [Equation] As for (cid:100) AP ↓
[Equation]
the difference can be decomposed into [Equation]
Similarly, we have ( e ) P → 0 . As for ( f ) , we have [Equation] According to the mean value theorem, there exists
[Equation] such that [Equation] Proposition 4.
For any 0 < δ < 1 , at least with probability of 1 − δ , we have [Equation] Proof. Denote X cn − , Y cn + , ( a ) , ( b ) , ( c ) as in the proof of Prop. 2.
Next, we focus on the term ( a ) .
Under the assumption of Prop. 2, X c n − can be viewed as an average of i.i.d. variables, thus according to Hoeffding’s inequality, for {+any (cid:15) > 
 we have+}
[Equation]
Therefore, for any c , 0 < δ < 1 , with probability at least 1 − δ , we have [Equation] where µ 1 = inf c µ c, 1 . If we further assume that X cn − is independent w.r.t.
different c , then considering all positive c , with probability at least 1 − δ/ 3 , we have [Equation] and similarly |
[Equation]
To sum up, with probability at least 1 − δ we have
[Equation] O (cid:32)(cid:114) log(6 n + /δ ) n + +(cid:114) log(6 n + /δ ) n − (cid:33) . B.2 Details of Simulation Experiments
Following [8], assume that scores are i [Figure] i.d. random variables drawn from one of three distributions: binormal, bibeta and uniform offset.
The hyperparameters choosen and the corresponding probability density functions are shown in Fig. For each kind of distribution, we draw 100 , 000 score points, where the ratio of positive points π = 0 .
Afterward, under different sampling-rate π 0 ∈ { 0 . 01 , 0 . 02 , 0 . 03 , 0 . 1 , 0 . 2 } , we repeatly sample several points from the above-mentioned points for 500 times, and report the corresponding mean value and standard derivation. The results are shown in Fig. 4 and Fig.
[Figure] which are consistent with our conclusions in the main paper.
[Figure] [Figure] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] π
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] b) π
[Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] C Proofs on Generalization via Stability C.1 Generalization by List Model Stability Theorem 1 ( Generalization via Model Stability ) .
Let a stochastic algorithm
A be listwise onaverage model ( (cid:15) + , (cid:15) − ) -stable
and
Asmp.
1 holds.
Then we have
[Equation] [Equation]
Proof. Let S , (cid:101) S be deﬁned as in Def. 1. First of all, according to the symmetry, we have [Equation]
where z ⊆ S and ˜ z ⊆ (cid:101) S . Then, since the gradients of ˆ f is upper bounded by G [Equation] we have
[Equation] proof is completed by denoting [Equation] C.2 List Model Stability of SOPRC
Lemma 1.
Let S , (cid:101) S , S
( i ) be constructed as Def. 1 and Asmp. 1, 2, 3 hold.
Let { w t } t and { w ( i ) t } t be produced by Alg. 1 with S and S ( i ) , respectively.
Denote L = max { L w , L v /n
+ , C φ B, G/ 2 , B (cid:48) (cid:96) } , m
( i ) t
[Equation]
1 N + (cid:80) y i = − 1 E S ,A (cid:104) m
( i ) t (cid:105) .
Then for all t ∈ [ T ] , by setting β t ≤ 2 C φ B/n + , we have
[Equation]
where I 3 is the 3 × 3 identity matrix and
[Equation] Proof.
For all i = 1 , · · · , N , let { w t } t , { v t } t , { z i t } t and { w (cid:48) t } t , { v (cid:48) t } t , { z (cid:48) i t } t be produced by
Alg. {+
+}
based on S and S ( i ) , respectively.
According to the update rules, for all t = 1 , · · · , T , we have [Equation] We discuss the second term on the right side in the following three cases. Case 1: ( x i , y i ) /
∈ z i t .
In this case, we have z i t =
z (cid:48)
i t .
According to the smoothness of f and
Lem.
4, we have
[Equation]
41) and (cid:107) v
[Equation]
(cid:107)
[Equation]
≤
[Equation]
(cid:48)
[Equation]
(cid:107)
[Equation]
β
[Equation]
(cid:0) h w (cid:48)
[Equation]
t )
[Equation]
− φ (cid:0) h w t ( z + i t ) (cid:1) (cid:107)
[Equation]
≤ (1 − β t ) (cid:107) v (cid:48)
[Equation]
− v t (cid:107)
[Equation]
+ C φ C h β t (cid:107) w
[Equation]
t − w t (cid:107)
[Equation]
≤ (1 − β
[Equation]
v (cid:48) t − v
[Equation]
(cid:107)
[Equation]
+ (cid:107) w (cid:48) t − w t (cid:107) 2 /n +
[Equation]
where the last inequation is due to β t ≤ C φ B n + .
Combining Eq.
(40), Eq.
(41) and Eq.
(42), we have
[Equation]
Case 2: ( x i , y i )
∈
z i t and y i = 1 .
In this case, we have z − i t
=
z (cid:48)− i t , thus
[Equation]
[Equation]
and
[Equation]
where the last inequality is due to β t ≤ 2 C φ B/n + .
By combining Eq.
(44) and Eq.
[Equation]
we have
[Equation]
Case 3: ( x i , y i )
∈
z i t and y i = − 1 .
In this case, we have z + i t =
z (cid:48)
+ i t , thus
[Equation]
where B (cid:48) (cid:96) =
(1 − π ) B (cid:96) /π .
Similar to case 1, Eq.
(42) still holds.
Combining it with Eq.
(47), we have
[Equation]
Next, we consider the expectation of (cid:107) w (cid:48) t +1 − w t
+1 (cid:107) 2 and (cid:107) v (cid:48)
t +1 − v t +1 (cid:107) 2 taking on A .
Note that
[Equation]
For y i = 1 , we have
E A
[Equation]
By setting L ≥ max { L w , L v /n + , C φ B, G/ 2 } and N + > 2 n +
[Equation]
the above result can be rewritten as
[Equation]
Similarly, for y i = − 1 , we have
[Equation]
By setting L ≥ max { L w
[Equation]
L v /n
+
[Equation]
C φ B
[Equation]
G, B
[Equation]
[Equation]
we have
[Equation]
Lemma 2.
{+Let z , z (cid:
+}
) ∈
Z n be two datasets that differ by exact one example and Asmp.
1 holds.
For all w ∈ Ω , v ∈ R N + , the gradient difference of w.r.t.
z and z (cid:48) are upper bounded:
(a) If z , z (cid:48) differ by a negative example, then
[Equation]
(b) If z , z (cid:48) differ by a positive example, then
[Equation]
Proof.
The proof of (b) is obvious.
For (a), let the different examples in z and z (cid:48) be ˜ x and ˜ x (cid:48) .
Denote c ( x ; v , (cid:96) )
=
ˆ E v ∼ v [ (cid:96) ( h w ( x ) − v ))] , and rewrite f as
[Equation]
[Equation]
π
[Equation]
Lemma 3.
For two datasets S , S ( i ) that differ by the i -th element, let { w t } t , { v t } t and { w (cid:48) t } t , { v (cid:48) t } t be produced by Alg.
1 with S and S ( i ) , respectively.
If y i = 1 , then for all t 0 ∈ [ N + /n
+ ] we have
[Equation]
[Equation]
where D (Ω) is the diameter of the hypothesis space, I t ( A ) :
= { i 1 , · · · , i t } is the set of indices selected by A at the ﬁrst t -th iterations.
Similarly, when y i = − 1 , for all t 0 ∈ [ N − /n − ] , we have
[Equation]
Proof.
Here we only need to prove the case y i = 1 .
According to the law of total probability, we have
[Equation]
Notice that
[Equation]
the proof is complete by bounding the ﬁrst term on the right side of Eq.
Theorem 2.
Let λ =
LC η
(1 + (cid:112) 1 − β 2 + β ) [Equation] and assumptions in Lem.
1 hold.
By setting η t ≤ C η t , β t = β (cid:16) 1 /n + and T ≤ N + , Alg. 1 is list on-average model stable with [Equation] Proof. According to Lem. 1 and Lem.
[Equation] we have [Equation]
where the last equation is due to the fact that for all t ≤ t 0 , z i t
= z (cid:48) i t , leading to w t = w (cid:48) t , v t = v (cid:48) t . Then we focus on (cid:81) Tt
= t 0 ( I 3 + R + t ) . Notice that all elements in this product are nonnegative, thus in the remainder of this proof, given two nonnegative matrics a , b with the same shape, we say a ≤ b if all elements in a are not larger than that in b . Denote [Equation]
then by setting η t ≤ C η /t we have
T (cid:89) t = t( I 3 + R + t ) ≤ T (cid:89) t = t  1 + 2 Lη t Lη t /N
+
Lη t
Lβ 1 /N +   
 =
{+T (cid:89) t = t(cid:34) 
+}
+ 2 Lη t
L (1 − β ) η t
Lη t
Lβ
N + 1 /N +   (cid:35) (cid:34)  0 1 /N +    (cid:35) = (cid:34)  0 N +    (cid:35) T (cid:89) t = t(cid:32)(cid:34)  0 1 /N +    (cid:35) (cid:34) 1 + 2 Lη t L (1 − β ) η t Lη t
Lβ N + 1 /N +   (cid:35)(cid:33) (cid:34)  0 1 /N +    (cid:35) = (cid:34)  0 N +    (cid:35) T (cid:89) t = t  I 3 +   2 Lη t L (1 − β ) η t
Lη t
Lβ/N + 1 / ( N + )   
 
 (cid:34)  0 1 /N +    (cid:35) ≤ (cid:34)  0 N +    (cid:35) T (cid:89)
[Equation]
[Equation] [Equation] where the last inequation is due to N + ≥ T . Notice that M ≥ 0 3 × 3 , thus we have
T (cid:89)
[Equation] [Equation] [Equation]
where the last eqution is obtained by diagonalizing M into Λ diag ( λ 1 , λ 2 , λ 3 )Λ − 1 , λ i are the eigenvalues of M and the columns of Λ are the corresponding eigen vectors. Combining Eq. (63) and Eq. (64), we have [ 1 0 0 ]
T (cid:89) t
= t( I 3 + R + t ) [ 0 0 1 ] (cid:62) ≤ [ 1 0 0 ] (cid:34)  0 N +    (cid:35) T (cid:89) t
= t( I 3 + M /t ) (cid:34)  0 1 /N +    (cid:35) (cid:34) (cid:35) ≤ [ 1 0 0 ]
T (cid:89) t
= t( I 3 + M /t ) (cid:34) (cid:35) ≤ [ 1 0 0 ]
Λ diag (cid:32)(cid:18)
T t 0 −(cid:19) λ, (cid:18)
T t 0 −(cid:19) λ, (cid:18)
T t 0 −(cid:19) λ 3 (cid:33) Λ −(cid:34) (cid:35) .
Speciﬁcally, we have λ 1 , 2 = LC η (1 ± (cid:112) 1 + β − β 2 ) , λ 3 = 0 , and Λ =    1 βN + β (1 − √ 1+ β − β 2 ) 1 − β β (1+ √ 1+ β − β 2 ) 1 − β − 2+ β β (1 − β ) N +  −
 , (66) leading to
[Equation]
[Equation]
Combining Eq.
(62), Eq.
(65), Eq. (69), we have N [Equation]
(cid:88)
[Equation]
(70) by choosing t 0 (cid:16) T λ 1 λ 1+1 · ( n + ) − 1 λ 1+1 , we have
[Equation]
[Equation]
Similarly, for y i = − 1 , let
[Equation] which shares the same eigenvalues with M , thus we have [Equation] C.3 Convergence Rate
Theorem 3.
Let Asmp.
1, 3, 4 hold.
By setting η t = 2 t +1 µ ( t +1) 2 and β t = β (cid:16) 1 /n + , we have [Equation]
Proof. First, according to the recurrence of v t , we have [Equation] For sake of the presentation, denote
[Equation]
According to the smoothness of F [Equation] we have [Equation]
By taking expectation of A on both sides [Equation] we have [Equation]
By using the PL condition and subtracting F ( w ∗ ) from both sides
[Equation]
we get
[Equation]
If we set the learning rate η t = 2(2 t +1) µ ( t +1)
[Equation]
we have 1 − µ · η t / 2 =
t 2 / ( t + 1) 2 , leading to the follwing result:
[Equation]
Applying the above result recursively
[Equation]
we get
[Equation]
Moreover, according to Lem. 4, we have
[Equation] and [Equation] [Equation]
From Eq.
(79), Eq.
(80), Eq.
(81), we have
E A [ F
[Equation]
w
[Equation]
+
[Equation]
[Equation] Lemma 4. Denote r ( s ) [-λ +1 2 λ +
-]= σ (cid:16) (1 − π ) /π
· ˆ E x ∼ z − [ (cid:96) 1 ( h w ( x + ) − h w ( x ))]
/s (cid:17) .
Assume ∇ r ( s ) is a L r -Lipschitz smooth function with regard to s , and (cid:96) 2 is L 2 -Lipschitz smooth, then the following facts hold: (a) (cid:107)∇ f ( w ; z , v 1 ) − ∇ f ( w ; z , v 2 ) (cid:107) 2 ≤ L r L 2 N + (cid:107) v 1 − v 2 (cid:107) 2 . (b) (cid:107) δ t 2 (cid:107) 22 ≤ (cid:16) L r L 2 C w G βN + (cid:17) 2 · (cid:80) t − 1 τ =1 η 2 τ / ( t − 1) .
(c) Assume E S
[ (cid:107) h w ( S ) − φ ( h w ( z ))
(cid:107) 2 ] is upper bounded by κ , then we have E A (cid:104) (cid:107) δ t 3 (cid:107) 22 (cid:105) ≤ L 2 r N + + L 22 L 2 r κ 2 ( N + ) 2 . Proof. First, we have ∇ f ( w ; z , v ) =
ˆ E [Equation]
z + [Equation] r [Equation]
ˆ E [Equation] ∼ [Equation]
[Equation]
2 ( h w ( x ) − v )] (cid:17)(cid:105) , according to the Lipschitz smoothness of ∇ r , we have [Equation] [Table]
[Equation]
[Table] [Equation] Denote the i -th element of v as v ( i ) , for any x we have [Equation]
Eq. (83) and Eq. (84) leads to conclusion (a) imediately. We then prove conclusion (b) .
According to the above derivation, we have
[Equation] where the right side could be upper bounded by the following recursion [Equation] Step (*) is due to the Cauchy-Schwarz inequality.
By iterating the above recursion, we get [Equation] The proof of (b) is complete due to Eq. (85) and Eq.
Now we turn to conclusion (c) . First of all, denote [Equation] where S is a dataset with size N and z is a randomly sampled subset of S , we will show that for any w ∈ Ω , E S (cid:2) (cid:107) ∆( S ) (cid:107) 22 (cid:3)
= O (1 /N + ) . Consider a dataset S (cid:48) with at most one example x 0 differs from S , ∆( S ) satisiﬁes the bounded differences condition: [Equation]
According to Lem. [Equation] we have [Equation]
[Equation] Furthermore, similar to Eq. [Equation] we have
[Equation] leading to [Equation] Lemma 5 (Efron-Stein Inequality [6]) .
Let X 1 , · · · , X n be independent random variables and let Z = f
( X 1 , · · · , X n ) , if f has the bounded differences property with constants c 1 , · · · , c n , then
[Equation]
Competitors.
To validate the advantages of the proposed method over the state-of-the art methods in image retrieval, we compare two types of competitors: 1) Pairwise Losses , including Contrastive Loss [27], Triplet Loss [32], Multi-Similarity (MS) Loss [71], Cross-Batch Memory (XBM) [72].
These methods construct loss functions with image pairs or triplets. 2) Ranking-Based Losses , including
SmoothAP [9], FastAP [12], DIR [58], BlackBox [33], and Area Under the ROC Curve Loss (AUROC) [76].
These methods directly optimize the ranking-based metrics such as AUPRC or AUROC.
We reimplement all the competitors on the same codebase to ensure that the main experimental setting of competitors is the same as ours, including model structure, data preprocessing and augmentation, learning rate schedule, testing pipeline, etc. The unique hyperparameters of competitors follows the optimal settings of the original paper.
Moreover, the optimizer used are slightly different: following previous work, Adam is used to train the competitors, while ours is trained with SGD to ensure the consistence with our theoretical analysis.
We also provide the results of ours trained with Adam in Tab. 2, which shows no signiﬁcant difference. Environments.
The proposed method and all competitors are implemented with Pytorch 1.8.2 [51]. All experiments are conducted on a single NVIDIA RTX 3090 GPU.
Quantitative results.
We provide full results with more k for the Recall@k metric in Tab. 3, 5, 7. In addition, since the mainstream work on image retrieval does not split a validation but instead trains models with all data except the test data (called trainval split here), we further provide evaluation results of some methods trained on the trainval split in Tab. 4, 6,
[Table]
[Table]
[Table]
[Table]
Qualitative results.
We show more mean PR curves in Fig.
6 and more convergence results in Fig.
These qualitative results are consistent with our main conclusions.
[Table]
[Table]
[Table]
[Figure]
[Figure]