ℓ ∞ -Bounds of the MLE in the BTL Model under General Comparison Graphs
Abstract
The Bradley-Terry-Luce (BTL) model is a popular statistical approach for estimating the global ranking of a collection of items using pairwise comparisons.
To ensure accurate ranking, it is essential to obtain precise estimates of the model parameters in the ℓ ∞ -loss.
The difficulty of this task depends crucially on the topology of the pairwise comparison graph over the given items.
However, beyond very few well-studied cases, such as the complete and
Erdös-Rényi comparison graphs, little is known about the performance of the maximum likelihood estimator (MLE) of the BTL model parameters in the ℓ ∞ -loss under more general graph topologies.
In this paper, we derive novel, general upper bounds on the ℓ ∞ estimation error of the BTL
MLE that depend explicitly on the algebraic connectivity of the comparison graph, the maximal performance gap across items and the sample complexity.
We demonstrate that the derived bounds perform well and in some cases are sharper compared to known results obtained using different loss functions and more restricted assumptions and graph topologies.
We carefully compare our results to Yan et al.
(2012), which is closest in spirit to our work.
We further provide minimax lower bounds under ℓ ∞ -error that nearly match the upper bounds over a class of sufficiently regular graph topologies.
Finally, we study the implications of our ℓ ∞ - bounds for efficient (offline) tournament design.
We illustrate and discuss our findings through various examples and simulations.
INTRODUCTION
Simultaneous or ‘global’ ranking of a set of items is a practical problem that arises naturally in a variety of domains.
For example, one may wish to ascertain a ‘best player’ or ‘best team’ in a given sports league.
Designing a principled statistical approach to global ranking of items is challenging due to data limitations and complex domain-specific relationships between the underlying items to be ranked.
A popular and practicable solution to estimating global ranking is to utilize pairwise comparison information across the items to be ranked, which is easily accessible across many application domains.
The BTL model (Bradley and Terry, 1952; Luce, 1959) is a popular statistical model for pairwise comparison data.
A similar model was also originally studied in Zermelo (1929).
The continued practical and theoretical interest in the BTL model stems from its relatively simple parametric form which provides a good balance between interpretability and tractability for theoretical analysis.
The BTL model is domain-agnostic, making it an ideal benchmarking tool across a variety of ranking applications e.g. sports analytics (Fahrmeir and Tutz, 1994; Masarotto and Varin, 2012; Cattelan et al., 2013), and bibliometrics (Stigler, 1994; Varin et al., 2016).
Formally, we can describe the BTL model as follows.
Suppose that we have n distinct items, each with a (fixed but unobserved) positive strength or preference score w ∗ i , i ∈ [ n ] , quantifying item i ’s propensity to beat other items in pairwise comparisons.
The BTL model assumes that the comparisons between different pairs are independent and the outcomes of comparisons between any given pair, say item i and item j , are i.i.d.
Bernoulli random variables, with winning probability p ij , defined as
[Equation]
A common reparametrization is to set, for each i , w ∗ i =
exp( θ ∗ i ) , where θ ∗ :
= ( θ ∗ 1 , . .
. , θ ∗ n ) ⊤ ∈ R n .
By convention, we assume that (cid:80) i ∈ [ n ] θ ∗ i = 0 for parameter identifiability.
From a theoretical perspective, much attention in the BTL literature has been paid to two popular estimators, namely the maximum likelihood estimator (MLE) and the spectral method (Jain et al., 2020).
Recently, Chen et al.
(2020) show that the MLE attains a sharper minimax rate of the Hamming top-
k loss compared to the spectral method.
In this paper, we thus focus on the MLE, which we formally define later in Section 2.
General pairwise comparison graphs
Given n items to be compared, the pairwise comparison scheme among them can be expressed through an undirected simple graph G ( V, E ) , where the vertex set V :=
[ n ] and the edge set E :=
{ ( i, j ) :
i and j are compared } is determined by the comparison scheme.
Correspondingly, if we define the directed edge set as E d := { ( i, j, k ) :
( i beats j ) k times } , then the induced directed simple graph
G ( V, E d ) is called a directed comparison graph.
It is a classical result (Ford, 1957; Simons and Yao, 1999; Hunter, 2004) that the BTL model is identifiable if and only if G ( V, E ) is connected, and the MLE of the model parameters exists and is consistent if and only if G ( V, E d ) is strongly connected.
Henceforth, comparison graph refers to the undirected pairwise comparison graph.
Typically one is interested in getting sharp bounds for the estimation risk, which could be based on a norm-induced metric
∥ ˆ θ − θ ∗ ∥
p or a ranking metric, e.g., Kendall’s tau distance (Kendall, 1938).
What makes risk analysis of BTL model estimators particularly challenging is a combination of the type of estimation risk loss considered, and the assumptions on the topology of G ( V, E ) .
Core questions of interest
Among all the metrics measuring uncertainty of estimators of BTL parameters, the ℓ ∞ -loss directly connects with ranking metrics, e.g. binary and Hamming top-
k (partial) ranking loss (see, e.g. Chen et al., 2019, 2020).
It is thus natural to study the MLE for the BTL parameters in the ℓ ∞ -loss, to better understand the risk optimality of the MLE and further justify its use for practical global and partial ranking problems.
In this spirit, Yan et al.
(2012) focus specifically on proving ℓ ∞ -error bounds for the BTL
MLE for general comparison graphs.
However, a notable limitation in their setting is that they impose a strictly dense comparison graph assumption, which may be impractical in many real world applications.
This leaves a gap in the literature, summarized in the following questions:
Core questions: For the BTL model, how does the MLE perform with respect to the ℓ ∞ loss, under much weaker assumptions on the pairwise comparison graph compared to Yan et al.
That is, assuming only that the comparison graph is connected.
Moreover, what are the implications of such bounds in applications?
Providing a sharp analysis to these questions with a detailed comparison to recent theoretical results in the BTL literature motivates our work in this paper.
Relevant and related literature
We give a brief overview of the work that addresses the challenge of comparison graph topology in ranking.
When the comparison graph is a complete graph, Simons and Yao (1999) give a high-probability upper bound for the ℓ ∞ loss, i.e., ∥ ˆ θ − θ ∗ ∥ ∞ and obtain the asymptotic distribution of the
MLE.
In the setting where the comparison graph follows the
Erdös-Rényi graph model, Chen and Suh (2015), Chen et al.
(2019), Chen et al.
and Han et al.
(2020) derive highprobability upper bounds for the ℓ ∞ loss.
Moreover, Chen et al.
(2019) show that both MLE and spectral method are minimax optimal in terms of the binary top-
k ranking loss, i.e., whether the items with the highest k out of n preference scores are perfectly identified; Chen et al.
(2020) consider a Hamming Loss for top- k items and show that the MLE is minimax optimal compared to the spectral method with differences arising in constant factors.
For a broader class of comparison graphs beyond complete and Erdös-Rényi graph, researchers have studied the explicit dependence of the estimation risk on graph topology.
In particular, Yan et al.
(2012) give a high-probability upper bound for the ℓ ∞ -loss for relatively dense graphs.
Hajek et al.
; Shah et al.
(2016) give a high probability upper bound for the ℓ 2 or Euclidean loss ∥
ˆ θ − θ ∗ ∥ 2 , establish upper and lower bounds of E
∥
ˆ θ − θ ∗ ∥ 2 and show the minimax optimality of the constraint MLE across a wide range of graph topologies.
Recently, Agarwal et al.
(2018) give sharp upper bounds for a novel spectral method in the ℓ 1 -loss ∥ ˆ π − π ∗ ∥ 1 for π ∗ =
w ∗ / ∥ w ∗ ∥ 1 instead of θ ∗ .
Hendrickx et al.
(2019, 2020) propose a weighted least square method to estimate w ∗ and prove a sharp upper bound for their estimator in E [sin 2 ( ˆ w , w ∗ )] or equivalently in E ∥
ˆ w / ∥ ˆ w ∥ 2 − w ∗ / ∥
w ∗ ∥ 2 ∥ 22 , in the sense that this upper bound matches a instance-wise lower bound up to constant factors.
Contributions
Our contributions in this paper are fourfold and are summarized as follows:
• Upper bounds: We derive a novel upper bound for the ℓ ∞ -error of the regularized MLE in BTL model allowing for general graph topology.
Our upper bounds hold under minimal assumptions on graph topologies, i.e., assuming only that the comparison graph is connected.
Given such generality, we show our ℓ ∞ bound is tighter than existing results under a broad range of graph topologies, and works well in general.
In particular, we carefully compare our work analytically and in simulation to Yan et al.
(2012), which is closest in spirit to our work.
A minor corollary of our techniques
results
in the state
of the art ℓ 2 -loss bounds for the Erdös-Rényi graph.
• Lower bounds: We derive minimax lower bounds for BTL parameter estimation in ℓ ∞ -loss.
We analyze specific graph topologies satisfying certain regularity connectivity conditions under which the BTL MLE is nearly minimax optimal.
• Implications for tournament design: We show that the BTL MLE in ℓ ∞ -loss satisfies a unique subadditivity property, and how our ℓ ∞ bounds can exploit this property for efficient (offline) tournament design.
• Extension to the unregularized BTL model: We also extend our upper bounds under ℓ ∞ -loss to the unregularized (‘vanilla’)
BTL MLE, which is also frequently used in practice.
Due to the more complicated form of the vanilla BTL
MLE upper bounds and space limitations, we present these analagous results and their proofs separately in Appendix A.7.
Henceforth, MLE refers to the regularized BTL
MLE unless stated otherwise.
In addition to our theoretical contributions a core aspect throughout our paper is to emphasize the interpretability of our results, the associated assumptions, and implications for practical ranking tasks.
Organization of the paper
The rest of the paper is organized as follows.
In Section 2, we present our main results for the upper bound in Theoremand an interpretation of the key components of the bound.
In Section 3, we discuss minimax lower bounds using the ℓ ∞ risk loss in Theorem 5.
In Section 4, we show some practical implications of our results in efficient tournament design from a ranking perspective.
In Section 5, we conduct extensive numerical simulations to validate the optimality of our bounds compared to related results in the literature.
Notation
We typically use lowercase for scalars, e.g., ( x, y, z, . . . )
, boldface lowercase for vectors, e.g., ( x , y , z , . . . ) , and boldface uppercase for matrices, e.g. ( X , Y , Z , . . . ) .
We denote the finite set { 1 , . . . , n } by [ n ] .
For asymptotics, we denote x n ≲ y n or x n = O ( y n ) and u n ≳ v n or u n = Ω( v n ) if ∀ n , x n ≤ c 1 y n and u n ≥ c 2 v n for some constants c 1 , c 2 > 0 .
We denote e i as a vector whose entries are all 0 except that the i -th entry is 1 .
a n = o ( b n ) means a n /b n → 0 as n → ∞ and conversely, a n = ω ( b n ) means b n /a n → 0 as n → ∞ .
We denote 1 n ∈ R n to be a vector of ones.
UPPER BOUNDS
Recall that given n items to be compared, the comparison scheme among them defines the comparison graph G ( V, E ) ,
where V = [ n ] and E = { ( i, j ) : i and j are compared } .
We denote the corresponding adjacency matrix as A ∈ R n × n , and its ( i, j ) th entry is A ij : =
1 { ( i, j ) ∈ E } .
The associated (unnormalized) graph Laplacian is the symmetric, positive-semidefinite matrix L A :
= D − A , where
D = diag( n 1 , . . . , n n ) , with n i := (cid:80) nj
=1 A ij the degree of node i .
It is well known that the smallest eigenvalue of
L A is 0 with an eigenvector 1 n .
Let λ 2 ( L A ) be the second smallest eigenvalue of L A , known as the algebraic connectivity of G (Das, 2004), then G is connected if and only if λ 2 ( L A ) > 0 .
Following the standard in the BTL literature we assume a that for each edge ( i, j ) of the comparison graph, the corresponding items i and j are compared L times, each leading to an independent outcome y ( l ) ij ∈ { 0 , 1 } , where l ∈ [ L ] .
If pairs are compared different number of times, we take L to be the smallest number of pairwise comparisons over the edge set, as a worst-case scenario.
The corresponding sample averages are denoted with ¯ y ij = 1 L (cid:80) Ll =1 y ( l ) ij and are sufficient statistics for the model parameters.
The ℓ 2 -regularized MLE is defined as
[Equation]
where ℓ ( θ ; y ) is the negative log-likelihood, given by
[Equation]
and t ∈ R (cid:55)→ ψ ( t )
= 1 / [1 + e − t ] the sigmoid function.
Under this notational setup, we are ready to state the ℓ ∞ upper bound of the BTL MLE in Theorem 1.
Theorem 1.
Assume the BTL model with parameter θ ∗ =
( θ ∗ 1 , . . .
, θ ∗ n ) ⊤ such that 1 ⊤ n θ ∗ = 0 and a comparison graph
G = G ([ n ] , E ) with adjacency matrix A , algebraic connectivity λ 2 ( L A ) and maximum and minimum degrees n max and n min .
Suppose that each pair of items ( i, j ) ∈ E are compared L times.
Let κ = max i,j | θ ∗ i − θ ∗ j | and κ E = max ( i,
j ) ∈ E | θ ∗ i − θ ∗ j | and set ρ ≥ c ρ κ − 2 e − 2 .
5 κ E n − 4 n 1 / 2 max .
Assume that G is connected or λ 2 ( L A ) > 0 .
Then with probability at least 1 − O ( n − 4 ) , the regularized MLE ˆ θ ρ from (2) satisfies
[Equation]
where λ 2 = λ 2 ( L A ) , r := κ E + log κ provided that
L ≤ n 8 e 5 κ E max { 1 , κ } , and L is large enough so that the right hand side of Equation (4) is smaller than a sufficiently small constant C > 0 .
In particular, if we set ρ = c ρ /κ
(cid:112) n max /L for some c ρ > 0 , then ∥ ˆ θ
[Equation]
θ
[Equation]
e
[Equation] [Equation]
As a brief sketch, the proof is based on a gradient descent procedure initialized at θ (0)
=
θ ∗ and the idea is to control ∥ θ ( T )
− ˆ θ ρ ∥ ∞ using the linear convergence property and ∥ θ ( T ) − θ ∗ ∥ ∞ using the leave-one-out technique in Chen et al.
and
Chen et al.
In fact, our work confirms that such a line of argument extends to more general graph topologies beyond the Erdös-Rényi graph, which is non-trivial.
The proof details can be found in Appendix A.2.
Interpretation of key terms
The upper bound in Equation (4) contains several distinct terms, which interact with each other in non-trivial ways and express different aspects of the intrinsic difficulty of the estimation task.
• The factor e κE λ 2 ( L A ) combines two sources of statistical hardness: the maximal gap in performance κ E among the ranked items over the edge set E , and the algebraic connectivity λ 2 ( L A ) of the comparison graph.
It is intuitively clear that the larger the performance gap among the compared items, the more difficult it is to accurately estimate the model parameters.
Furthermore, the smaller the algebraic connectivity, the less connected the comparison graph is, due to the presence of bottlenecks 1 .
This in turn will increase the chance of obtaining a highly erroneous ranking or of gathering data from which a global ranking cannot be elicited at all.
The minimal and maximal degrees n min and n max further quantify the impact of the connectivity of the comparison graph.
• We note that the factor 1 λ 2 ( L A ) can be equivalently replaced with 1 λ 2 ( I ) (see Lemma 2 in Appendix A.2).
Here, I := ∇ 2 ℓ 0 ( θ ∗ ; y ) is the Fisher information matrix at θ ∗ and λ 2 ( I ) its smallest non-zero eigenvalue.
The fact that the bound depends on the Fisher information is not too surprising.
This is so, since this quantity in exponential families quantifies the curvature of the likelihood and the intrinsic difficulty of estimating θ ∗ .
• Our bounds depend on both κ and κ E , which is nonstandard in the literature.
By definition, κ E ≤ κ and in many cases, κ E can be much smaller than κ .
We discuss this further in Section 5.
• The term r :=
κ E + log κ shows the impact of large κ and κ E .
When κ ≲ n and κ E ≲ log n , r is negligible.
We will consider this parameter range throughout the paper unless stated otherwise.
• The term (cid:112) nL describes explicitly the impact of a high-dimensional parameter space on the estimation problem in relation to L , the number of samples for each comparison, which can be thought of as a measure of the sample size required for each of the n parameters.
The inverse root dependence on L is to be expected and, we conjecture, not improvable.
Remark 1 .
In the case of dense graphs, e.g., complete graphs, λ 2 ( L A ) is large enough so that even L = 1 will ensure a consistent estimator as n → ∞ .
But for sparse graphs,
L needs to be larger to compensate for weaker connectivity.
The assumption that L ≤ n 8 e 5 κ E max { 1 , κ } is a technical condition.
There is nothing special in the exponent for n .
Any fixed number larger than 8 can be used which will only affect the constants in the bounds.
The condition
L ≤ n 8 e 5 κ E max { 1 , κ } may seem counter-intuitive, since it places an upper bound on the sample size.
But a control over L is needed because as L gets larger, the optimal choice of the regularization parameter ρ = c ρ 1 κ (cid:112) n max L gets smaller and, accordingly, the convergence rate of the gradient descent procedure upon which our proof is based degrades.
The optimal choice ρ = c ρ 1 κ (cid:112) n max L depends on κ , which is unknown before an estimator is produced, however, one can set ρ =
c ρ (cid:112) n max L and the upper bound will only change by a factor max { 1 , κ } in the first term of
Equation (6).
2.1 COMPARISON TO OTHER WORK
To the best of our knowledge, Yan et al.
; Hajek et al.
; Shah et al.
; Negahban et al.
(2017); Agarwal et al.
; Hendrickx et al.
(2019, 2020) are the only existing papers that study estimation error for the BTL model on a comparison graph with general topology.
Since Negahban et al.
(2017); Agarwal et al.
; Hendrickx et al.
(2019, 2020) estimate the the preference scores w ∗ rather than θ ∗ , we cannot directly compare our results with theirs because there is no tight two-sided relationship between their metrics of error and ours.
Therefore, here we only compare our results to those in Yan et al.
; Hajek et al.
Shah et al.
(2016), as is summarized in Table 1.
We include the comparison to the other four papers in Appendix A.1.
ℓ ∞ loss: Yan et al.
(2012) establish an ℓ ∞ -bound depending on n ij , the number of common neighbors of item i and item j in the comparison graph, under a strong assumption that n ij ≥ cn for some constant c ∈ (0 , 1) .
This constraint on graph topology is stronger than ours since it requires the graph to be dense.
In particular, when the comparison graph comes from an Erdös-Rényi model ER ( n, p ) , min i,j n ij ≍ np 2 .
Then the conditions in Yan et al.
(2012) requires p to be bounded away from 0 and their bound becomes e κ p (cid:113) log n npL , while our bound is e 2 κE √ p (cid:113) log n npL .
Our bound is tighter for moderate or small κ E , and importantly, allows p to vanish.
Furthermore, in Section 5, we show by some specific examples that min i,j n ij could be 0 even for many fairly dense graphs, to illustrate that the Yan et al.
(2012) upper bound cannot apply to many realistic settings.
ℓ 2 loss: Hajek et al.
; Shah et al.
(2016) consider constrained MLE
ˆ θ := min ∥ θ ∥ ∞ ≤ B ℓ 0 ( θ ) for a known parameter B such that ∥ θ ∗ ∥ ∞ ≤ B .
Setting aside the fact that their results require stricter conditions than ours, our ℓbound is tighter than theirs for general parameter settings with moderate B, κ and for a broad range of graphs with moderate λ 2 ( L A ) , i.e., not too sparse or irregular.
[Table]
[Equation]
[Table]
Yan et al
[Equation]
[Table]
Our work
See Theorem
Hajek et al.
[Equation]
Shah et al
[Equation] [Equation]
Our work
[Equation]
[Table]
[Equation]
[Table]
[Equation]
We re-emphasize that Hendrickx et al.
(2020) also provide upper bounds for a general fixed comparison graph that matches an instance-wise lower bound, for their parameter of interest w ∗ :
= ( e θ ∗ 1 , . .
, e θ ∗ n ) ⊤ , instead of θ ∗ .
However, their error metric, i.e., sin( ˆ w , w ∗ ) , is quite different from other similar papers in the BTL literature, including our work.
As such, it is not clear how to compare to their results.
Furthermore, as noted in Section 1, from the perspective of ranking, an entry-wise metric like ∥·∥ ∞ is more informative than vector-level metrics like ∥ · ∥ 2 and sin( · , · ) .
2.2 SPECIAL CASES OF GRAPH TOPOLOGIES
We can check some common types of comparison graph topologies and see in what order the necessary sample complexity N comp = | E | L needs to be to achieve consistency, i.e., ∥ ˆ θ − θ ∗ ∥
∞
= o (1) .
The results are summarized in
Table 2.
For path and star graphs, we used the specialized bounds in Propositions 3 and 4.
As shown in Table 2, our
Graph
N comp (Yan et al., 2012)
N comp (Our work)
Complete Ω( n 2 ) Ω( n 2 )
Bipartite
N/A Ω( n 2 )
Path
N/A ω ( e 2 κ E n
[Table]
n )
Star
[Table] [Table]
bound now applies to a much broader class of graph topologies under the ℓ ∞ -norm compared to Yan et al.
Remark 2 .
For the path graph, star graph, and barbell graph, the necessary sample complexity induced by directly applying our ℓ ∞ bound is larger than the sample complexity induced by the ℓ 2 bound in Shah et al.
(2016), though they require more stringent conditions than ours.
Thus we provide specialized sharp upper bounds in the case of path and star graph in Proposition 3 and 4.
Additionally, in Section 4, we illustrate that by applying a unique sub-additivity property of ℓ ∞ -loss, we can achieve a much smaller sample complexity in graphs with bottlenecks like the barbell graph.
Erdös-Rényi graph: By applying a union bound on λ 2 ( L A ) , n max , and n min to the sample-wise bounds in Theorem 1, we obtain a corollary in the setting where the comparison graph follows the Erdös-Rényi model ER ( n, p ) .
Corollary 2 (Erdös-Rényi graph) .
As a corollary to Theorem 1, suppose that the comparison graph comes from an
Erdös-Rényi graph ER ( n, p ) , then under the same conditions, with probability at least 1 − O ( n − 4 ) , it holds that
[Equation]
The full form of Corollary 2 with a proof can be found at the end of Appendix A.2.
For the Erdös-Rényi comparison graph ER ( n, p ) , the tightest ℓ ∞ -norm error bound e 2 κ (cid:113) log n npL is proved in Chen et al.
and Chen et al.
Han et al.
(2020) establish an ℓ ∞ -norm upper bound of e 2 κ (cid:113) log n np · log n log( np ) .
Negahban et al.
(2017) obtain an ℓ 2norm upper bound of e 4 κ log n pL and a lower bound of e − κ 1 pL .
Thus the derived ℓ 2 -bound in Corollary 2 in Erdös-Rényi case is minimax optimal.
In this case our derived ℓ ∞ -bound cannot achieve the rate established in Chen et al.
, Chen et al.
(2020), though our ℓ 2 -bound exhibits the optimal rate proved in Negahban et al.
The reason why our bound does not imply the optimal ℓ ∞ -rate under a Erdös-Rényi comparison graph is that our bound is a sample-wise bound and thus cannot leverage some regular property of Erdös-Rényi graph beyond algebraic connectivity and degree homogeneity that is exhibited with high probability.
Tree graphs: For extremely sparse graphs like tree graphs, the general upper bound in Theorem 1 is loose compared to the lower bound in Theorem 5.
Therefore, we separately prove some sharp upper bounds for path and star graphs as a complement to our general theory, in these frequently studied cases.
For example, single-elimination sports tournaments are commonly designed as a binary tree graph.
By the spectral property of path and star graphs (see Appendix A.6), one can verify that the upper bounds in both norms match the ℓ ∞ lower bound in Theorem 5 and the ℓ 2 lower bound in Shah et al.
(2016), up to √ log n and e 2 κ E factors.
Proposition 3 (Path graph) .
Suppose the comparison graph is a path graph ([ n ] , E ) with E
= { ( i, i + 1) } i ∈
[ n − 1] and
L > ce 2 κ E n log n for some universal constant c , then with probability at least 1 − n − 4 , the vanilla MLE ˆ θ 0 satisfies
[Equation]
Proposition 4 (General tree graph) .
Suppose the graph is a tree graph ([ n ] , E ) where each item i and j are compared
L times such that L > ce
2 κ E n log n for some universal constant c .
Then with probability at least 1 − n − 4 , the vanilla
MLE ˆ θ 0 satisfies
[Equation]
where D := max i,j | path( i, j ) | is the diameter.
In particular, for star graph, the upper bound is given by D = 1 .
The full form of Proposition 3 and Proposition 4 with proofs are found in Appendix A.2.
Briefly, the proofs leverage the closed-form solution of vanilla MLE under the tree graph.
LOWER BOUNDS
In this section, we derive a minimax lower bound for the ℓ ∞ loss.
Towards that end, we first introduce some new notation.
Let N comp be the total number of comparisons that have been observed, so in our setting, N comp = | E | L where | E | is number of edges in the comparison graph G .
Denote the two items involved in the i -th comparison as ( i 1 , i 2 ) such that i 1 < i 2 .
Let ˜ L A =
[Equation]
be the normalized graph Laplacian with pseudo inverse ˜
L † A and eigenvalues 0 = λ 1 ( ˜ L A ) ≤ λ 2 ( ˜ L A ) ≤ · · · ≤ λ n ( ˜ L A ) .
With the main notation in place, our minimax lower bound is summarized in the following result.
Theorem 5.
Assume that the comparison graph G is connected and
the sample size N comp ≥ c 2 tr ( ˜ L † A ) e 2 κ κ 2 , any estimator (cid:101) θ based on N comp comparisons with outcomes from the
BTL model satisfies
[Equation]
where Θ κ = { θ ∈ R n : 1 ⊤ n θ = 0 ,
∥ θ ∥ ∞ ≤ κ } .
The proof of Theorem 5 largely leverages the lower bound construction from Theorem 2 in Shah et al.
The main modification in adapting it to our setting is to construct an ℓ ∞ -packing set.
This is done by utilizing the tight topological equivalence of ℓ ∞ and ℓ 2 norms in finite dimensions.
We can compare this lower bound with the upper bound in
Theorem 1.
In our setting, the comparisons distribute evenly over all pairs, so N comp = | E | L , and λ i ( ˜ L A )
= 1 | E | λ i ( L A ) .
Thus, given a comparison graph with λ 2 ( ˜ L A ) ≍ 1 n , the lower bound becomes
[Equation]
In ER ( n, p ) case, this lower bound becomes
[Equation]
which matches the upper bound in Chen et al.
For some “regular” graph topology with λ 2 ( ˜ L A ) ≍ 1 n like complete graph, expander graph with ϕ = Ω( n ) and complete bipartite graph with two partition sets of size Ω( n ) , the upper bound becomes
[Equation]
Therefore, when the comparison graph topology is sufficiently regular, our upper bound matches the lower bound up to a log n factor and a factor of e 3 κ .
As a final remark,
Negahban et al.
(2017) show that the minimax lower bound for ℓ 2 -loss and Erdös-Rényi comparison graph ER ( n, p ) is e − κ 1 pL , which matches our ℓ 2 upper bound up to a factor of e 2 κ .
IMPLICATIONS FOR TOURNAMENT DESIGN
In this section, we discuss how our results can be leveraged to construct more efficient tournament design from a ranking perspective in sports leagues.
As discussed in Section 2.2, for some comparison graphs with small λ 2 ( L A ) , the requirement on L and N comp for consistency is stringent.
However, as we show next, we can significantly relax the requirement on the sample complexity N comp by adaptively varying the number pairwise comparisons observed over different subsets of the items in a manner that leverages different degrees of connectivity of the comparison graphs.
The basic idea is that model parameters corresponding to a subset of items inducing a highly connected sub-graph require relatively few observations.
On the other hand, the outcomes of comparisons with items corresponding to nodes of the comparison graph that are part of a “graph bottleneck” are especially important in yielding accurate global ranking and, therefore, should be more heavily sampled (in the sense of having a larger number L of observations).
The case of a Barbell graph consisting of two complete subgraphs connected by few “bridge” edges (as is shown in
Figure 2) is an extreme illustration of this situation and will be discussed below.
In this case, it is clear that the parameters corresponding the items adjacent to the bridge edges ought to be estimated with higher accuracy and therefore, for those items L should be set larger.
Furthermore, it is possible to estimate the model parameters separately over different sub-graphs and combine these estimators in a way that could lead to an improved rate, compared to a joint or omnibus estimator.
Indeed, the next result shows that the ℓ ∞ -error rate of the combined estimator is bounded by the sum of the error rates for estimating the parameters of the individual sub-graphs.
Formally, let I 1 , I 2 , I 3 be three subsets of [ n ] such that ∪ 3 j =
I j = [ n ] and, for each j ̸ = k , I
j ̸⊆
I k and for i = 1 , 2 ,
I i ∩ I 3 ̸ = ∅ .
Assume that the sub-graphs induced by I j ’s are connected and the number of comparisons for all pairs can be different across sub-graphs.
Let θ ∗ be the vector of preference scores in the BTL model over n items and ˆ θ ( j ) be the MLE of θ ∗ ( j ) ∈ R | I j | for the BTL model involving only items in I j , j = 1 , 2 , 3 .
Also define the augmented version ˜ θ ( j ) ∈ R n such that ˜ θ ( j ) ( I j )
=
ˆ θ ( j ) .
Now take two nodes t 1 ∈ I 1 ∩ I 3 , t 2 ∈ I 2 ∩ I 3 , and let δ 3 = ˜ θ (1) ( t 1 ) − ˜ θ (3) ( t 1 ) , δ 2 = ˜ θ (3) ( t 2 ) − ˜ θ (2) ( t 2 ) .
An ensemble estimator add-MLE ˆ θ ∈ R n is a vector such that ˆ θ (
[Equation]
) =
[Equation] θ [Equation] [Equation]
=
[Equation] + [Equation] 2 , and ˆ θ ( S 3 ) =
˜ θ (3) ( S 3 ) + δ 3 , where S 2 =
I 2 \ I 1 and S
= I 3 \ ( I 1 ∪ I 2 ) .
Notice that the value of ˆ θ depends on the choice of t 1 , t 2 , but the estimation error of all ensemble estimators can be well-bounded, as is shown in Proposition 6.
Proposition 6 (Subadditivity of ℓ ∞ -loss in BTL) .
Under the setting above, for any add-MLE ˆ θ ∈ R n based on ˆ θ (1) , ˆ θ (2) , ˆ θ (3) , it holds that
[Equation]
where d ∞ ( v 1 , v 2 ) :
=
[Equation]
avg
[Equation] and [Equation] for x ∈ R n .
The proof of Proposition 6 is found in Appendix A.5.
For some types of graph topologies the above result can be used to devise a divide-and-conquer strategy for estimating the model parameters with better sample complexity than that of an omnibus estimator, i.e., the joint-MLE in our setting.
Indeed, as discussed in Section 2.2, for a barbell graph containing two size- n/ 2 complete sub-graphs connected by a single edge, we need N comp =
Ω( n 5 log n ) for an o (1) error bound of the joint-MLE.
From a practical perspective, we note that such a divide and conquer strategy gives flexibility in the number of comparisons in each sub-graph.
For example, if we set L = 1 for the two complete sub-graphs to get
MLEs ˆ θ 1 , ˆ θ 2 , and set L = n for the two items linking the two sub-graphs to get an MLE ˆ θ 3 , and combine them by shifting ˆ θ 2 by the difference of two entries of ˆ θ 3 , then a total sample complexity reduction to N comp = Ω( n 2 ) will ensure ℓ ∞ -norm error of order
O
( e 2 κ E √ log n / √ n ) =
o (1) , because for a complete graph of size m , the ℓ ∞ -norm error is O ( e 2 κ E √ log m / √ mL ) .
In Example 7 and Appendix A.4.1, we show some simulation results illustrating the advantage of using subadditivity in estimation, where we generalize the add-MLE to Island graph and Barbell graph with multiple bridge edges that can have more than 3 dense sub-graphs.
Note that such flexible tournament design is similar to the idea of active ranking (Heckel et al., 2019; Ren et al., 2019), but there is still a substantial difference between our setting and active ranking.
Active ranking assumes that one can design the tournament in an online manner, so that the next pair of items to be compared is determined by the newest outcomes of comparisons.
However, in practice many tournaments can only be designed offline , i.e., before any games are played.
Under this common setting, our ℓ ∞ - subadditivity property provides a useful offline approach to efficient tournament design.
EXAMPLES AND SIMULATIONS
In this section, we conduct numerical experiments on simulated data with two main goals.
First, we illustrate the utility of the subadditivity property in Proposition 6 in the case of Island graphs (see Example 7).
Second, we demonstrate the relative tightness of our ℓ ∞ upper bound compared to
Yan et al.
(2012), since their work is closest in spirit to ours.
Specifically, we compare the two bounds in a setting where analytical comparison is not directly feasible (see Example 8).
All of our reproducible code is openly accessible 2 .
In the BTL model, the maximal winning probability is p max ( κ )
=
(1 + e − κ ) .
To get a sense, p max (2 . 20) = 0 .
900 , p max (4 . 59)
= 0 .
A winning probability larger than 0.99 is fairly rare in practice, so it would not be too constraining to set κ = 2 . 2 in our simulation.
But analytically our result allows κ to diverge with n .
In our experiments, we set θ ∗ i
= θ ∗ 1 +( i − 1)
δ for i > 1 with δ =
κ/ ( n − 1) .
We additionally assign θ ∗ 1 to ensure that 1 ⊤ n θ ∗ = 0 , for parameter identifiability.
Under this setting, for some special graphs, e.g., the Island graph in Example 7, κ E can be much smaller than κ , showing an advantage of our upper bound in representing the dependency on the maximal performance gap κ E along the edge set, rather than κ the whole vertex set.
However, there may be some cases where the majority of edges have small performance gaps and only a few edges have large gaps.
Here, the control in the upper bound purely by κ E can again be loose.
An interesting future direction is to make upper bounds tighter in such cases by including more structural parameters, like the proportion of small-gap edges.
We include some illustrative examples in
Appendix A.4.
[Figure]
l - i n f i n i t
y e
rr o r
[Figure]
Example 7 (Graph with min i,j n ij = 0 ) .
In this case, we intend to illustrate that min i,j n ij could be 0 or quite close to 0 for even fairly dense graphs, making the upper bound in
Yan et al.
(2012) less effective.
Consider a 3-Island comparison graph G with n nodes.
The induced sub-graphs on node sets V 1 , V 2 , V 3 with | V i | =
n i are complete graphs, where
V 1 ∩ V 3 = ∅ , V 1 ∪ V 2 ∪ V 3 =
[ n ] , and V i ∩ V 2 ̸ = ∅ for i = 1 , 3 .
There is no edge except for those within V 1 , V 2 , V 3 .
This graph G is connected, and can be fairly dense if we make nlarge, but min i,j n ij = 0 always holds since V 1 ∩ V 3 = ∅ and the two induced sub-graphs are complete.
See Figureleft panel for a visualization of the adjacency matrix of such a graph.
We can also consider more general Island graphs.
A general Island graph is determined by n , the size of the graph, n island , the size of island sub-graphs, and n overlap , the number of overlapped nodes between islands.
Each island subgraph is a complete graph, and there is no edge outside islands.
For Island graphs, it holds that min i,j n ij =and κ E ≈ κ · n island /n .
Figure 1 top panel shows the adjacency matrix of two Island graphs.
Figure 1 bottom panel shows the comparison of the ℓ ∞ -error of the jointMLE and the add-MLE (see the detailed definition in Appendix A.4) while varying the difference in the average of preference scores of each island sub-graph, where we set n island =
50 , n overlap = 5 , L = 10 .
Every point on the lines is the average of 100 trials.
It can be seen that the add-MLE by the divide-and-conquer strategy largely dominates the joint-MLE in ℓ ∞ -error.
In Example 7, we show a common family of graphs which is fairly dense while min i,j n ij = 0 , so that the upper bound in Yan et al.
(2012) does not hold.
Next in Example 8 we consider another family of graphs where their upper bound holds but still relatively looser than our bound.
Example 8 (Barbell graph with random bridge edges) .
Consider a generalized Barbell graph G containing n = n 1 + nnodes, where the induced sub-graph on nodes { 1 , · · · , n 1 } and { n 1 + 1 , · · · , n } are complete graphs, and the two subgraphs are connected by some bridge edges ( i, j ) for some 1 ≤ i ≤ n 1 and n 1 + 1 ≤ j ≤ n .
Denote the set of bridge edges as E l , then | E l | / ( n 1 n 2 ) quantifies the connectivity of G : the larger | E l | / ( n 1 n 2 ) is, the denser or more regular
G is.
[Figure]
r a t
i o : o u r s / y a n
The curve is obtained as the average of 100 trials with one standard deviation shown by the colored area.
In Figure 2 we show a comparison of the real ℓ ∞ -loss ∥ ˆ θ − θ ∗ ∥ ∞ , and the upper bounds of ℓ ∞ -error in Yan et al.
(2012) and our paper.
We include this relative comparison to numerically demonstrate that our bound is in general tighter than Yan et al.
(2012), since there is no known analyt- ical relationship between min i,j n ij and λ 2 ( L A ) for general graphs.
In our experiment, we set n 1 = n 2 = n s , L = 10 , and randomly link | E l | = n 1 n 2 p edges between the two complete sub-graphs, and vary n s from 50 to 1000 with p = 3 log( n s ) /n s .
Every point on the line is the average of 100 trials.
It can be seen that our upper bound has a faster vanishing rate, compared to Yan et al.
(2012) for this simulated scenario.
This is evident as the plotted ratio of our upper bound relative to the upper bound in Yan et al.
(2012) has a steady decreasing trend, as n increases.
It should be noted that there are leading constant factors in both upper bounds, and for convenience we set them to be 1 for both bounds.
Thus, one should focus on the trend of the curve rather than the magnitude of the ratio in Figure 2.
DISCUSSION
In this work we provide a sharp risk analysis of the MLE for the BTL global ranking model, under a more general graph topology, in the ℓ ∞ -loss.
This addresses a major gap in the BTL literature, in extending the comparison graph to more general and thus more practical settings, compared to dense graph setting in Yan et al.
Specifically we derive a novel upper bound for the ℓ ∞ and ℓ 2 -loss of the
BTL-MLE, showing explicit dependence on the algebraic connectivity of the graph, the sample complexity, and the maximal performance gap between compared items.
We also derive lower bounds for the ℓ ∞ -loss and analyze specific topologies under which the MLE is nearly minimax optimal.
We also show that the ℓ ∞ -loss satisfies a unique subadditivity property for the BTL MLE and utilize our derived bounds for efficient tournament design.
We note that our upper bound is suboptimal in the cases where the graph topology is extremely sparse or irregular.
Although we provide sharp upper bounds for path and star graphs as separate propositions, we still miss optimality other graph topologies.
A good future direction would be to optimize the upper and lower bounds in such comparison graph regimes.
Another promising direction is to extend this analysis to the multi-user ranking models as in Jin et al.
Acknowledgments
We would like to thank Heejong Bong from the Carnegie
Mellon University (CMU) Department of Statistics & Data
Science, for his valuable feedback and discussions during this work.
We would also like to thank the anonymous reviewers for their feedback which greatly helped improve our exposition.
In Advances in Neural Information Processing
Montreal, Quebec, Canada , pages 1475–
URL http://papers.nips.cc/paper/
PMLR http://proceedings mlr press/v97/hendrickx19a.html .
PMLR
URL http://proceedings.
mlr.press/v119/hendrickx20a.html .
, Proceedings of the Thirty
Sixth Conference on Uncertainty in Artificial Intelligence,
UAI 2020, virtual online, August 3-6, 2020 , volume 124 of
Proceedings of Machine Learning Research , pages 609– 618.
AUAI Press, 2020.
URL http://proceedings.
mlr.press/v124/jain20a.html .
Tao Jin, Pan Xu, Quanquan Gu, and Farzad Farnoud.
Rank aggregation via heterogeneous thurstone preference models
In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020,
New York, NY, USA, February 7-12, 2020 , pages 4353– 4360.
AAAI Press, 2020.
URL https://ojs.aaai.org/ index.php/AAAI/article/view/5860 .
Biometrika
ISSN 00063444.
URL http://www.jstor.org/stable/2332226 .
doi: 10.
URL https: //proceedings.neurips.cc/paper/2019/file/ be3087e74e9100d4bc4c6268cdbe8456-Paper.pdf .
Balakrishnan
Journal of Machine Learning Research
URL http://jmlr.org/papers/v17/15-189.
html .
doi
10.1214/aos/
URL https org/
1214/aos/1018031267 .
ISSN 0964-1998.
doi: 10.1111/rssa.12124.
URL https://doi.org/10.1111/rssa.12124 .
ISSN
URL http://www.jstor.org/stable/24309985 .
BF https://doi.org/10.