Detecting Abrupt Changes in Sequential Pairwise Comparison Data
Anonymous Author(s)
AffiliationAddress email
Abstract
The Bradley-Terry-Luce (BTL) model is a classic and very popular statisticalapproach for eliciting a global ranking among a collection of items using pairwisecomparison data.
In applications in which the comparison outcomes are observedas a time series, it is often the case that data are non-stationary, in the sense thatthe true underlying ranking changes over time.
In this paper we are concernedwith localizing the change points in a high-dimensional BTL model with piecewise constant parameters.
We propose novel and practicable algorithms based ondynamic programming that can consistently estimate the unknown locations ofthe change points.
We provide consistency rates for our methodology that dependexplicitly on the model parameters, the temporal spacing between two consecutivechange points and the magnitude of the change.
We corroborate our findings withextensive numerical experiments and a real-life example
Introduction
Pairwise comparison data are among the most common types of data collected for the purpose ofeliciting a global ranking among a collection of items or teams.
The Bradley-Terry-Luce model(Bradley and Terry, 1952; Luce, 1959) is a classical and popular parametric approach to modelpairwise comparison data and to obtain an estimate of the underlying ranking.
The Bradley-TerryLuce model and its variants have been proven to be powerful approaches in many applications,including sports analytics (Fahrmeir and Tutz, 1994; Masarotto and Varin, 2012;
Cattelan et al., 2013),bibliometrics (Stigler, 1994; Varin et al., 2016), search analytics (Radlinski and Joachims, 2007;
Agresti, 2013), and much more.
To introduce the BTL model, suppose that we are interested in ranking n distinct items, each with a(fixed but unobserved) positive preference score w i , i ∈
[ n ] , quantifying its propensity to beat otheritems in a pairwise comparison.
The BTL model assumes that the outcomes of the comparisonsbetween different pairs are independent Bernoulli random variables such that, for a given pair ofitems, say i and j in [ n ] := { 1 , . . . , n } , the probability that i is preferred to (or beats) j is
[Equation]
A common reparametrization is to set w ∗ i =
exp( θ ∗ i ) for each i , where θ ∗ :
= ( θ ∗ 1 , . .
. , θ ∗ n ) ⊤ ∈ R n .
To ensure identifiability it is further assumed that (cid:80) i ∈ [ n ] θ ∗ i = 0 .
The properties and performance of the BTL model have been thoroughly studied under the assumptionthat the outcomes of all the pairwise comparisons are simultaneously available and follow the same
BTL model.
In many applications however, it is very common to observe pairwise comparison datasequentially (i.e. one at a time), with time stamps over multiple time periods.
In these cases, it is unrealistic to assume that observations with different time stamps come from the same distribution.
For instance, in sports analytics, the performance of teams often changes across match rounds, and
Fahrmeir and Tutz (1994) utilized a state-space generalization of the BTL model to analyze sporttournaments data.
Ranking analysis with temporal variants has also become increasingly importantbecause of the growing needs for models and methods to handle time-dependent data.
A series ofresults in this direction can be found in Glickman (1993), Glickman and Stern (1998), Cattelan et al.(2013), Lopez et al.
, Maystre et al.
Bong et al.
(2020), Karlé and Tyagi (2021) andreferences therein.
Much of the aforementioned literature on time-varying BTL model postulates thattemporal changes in the model parameters are smooth functions of time and thus occur gradually ona relatively large time scale.
However, there are instances in which it may be desirable to insteadmodel abrupt changes in the underlying parameters and estimate the times at which such change hasoccurred.
These change point settings, which, to the best of our knowledge, have not been consideredin the literature, and are the focus of this paper.
Contributions
We make the following methodological and theoretical contributions.
• Novel change point methodology.
We develop a computationally efficient methodology to
consistently estimate the change points for a time-varying BTL model with piece-wise constantparameters.
Our baseline procedure Algorithm 1 consists of a penalized maximum likelihood
estimator of the BTL model under an ℓ 0 penalty, and can be efficiently implemented via dynamic
programming.
We further propose a slightly more computationally expensive two-step procedure in
Algorithm 2 that takes as input the estimator returned by our baseline procedure and delivers a more
precise estimator with provably better error rates.
We demonstrate through simulations and a real life
example the performance and practicality of the procedure we develop.
• Theoretical guarantees.
We obtain finite sample error rates for our procedures that dependexplicitly on all the parameters at play: the dynamic range of the BTL model and the number of itemsto be compared, the number of change points
, the smallest distance between two consecutive changepoints and the minimal magnitude of the difference between the model parameters at two consecutivechange points.
Our results hold provided that a critical signal-to-noise ratio condition involving allthe relevant parameters is satisfied.
We conjecture that this condition is optimal in an informationtheoretic sense.
Both the signal-to-noise ratio condition and the localization rates we obtain exhibit aquadratic dependence on the number of items to be compared, which matches the sample complexitybound for two sample testing for the BTL model recently derived by Rastogi et al.
We emphasize that the change point setting we consider have not been previously studied and bothour methodology and the corresponding theoretical guarantees appear to be the first contribution ofits kind in this line of work.
69 Related work
Change point detection is a classical problem in statistics that dates back to 1940s (Wald, 1945; Page,1954).
Contributions in the 1980s established asymptotic theory for change point detection methods(Vostrikova, 1981; James et al., 1987; Yao and Au, 1989).
Most of the classical literature studied theunivariate mean model.
Recently with more advanced theoretical tools developed in modern statistics,more delicate analysis of change point detection came out in high-dimensional mean models (Jirak,2015; Aston and Kirch, 2018; Wang and Samworth, 2018), covariance models (Aue et al., 2009;
Avanesov and Buzun, 2018; Wang et al., 2021b), high-dimensional regression models (Rinaldo et al.
Wang et al., 2021c), network models (Wang et al., 2021a), and temporally-correlated timesseries (Cho and Fryzlewicz, 2015; Preuss et al., 2015; Chen et al., 2021; Wang and Zhao, 2022).
Although change point detection has already been extensively studied in many different settings, littleis known about the case of pairwise comparison data.
Höhle (2010) numerically study the CUSUMmethod for online change point detection in logit models and BTL models without giving theoreticalguarantees.
We aim to fill the gap in the literature and propose a theoretically trackable approach
Model and assumptions
Below we introduce the time-varying BTL model with piece-wise constant coefficients that we aregoing to study and the sampling scheme for collecting pairwise comparison data over time.
We assume throughout that data are collected as a time series indexed by t ∈ [ T ] := { 1 , . . .
, T } that,at each time point t , a single pairwise comparison among a collection of n items is observed.
Thedistinct pair ( i t , j t ) ∈ [ n ] 2 of items to be compared at time t is randomly chosen from the n items,independently over time.
That is,
[Equation]
For each t , let y t ∈ { 0 , 1 } denote the outcome of the comparison between i t and j t , where y t = 1indicates that i t beats j t in the comparison.
We assume that y t follows the BTL model (1.1), i.e.
[Equation]
where θ ∗ ( t ) = ( θ ∗ 1 ( t ) , . . .
θ ∗ n ( t )) is, a possibly time-varying, parameter that belongs to the set
[Equation]
for some B > 0 .
In the recent literature on the BTL model, the parameter B is referred to as thedynamic range (see, e.g., Chen et al., 2019) which readily implies a bound on the smallest possibleprobability that an item is beaten by any other item.
Indeed, it follows from (2.2) and (2.3) that
[Equation]
Remark 1 .
The quantity p lb have appeared in several equivalent forms in the BTL literature, e.g.,max i,j ∈ [ n ] w ∗ i w ∗ j (Simons and Yao, 1999; Negahban et al., 2017) and e 2 B (Li et al., 2022).
The minimalwinning probability p lb can quantify the difficulty in estimating the model parameters, with a smallp lb implying that some items are systematically better than others, a fact that is known to lead tonon-existence of the MLE (see, e.g. Ford, 1957) and to hinder parameter estimability.
In the BTLliterature the dynamic range B and, as a result, the quantity p lb are often treated as known constantsand thus omitted (Shah et al., 2016; Chen et al., 2020), a strong assumption that results in an implicitregularization but potentially hides an important feature of the model.
As argued in Bong and Rinaldo(2022), in high-dimensional settings this may not be realistic.
We will allow for the possibility of avarying B and p lb , and keep track of the effect of these parameters on our consistency rates.
It is convenient to rewrite (2.2) in a different but equivalent form that is reminiscent of logisticregression and will facilitate our analysis.
One can express the fact that, at time t , the items i t and j tare randomly selected to be compared using a random n -dimensional vector x ( t ) that is uniformlydrawn from the sets of all vectors in {− 1 , 0 , 1 } n with exactly two-non-zero entries of opposite sign,namely x i t ( t )
=
and x j t ( t ) = − 1 .
Then equation (2.2) can be written as
[Equation]
(2.5) where ψ ( x ) = 1 1+ e − x is the sigmoid function.
For any time interval I ⊂ [ T ] we then assume thatthe data take the form of an i.i.d.
sequence { ( x ( t ) , y t ) } t ∈I , where each x ( t ) is an i.i.d.
draw from{− 1 , 0 , 1 } n with aforementioned properties and, conditionally on x ( t ) , y t is a Bernoulli randomvariable with success probability (2.2).
The negative log-likelihood of the data is then given by
L ( θ , I ) = (cid:88) t ∈I ℓ t ( θ ) , where ℓ t ( θ ) :=
ℓ ( θ ; y t , x ( t )) = − y t x ( t ) ⊤ θ + log[1 + exp( x ( t ) ⊤ θ )] .
For a time interval I , we can define a random comparison graph G ( V, E ) with vertex set V :=
[ n ]and edge set E :=
{ ( i, j ) :
i and j are compared } .
It is well-known that the topology of G ( V, E )plays an important role in the estimation of BTL parameters (Shah et al., 2016).
Under assumption(2.1), the comparison graph over I follows the random graph model G ( n, |I| ) , which has |I| edgesrandomly picked from the full edge set E full :=
{ ( i, j ) : 1 ≤ i < j ≤ n } with replacement.
Therefore, the process { ( x ( t ) , y t ) } t ∈I is stationary as long as θ ∗ ( t ) is unchanged over I .
In the change point BTL model we assume that, for some unknown integer K ≥ 1 , there exist K + 2points { η k } K +1 k =
such that 1 = η 0 < η 1 < · · · < η K < η K +
=
T and θ ∗ ( t ) ̸
= θ ∗ ( t − 1) whenevert ∈ { η k } k ∈ [ K ] .
Define the minimal spacing ∆ between consecutive change points and the minimaljump size κ as
[Equation]
As we mentioned in the introduction, the goal of change point localization is to produce an estimatorof the change points { ˆ η k } k ∈ [ ˆ K ] such that, with high-probability as T → ∞ , we recover the correctnumber of change points and the localization error is a vanishing fraction of the minimal distancebetween change points, i.e. that
[Equation]
In change point literature, estimators satisfying the above conditions are called consistent.
In the nextsection we will present two change point estimators and prove their consistency.
Main results
To estimate the change points, we solve the following regularized maximum likelihood problem overall possible partitions P of the time course [ T ] :
[Equation]
where L ( θ , I ) is the negative log-likelihood function for the BTL model defined in (2.6) and γ > 0is an user-specified tuning parameter.
Here a partition P is defined as a set of integer intervals:
[Equation]
1 ) , [ p 1 , p 2 ) , . . .
, [ p K P , T ] } , 1 < p 1 < p 2 < · · · < p K P < T.
[Equation]
With ˜ K
= K ˆ P = | ˆ P| − 1 , the estimated change points { ˜ η k } k ∈ ˜
K are then induced by ˜ η k = ˆ p k ,k ∈ [ ˜ K ] .
The optimization problem (3.1) has an ℓ 0 -penalty, and can be solved by a dynamicprogramming algorithm described in Algorithm 1 with O
( T 2 C ( T )) complexity (Friedrich et al.
,2008; Rinaldo et al., 2021), where C ( T ) is the complexity of solving min θ L ( θ , [1 , T ]) .
In this section, we will demonstrate that the estimator returned by Algorithm 1 is consistent.
Towardsthat goal, we require the following signal-to-noise ratio condition involving the parameters ∆ , κ and
B , n and the sample size T .
Assumption 3.1 (Signal-to-noise ratio) .
Let { ( x ( t ) , y t ) } t ∈ [ T ] be i.i.d.
observations generated frommodel (2.1) and (2.5) with parameters { θ ∗ ( t ) } ⊂ Θ B defined in (2.3).
We assume that for a divergingsequence {B T } T ∈ Z + ,
[Equation]
where we recall that p lb :
= e − 2 B 1+ [Equation] 2 B .
The formulation of signal-to-noise ratio conditions involving all the parameters of the model hasbecome a staple of modern change point analysis literature.
To provide some intuition, the term∆ κ 2 is a proxy for the strength of the signal of change points in the sense that the localization anddetection problems are expected to become easier, as the magnitude of the jumps and the spacingbetween change points increase.
On the other hand, the right hand side of Equation (3.3) collectsterms that impact negatively the difficulty of the problem: the smaller the minimal win probability p lb ,the larger the number of items n to compare and the number of change points K , the more difficult itis to estimate the change points.
Remark
(On the sharpness of the signal-to-noise ratio condition) .
We will now argue that therequirement (3.1) imposed by the signal-to-noise ratio (SNR for brevity) is reasonably sharp byrelating it to the sample complexity of a two-sample testing problem.
To that effect, consider thesimplified setting in which there is only one change point at time ∆ =
T/ 2 .
In this case, it can beshown that the SNR condition (3.1) becomes (see Proposition B.5)
[Equation]
i.e. the dependence on the dynamic range B is through p − 2 lb instead of p − 4 lb .
It stands to reason thatestimating the unknown change point ∆ should be at least as hard as testing the null hypothesis thatthere exists a change point at time ∆ .
Indeed, this testing problem should be easier because ∆ hasbeen revealed and because, in general, testing is easier than estimation.
This can in turn be cast as atwo-sample testing problem of the form
[Equation]
where ϵ > 0 is to be specified, θ (1) and θ (2) are the BTL model parameters for the first and the last∆ observations respectively and, for i ∈ { 1 , 2 } , P ( θ ( i ) ) is the n × n matrix of winning probabilitiescorresponding to the BTL model parameter
θ ( i ) as specified by (2.2).
To see how one arrives at (3.5),we have that, by Proposition B.4,
[Equation]
[Equation] Thus, a change point setting with ∥ θ (1) − θ (2) ∥ {+
+} = κ 2 , translates into the testing problem (3.5)with ϵ 2 = κ 2 p 2 lb / (16 n ) .
By Theorem 7 of Rastogi et al.
(2020), there exists an algorithm that willreturn a consistent test for (3.5) based on two independent samples of size N if N ≥ cn 2 log( n ) 1 nϵ 2 .
When we apply this result to the simplified change point settings described above (by replacing N and ϵ 2 with ∆ and κ 2 p 2 lb / (16 n ) respectively)
we conclude that the sample complexity bound of Theorem 7 of Rastogi et al.
(2020) corresponds, up to constants, to the above SNR condition (3.4)save for the terms log( T ) and B T .
Thus, we conclude that the assumed SNR condition for changepoint localization is essentially equivalent to the sample complexity needed to tackle the simplertwo-sample testing problem, an indication that our assumption is sharp.
Finally, we take notice that, when there are multiple change points, in our analysis it appears necessaryto strengthen the signal-to-noise ratio condition (3.4) to (3.1) by requiring a dependence on p − 4 lb .
We are now ready to present our first consistency result.
Theorem 3.2.
Let { ˜ η k } k ∈
[ ˜ K ] be the estimates of change points from Algorithm 1 with the tuningparameter γ =
C γ p − 2 lb ( K + 1) n log( Tn ) where C γ is a universal constant.
Under Assumption 3.1we have
[Equation]
where C P > 0 is a universal constant that depends on C γ .
Theorem 3.2 gives a high-probability upper bound for the localization error of the output { ˜ η k } k ∈ [ ˜ K ]of Algorithm 1.
By Assumption 3.1, it follows that as T → ∞ , with high probability,
[Equation]
where we use the singal-to-noise ratio assumption ∆ · κ 2 ≥ B T p − 4 lb Kn 2 log( Tn ) in the last inequalityand the fact that B T diverges in the final step.
This implies that the estimators { ˜ η k } k ∈ [ ˜ K ] areconsistent.
Moreover, when K = 0 or there is no change point, it is guaranteed that with highprobability, Algorithm 1 will return an empty set.
We summarize this property as Proposition B.6and include it in Appendix B.2 due to the limit of space.
Inspired by previous works (Wang et al., 2021a; Rinaldo et al., 2021), we can further improve thelocalization error by applying a local refinement procedure as described in Algorithm 2 to { ˜ η k } k ∈ [ ˜ K ] .
This methodology takes as input any preliminary estimator of the change points that estimates thenumber of change points correctly with a localization error that is a (not necessarily vanishing)fraction of the minimal spacing ∆ , and returns a new estimator with a provably smaller localizationerror.
A natural preliminary estimator is the one returned in Algorithm 1.
The next result derives theimproved localization rates delivered by the local refinement step.
The two improvements are theelimination of the term K in the rate and a better dependence on p lb .
INPUT: Data { ( x ( t ) , y t ) } t ∈
[ T ] , tuning parameter γ .
Set S = ∅ , p = − 1 T , b = ( γ, ∞ , . . . , ∞ ) ∈ R T .
Denote b i to be the i -th entry of b .
for r in { 2 , . . .
, T } do for l in { 1 , . . .
, r − 1 } do b ← b l + γ + L (ˆ θ ( I ) , I ) where I =
( l, . . .
, r ]; if b < b r then b r ← b ; p r ← l .
To compute the change point estimates from p ∈ N T , k ← T .
while k > 1 do h ← p k ; S = S ∪ h ; k ← h .
OUTPUT:
The estimated change points S =
{ ˜ η k } k ∈ ˜ K .
Algorithm 2: Local Refinement.
INPUT: Data { ( x ( t ) , y t ) } t ∈
[ T ] , { (cid:101) η k } k ∈
[ (cid:101) K ] , ( (cid:101) η 0 , (cid:101) η (cid:101) K +1 )
← (1 , T ) .
for k = 1 , . . .
, (cid:101) K do ( s k , e k ) ← (2 (cid:101)
η k − 1 / 3 + (cid:101) η k / 3 , (cid:101) η k / 3 + 2 (cid:101)
η k +1 / 3)
; ˆ η k ← arg min η ∈{ s k +1 ,...,e k − 1 } (cid:40) min θ (1) ∈
Θ B η (cid:88) t =
s k + ℓ t ( θ (1) )
+ min θ (
) ∈
Θ B e k (cid:88) t =
η + ℓ t ( θ (2) )
(cid:41) ; (3.9)
OUTPUT: { ˆ η k } k ∈ [ (cid:101) K ] .
Theorem 3.3.
Let { ˆ η k } k ∈
[ ˆ K ] be the output of Algorithm 2 with input { ˜ η k } k ∈ [ ˆ K ] returned by Algorithm 1.
Under Assumption 3.1, for all sufficiently large T we have
[Equation]
) where C P > 0 is a universal constant that depends on C γ .
Remark 3 .
By “sufficiently large T ” in the theorem statement, we mean that T should be largeenough to make max k ∈
[ K ] | ˆ η k − η k | ≤ ∆ / 5 (see Proposition B.3 in Appendix B for details).
Such Texists because of Equation (3.8) and the fact that B T is diverging in T .
We conjecture that the rate (3.10) resulting from the local refinement procedure is, aside possiblyfrom a logarithmic factor, minimax optimal.
Experiments
In this section, we study the numerical performance of our newly proposed method based on acombination of dynamic programming with local refinement, which we will refer to as DPLR; see
Algorithms 1 and 2. We note that the detection of multiple change points in pairwise comparison datahas not been studied before, as Höhle (2010) only focus on single change point detection for pairwisecomparison data, so we are not aware of any existing competing methods in the literature.
Thus,we develop a potential competitor based on the combination of Wild Binary Segmentation (WBS)(Fryzlewicz, 2014), a popular method for univariate change point detection, and the likelihood ratioapproach studied in Höhle (2010).
We will call this potential competitor WBS-GLR (GLR standsfor generalized likelihood ratio).
Due to the limit of space, we include the detail of WBS-GLR in Appendix A.1, and results of additional experiments in Appendix A.2.
All of our simulation results show that our proposed method DPLR outperforms WBS-GLR inthe sense that DPLR gives more accurate change point estimates with similar running time.
Each experiment is run on a virtual machine of Google Colab with Intel(R)
Xeon(R) CPU of 2 cores 2.
GHz and 12GB RAM.
All of our reproducible code is openly accessible 1 .
Simulation Settings.
Suppose we have K change points { η k } k ∈
[ K ] in the sequential pairwisecomparison data, with η 0 = 1 .
We can use θ ∗ ( η k ) to represent the value of true parameters afterthe change point η k .
To begin, we define θ ∗ i ( η 0 ) as follows.
For 1 < i ≤ n , we set θ ∗ i ( η 0 )
=
θ ∗ 1 ( η 0 )
+( i − 1) δ with some constant δ .
In each experiment, we set δ first and then set θ ∗ 1 ( η 0 ) to make1 ⊤ n θ ∗ ( η 0 ) =
For a given n , we set δ = 1 n − 1 ψ − 1 ( p ) =
1 n − 1 log( p 1 − p ) where ψ − 1 is the inversefunction of ψ and p = 0 .
Recall that P ij = ψ
( θ i − θ j ) is the winning probability, so the value of δguarantees that the maximum winning probability is 0.9.
We consider three types of changes:
Type I (reverse): θ ∗ i ( η k )
=
θ ∗ n +1 − i ( η 0 ) .Type
II (block-reverse): θ ∗ i ( η k ) =
θ ∗ [ n 2 ]+1 − i ( η 0 ) for i ≤ [ n 2 ] ; θ ∗ i ( η k )
=
θ ∗ [ n 2 ]+ n +1 − i ( η 0 ) for i > [ n 2 ] .Type III (block exchange): θ ∗ i ( η k ) =
θ ∗ i +[ n 2 ] ( η 0 ) for i ≤ [ n 2 ] ; θ ∗ i ( η k )
= θ ∗ i − [ n 2 ] ( η 0 ) for i > [ n 2 ] .
We consider four simulation settings.
For each setting, we have T =
( K + 1)∆ and the changepoints locate at η i = i
∆
for i ∈
[ K ] .
To describe the true parameter at each change point, we use anordered tuple.
For instance, (I, II, III, I) means that K
= 4 and the true parameters at η 1 , η 2 , η 3 , η 4are determined based on θ ∗ ( η 0 ) and the change type I, II, III, and I, respectively. [Table] < [Table]
= K
[Table]
> [Table] K < K , ˆ K = K , and ˆ K > K respectively. For the constrained MLE in Equation (3.1), we use the function in sklearn for fitting the ℓ 2penalized logistic regression, as it is well-known that the constrained and the penalized estimators forgeneralized linear models are equivalent.
For both DPLR and WBS-GLR, we use λ = 0 .
For M ,the number of random intervals in WBS-GLR, we set it to be 50 as a balance of time and accuracy.
For both methods, we use cross-validation to choose the tuning parameter γ .
Given the sequentialpairwise comparison data in each trial, we use samples with odd time indices as training data and eventime indices as test data.
For each tuning parameter, the method is applied to the training data to getestimates of change points.
Then a BTL model is fitted to the test data for each interval determinedby the estimated change points.
The tuning parameter and the corresponding change point estimatorswith the minimal test error (negative loglikelihood) are selected.
We run 100 trials for each setting.
Results.
To measure the localization errors, we use the Hausdorff distance H ( { ˆ η i } i ∈ [ ˆ K ] , { η i } i ∈
[ K ] )between the estimated change points { ˆ η i } i ∈ [ ˆ K ] and the true change points { η i } i ∈
[ K ] .
The Hausdorffdistance H ( S 1 , S 2 ) between two sets of scalars is defined as
[Equation]
The results are summarized in Table 1, where we use H (ˆ η, η ) to denote the localization error forbrevity.
As we can see, our proposed method DPLR gives more accurate localization with similarrunning time compared to the potential competitor WBS-GLR [Table]
Application: the National Basketball Association games
[Table]
We study the game records of the National Basketball Association (NBA) 2 .
Usually a regular NBAseason begins in October and ends in April of the next year, so in what follows, a season is namedby the two years it spans over.
The original data contains all game records of NBA from season1946-1947 to season 2015-2016.
We focus on a subset of 24 teams founded before 1990 and seasonsfrom season 1980-1981 to season 2015-2016.
All code of analysis is available online with the data 3 .
We start with an exploratory data analysis and the results show strong evidence for multiple changepoints 4 .
Therefore, we apply our method DPLR to the dataset to locate those change points.
Weuse the samples with odd time indices as training data and even time indices as test data, and usecross-validation to choose the tuning parameter γ .
To interpret the estimated change points, we fit the BTL model on each subset splitted at change pointestimates separately.
The result is summarized in Table 2.
Several teams show significant jumps inthe preference scores and rankings around change points.
Apart from this quantitative assessment,the result is also firmly supported by memorable facts in NBA history, and we will name a few here.
In 1980s, Celtics was in the “Larry Bird” era with its main and only competitor “Showtime” Lakers.
Then starting from 1991, Michael Jordan and Bulls created one of the most famous dynasties in NBAhistory.
1998 is the year Michael Jordan retired, after which Lakers and Spurs were dominatingduring 1998-2009 with their famous cores “Shaq and Kobe” and “Twin Towers”.
The two teamstogether won 8 champions during these seasons.
S2010-S2012 is the well-known “Big 3” era of Heat.
Meanwhile, Spurs kept its strong competitiveness under the lead of Timothy Duncan.
From 2013,with the arise of super stars Stephen Curry and Klay Thompson, Warriors started to take the lead.
Conclusions
We have formulated and investigate a novel change point analysis problem for pairwise comparisondata based on a high-dimensional BTL model.
We have developed a novel methodology that yieldsconsistent estimators of the change points, and establish theoretical guarantees with nonasymptoticlocalization error.
To the best of our knowledge, this is the first work in the literature that addressesin both a methodological and theoretically sound way multiple change points in ranking data.
Although we filled a big gap in the literature, there remain many open and interesting problems forfuture work.
First, we only consider pairwise comparison data modeled by the BTL model.
Of course,there are other popular ranking models for general ranking data, e.g., the Plackett-Luce model(Luce,1959; Plackett, 1975), Stochastically Transitive models(Shah et al., 2017), and the Mallows model(Tang, 2019).
It would be interesting to see that for those models how different the method andtheory would be from our settings.
Second, we have focused on retrospective setting of change pointdetection and passive setting of ranking.
On the other hand, online change point detection (Vovk,2021) and active ranking (Heckel et al., 2019; Ren et al., 2021) are widely used in practice.
Thus, itwould be interesting to consider the online or active framework in change point detection for rankingdata.
Third, in the recent change point detection literature, incorporating temporal dependence is ofgrowing interest (Chen et al., 2021;
Wang and Zhao, 2022), so investigating how temporal dependencein the pairwise comparison data can affect our results seems like a worthwhile direction.
Lastly, weassume that the compared pairs are randomly sampled from the full edge set, or the complete graph,making the comparison graph similar to an Erdös-Rényi graph.
Although this setting is common inthe literature (Chen et al., 2019, 2020), it does not explicitly show the impact of the graph topologyof the comparison graph on ranking.
Therefore, it would be very interesting to generalize our resultsto explicitly show the effect of the topology of the sampling graph.
At last, we discuss potential societal impacts of our work.
The BTL model does have applicationswith potentially undesirable societal impacts, e.g., sports-betting (McHale and Morton, 2011), whichcould amplify the negative impacts of gambling.
We recommend using our method for researchpurposes rather than gambling-driven purposes. REFERENCES
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See theend of Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix B
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] See Sections and 5 for details including a link to our anonymized reproducible code repository
i
e.
[Table]
://anonymous.
[Table]
.science/
[Table]
{+/CPD_BT-
+}
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We report standard errors of 100 trials, see Section
(d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[Yes] See the second paragraph of
Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets.. (a) If your
work uses existing assets did you cite the creators?
[Yes] We cited all relevant
works
of
the algorithms we used at places they first occur in our paper
We also cite thecollector of the data we used in Section 5.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]