On A Mallows-type Model For (Ranked) Choices
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In a preference learning setting, every participant chooses an ordered list of k mostpreferred items among a displayed set of candidates.
(The set can be different forevery participant.)
We identify a distance-based ranking model for the population’spreferences and their (ranked) choice behavior.
The ranking model resembles the Mallows model but uses a new distance function called Reverse Major Index (RMJ).
We ﬁnd that despite the need to sum over all permutations, the RMJ-based rankingdistribution aggregates into (ranked) choice probabilities with simple closed-formexpression.
We develop effective methods to estimate the model parameters andshowcase their generalization power using real data, especially when there is alimited variety of display sets.
Introduction
How to aggregate the population’s preferences from their (ranked) choices out of different choicesets?
This question is of interest to many communities, such as economics, business, and computerscience.
A concrete setting is a platform that wishes to learn customer preferences over a universeof n product prototypes.
The platform is able to display different subsets of versions to differentcustomers, who then provide feedback in the form of a top- k ranked list of their most preferred itemswithin the subset they see (hereafter referred to as “ranked choices").
The population’s (ranked) choice behavior could be summarized as a (ranked) choice model Pr( ⇡ k | S ) ,which speciﬁes the probability that a randomly drawn participant choosing a top- k list ⇡ k from thedisplay set S .
An economically rationalizable and yet very general way to model (ranked) choicesis to use probabilistic ranking models .
That is, given a probability distribution  over preferencerankings, Pr( ⇡ k | S ) equals the probability that a randomly drawn participant would place ⇡ k as thetop- k positions among the items in S .
A popular family of ranking models is distance-based , which is the conceptual analog of Gaussiandistribution for scalars; see [1].
A distance-based ranking model is speciﬁed by a modal (central)ranking ⇡ ⇤ and a dispersion parameter q 2 (0 , 1) .
Given a ranking ⇡ , its probability of being sampledis proportional to q d ( ⇡ ⇤ , ⇡ ) .
Here d ( ⇡ 1 , ⇡ 2 ) is a distance function that describes the discrepancy ofranking ⇡ 1 from ⇡ 2 , and different distance functions lead to different models.
The most representativedistance-based ranking model is the Mallows model ([2]), which uses the Kendall-Tau distance as itsdistance function.
It has been studied extensively in the literature regarding topics such as sampling,estimation from sampled (partial) rankings, and learning in a Mallows mixture setting; see [3] andreferences therein.
Due to its popularity and the fact that every distance-based ranking model onlydiffers from the Mallows by distance function, we will also refer to a distance-based ranking modelas a Mallows-type model interchangeably in the sequel.
Mallows-type models could be used as “kernels" to “smooth out" the distribution over rankings.
As such, they could help mitigate the overﬁtting issues of generic (i.e., nonparametric) rankingmodels, which are typically overparameterized.
Yet, Mallows models can be highly expressive in amixture setting: A mixture of Mallows-type models can approximate any probability distributionover rankings by an arbitrary precision (as q tends to zero and the number of clusters tends to inﬁnity).
Therefore, the Mallows-type model family is a helpful tool to balance capturing (complex) preferenceheterogeneity across individuals and regularizing the ranking distribution for better out-of-samplepredictions.
A representative work is by [4], who use the Mallows model to aggregate customerpreferences from their choices (i.e., k = 1 ).
Despite the theoretical elegance, the main challenge in applying Mallows-type ranking models to(ranked) choice modeling is analytical and computational tractability.
More speciﬁcally, Pr( ⇡ k | S )is calculated from summation over all rankings subject to nontrivial conditions.
Therefore, evenif a Mallows-type ranking model has a simple structure for  , the resulting choice probabilities{ Pr( ⇡ k | S ) } can be difﬁcult to obtain even when k = 1 .
The state-of-art results are obtained by [4],who develop polynomial-time numerical algorithms to compute { Pr( ⇡ k | S ) } for k = 1 under the Mallows model.
The estimation problem is even more difﬁcult, which involves ﬁnding the centralranking ⇡ ⇤ and dispersion parameters q that best explain the (ranked) choice data.
To the best of ourknowledge, no effective methods to estimate any Mallows-type ranking model from (ranked) choicedata are known.
(Perhaps the best method to date is again by [4], who use a “Mallows smoothing"heuristic to conduct the estimation when k = 1 , which we will discuss later.)
Summary of results and contributions.
This paper identiﬁes and studies a new distance-based(i.e.
, Mallows-type) ranking model.
It is the same as the Mallows model except that it builds on anew distance function (i.e., smoothing kernel), which we call reverse major index (RMJ) .
Unlike the
Mallows model’s Kendall-Tau distance (which weighs all pairwise disagreements equally), RMJ putsmore weight on top-position deviations.
The RMJ-based ranking model is a small conceptual deviation from the Mallows model (and, inparticular, enjoys the desired properties mentioned above, such as rationalizability, smoothing, andexpressive power); see Section 2.
However, this twist brings a signiﬁcant advantage in both analyticaland computational simplicity.
Speciﬁcally, we solve a list of problems under the RMJ-based rankingdistribution.
That includes:
• Characterizing (ranked) choice probabilities : Given k  1 , calculating { Pr( ⇡ k | S ) } ;•
Sampling :
Given k  1 , efﬁciently sampling a top- k list ⇡ k ; • Parameter learning : Estimating the central ranking ⇡ ⇤ and dispersion parameter q from the (ranked) choice data through a maximum likelihood estimator (MLE) formulation ;
• Learning in a mixture setting :
Assuming that there are multiple clusters of participant preferences, learning the central ranking and dispersion parameter for each cluster fromchoice data.
The solutions to the problems above can be implemented relatively easily.
First, we are able to obtain{ Pr( ⇡ k | S ) } in simple and closed-form expressions for all k  1 ; see Theorems 1 and 4.
This is incontrast to, say, its counterpart for the Mallows model ([4]), and we view it as our main theoreticalachievement.
Second, the sampling can be done in O ( nk ) time directly; see Lemma 2.
Third, theestimation problem can be reduced to a well-studied ranking-aggregation-type formulation; see (5)and Theorem 5.
Many off-the-shelf tools are available.
For example, it admits a PTAS and can bepractically solved via a linear integer programming formulation.
The estimation is guaranteed torecover the model parameters asymptotically under mild conditions on the coverage of display sets;see Theorem 2.
This stands in contrast to Mallows Smoothing by [4], which cannot recover the
Mallows central ranking even under sufﬁcient coverage; see Theorem 3.
Finally, the learning problemin a mixture setting can be solved using the standard Expectation-Maximization (EM) algorithm.
We demonstrate the practical effectiveness of our methodology on two data sets on customer preference over different types of sushi.
When k = 1 , we compare it with two representative ranking-basedchoice models: one based on the Mallows and the other on Plackett-Luce (which leads to the Multinomial Logit choice model).
Our tools display superb generalization power, especially when thereis a limited variety of display sets in the choice data.
When k > 1 , we demonstrate the robustnessof the methodology.
(It is difﬁcult to ﬁnd direct comparisons for prediction power.)
Speciﬁcally,as long as the underlying population keeps the same, different top-
k lists aggregate into the same central rankings.
We also show that our methodology can handle a relatively large number of items( n = 100 ) by achieving high precision solutions ( < 2% optimality gap) in a reasonable time (< 5minute solving time).
Related literature.
Our paper is most closely related to the (extensive) literature on Mallows-typeranking models and their applications to preference and choice modeling (e.g., [1, 5, 3, 6, 7, 8, 9, 4, 10]to name a few.)
Besides the new distance function and tractability results explained above, the mostnotable difference of this paper is the feedback structure .
In our paper, every participant choosesfrom an arbitrary display set (as opposed to pairwise or the full display set only), and their feedbackis in the form of a top- k ranked list (as opposed to single choices or full rankings).
To the best ofour knowledge, the combination of both generalities makes the setting the ﬁrst of its kind.
It is alsoworth mentioning that the analytical and computational simplicity of our model makes it a convenientbuilding block for further the subsequent optimization problems (e.g., new product introduction).
Itis an advantage that many common methods (e.g., Monte-Carlo algorithms) do not have.
Also related is the literature on learning to rank (e.g., [12, 13]).
We wish to stress that we do notmerely ﬁt a ranking from the data but also the uncertainty quantiﬁcation of the estimate (and thewhole choice model { Pr( ⇡ k | S ) } ).
Finally, our paper is also closely related to that of [11].
They studya choice model, called the “ordinal attraction model" (OAM), that emerges from an active learningproblem of consumer preferences.
We rationalize the (surprisingly simple) OAM by showing that itis equivalent to the aggregated choice probabilities { Pr( ⇡ k | S ) } from the RMJ-based ranking modelwhen k = 1 .
We view it as a nontrivial observation in its own right.
A Ranking Model Based on Reverse Major Index
Preliminary.
We consider a universe of n candidates (hereafter referred to as “items"), representedby [ n ] = { 1 , 2 , . . . , n } .
We assume that every participant has a strict preference over them items,represented by a ranking (permutation).
We use a bijection ⇡ : [ n ] ! [ n ] to represent a rankingwhere ⇡ ( i ) is the i th most preferred item.
We sometimes ﬁnd the notation  := ⇡  1 helpful, where ( x ) is the position of item x in the ranking  .
We also use x  ⇡ y if item x is preferred to item yunder ⇡ , i.e., ⇡  1 ( x ) <
⇡  1 ( y ) .
For example, ⇡ = (3 , 1 , 2) means 3  ⇡ 1  ⇡
and it correspondsto a “  " notation of  = (2 , 3 , 1) .
Finally, we use e to represent the identity ranking, ⌃ for the set ofall rankings over n items, ⇡ k ( k  1) for a top- k list, and ⌃ k for the set of all top- k rankings.
Mallows-type ranking models.
Given the distance function d ( · , · ) , the probability mass function ofthe ranking for a Mallows-type model can be written as
[Equation]
where ⇡ ⇤ is the central ranking, q is the dispersion parameter, and
P ˜ ⇡ q d ( ⇡ ⇤ , ˜ ⇡ ) is the normalizationconstant.
Intuitively, the Mallows-type model deﬁnes a population of participants whose preferencesare "similar" as they are centered around a common ranking, where the probability for deviationsthereof decreases exponentially.
Different distance functions correspond to different models.
A common requirement for a validdistance function is that it is invariant to “relabeling."
Formally, that means d ( · , · ) is left-invariantunder the ranking composition, i.e., d ( ⇡ 1 , ⇡ 2 ) =
d ( ⇡⇡ 1 , ⇡⇡ 2 ) for every ⇡ , ⇡ 1 , ⇡ 2 .
This invarianceproperty enables to make the following conventions without loss of generality: First, we assume thatthe items have been properly relabeled so that the central ranking ⇡ ⇤ =
e (unless otherwise speciﬁed).
Second, we may use d ( ⇡ ) as shorthand notation for d ( e, ⇡ ) , which fully represents a distance functionwith the knowledge that d ( ⇡ 1 , ⇡ 2 ) =
d ( e, ⇡  1 1 ⇡ 2 ) .
The Mallows (Kendall’s Tau distance based) ranking model.
Commonly studied distance functionsinclude the Spearman’s rank correlation, Spearman’s Footrule, and the Kendall’s Tau distance ([1]).
Among those three, Kendall’s Tau corresponds to the Mallows model and is deﬁned as
[Equation]
It measures a ranking’s total number of pairwise disagreements (with the identity ranking e ).
For example, consider the ranking ⇡ = (4 , 2 , 1 , 3) .
There are four pairwise disagreements:{ (4 , 2) , (4 , 1) , (4 , 3) , (2 , 1) } .
Therefore, d K ((4 , 2 , 1 , 3))
= 4 .
As an appealing property, the Kendall’s tau distance leads to a tractable expression for  .
Othercommon distances do not have this since the normalization constant involves summing over n !
rankings.
The constant under the Mallows model can be expressed as (see [14]):
[Equation]
The RMJ based ranking model.
In this paper, we identify a new distance function, which we callreverse major index (RMJ) .
It is deﬁned as
[Equation]
Compared to Kendall’s Tau in (1), RMJ focuses on adjacent disagreements and puts more weighton top-position disagreements.
For example, consider the ranking ⇡ = (4 , 2 , 1 , 3) again.
The onlyadjacent disagreements are { (4 , 2) , (2 , 1) } .
Therefore, after including the weights in (3), we haved R ((4 , 2 , 1 , 3))
= 3+2 = 5 .
The name of RMJ is inspired by the major index from the combinatoricsliterature ([15]), which is deﬁned as d M ( ⇡ ) =
[-= (cid:80) n − 1 i =
 I { π ( i ) > π ( i + 1) } · i.-][Equation]
Discussion: Kendall’s Tau vs. RMJ.
Both metrics are conceptually similar: they measure a ranking’sdeviation from the identity.
They coincide in many intuitive cases.
For example, in the most extremeones we have d K ( e ) =
d R ( e ) = 0 and d K (( n, . . .
=
d R (( n, . . . , 1))
= n ( n  1) / 2 .
Therefore,both can be used as “kernels" to smooth out the distribution over rankings.
However, they emphasizethe deviation in a subtly different way.
Therefore, we can also ﬁnd subtle rankings where their valuesare different.
For example, d K ((4 , 2 , 1 , 3))
= 4 but d R ((4 , 2 , 1 , 3))
= 5 .
It is difﬁcult to tell which kernel is “better" from an axiomatic approach as both distance functionssatisfy the basic axioms for ranking distances ([1]).
Nevertheless, we will show later that RMJ-basedranking model leads to simple and estimatable (ranked) choice probabilities without compromisingthe desirable properties of Mallows, such as rationalizability and ﬂexibility for a mixture setting.
Moreover, regarding its ability to describe preference distributions that may occur in practice, we willdemonstrate its descriptive and predictive power in a case study with real preference data.
Therefore,we believe that RMJ produces a promising tool that is more tailed to the application of learningpopulation preferences from (ranked) choice data.
Analysis of the (Ranked) Choice Model { Pr( ⇡ k | S ) }
In this section, we will characterize the (ranked) Choice Model { Pr( ⇡ k | S ) } aggregated from the
RMJ-based ranking model, formally deﬁned as
[Equation]
I
⇡ k ˜ ⇡ , S ) means - k list ⇡ k
˜ ⇡
S
We will also discuss how to estimate the ranked choice model parameters from data.
In the sequel,we will write d ( · ) =
d R ( · ) since RMJ is the distance function of interest.
3.1 The k = 1 case
When k = 1 , a top- k list model reduces to a choice model, which connects to a richer literature (e.g.,[4]).
Therefore, it is worth a separate discussion, which also helps build intuition for the k > 1 case.
Choice probabilities.
Our main result for k = 1 is summarized below, which characterizes thechoice probabilities under the RMJ-based ranking model.
Theorem 1 Let a display set S
= { x 1 , x 2 , . . .
, x M } be such that x 1 < x 2 < · · · < x M .
Then
[Equation]
In other words, for every display set, all the items within the display set are re-ranked so that theirchoice probabilities decay exponentially fast according to their relative ranking within the displayset.
Noticeably, the choice probabilities in (4) are (much) simpler than that induced by the Mallowsmodel, which even needs a Fast Fourier Transform to evaluate in O ( n 2 log n ) time ([4]).
Also,(4) rediscovers the “Ordinal Attraction Model" (OAM) in [11], which could also be viewed as a“multiwise" generalization of pairwise noisy comparison models (e.g., [16, 17]).
OAM gets its namebecause the “attractiveness" (i.e., choice probability) of an item within a display set S only dependson its relative position in S and therefore is only a function of the “ordinal" information.
While OAMemerges from an active learning problem of consumer preferences, we “rationalize" it by showing thatit can be aggregated from the RMJ-based ranking model, which we believe is a nontrivial observationin its own right.
In the sequel, we will follow its convention and refer to the choice model deﬁned in
Theorem 1 as OAM.
Proof outline and key intermediate results for Theorem 1.
Let us introduce a few notations fortop-
k (sub-) rankings.
Given a top- k ranking ⇡ k 2 ⌃ k , we could deﬁne RMJ for ⇡ k by truncatingthe index at position k , i.e., d ( ⇡ k ) =
P k  1 i =
I { ⇡ k ( i ) > ⇡ k
( i + 1) } · ( n  i ) .
In addition, let 191 R ( ⇡ k ) :
= { ⇡ k ( i ) : i = 1 , . .
, k } be the set of top- k items and R c ( ⇡ k ) :
= [ n ] \ R ( ⇡ k ) its complement.
Finally, let L ( ⇡ k ) :
= | { x 2 R c ( ⇡ k ) : x < ⇡ k ( k ) } | be the number of items that are (i) not includedin ⇡ k and (ii) having smaller indices than (i.e., preferred under ⇡ ⇤ = e ) item ⇡ k ( k ) .
For example,suppose n = 7 and ⇡ 4 = (7 , 4 , 6 , 2) .
Then d ( ⇡ 4 ) = 6 + 4 = 10 , R ( ⇡ 4 ) = { 2 , 4 , 6 , 7 } , R c ( ⇡ 4 ) ={ 1 , 3 , 5 } , ⇡ 4 (4) = 2 , and L ( ⇡ 4 ) = |{ 1 }| = 1 .
Given two subrankings ⇡ k 2 ⌃ k and ⇡ k 0 2 ⌃ k 0 with k  k 0 , we write ⇡ k ✓ ⇡ k 0 if they are 197 compatible, i.e., ⇡ k ( i ) = {+⇡ k 
+}
( i ) for all i = 1 , . .
k .
Our ﬁrst result concerns extending the domainof  to top-
k rankings, formally deﬁned as  ( ⇡ k ) :
=
P ˜ ⇡  (˜ ⇡ ) I { ⇡ k ✓ ˜ ⇡ } .
Lemma 1 (Probability distribution of top- k rankings)  ( ⇡ k ) = q [Equation]
⇡
[Equation] + [Equation] ⇡ k ) · ( n
[Equation] [Equation] [Equation] The signiﬁcance of the result above is that we could think of a ranking ⇡ as a stochastic process on a“tree" with depth n and n ! leaves.
While {  ( ⇡ ) : ⇡ 2 ⌃ } describe the probability distribution over theleaves, for every k , {  ( ⇡ k ) : ⇡ k 2 ⌃ k } describe the probability distribution over the nodes at level k .
As a consequence of Lemma 1, we can write out how to randomly sample a ranking ⇡ under the
RMJ-based ranking model from the top to the bottom position.
Given a top- k ranking ⇡ k and an itemz , we write ⇡ k  z as the concatenation of ⇡ k and item z .
For example, suppose [-π 
-]⇡ 3 = (5 , 2 , 4) , then⇡ 3  3 = (5 , 2 , 4 , 3) .
Lemma 2 (Random ranking generation)
Given a top-k ranking ⇡ k such that ⇡ k ( k )
=
z , the conditional probability for the ( k + 1) -positioned item is
[Equation]
where h ( y | z )
= ( y  z if y > z n  k
P x 2 R c ( ⇡ k ) I { y < x < z } if y < z .
Finally, Lemmas 1 and 2 leads to the following result based on an induction argument.
Lemma 3 Let a display set S = { x 1 , x 2 , . . .
, x M } be such that x 1 < x 2 < · · · < x M and a top- k 210 ranking ⇡ k be such that ⇡ k = ⇡ k  1  z and R ( ⇡ k  1 ) \ S = ; .
Then conditional on a participant’s 211 top- k preference list is ⇡ k , the probability that (s)he will choose item x i out of display set S is
[Equation]
[Table]
[Equation]
if z > x i and z
[Equation]
[Equation]
[Table]
[Equation]
if z < x i and z
[-( z | S ) −
 + q + ... + q M −if-][Equation]
, if z = x i , if z [Equation]
\ { x i } . where p ( z | S ) := [Equation]
Note that Lemma 3 could be viewed as a generalization of Theorem 1 by setting ⇡ k = ; (whichcorresponds to z = 0 ).
Parameter learning from choice data.
In the parameter learning problem, we are endowed withchoice data H T
= ( S 1 , x 1 , . . .
, S T , x T ) , where S t is the display set shown to the t th participant andx t is his/her choice.
Following [11], the maximum likelihood estimator (MLE) for the central ranking⇡ ⇤ can be obtained from following choice aggregation problem
[Equation]
It has a further linear integer programming formulation.
Let w ij :=
[-(cid:80) Tt =
-][Equation]
S t and x t = i } 220 be the number of times that item i and item j are displayed together and item i is chosen.
Intuitively,a positive w ij  w ji is an indication that item i should be preferred to item j .
Invoking Proposition 3in [11], the integer programming could be written as following:
[Equation]
In the formulation above, the solution ˆ x is such that ˆ x ij = 1 if i  ˆ ⇡ j under the MLE ˆ ⇡ .
Computationally, this integer programming is an instance of the well-studied feedback arc set problemon tournaments .
Therefore, it admits a polynomial-time approximation scheme (PTAS).
From amore practical side, the central ranking  ⇤ can be effectively obtained using off-the-shelf integerprogramming solvers and with many speeding-up heuristics; see [11] for more discussion.
The
MLE for dispersion parameter q can be subsequently obtained from an one-dimensional (convex)optimization problem ˆ ↵ 2 argmin ↵ [Equation]
↵ [Equation]
P [Equation] e  j ↵ } so thatˆ ↵ =  ln
ˆ q .
It is also worth noting that the MLE framework can be easily extended to learning in amixture setting using the standard EM algorithm, which we refer to the supplementary materials formore details.
Let us conclude this section by providing some theoretical understanding of whether we could recoverthe ground truth values of ⇡ ⇤ and q at least asymptotically.
Intuitively, this should depend on the“coverage" of display sets: if only a pair { x, y } is repeatedly, there is no hope of recovering the fullranking ⇡ ⇤ .
It turns out that the parameters can be recovered as long as every pair is “covered" bysome display set.
Theorem 2 The MLE (ˆ ⇡ , ˆ q ) is an consistent estimator if and only if for every pair of items { i, j } ,there exists a display set S that includes { i, j } and is displayed inﬁnite many times.
[-=
-]
We would like to mention that the coverage condition is fairly mild.
For example, as long as the fulldisplay set is displayed sufﬁciently many times, the RMJ-implied central ranking can eventually berecovered.
This stands in contrast to its counterpart in the Mallows model.
Since the Mallows model does not produce simple choice probabilities, the MLE from choice data is rather difﬁcult to obtain.
Perhaps the best method to date is by [4], who use a “Mallows smoothing" heuristic.
The followingresult reveals that the heuristic can be unstable or fail to recover the underlying Mallows parameterseven under sufﬁcient coverage of display sets.
Theorem 3 Even if all display sets with sizes larger than two are displayed inﬁnite times, theestimator from the Mallows Smoothing heuristic is not consistent.
The intuition behind the result above is that Mallows Smoothing needs a “choice to ranking" step.
That is, it needs to ﬁrst ﬁnd a distribution over rankings, denoted by ˆ  2  ( ⌃ ) , that aggregates intochoice probabilities to match the empirical choice probabilities { ˆPr( x | S ) } .
Formally, it correspondsto solving the system of linear equations
[Equation]
for all display set S in the data and x 2 S .
Since there are vastly more variables than equations, thesolutions in general form a polytope rather than a singleton.
More importantly, we ﬁnd that therecan be solutions ˆ  1 and ˆ  2 that aggregate to different Mallows models in the “ranking aggregation"step of the heuristic.
Therefore, in general, this heuristic can produce non-unique results and thus beinconsistent.
3.2 The general ( k  1 ) case
In the general ( k  1 ) case, the ranked choice probabilities and the corresponding estimation remainparsimonious.
Roughly speaking, the ranked choice model behaves like a generalized OAM where
Pr( ⇡ k | S ) depends on the relative rankings of the items in ⇡ k among the set S .
The tractability comesfrom the sole dependence on the relative ranking.
That is, to obtain Pr( ⇡ k | S ) , one could ﬁrst treat Sas the “full display set" for a subuniverse of items, then re-rank all the items within the display set S ,and ﬁnally just apply Lemma 1.
Choice probabilities.
Given a display set S , let d S ( · ) , L S ( · ) be the originally deﬁned d ( · ) and L ( · )functions, but treating the display set S as the universe.
Formally, d S ( ⇡ k ) :=
P k  1 i {+=
+} I { ⇡ k ( i ) > 268 ⇡ k ( i + 1) } · ( | S |  i ) and L S ( ⇡ k ) :
= | { x 2 R c ( ⇡ k ) \ S : x < ⇡ k ( k ) } | .
For example, suppose 269 again that n = 7 and ⇡ 4 = (7 , 4 , 6 , 2) , and let S = { 2 , 3 , 4 , 5 , 6 , 7 } .
Recall that d ( ⇡ 4 ) = 6 + 4 =10 and L ( ⇡ 4 )
= |{ 1 }| = 1 .
In comparison, d S ( ⇡ 4 )
= 5 + 3 = 8 and L S ( ⇡ 4 ) = | ; | = 0 .
Theorem 4 Given a display set S and a top- k ranking ⇡ k such that R ( ⇡ k ) ✓ S , we have
[Equation]
Note that Theorem 4 generalizes both Theorem 1 and Lemma 1 (but in different ways).
It reduces to
Theorem 1 by setting k = 1 : Given ⇡ 1 = ( z ) where z 2 S , we have d S ( ⇡ 1 ) = 0 , L S ( ⇡ 1 )
= |{ x {+
+}
S \ { z } : x < z }| the relative ranking of z in display set S , and ( | S |  k,q ) ( | S | ,q ) = 1 + q + · · · + q | S |  1the appropriate normalizing constant where k = 1 .
In addition, Theorem 4 reduces to Lemma 1 bysetting S =
[ n ] , as can be easily veriﬁed.
Parameter learning from ranked choice data.
In the parameter learning problem, we are endowedwith choice data H T =
( S 1 , ⇡ 1 k , . . .
, S T , ⇡ Tk ) , where S t is the display set shown to the t th participantand ⇡ tk = ( x t 1 , . . .
, x tk ) are his/her top- k choices ranked from the most preferred to the k th preferred.
It turns out that (somewhat surprisingly) in the k  1 case, the central ranking ⇡ ⇤ can be estimatedfrom the same integer programming (5) but just with a generalized re-deﬁnition of the weightparameters { w ij } .
Theorem 5 The MLE for the central ranking ⇡ ⇤ , ˆ ⇡ , can be obtained from the integer program (5)with ˆ x ij = 1 iff i  ˆ ⇡ j and a generalized deﬁnition of { w ij } below
[Equation]
Intuitively, a positive w ij  w ji is still an indication that item i should be preferred to item j , butnow taking into the consideration that every participant’s response is actually a ranked list ratherthan a single choice.
Practically speaking, the { w ij } can be maintained in a relatively simple manner.
Suppose a display set S and a ranked choice ⇡ k are given.
Then w ⇡ k ( ` ) , ⇡ k ( ` +1) will be added by| S |  ` for all ` = 1 , . . .
, k  1 .
(This captures the ranking information for items included in ⇡ k .)
In addition, w ⇡ k ( k ) ,j will be added by 1 for all ` 2 { R c ( ⇡ k ) \ S } .
(This captures the rankinginformation between ⇡ k ( k ) and all items that are not included in ⇡ k .)
For example, suppose n = 6and k
= 3 .
The ranked choice data is such that S 1 = { 1 , 2 , 3 , 4 , 5 } and ⇡ 1 k = (3 , 1 , 2) .
Then we havew 3 , 1 = 4 , w 1 , 2 = 3 , w 2 , 4 = w 2 , 5 = 1 and all the other w ij = 0 .
Discussion: model limitation.
A few more words on the ranked choice model { Pr ( ⇡ k | S ) } .
The 295 speciﬁc form gives it great simplicity, but as any other parametric model, it may be vulnerable tomodel missspeciﬁcation issues in practice.
In particular, since the model is calculated from a unimodalranking distribution, this model (in its “vanilla" form) is best suited when there is an approximately“consensus" ranking in the population.
In our numerical studies we will demonstrate how we caneffectively mitigate those issues by considering learning the model parameters in a mixture setting tobetter capture the preference heterogeneity in the population.
Numerical Experiments
We investigate the performance of our methodology on two anonymous survey data sets regardingsushi preferences ([18]):
The ﬁrst one consists of 5,000 full preference rankings over 10 kindsof sushi, while the second one consists of 5,000 top- 10 rankings over 100 kinds.
Throughout thenumerical experiments, we use a workstation with dual Intel Xeon Gold 6244 CPU (3.6 GHz and 32cores in total), no GPUs, and 754 GB of memory.
Experiment 1: Top- 1 choice.
When k = 1 , a top- k list corresponds to a choice model, whichconnects to a richer literature (e.g., [19, 4, 11]).
We compare our prediction power with tworepresentative ranking models: Mallows and Plackett-Luce.
(The previous is distance-based.
Thelatter is not but leads to the famous MNL choice model.)
We use the Mallows Smoothing (MS)heuristic to estimate a Mallows model and MLE to estimate a Plackett-Luce.
For all three rankingsmodels (Mallows, PL, and ours), we also use an EM algorithm to perform parameter learning in amixture setting.
Since the estimation and generalization performances can depend on the display sets, let us deﬁne aninstance of the comparison by {S train , S test , C} .
Here S train (resp.
S test ) is collection of display 316 sets in the training (resp.
test) data, and C is the number of clusters.
In every instance, we split thedata into 4000 participants in training and 1000 in testing.
We generate empirical training (resp.testing) choice data by ﬁrst enumerating all display sets in S train (resp.
S test ) and then record everyparticipant’s favorite type of sushi for every display set in the traing (resp. testing) data.
Finally, weuse the log-likelihood as performance metric.
We will use term explanation power (resp.
predictionpower ) to refer the performance metric evaluated on the training (resp. testing) data.
We use two conﬁgurations for S train and S test , respectively.
For S train , we take it to be ei- 323 ther the full set or a collection of three randomly generated sets, which are realized to be{{ 1 , 3 , 4 , 7 , 8 , 10 } , { 2 , 4 , 5 , 6 , 8 , 9 } , [10] } .
For S test , we take it to be either the collection of allpairwise sets or the collection of all display sets with sizes at least two.
Note that S train has low variability and many elements in S test do not appear in S train , making the experiment emphasizing more 327 on the generalization power .
We take C 2 { 1 , 2 , 4 , 6 , 15 } , therefore running a total of 2 ⇥ 2 ⇥ 5 = 20 328 instances.
We summarize our results in Figure 1.
We ﬁnd that our method has supreme performance comparedto MS and MNL across the settings.
Intuitively, MS and MNL underperform in different ways.
MS is vulnerable to the identiﬁability issue in the “choice-to-rank" step, for which we providetheoretical understanding in Theorem 3.
Our numerical experiments conﬁrm the theoretical insight:
We randomize over the extreme points of the polytope of solutions to (6), each of which leads to avalid output of the MS heuristic.
Every box plot in Figure 1 contains a summary of the performancesof those outputs.
It is clear that the performances (both in training and testing) span wide ranges.
Inthe meantime, MNL is vulnerable to both the model risk from its speciﬁc parametric form (reﬂectedin the limited increase of explanation power when considering more clusters) and the sample risk of overﬁtting (reﬂected in the difference in performance in training vs. testing, especially when trainedon the full display set only).
[Figure]
In
Experiment 2: Top- k choice.
In this experiment, we conduct estimation on top- 1 , top- 2 and top- 3ranked choices constructed from the ﬁrst (i.e., 10-sushi) data set.
The collection of display sets istaken to be all display sets with sizes at least k .
We ﬁnd that the estimation results are relativelyrobust: all three instances of experiments produce the same estimated RMJ-implied central rankingˆ ⇡
= (8 , 5 , 6 , 3 , 2 , 1 , 4 , 9 , 7 , 10) .
Experiment 3: 100 sushi types.
We apply our method on the large second 100 sushi data set.
In thisexperiment, we offer the full set (100 sushi) to 5000 individuals and record their top- 10 choices.
Sincethe problem size is larger, we use the LP-Rand-Pivot heuristic by [11] based on LP relaxation.
Webootstrap 10 times (each time drawing 10000 samples) and record the running times and optimalitygaps in the Table 1.
We ﬁnd that our method can reasonably handle the 100-sushi-type setup: We canobtain < 2% optimality gap within 5 minutes (excl.
model building time).
[Table]
Conclusions
We identify a novel distance-based (Mallows-type) ranking model.
It aggregates into simple probability distributions for top- k subrankings among an arbitrary display set S .
In addition, it facilitateseffective parameter learning through the MLE formulation.
This is the ﬁrst distance-based rankingmodel with such properties (even for k = 1 ) to the best of our knowledge.
This ranking model can be used to model population preferences and provide a rationalizable way tomodel their ranked choices from given display sets.
We demonstrate its practical value using realpreference data.
For example, under a mixture setting with only a few clusters, it shows promisingprediction power, especially when there is a limited variety in the display sets.
For future steps, we believe our work can serve as the “infrastructure" for a range of business-relateddecision problems, such as new product introduction, crowdsourcing, and marketing research, amongothers.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes] We have double checked every claim and they arebacked by the theorems/lemmas formally stated in the paper.(b)
Did you describe the limitations of your work?
[Yes] See end of Section 3 named“Discussion: model limitation.
"(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Themain negative societal impact that we can think of is that the RMJ-based rankingmodel (in its vallina form) tends to aggregate into a social “consensus ranking, whichunder estimates preference heterogeneity.
This should be mitigated by acknowledgingmultiple preference clusters using a mixture setting.
For details, please see the end ofSection 3 named “Discussion: model limitation.
"(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes] We have carefully read the ethics guidelines and ensured that we followthe guidelines.2.
If you are including theoretical results.
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 2.
(b) Did you include complete proofs of all theoretical results?
[Yes] Proofs of our theoreti-
cal results can be found in the supplemental material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] They are
included in the supplemental material.
However, we wish to clarify that our code is not necessarily “executable" as is since it has dependencies (i.e., Gurobi, which is a
paid-for mathematical programming solver) going beyond the algorithm itself.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Yes to all applicable one, i.e., the performances for MS;
see Section 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We included the computation
time in Experiment 3, which was speciﬁcally designed to demonstrate the speed of
the methods on the 100-sushi data set.
For other experiments (especially Experiment
1), a large amount of compute is devoted to ﬁnding the extreme points of the solution
polytope to (6).
This is more about understanding the statistical property of MS and
hence the speed is a less relevant issue.
For computational resources, please see the
beginning of Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We use a publicly
available sushi dataset and cite the creators [18].
(b) Did you mention the license of the assets?
[Yes] The data we use is public.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
We didn’t include any new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
This is a public available dataset.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Yes] Please see Section 4.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]