On A Mallows-type Model For (Ranked) Choices
Anonymous Author(s)
AffiliationAddress email
Abstract
We consider a preference learning setting where every participant chooses anordered list of k most preferred items among a displayed set of candidates.
(Theset can be different for every participant.)
We identify a distance-based rankingmodel for the population’s preferences and their (ranked) choice behavior.
Theranking model resembles the Mallows model but uses a new distance functioncalled Reverse Major Index (RMJ).
We find that despite the need to sum over allpermutations, the RMJ-based ranking distribution aggregates into (ranked) choiceprobabilities with simple closed-form expression.
We develop effective methods toestimate the model parameters and showcase their generalization power using realdata, especially when there is a limited variety of display sets
Introduction
How to aggregate the population’s preferences from their (ranked) choices out of different choicesets?
This question is of interest to many communities, such as economics, business, and computerscience.
A concrete setting is a platform that wishes to learn customer preferences over a universeof n product prototypes.
The platform is able to display different subsets of versions to differentcustomers, who then provide feedback in the form of a top- k ranked list of their most preferred itemswithin the subset they see (hereafter referred to as “ranked choices").
The population’s (ranked) choice behavior could be summarized as a (ranked) choice model Pr( π k | S ) ,which specifies the probability that a randomly drawn participant choosing a top- k list π k from thedisplay set S .
An economically rationalizable and yet very general way to model (ranked) choicesis to use probabilistic ranking models .
That is, given a probability distribution λ over preferencerankings, Pr( π k | S ) equals the probability that a randomly drawn participant would place π k as thetop- k positions among the items in S .
A popular family of ranking models is distance-based , which is the conceptual analog of Gaussiandistribution for scalars; see [1].
A distance-based ranking model is specified by a modal (central)ranking π ∗ and a dispersion parameter q ∈ (0 , 1) .
Given a ranking π , its probability of being sampledis proportional to q d ( π ∗ ,π ) .
Here d ( π 1 , π 2 ) is a distance function that describes the discrepancy ofranking π 1 from π 2 , and different distance functions lead to different models.
The most populardistance-based ranking model is the Mallows model ([2]), which uses the Kendall-Tau distance as itsdistance function.
It has been studied extensively in the literature regarding topics such as sampling,estimation from sampled (partial) rankings, and learning in a Mallows mixture setting; see [3] andreferences therein.
Due to its popularity and the fact that every distance-based ranking model onlydiffers from the Mallows by distance function, we will also refer to a distance-based ranking modelas a Mallows-type model interchangeably in the sequel.
Mallows-type models could be used as “kernels" to “smooth out" the distribution over rankings.
As such, they could help mitigate the overfitting issues of generic (i.e., nonparametric) rankingmodels, which are typically overparameterized.
Yet, Mallows models can be highly expressive in amixture setting: A mixture of Mallows-type models can approximate any probability distributionover rankings by an arbitrary precision (as q tends to zero and the number of clusters tends to infinity).
Therefore, the Mallows-type model family is a helpful tool to balance capturing (complex) preferenceheterogeneity across individuals and regularizing the ranking distribution for better out-of-samplepredictions.
A representative work is by Antoine et al.
[4], who use the Mallows model to aggregatecustomer preferences from their choices (i.e., k = 1 ).
Despite the theoretical elegance, the main challenge in applying Mallows-type ranking models to(ranked) choice modeling is analytical and computational tractability.
More specifically, Pr( π k | S )is calculated from summation over all rankings subject to nontrivial conditions.
Therefore, evenif a Mallows-type ranking model has a simple structure for λ , the resulting choice probabilities{ Pr( π k | S ) } can be difficult to obtain even when k = 1 .
The state-of-art results are obtained by
Antoine et al.
[4], who develop polynomial-time numerical algorithms to compute { Pr( π k | S ) } fork = 1 under the Mallows model.
The estimation problem is even more difficult, which involvesfinding the central ranking π ∗ and dispersion parameters q that best explain the (ranked) choice data.
To the best of our knowledge, no effective methods to estimate any Mallows-type ranking modelfrom (ranked) choice data are known.
(Perhaps the best method to date is again by Antoine et al.[4], who use a “Mallows smoothing" heuristic to conduct the estimation when k = 1 , which we willdiscuss later.)
Summary of results and contributions.
This paper identifies and studies a new distance-based(i.e.
, Mallows-type) ranking model.
It is the same as the Mallows model except that it builds on anew distance function (i.e., smoothing kernel), which we call reverse major index (RMJ) .
Unlike the
Mallows model’s Kendall-Tau distance (which weighs all pairwise disagreements equally), RMJ putsmore weight on top-position deviations.
The RMJ-based ranking model is a small conceptual deviation from the Mallows model (and, inparticular, enjoys the desired properties mentioned above, such as rationalizability, smoothing, andexpressive power); see Section 2.
However, this twist brings a significant advantage in both analyticaland computational simplicity.
Specifically, we solve a list of problems under the RMJ-based rankingdistribution.
That includes:
• Characterizing (ranked) choice probabilities : Given k ≥ 1 , calculating { Pr( π k | S ) } ;•
Sampling :
Given k ≥ 1 , efficiently sampling a top- k list π k ;• Parameter learning : Estimating the central ranking π ∗ and dispersion parameter q from the(ranked) choice data through a maximum likelihood estimator (MLE) formulation ;• Learning in a mixture setting :
Assuming that there are multiple clusters of participantpreferences, learning the central ranking and dispersion parameter for each cluster fromchoice data.
The solutions to the problems above can be implemented relatively easily.
First, we are able to obtain{ Pr( π k | S ) } in simple and closed-form expressions for all k ≥ 1 ; see Theorems 1 and 4.
This is incontrast to, say, its counterpart for the Mallows model ([4]), and we view it as our main theoreticalachievement.
Second, the sampling can be done in O ( nk ) time directly; see Lemma 2.
Third, theestimation problem can be reduced to a well-studied ranking-aggregation-type formulation; see (5)and Theorem 5.
Many off-the-shelf tools are available.
For example, it admits a PTAS and can bepractically solved via a linear integer programming formulation.
The estimation is guaranteed torecover the model parameters asymptotically under mild conditions on the coverage of display sets;see Theorem 2.
This stands in contrast to Mallows Smoothing by [4], which cannot recover the
Mallows central ranking even under sufficient coverage; see Theorem 3.
Finally, the learning problemin a mixture setting can be solved using the standard Expectation-Maximization (EM) algorithm.
We demonstrate the practical effectiveness of our methodology on two data sets on customer preference over different types of sushi.
When k = 1 , we compare it with two representative ranking-basedchoice models: one based on the Mallows and the other on Plackett-Luce (which leads to the Multinomial Logit choice model).
Our tools display superb generalization power, especially when thereis a limited variety of display sets in the choice data.
When k > 1 , we demonstrate the robustnessof the methodology.
(It is difficult to find direct comparisons for prediction power.)
Specifically, as long as the underlying population keeps the same, different top-
k lists aggregate into the samecentral rankings.
We also show that our methodology can handle a relatively large number of items( n = 100 ) by achieving high precision solutions ( < 2% optimality gap) in a reasonable time (< 5minute solving time).
Related literature.
Our paper is most closely related to the (extensive) literature on Mallows-typeranking models and their applications to preference and choice modeling (e.g., [1, 5, 3, 6, 7, 8, 9, 4, 10]to name a few.)
Besides the new distance function and tractability results explained above, the mostnotable difference of this paper is the feedback structure .
In our paper, every participant chooses froman arbitrary display set (as opposed to pairwise or the full display set only), and their feedback is inthe form of a top- k ranked list (as opposed to single choices, e.g., [11, 12], or full rankings).
To thebest of our knowledge, the combination of both generalities makes the setting the first of its kind.
2 Itis also worth mentioning that the analytical and computational simplicity of our model makes it aconvenient building block for subsequent optimization problems (e.g., new product introduction).
Itis an advantage that many common methods (e.g., Monte-Carlo algorithms) do not have.
Also related is the literature on learning to rank (e.g., [14, 15]).
We wish to stress that we do notmerely fit a ranking from the data but also the uncertainty quantification of the estimate (and thewhole choice model { Pr( π k | S ) } ).
Finally, our paper is also closely related to that of [13].
They studya choice model, called the “ordinal attraction model" (OAM), that emerges from an active learningproblem of consumer preferences.
We rationalize the (surprisingly simple) OAM by showing that itis equivalent to the aggregated choice probabilities { Pr( π k | S ) } from the RMJ-based ranking modelwhen k = 1 .
We view it as a nontrivial observation in its own right.
A Ranking Model Based on Reverse Major Index
Preliminary.
We consider a universe of n candidates (hereafter referred to as “items"), representedby [ n ] = { 1 , 2 , . . . , n } .
We assume that every participant has a strict preference over these items,represented by a ranking (permutation).
We use a bijection π : [ n ] → [ n ] to represent a rankingwhere π ( i ) is the i th most preferred item.
We sometimes find the notation σ := π − 1 helpful, whereσ ( x ) is the position of item x in the ranking σ .
We also use x ≻ π y if item x is preferred to item yunder π , i.e., π − 1 ( x ) <
π − 1 ( y ) .
For example, π = (3 , 1 , 2) means 3 ≻
π 1 ≻ π
and it correspondsto a “ σ " notation of σ = (2 , 3 , 1) .
Finally, we use e to represent the identity ranking, Σ for the set ofall rankings over n items, π k ( k ≥ 1) for a top- k list, and Σ k for the set of all top- k rankings.
Mallows-type ranking models.
Given the distance function d ( · , · ) , the probability mass function ofthe ranking for a Mallows-type model can be written as
[Equation]
where π ∗ is the central ranking, q is the dispersion parameter, and (cid:80) ˜
π q d ( π ∗ , ˜ π ) is the normalizationconstant.
Intuitively, the Mallows-type model defines a population of participants whose preferencesare "similar" as they are centered around a common ranking, where the probability for deviationsthereof decreases exponentially.
Different distance functions correspond to different models.
A common requirement for a validdistance function is that it is invariant to “relabeling."
Formally, that means d ( · , · ) is left-invariantunder the ranking composition, i.e., d ( π 1 , π 2 ) =
d ( ππ 1 , ππ 2 ) for every π, π 1 , π 2 .
This invarianceproperty enables to make the following conventions without loss of generality: First, we assume thatthe items have been properly relabeled so that the central ranking π ∗ =
e (unless otherwise specified).
Second, we may use d ( π ) as shorthand notation for d ( e, π ) , which fully represents a distance functionwith the knowledge that d ( π 1 , π 2 ) =
d ( e, π − 1 1 π 2 ) .
The Mallows (Kendall’s Tau distance based) ranking model.
Commonly studied distance functionsinclude the Spearman’s rank correlation, Spearman’s Footrule, and the Kendall’s Tau distance ([1]).
Among those three, Kendall’s Tau corresponds to the Mallows model and is defined as
[Equation]
It measures a ranking’s total number of pairwise disagreements (with the identity ranking e ).
For example, consider the ranking π = (4 , 2 , 1 , 3) .
There are four pairwise disagreements:{ (4 , 2) , (4 , 1) , (4 , 3) , (2 , 1) } .
Therefore, d K ((4 , 2 , 1 , 3))
= 4 .
As an appealing property, Kendall’s tau distance leads to a tractable expression for λ .
Other commondistances do not have this since the normalization constant involves summing over n !
rankings.
Theconstant under the Mallows model can be expressed as (see [16]):
[Equation]
The RMJ-based ranking model.
In this paper, we identify a new distance function, which we callreverse major index (RMJ) .
It is defined as
[Equation]
Compared to Kendall’s Tau in (1), RMJ focuses on adjacent disagreements and puts more weighton top-position disagreements.
For example, consider the ranking π = (4 , 2 , 1 , 3) again.
The onlyadjacent disagreements are { (4 , 2) , (2 , 1) } .
Therefore, after including the weights in (3), we haved R ((4 , 2 , 1 , 3))
= 3+2 = 5 .
The name of RMJ is inspired by the major index from the combinatoricsliterature ([17]), which is defined as d M ( π )
= (cid:80) n − 1 i
=
I { π ( i ) >
π ( i + 1) } · i.
Discussion: Kendall’s Tau vs. RMJ.
Both metrics are conceptually similar: they measure a ranking’sdeviation from the identity.
They coincide in many intuitive cases.
For example, in the most extremeones we have d K ( e ) =
d R ( e ) = 0 and d K (( n, . . .
=
d R (( n, . . . , 1))
= n ( n − 1) / 2 .
Therefore,both can be used as “kernels" to smooth out the distribution over rankings.
However, they emphasizethe deviation in a subtly different way.
Therefore, we can also find subtle rankings where their valuesare different.
For example, d K ((4 , 2 , 1 , 3))
= 4 but d R ((4 , 2 , 1 , 3))
= 5 .
It is difficult to tell which kernel is “better" from an axiomatic approach as both distance functionssatisfy the basic axioms for ranking distances ([1]).
Nevertheless, we will show later that the
RMJ-based ranking model leads to simple and estimatable (ranked) choice probabilities withoutcompromising the desirable properties of Mallows, such as rationalizability and flexibility for amixture setting.
Moreover, regarding its ability to describe preference distributions that may occur inpractice, we will demonstrate its descriptive and predictive power in a case study with real preferencedata.
Therefore, we believe that RMJ produces a promising tool that is more tailed to the applicationof learning population preferences from (ranked) choice data.
Analysis of the (Ranked) Choice Model { Pr( π k | S ) }
In this section, we will characterize the (ranked) Choice Model { Pr( π k | S ) } aggregated from the
RMJ-based ranking model, formally defined as
[Equation]
where I ( π k , ˜ π, S ) means the top- k list π k is compatible with the ranking ˜ π in the set S .
That is,π k ( i ) ∈ S and π k ( i ) ≻
˜ π x for all position i ∈ { 1 , . . .
, k } and item x ∈ S \ { π k (1) , . .
, π k ( i ) } .
Note that the summation is over rankings with nontrivial conditions.
Therefore it is unclear in prioriwhether any distance-based ranking model can aggregate into a tractable Pr( π k | S ) (even for k = 1 ).
We will also discuss how to estimate the ranked choice model parameters from data.
In the sequel,we will write d ( · ) =
d R ( · ) since RMJ is the distance function of interest.
3.1 The k = 1 case
When k = 1 , a top- k list model reduces to a choice model, which connects to a richer literature (e.g.,[4]).
Therefore, it is worth a separate discussion, which also helps build intuition for the k > 1 case.
Choice probabilities.
Our main result for k = 1 is summarized below, which characterizes thechoice probabilities under the RMJ-based ranking model.
Theorem 1 Let a display set S
= { x 1 , x 2 , . . .
, x M } be such that x 1 < x 2 < · · · < x M .
Under the
RMJ-based ranking model
[Equation]
In other words, for every display set, all the items within the display set are re-ranked so that theirchoice probabilities decay exponentially fast according to their relative ranking within the displayset.
Noticeably, the choice probabilities in (4) are (much) simpler than that induced by the Mallowsmodel, which even needs a Fast Fourier Transform to evaluate in O ( n 2 log n ) time ([4]).
Also,(4) rediscovers the “Ordinal Attraction Model" (OAM) in [13], which could also be viewed as a“multiwise" generalization of pairwise noisy comparison models (e.g., [18, 19]).
OAM gets its namebecause the “attractiveness" (i.e., choice probability) of an item within a display set S only dependson its relative position in S and therefore is only a function of the “ordinal" information.
While OAMemerges from an active learning problem of consumer preferences, we “rationalize" it by showing thatit can be aggregated from the RMJ-based ranking model, which we believe is a nontrivial observationin its own right.
In the sequel, we will follow its convention and refer to the choice model defined in
Theorem 1 as OAM.
Proof outline and key intermediate results for Theorem 1.
Let us introduce a few notations fortop-
k (sub-) rankings.
Given a top- k ranking π k ∈ Σ k , we could define RMJ for π k by truncatingthe index at position k , i.e., d ( π k ) =
(cid:80)
k − 1 i
=
I { π k ( i ) >
π k ( i + 1) } · ( n − i ) .
In addition, let
R ( π k ) :
=
{ π k ( i ) : i = 1 , . . .
, k } be the set of top- k items and R c ( π k ) :
= [ n ] \ R ( π k ) its complement.
Finally, let L ( π k ) :
= | { x ∈ R c ( π k ) : x < π k ( k ) } | be the number of items that are (i) not includedin π k and (ii) having smaller indices than (i.e., preferred under π ∗ = e ) item π k ( k ) .
For example,suppose n = 7 and π 4 =
Then d ( π 4 ) = 6 + 4 = 10 , R ( π 4 ) = { 2 , 4 , 6 , 7 } , R c ( π 4 ) ={ 1 , 3 , 5 } , π 4 (4) = 2 , and L ( π 4 ) =
|{ 1 }| = 1 .
Given two subrankings π k ∈ Σ k and π k ′ ∈
Σ k ′ with k ≤ k ′ , we write π k ⊆ π k ′ if they arecompatible, i.e., π k ( i ) =
π k ′ ( i ) for all i = 1 , . . .
, k .
Our first result concerns extending the domainof λ to top-
k rankings, formally defined as λ ( π k ) :
= (cid:80) ˜
π λ (˜ π ) I { π k ⊆ ˜ π } .
Lemma 1 (Probability distribution of top- k rankings) λ ( π k ) =
q d ( π k )+ L ( π k ) · ψ ( n − k,q ) ψ ( n,q ) .
The significance of the result above is that we could think of a ranking π as a stochastic process on a“tree" with depth n and n ! leaves.
While { λ ( π ) : π ∈ Σ } describe the probability distribution over theleaves, for every k , { λ ( π k ) : π k ∈ Σ k } describe the probability distribution over the nodes at level k .
As a consequence of Lemma 1, we can write out how to randomly sample a ranking π under the
RMJ-based ranking model from the top to the bottom position.
Given a top- k ranking π k and an itemz , we write π k ⊕ z as the concatenation of π k and item z .
For example, suppose π
=
(5 , 2 , 4) , thenπ 3 ⊕ 3 = (5 , 2 , 4 , 3) .
Lemma 2 (Random ranking generation)
Given a top-k ranking π k such that π k ( k )
=
z , the conditional probability for the ( k + 1) -positioned item is
[Equation]
where h ( y | z )
= (cid:40)(cid:80) x ∈ R c
( π k ) I { z < x ≤ y } if y >
z
[Equation]
if
[Equation]
< z .
Finally, Lemmas 1 and 2 leads to the following result based on an induction argument.
Lemma 3 Let a display set S = { x 1 , x 2 , . . .
, x M } be such that x 1 < x 2 < · · · < x M and a top- kranking π k be such that π k = π k − 1 ⊕ z
and R ( π k − 1 ) ∩ S = ∅ .
Then conditional on a participant’s top-
k preference list is π k , the probability that (s)he will choose item x i out of display set S is
Pr ( { x i | S }| π
[Equation]
) :=
[Equation]
λ
[Equation]
=   q M − p ( z | S )
+ i −1+ q
+ ... +
q M −if
z > x i and z ̸∈ S, q i − p
( z | S ) −
+ q + ...
+ q M −if
z
< x i and z ̸∈ S, if z = x i , if z ∈ S \ { x i } .
where p ( z | S ) := (cid:80) x ∈ S I { x < z } .
Note that Lemma 3 could be viewed as a generalization of Theorem 1 by setting π k = ∅ (whichcorresponds to z = 0 ).
Parameter learning from choice data.
In the parameter learning problem, we are endowed withchoice data H T
= ( S 1 , x 1 , . . .
, S T , x T ) , where S t is the display set shown to the t th participant andx t is his/her choice.
Following [13], the maximum likelihood estimator (MLE) for the central rankingπ ∗ can be obtained from following choice aggregation problem
[Equation]
It has a further linear integer programming formulation.
Let w ij :=
(cid:80) Tt
=
I {{ i, j } ⊆
S t and x t = i }be the number of times that item i and item j are displayed together and item i is chosen.
Intuitively,a positive w ij − w ji is an indication that item i should be preferred to item j .
Invoking Proposition 3in [13], the integer programming could be written as follows:
[Equation]
In the formulation above, the solution ˆ x is such that ˆ x ij = 1 if i ≻ ˆ π j under the MLE ˆ π .
Computationally, this integer programming is an instance of the well-studied feedback arc set problemon tournaments .
Therefore, it admits a polynomial-time approximation scheme (PTAS).
From amore practical side, the central ranking σ ∗ can be effectively obtained using off-the-shelf integerprogramming solvers and with many speeding-up heuristics; see [13] for more discussion.
The
MLE for dispersion parameter q can be subsequently obtained from an one-dimensional (convex)optimization problem ˆ α ∈ argmin α { α (cid:80)
Tt =1 (cid:80) x ∈ S t I { x ≻ ˆ π x t } + (cid:80) Tt =1 log (cid:80)
| S t |− 1 j = e − jα } so thatˆ α = − ln
ˆ q .
It is also worth noting that the MLE framework can be easily extended to learning in amixture setting using the standard EM algorithm, which we refer to the supplementary materials formore details.
Let us conclude this section by providing some theoretical understanding of whether we could recoverthe ground truth values of π ∗ and q at least asymptotically.
Intuitively, this should depend on the“coverage" of display sets: if only a pair { x, y } is repeatedly, there is no hope of recovering the fullranking π ∗ .
It turns out that the parameters can be recovered as long as every pair is “covered" bysome display set.
Theorem 2 Consider a sequence of display sets { S t } .
Suppose every pair of items { i, j } ⊆
[ n ]is displayed infinitely often.
That is, (cid:80)
Tt
=
I (cid:8) { i, j ∈ S t } (cid:9) → ∞ as T → ∞ .
Then the MLE(ˆ π, ˆ q ) =
(ˆ π ( H T ) , ˆ q ( H T )) is an consistent estimator.
That is, (ˆ π, ˆ q ) → ( π ∗ , q ∗ ) almost surely as T → ∞ .
Conversely, if there exists a pair of items { i, j } that is only displayed finitely often, thenthere exists a tie-breaking rule for MLE so that ˆ π ̸→ π ∗ with positive probability.
We would like to mention that the coverage condition is fairly mild.
For example, as long as the fulldisplay set is displayed sufficiently many times, the RMJ-implied central ranking can eventually berecovered.
This stands in contrast to its counterpart in the Mallows model.
Since the Mallows modeldoes not produce simple choice probabilities, the MLE from choice data is rather difficult to obtain.
Perhaps the best method to date is by [4], who use a “Mallows smoothing" heuristic.
The followingresult reveals that the heuristic can be unstable or fail to recover the underlying Mallows parameterseven under sufficient coverage of display sets.
Theorem 3 Even if all display sets with sizes larger than two are displayed infinite times, theestimator from the Mallows Smoothing heuristic is not consistent.
The intuition behind the result above is that Mallows Smoothing needs a “choice to ranking" step.
That is, it needs to first find a distribution over rankings, denoted by ˆ λ ∈ ∆(Σ) , that aggregates intochoice probabilities to match the empirical choice probabilities { ˆPr( x | S ) } .
Formally, it correspondsto solving the system of linear equations
[Equation] [Equation]
for all display set S in the data and x ∈ S .
Since there are vastly more variables than equations, thesolutions in general form a polytope rather than a singleton.
More importantly, we find that therecan be solutions ˆ λ 1 and ˆ λ 2 that aggregate to different Mallows models in the “ranking aggregation"step of the heuristic.
Therefore, in general, this heuristic can produce non-unique results and thus beinconsistent.
3.2 The general ( k ≥ 1 ) case
In the general ( k ≥ 1 ) case, the ranked choice probabilities and the corresponding estimation remainparsimonious.
Roughly speaking, the ranked choice model behaves like a generalized OAM where
Pr( π k | S ) depends on the relative rankings of the items in π k among the set S .
The tractability comesfrom the sole dependence on the relative ranking.
That is, to obtain Pr( π k | S ) , one could first treat Sas the “full display set" for a subuniverse of items, then re-rank all the items within the display set S ,and finally just apply Lemma 1.
Choice probabilities.
Given a display set S , let d S ( · ) , L S ( · ) be the originally defined d ( · ) and L ( · )functions, but treating the display set S as the universe.
Formally, d S ( π k ) :
= (cid:80) k − 1 i =1 I { π k ( i ) >
π k ( i + 1) } · ( | S | − i ) and L S ( π k ) :
= | { x ∈ R c ( π k ) ∩ S : x < π k ( k ) } | .
For example, supposeagain that n = 7 and π 4 = (7 , 4 , 6 , 2) , and let S = { 2 , 3 , 4 , 5 , 6 , 7 } .
Recall that d ( π 4 )
= 6 + 4 =10 and L ( π 4 )
= |{ 1 }| = 1 .
In comparison, d S ( π 4 )
= 5 + 3 = 8 and L S ( π 4 ) =
|∅| = 0 .
Theorem 4 Given a display set S and a top- k ranking π k such that R ( π k ) ⊆
S , we have
[Equation]
Note that Theorem 4 generalizes both Theorem 1 and Lemma 1 (but in different ways).
It reduces to
Theorem 1 by setting k = 1 :
Given π 1 = ( z ) where z ∈ S , we have d S ( π 1 ) = 0 , L S ( π 1 ) =
|{ x ∈
S \ { z } : x < z }| the relative ranking of z in display set S , and ψ ( | S |− k,q ) ψ ( | S | ,q ) = 1 + q + · · · + q | S |− 1the appropriate normalizing constant where k = 1 .
In addition, Theorem 4 reduces to Lemma 1 bysetting S =
[ n ] , as can be easily verified.
Parameter learning from ranked choice data.
In the parameter learning problem, we are endowedwith choice data H T =
( S 1 , π 1 k , . . .
, S T , π Tk ) , where S t is the display set shown to the t
th participantand π
tk
=
( x t 1 , . . .
, x tk ) are his/her top- k choices ranked from the most preferred to the k th preferred.
It turns out that (somewhat surprisingly) in the k ≥ 1 case, the central ranking π ∗ can be estimatedfrom the same integer programming (5) but just with a generalized re-definition of the weightparameters { w ij } .
Theorem 5 The MLE for the central ranking π ∗ , ˆ π , can be obtained from the integer program (5)with ˆ x ij = 1 iff i ≻
ˆ π j and a generalized definition of { w ij } below
[Equation]
Intuitively, a positive w ij − w ji is still an indication that item i should be preferred to item j , butnow taking into consideration that every participant’s response is actually a ranked list rather thana single choice.
Practically speaking, the { w ij } can be maintained in a relatively simple manner.
Suppose a display set S and a ranked choice π k are given.
Then w π k ( ℓ ) ,π k ( ℓ +1) will be added by| S | − ℓ for all ℓ = 1 , . . .
, k − 1 .
(This captures the ranking information for items included in π k .)
In addition, w π k ( k ) ,j will be added by 1 for all ℓ ∈ { R c ( π k ) ∩ S } .
(This captures the rankinginformation between π k ( k ) and all items that are not included in π k .)
For example, suppose n = 6and k
= 3 .
The ranked choice data is such that S 1 = { 1 , 2 , 3 , 4 , 5 } and π 1 k = (3 , 1 , 2) .
Then we havew 3 , 1 = 4 , w 1 , 2 = 3 , w 2 , 4 = w 2 , 5 = 1 and all the other w ij = 0 .
Discussion: model limitation.
A few more words on the ranked choice model { Pr ( π k | S ) } .
Thespecific form gives it great simplicity, but as with any other parametric model, it may be vulnerableto model missspecification issues in practice.
In particular, since the model is calculated from aunimodal ranking distribution, this model (in its “vanilla" form) is best suited when there is anapproximate “consensus" ranking in the population.
In our numerical studies, we will demonstratehow we can effectively mitigate those issues by considering learning the model parameters in amixture setting to better capture the preference heterogeneity in the population.
Numerical Experiments
We investigate the performance of our methodology on two anonymous survey data sets regardingsushi preferences ([20]):
The first one consists of 5,000 full preference rankings over 10 kindsof sushi, while the second one consists of 5,000 top- 10 rankings over 100 kinds.
Throughout thenumerical experiments, we use a workstation with dual Intel Xeon Gold 6244 CPU (3.6 GHz and 32cores in total), no GPUs, and 754 GB of memory.
Experiment 1: Top- 1 choice.
When k = 1 , a top- k list corresponds to a choice model, whichconnects to a richer literature (e.g., [21, 4, 13]).
We compare our prediction power with tworepresentative ranking models: Mallows and Plackett-Luce.
(The previous is distance-based.
Thelatter is not but leads to the famous MNL choice model.)
We use the Mallows Smoothing (MS)heuristic to estimate a Mallows model and MLE to estimate a Plackett-Luce.
For all three rankingsmodels (Mallows, PL, and ours), we also use an EM algorithm to perform parameter learning in amixture setting.
Since the estimation and generalization performances can depend on the display sets, let us define aninstance of the comparison by {S train , S test , C} .
Here S train (resp.
S test ) is collection of displaysets in the training (resp.
test) data, and C is the number of clusters.
In every instance, we split thedata into 4000 participants in training and 1000 in testing.
We generate empirical training (resp.testing) choice data by first enumerating all display sets in S train (resp.
S test ) and then record everyparticipant’s favorite type of sushi for every display set in the training (resp. testing) data.
Finally, weuse the log-likelihood as performance metric.
We will use term explanation power (resp.
predictionpower ) to refer the performance metric evaluated on the training (resp. testing) data.
We use two configurations for S train and S test , respectively.
For S train , we take it to be either the full set or a collection of three randomly generated sets, which are realized to be{{ 1 , 3 , 4 , 7 , 8 , 10 } , { 2 , 4 , 5 , 6 , 8 , 9 } , [10] } .
For S test , we take it to be either the collection of allpairwise sets or all display sets with sizes at least two.
Note that S train has low variability andmany elements in S test do not appear in S train , making the experiment emphasizing more on thegeneralization power .
We take C ∈ { 1 , 2 , 4 , 6 , 15 } .
We summarize our results in Figure 1.
We find that our method has favorable performance comparedto MS and MNL across the settings.
Intuitively, MS and MNL underperform in different ways.
MS is vulnerable to the identifiability issue in the “choice-to-rank" step, for which we providetheoretical understanding in Theorem 3.
Our numerical experiments confirm the theoretical insight:
We randomize over the extreme points of the polytope of solutions to (6), each of which leads to avalid output of the MS heuristic.
Every box plot in Figure 1 contains a summary of the performancesof those outputs.
It is clear that the performances (both in training and testing) span wide ranges.
Inthe meantime, MNL is vulnerable to both the model risk from its specific parametric form (reflectedin the limited increase of explanation power when considering more clusters) and the sample risk of overfitting (reflected in the difference in performance in training vs. testing, especially when trainedon the full display set only); see more details in Appendix F.
[Figure]
Experiment 2: Top- k choice.
In this experiment, we perform a robustness check based on thecriterion that a good model should return similar central rankings under different feedback structures,i.e., different k .
Specifically, we conduct estimation on top- 1 , top- 2 and top- 3 ranked choicesconstructed from the first 10-sushi data set.
The collection of display sets is taken to be all displaysets with sizes at least k .
We find that all three experiment instances produce the same estimated
RMJ-implied central ranking ˆ π = (8 , 5 , 6 , 3 , 2 , 1 , 4 , 9 , 7 , 10) .
Such robustness stands in contrast withother more naive methods, such as various versions of Borda count; see more details in Appendix F.
We believe it presents evidence that our methodology is learning sensible information from the data.
Experiment 3: 100 sushi types.
In this experiment, we wish to show how effective our method is fora relatively large number of items.
In the data, each of the 5000 individuals indicates their top- 10choices out of 100 types of sushi.
We use the LP-Rand-Pivot speed-up heuristic by [13] based on
LP relaxation to train a single-cluster RMJ-based model.
We bootstrap 10 times (each time drawing10000 samples) and record the running times and optimality gaps in the Table 1.
We find that we canobtain < 2% optimality gap within 5 minutes (excl.
model building time).
[Table]
Conclusions
We identify a novel distance-based (Mallows-type) ranking model.
It aggregates into simple probability distributions for top- k subrankings among an arbitrary display set S .
In addition, it facilitateseffective parameter learning through the MLE formulation.
This is the first distance-based rankingmodel with such properties (even for k = 1 ) to the best of our knowledge.
This ranking model can be used to model population preferences and provide a rationalizable way tomodel their ranked choices from given display sets.
We demonstrate its practical value using realpreference data.
For example, under a mixture setting with only a few clusters, it shows promisingprediction power, especially when there is a limited variety in the display sets.
For future steps, we believe our work can serve as the “infrastructure" for a range of business-relateddecision problems, such as new product introduction, crowdsourcing, and marketing research, amongothers.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] We have double checked every claim and they arebacked by the theorems/lemmas formally stated in the paper.(b)
Did you describe the limitations of your work?
[Yes] See end of Section 3 named
“Discussion: model limitation."
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
The
main negative societal impact that we can think of is that the RMJ-based rankingmodel (in its vallina form) tends to aggregate into a social “consensus ranking, whichunder estimates preference heterogeneity.
This should be mitigated by acknowledgingmultiple preference clusters using a mixture setting.
For details, please see the end ofSection 3 named “Discussion: model limitation."
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] We have carefully read the ethics guidelines and ensured that we follow
the guidelines.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 2.
(b) Did you include complete proofs of all theoretical results?
[Yes] Proofs of our theoreti-
cal results can be found in the supplemental material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] They are
included in the supplemental material.
However, we wish to clarify that our code isnot necessarily “executable" as is since it has dependencies (i.e., Gurobi, which is a
paid-for mathematical programming solver) going beyond the algorithm itself.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Yes to all applicable one,
i
e., the performances for MS;
see Section 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] We included the computation
time in Experiment 3, which was specifically designed to demonstrate the speed of
the methods on the 100-sushi data set.
For other experiments (especially Experiment
1), a large amount of compute is devoted to finding the extreme points of the solution
polytope to (6).
This is more about understanding the statistical property of MS and
hence the speed is a less relevant issue.
For computational resources, please see the
beginning of Section 4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We use a publicly
[Table]
available sushi dataset and cite the creators [20].
[Table]
(b) Did you mention the license of the assets?
[Yes] The data we use is public.
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Table]
We didn’t include any new assets.
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
This is a public available dataset.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[Yes] Please see Section 4.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Table]