Original: Efﬁcient and Accurate Top- K Recovery from Choice Data
Revised: Efficient and Accurate Top- K Recovery from Choice Data

Original: Abstract
Revised: Abstract

Original: The intersection of learning to rank and choice modeling is an active area of research with applications in e-commerce, information retrieval and the social sciences.
Revised: The intersection of learning to rank and choice modeling is an active area of research with applications in e-commerce, information retrieval and the social sciences.

Original: In some applications such as recommendation systems, the statistician is primarily interested in recovering the set of the top ranked items from a large pool of items as efﬁciently as possible using passively collected discrete choice data , i.e., the user picks one item from a set of multiple items.
Revised: In some applications such as recommendation systems, the statistician is primarily interested in recovering the set of the top ranked items from a large pool of items as efficiently as possible using passively collected discrete choice data , i.e., the user picks one item from a set of multiple items.

Original: Motivated by this practical consideration, we propose the choice-based Borda count algorithm as a fast and accurate ranking algorithm for top K -recovery i.e., correctly identifying all of the top K items.
Revised: Motivated by this practical consideration, we propose the choice-based Borda count algorithm as a fast and accurate ranking algorithm for top K -recovery i.e., correctly identifying all of the top K items.

Original: We show that the choice-based
Revised: We show that the choice-based

Original: Borda count algorithm has optimal sample complexity for top- K recovery under a broad class of random utility models .
Revised: Borda count algorithm has optimal sample complexity for top- K recovery under a broad class of random utility models .

Original: We prove that in the limit, the choice-based Borda count algorithm produces the same top- K estimate as the commonly used
Revised: We prove that in the limit, the choice-based Borda count algorithm produces the same top- K estimate as the commonly used

Original: Maximum Likelihood Estimate method but the former’s speed and simplicity brings considerable advantages in practice.
Revised: Maximum Likelihood Estimate method but the former’s speed and simplicity brings considerable advantages in practice.

Original: Experiments on both synthetic and real datasets show that the counting algorithm is competitive with commonly used ranking algorithms in terms of accuracy while being several orders of magnitude faster.
Revised: Experiments on both synthetic and real datasets show that the counting algorithm is competitive with commonly used ranking algorithms in terms of accuracy while being several orders of magnitude faster.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: The research on discrete choice modeling and learning to rank has received a lot of interest in recent years thanks to the growing availability of discrete choice data generated by e-commerce platforms, search engines and the social sciences.
Revised: The research on discrete choice modeling and learning to rank has received a lot of interest in recent years thanks to the growing availability of discrete choice data generated by e-commerce platforms, search engines and the social sciences.

Original: In the discrete choice setting, when presented with a set of items, also referred to as menu , the user picks the most preferred item.
Revised: In the discrete choice setting, when presented with a set of items, also referred to as menu , the user picks the most preferred item.

Original: Discrete choice data is an intermediate between pairwise comparison data and full ranking data.
Revised: Discrete choice data is an intermediate between pairwise comparison data and full ranking data.

Original: In many settings such as e-commerce and political surveys, a large quantity of passively collected data is in the form of discrete choice data, e g., consumers choosing to buy a product when presented with a catalogue of items, voters picking a favorite candidate from a pool of candidates.
Revised: In many settings such as e-commerce and political surveys, a large quantity of passively collected data is in the form of discrete choice data, e g., consumers choosing to buy a product when presented with a catalogue of items, voters picking a favorite candidate from a pool of candidates.

Original: In this paper we focus on the problem of learning to rank using choice data.
Revised: In this paper we focus on the problem of learning to rank using choice data.

Original: Speciﬁcally, we are interested in the topK recovery problem, i.e., identifying the set of the top K items out of a universe of n items, using passively collected choice data .
Revised: Specifically, we are interested in the topK recovery problem, i.e., identifying the set of the top K items out of a universe of n items, using passively collected choice data .

Original: This problem has many useful applications.
Revised: This problem has many useful applications.

Original: For example, in e-commerce applications, marketers are interested in ﬁnding the set of the best items based on how consumers make purchasing decisions.
Revised: For example, in e-commerce applications, marketers are interested in finding the set of the best items based on how consumers make purchasing decisions.

Original: In the social sciences, political scientists are interested in determining the most preferred candidates among a pool of candidates using survey questionnaires.
Revised: In the social sciences, political scientists are interested in determining the most preferred candidates among a pool of candidates using survey questionnaires.

Original: To ground our theoretical discussions, we posit that the choice data is generated according to a probabilistic choice model- when presented with a menu of items S , the user makes a non-deterministic decision, picking a single item i from S with some probability p i | S .
Revised: To ground our theoretical discussions, we posit that the choice data is generated according to a probabilistic choice model- when presented with a menu of items S , the user makes a non-deterministic decision, picking a single item i from S with some probability p i | S .

Original: More speciﬁcally, we assume our choice model falls within the class of Random Utility Models with Independently and Identically Distributed noise (IID-RUMs), described in detail in section (2).
Revised: More specifically, we assume our choice model falls within the class of Random Utility Models with Independently and Identically Distributed noise (IID-RUMs), described in detail in section (2).

Original: IIDRUMs are an expressive and ﬂexible framework that can be used to model pairwise comparison data, discrete choice data as well as full ranking data.
Revised: IIDRUMs are an expressive and flexible framework that can be used to model pairwise comparison data, discrete choice data as well as full ranking data.

Original: For example, the Multinomial Logit (MNL) model is one of the most commonly used
Revised: For example, the Multinomial Logit (MNL) model is one of the most commonly used

Original: IID-RUMs to model discrete choice data [Train, 2009].
Revised: IID-RUMs to model discrete choice data [Train, 2009].

Original: Our motivation: While expressive, random utility models also pose hard computational problems.
Revised: Our motivation: While expressive, random utility models also pose hard computational problems.

Original: For example, many models within the class of IID-RUMs with the few exceptions such as the MNL model do not admit analytical expression for the choice probabilities (while the pairwise comparison probabilities can be evaluated easily), limiting inference to MCMC-based algorithms.
Revised: For example, many models within the class of IID-RUMs with the few exceptions such as the MNL model do not admit analytical expression for the choice probabilities (while the pairwise comparison probabilities can be evaluated easily), limiting inference to MCMC-based algorithms.

Original: However, samplingbased algorithms can be time inefﬁcient when running on large choice datasets with many items and menus.
Revised: However, samplingbased algorithms can be time inefficient when running on large choice datasets with many items and menus.

Original: Furthermore, most classical inference algorithms assume a parametric model generating the choice data.
Revised: Furthermore, most classical inference algorithms assume a parametric model generating the choice data.

Original: In practice, it is often hard to verify if the data comes from a speciﬁc parametric model.
Revised: In practice, it is often hard to verify if the data comes from a specific parametric model.

Original: Therefore, developing efﬁcient ranking algorithms that are robust to model misspeciﬁcation is of timely interest.
Revised: Therefore, developing efficient ranking algorithms that are robust to model misspecification is of timely interest.

Original: Motivated by these considerations, we study the generalization of a simple yet powerful counting algorithm for ranking-
Revised: Motivated by these considerations, we study the generalization of a simple yet powerful counting algorithm for ranking-

Original: Borda count - to the discrete choice setting.
Revised: Borda count - to the discrete choice setting.

Original: The
Revised: The

Original: Borda count algorithm itself has a long history, dating back to the 18th century and its analysis has been instantiated in various contexts such ranking from pairwise comparisons in Rajkumar and Agarwal [2014], Shah and Wainwright [2017].
Revised: Borda count algorithm itself has a long history, dating back to the 18th century and its analysis has been instantiated in various contexts such ranking from pairwise comparisons in Rajkumar and Agarwal [2014], Shah and Wainwright [2017].

Original: Our work, however, is the ﬁrst to study the theoretical guarantees of Borda count in the discrete choice setting under a broad class of discrete choice models.
Revised: Our work, however, is the first to study the theoretical guarantees of Borda count in the discrete choice setting under a broad class of discrete choice models.

Original: Our
Revised: Our

Original: contributions:
Revised: contributions

Original: • In Section 4 and Section 5, we show that the choicebased Borda count algorithm needs θ ( n log n ) samples in order to exactly recover all of the top K items using choice data.
Revised: • In Section 4 and Section 5, we show that the choicebased Borda count algorithm needs θ ( n log n ) samples in order to exactly recover all of the top K items using choice data.

Original: We further show that this sample complexity is optimal for a broad class of IID-RUMs.
Revised: We further show that this sample complexity is optimal for a broad class of IID-RUMs.

Original: This hinges on a fundamental property shared by many IIDRUMs which we term Borda consistency .
Revised: This hinges on a fundamental property shared by many IIDRUMs which we term Borda consistency .

Original: • In Section 6, we study the effect of the menu size m on the sample complexity for top K recovery.
Revised: • In Section 6, we study the effect of the menu size m on the sample complexity for top K recovery.

Original: For the special case of the MNL model, which is a common assumption in the ranking literature, we present an asymptotic characterization of the optimal sample complexity for top K recovery in terms of m .
Revised: For the special case of the MNL model, which is a common assumption in the ranking literature, we present an asymptotic characterization of the optimal sample complexity for top K recovery in terms of m .

Original: This bound monotonically decreases, but at a decreasing rate, with m .
Revised: This bound monotonically decreases, but at a decreasing rate, with m .

Original: This suggests that there is a beneﬁt to increasing the menu size but such beneﬁt comes with diminishing returns.
Revised: This suggests that there is a benefit to increasing the menu size but such benefit comes with diminishing returns.

Original: To the best of our knowledge, this result is the ﬁrst of its kind in the choice modeling and ranking literature.
Revised: To the best of our knowledge, this result is the first of its kind in the choice modeling and ranking literature.

Original: • In Section 7, we study the connections between the choice-based Borda count algorithm and two commonly used top- K recovery algorithms: Maximum Likelihood Estimate under MNL assumption (MNLMLE and Spectral Ranking [Negahban et al., 2017, Maystre and Grossglauser, 2015, Agarwal et al., 2018].We prove
Revised: • In Section 7, we study the connections between the choice-based Borda count algorithm and two commonly used top- K recovery algorithms: Maximum Likelihood Estimate under MNL assumption (MNLMLE) and Spectral Ranking [Negahban et al., 2017, Maystre and Grossglauser, 2015, Agarwal et al., 2018].

Original: that the choice-based Borda count algorithm and

Original: MNL

Original: MLE produce the

Original: same top- K estimate

Original:  in the limit of inﬁnite data, even if the data has not been generated by an IID-RUM.
Revised: We prove that the choice-based Borda count algorithm and MNL-MLE produce the same top- K estimate in the limit of infinite data, even if the data has not been generated by an IID-RUM.

Original: On the other hand, Spectral Ranking does not in general give the same estimate as the choice-based Borda count algorithm MNL-
Revised: On the other hand, Spectral Ranking does not in general give the same estimate as the choice-based Borda count algorithm/MNL-MLE even with infinite data.

Original: MLE

Original: even with inﬁnite data

Original: • In Section 8, We show through empirical experiments that the choice-based Borda count algorithm is com - petitive in terms of accuracy with both MNL
Revised: • In Section 8, We show through empirical experiments that the choice-based Borda count algorithm is competitive in terms of accuracy with both MNL-MLE and Spectral Ranking while being several orders of magnitude faster.

Original: MLE

Original: and

Original: Spectral Ranking while being several orders of magnitude faster

Original: This highlights the advantage of the choicebased Borda count algorithm in applications where the statistician is primarily interested in efﬁciently and accurately identifying the top items.
Revised: This highlights the advantage of the choicebased Borda count algorithm in applications where the statistician is primarily interested in efficiently and accurately identifying the top items

Original: 1.1 RELATED WORKS
Revised: 1.1 RELATED WORKS

Original: Our work falls within the literature on learning to rank under
Revised: Our work falls within the literature on learning to rank under

Original: Random Utility Models (RUMs).
Revised: Random Utility Models (RUMs).

Original: There has been a substantial amount of work on learning to rank under Random
Revised: There has been a substantial amount of work on learning to rank under Random

Original: Utility Models and mixtures of Random Utility Models using full ranking data [Parkes et al., 2012, Azari Souﬁani et al., 2013b,a, Souﬁani et al., 2014, Zhao et al., 2016, Zhao and Xia, 2019].
Revised: Utility Models and mixtures of Random Utility Models using full ranking data [Parkes et al., 2012, Azari Soufiani et al., 2013b,a, Soufiani et al., 2014, Zhao et al., 2016, Zhao and Xia, 2019].

Original: Furthermore, most classical ranking methods assume that the data is generated by a well speciﬁed
Revised: Furthermore, most classical ranking methods assume that the data is generated by a well specified

Original: RUM.
Revised: RUM.

Original: To the best of our knowledge, our paper is the ﬁrst to propose a method for top-
Revised: To the best of our knowledge, our paper is the first to propose a method for top-

Original: K ranking under a broad class of
Revised: K ranking under a broad class of

Original: RUMs using passively collected choice data alone .
Revised: RUMs using passively collected choice data alone .

Original: The related literature on ranking from pairwise comparisons is vast and we can only refer the interested reader to adjacent problems such as active top- K recovery from pairwise comparisons [Busa-Fekete et al., 2013, Agarwal et al., 2017,
Revised: The related literature on ranking from pairwise comparisons is vast and we can only refer the interested reader to adjacent problems such as active top- K recovery from pairwise comparisons [Busa-Fekete et al., 2013, Agarwal et al., 2017,

Original: Mohajer et al., 2017, Falahatgar et al., 2017, 2018, Heckel et al., 2019]; top- K recovery from pairwise comparisons
Revised: Mohajer et al., 2017, Falahatgar et al., 2017, 2018, Heckel et al., 2019]; top- K recovery from pairwise comparisons

Original: [Chen and Suh, 2015, Shah and Wainwright, 2017, Chen et al., 2019]; top- K recovery from m -wise sorted data (full rankings among some m items) [Jang et al., 2017, Chen et al., 2020].
Revised: [Chen and Suh, 2015, Shah and Wainwright, 2017, Chen et al., 2019]; top- K recovery from m -wise sorted data (full rankings among some m items) [Jang et al., 2017, Chen et al., 2020].

Original: Closest to our work is the analysis of Borda count by Shah and Wainwright [2017] who showed that it is optimal for top- K recovery from pairwise comparisons .
Revised: Closest to our work is the analysis of Borda count by Shah and Wainwright [2017] who showed that it is optimal for top- K recovery from pairwise comparisons .

Original: Our work complements theirs by showing that the choice-based Borda count is optimal even in the general choice setting .
Revised: Our work complements theirs by showing that the choice-based Borda count is optimal even in the general choice setting .

Original: To this end, we obtain in Section 4 sample complexity upper and lower bounds that are both more general and reﬁned than those given by Shah and Wainwright [2017].
Revised: To this end, we obtain in Section 4 sample complexity upper and lower bounds that are both more general and refined than those given by Shah and Wainwright [2017].

Original: We also study in Section 6 the effect of the menu size on the sample complexity.
Revised: We also study in Section 6 the effect of the menu size on the sample complexity.

Original: To the best of our knowledge, our paper presents the ﬁrst asymptotic characterization of the sample complexity for ranking from choice data in terms of the menu size under the very commonly used MNL model.
Revised: To the best of our knowledge, our paper presents the first asymptotic characterization of the sample complexity for ranking from choice data in terms of the menu size under the very commonly used MNL model.

Original: Operating on m-wise sorted data , Jang et al.
Revised: Operating on m-wise sorted data , Jang et al.

Original: [2017] showed that the optimal sample complexity for top- K recovery under the Plackett Luce model 1 scales with O ( 1 m ) .
Revised: [2017] showed that the optimal sample complexity for top- K recovery under the Plackett Luce model 1 scales with O ( 1 m ) .

Original: Our results complement theirs by showing that the sample complexity for top- K recovery from discrete choice data scales as O (1 + 1 m ) .
Revised: Our results complement theirs by showing that the sample complexity for top- K recovery from discrete choice data scales as O (1 + 1 m ) .

Original: Furthermore, the choice-based Borda count algorithm is different from the
Revised: Furthermore, the choice-based Borda count algorithm is different from the

Original: Spectral-MLE algorithm studied there which is specialized to the Plackett-Luce model.
Revised: Spectral-MLE algorithm studied there which is specialized to the Plackett-Luce model.

Original:  NOTATIONS AND PROBLEM FORMULATION
Revised:  NOTATIONS AND PROBLEM FORMULATION

Original: Let there be n items in the universe.
Revised: Let there be n items in the universe.

Original: Each item i has a deterministic and hidden utility, also referred to as partworth,
Revised: Each item i has a deterministic and hidden utility, also referred to as partworth,

Original: U i for i = 1 , . . . , n .
Revised: U i for i = 1 , . . . , n .

Original: Let us assume the non-degenerate case where no two items have identical partworths.
Revised: Let us assume the non-degenerate case where no two items have identical partworths.

Original: Without loss of generality, we also assume that U max
Revised: Without loss of generality, we also assume that U max

Original: =
Revised: =

Original: U 1 > U 2 > . . .
Revised: U 1 > U 2 > . . .

Original: > U n =
Revised: > U n =

Original: U min > 0 .
Revised: U min > 0 .

Original: Let S ∗ K = { 1 , . .
Revised: Let S ∗ K = { 1 , . .

Original: , K } denote the set of K items with the highest parthworths.
Revised: , K } denote the set of K items with the highest parthworths.

Original: Items are presented to the consumer in a set S , also referred to as menu, of size at least 2.
Revised: Items are presented to the consumer in a set S , also referred to as menu, of size at least 2.

Original: When S is presented to the consumer, the perceived utility of each item i ∈ S is the sum of its parthworth and a random noise term: X i =
Revised: When S is presented to the consumer, the perceived utility of each item i ∈ S is the sum of its parthworth and a random noise term: X i =

Original: U i + (cid:15) i where the (cid:15)
Revised: U i + ϵ

Original: i  ’s are independently and identically distributed according to an unknown universal noise distribution D .
Revised: i where the ϵ i ’s are independently and identically distributed according to an unknown universal noise distribution D .

Original: The consumer then picks the item i with the highest perceived utility among all the items in S .
Revised: The consumer then picks the item i with the highest perceived utility among all the items in S .

Original: Such a choice model is referred to as a random utility model with independent and identically distributed noise (IID-RUM).
Revised: Such a choice model is referred to as a random utility model with independent and identically distributed noise (IID-RUM).

Original: In short, a choice model ρ within the class of IID-RUMs is parametrized by a set of partworths { U 1 , . . .
Revised: In short, a choice model ρ within the class of IID-RUMs is parametrized by a set of partworths { U 1 , . . .

Original: U n } and noise distribution D .
Revised: U n } and noise distribution D .

Original: As an overload of notation, we will also use ρ ( i | S ) to denote the probability that a consumer picks item i from menu S under choice model ρ .
Revised: As an overload of notation, we will also use ρ ( i | S ) to denote the probability that a consumer picks item i from menu S under choice model ρ .

Original: By deﬁnition, ρ ( i | S ) =
Revised: By definition, ρ ( i | S ) =

Original: P ( X i >
Revised: P ( X i >

Original: X k ∀ k ∈ S \{ i } ) .
Revised: X k ∀ k ∈ S \{ i } ) .

Original: For simplicity, we consider a ﬁxed menu size m .
Revised: For simplicity, we consider a fixed menu size m .

Original: However, our analysis can be easily extended to account for a mixture of menu sizes.
Revised: However, our analysis can be easily extended to account for a mixture of menu sizes.

Original: A choice sample is a (menu, item) tuple ( S, y ) where the consumer chooses item y from menu S .
Revised: A choice sample is a (menu, item) tuple ( S, y ) where the consumer chooses item y from menu S .

Original: A choice dataset is a set of choice samples.
Revised: A choice dataset is a set of choice samples.

Original: A top- K recovery algorithm takes in a choice dataset and returns an estimate of the top
Revised: A top- K recovery algorithm takes in a choice dataset and returns an estimate of the top

Original: K items, ˆ S K .
Revised: K items, ˆ S K .

Original: The goal is to exactly recover the top K items and the performance metric of interest is the 0-1 loss:
Revised: The goal is to exactly recover the top K items and the performance metric of interest is the 0-1 loss:

Original: L 01 (ˆ S K , S [Equation] K ) = 1 [
Revised: L 01 (

Original: [Equation]
Revised: [Equation]

Original: S K = S [Equation] K ] 2 . We emphasize that as opposed to the top- K ranking problem, the objective of the top- K recovery prolem is to accurately identify the set of the top K items, while allowing for mis-ranking among these items.
Revised: We emphasize that as opposed to the top- K ranking problem, the objective of the top- K recovery prolem is to accurately identify the set of the top K items, while allowing for mis-ranking among these items.

Original:  THE CHOICE-BASED BORDA COUNT ALGORITHM
Revised:  THE CHOICE-BASED BORDA COUNT ALGORITHM

Original: As discussed previously, the general counting approach referred to as Borda count has a long history and has been instantiated in various contexts such as ranking from pairwise comparisons.
Revised: As discussed previously, the general counting approach referred to as Borda count has a long history and has been instantiated in various contexts such as ranking from pairwise comparisons.

Original: Here, we instantiate the Borda count approach to the more general discrete choice setting.
Revised: Here, we instantiate the Borda count approach to the more general discrete choice setting.

Original: This is shown in Algorithm 1.
Revised: This is shown in Algorithm 1.

Original: As would be expected, the algorithm essentially tallies the number of observed ‘wins’ by each item and ﬁnally ranking the items by their number of wins, returning the top K items.
Revised: As would be expected, the algorithm essentially tallies the number of observed ‘wins’ by each item and finally ranking the items by their number of wins, returning the top K items.

Original: As with other versions of the Borda count approach, the algorithm is simple and easy to implement; and very efﬁcient in practice.
Revised: As with other versions of the Borda count approach, the algorithm is simple and easy to implement; and very efficient in practice.

Original: This makes the choice-based Borda count algorithm appropriate in settings where the statistician is primarily interested in efﬁciently and accurately recovering the set of the top items from a large pool of choice data.Algorithm
Revised: This makes the choice-based Borda count algorithm appropriate in settings where the statistician is primarily interested in efficiently and accurately recovering the set of the top items from a large pool of choice data.

Original: Input: Choice dataset B = { ( S l , y l ) } Nl =
Revised: Input: Choice dataset B = { ( S l , y l ) } Nl =

Original: Output: Top- K estimate ˆ S K 1:
Revised: Output: Top- K estimate ˆ S K 1:

Original: For each item i = 1 , . . . , n 2:
Revised: For each item i = 1 , . . . , n 2:

Original: Compute the number of times i gets chosen: 3: ˆ W i :=
Revised: Compute the number of times i gets chosen: 3: ˆ W i :=

Original: (cid:80) Nl =1 1 [ y l = i ] 4:
Revised: (cid:80) Nl =1 1 [ y l = i ] 4:

Original: Return the set of K items corresponding to the highest ˆ
Revised: Return the set of K items corresponding to the highest ˆ

Original: W i ’s. Ties are broken arbitrarily.
Revised: W i ’s. Ties are broken arbitrarily.

Original:  SAMPLE COMPLEXITY BOUND
Revised:  SAMPLE COMPLEXITY BOUND

Original: In this section, we present the sample complexity of the choice-based Borda count algorithm for top- K recovery.
Revised: In this section, we present the sample complexity of the choice-based Borda count algorithm for top- K recovery.

Original: We ﬁrst formalize our sampling model in Section 4.1.
Revised: We first formalize our sampling model in Section 4.1.

Original: In Section 4.2, we characterize the class of IID-RUMs under which the choice-based Borda count can successfully identify all of the top K items via a theoretical quantity we term the generalized Borda score .
Revised: In Section 4.2, we characterize the class of IID-RUMs under which the choice-based Borda count can successfully identify all of the top K items via a theoretical quantity we term the generalized Borda score .

Original: The main theorems on the sample complexity of the choice-based Borda count algorithm are presented in Section 4.3.
Revised: The main theorems on the sample complexity of the choice-based Borda count algorithm are presented in Section 4.3.

Original: 4.1 THE SAMPLING MODEL
Revised: 4.1 THE SAMPLING MODEL

Original: Let C ( m ) be the set of all menus of size m ≥ 2 (i.e., | C ( m ) | = (cid:0) nm (cid:1) ).
Revised: Let C ( m ) be the set of all menus of size m ≥ 2 (i.e., | C ( m ) | = (cid:0) nm (cid:1) ).

Original: Additionally, let C ( m )
Revised: Additionally, let C ( m )

Original: i be the set of all menus of size m containing item
Revised: i be the set of all menus of size m containing item

Original: i (i.e., |C ( m )
Revised: i (i.e., |C ( m )

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: We consider a multiple-round uniform sampling model with
Revised: We consider a multiple-round uniform sampling model with

Original: R rounds of sampling in total.
Revised: R rounds of sampling in total.

Original: In each round r = 1 , . . .
Revised: In each round r = 1 , . . .

Original: , R , each menu S ∈ C ( m ) is independently offered with probability p > 0 .
Revised: , R , each menu S ∈ C ( m ) is independently offered with probability p > 0 .

Original: Let ˆ C ( m,r ) denote the set of menus of size m that are offered in round r .
Revised: Let ˆ C ( m,r ) denote the set of menus of size m that are offered in round r .

Original: If offered menu S , the user responds with a random choice y ( r ) S , where
Revised: If offered menu S , the user responds with a random choice y ( r ) S , where

Original: [Equation]
Revised: [Equation]

Original: It is easy to check that we have, in expectation, pR (cid:0) nm (cid:1) samples over R rounds.
Revised: It is easy to check that we have, in expectation, pR (cid:0) nm (cid:1) samples over R rounds.

Original: As a practical example, this sampling procedure can be used to design online political surveys.
Revised: As a practical example, this sampling procedure can be used to design online political surveys.

Original: Suppose that there are
Revised: Suppose that there are

Original: R voters willing to take part in answering survey questionnaires to determine support for n political candidates.
Revised: R voters willing to take part in answering survey questionnaires to determine support for n political candidates.

Original: Fix a ballot size m .
Revised: Fix a ballot size m .

Original: For each voter, each ballot of size m is independently presented to that user with probability p .
Revised: For each voter, each ballot of size m is independently presented to that user with probability p .

Original: For each ballot, the voter picks one favourite candidate.
Revised: For each ballot, the voter picks one favourite candidate.

Original: 4.2 THE GENERALIZED BORDA SCORE
Revised: 4.2 THE GENERALIZED BORDA SCORE

Original: For each item i , deﬁne the following theoretical quantity, which we term the generalized Borda score :
Revised: For each item i , define the following theoretical quantity, which we term the generalized Borda score :

Original: [Equation]
Revised: [Equation]

Original: Intuitively, the generalized Borda score is the expected probability that an item i is chosen from a menu S where S is uniformly sampled from C ( m ) i .
Revised: Intuitively, the generalized Borda score is the expected probability that an item i is chosen from a menu S where S is uniformly sampled from C ( m ) i .

Original: Note that τ ( m ) i ∈ [0 , 1] for all i ∈ [ n ] .
Revised: Note that τ ( m ) i ∈ [0 , 1] for all i ∈ [ n ] .

Original: The generalized Borda score is interesting to us because for a large class of IID-RUMs , the order among the items with respect to the generalized Borda scores is the same as that with respect to the partworths.
Revised: The generalized Borda score is interesting to us because for a large class of IID-RUMs , the order among the items with respect to the generalized Borda scores is the same as that with respect to the partworths.

Original: Therefore, it sufﬁces to rank the items by their generalized Borda scores to recover the items with the highest partworths.
Revised: Therefore, it suffices to rank the items by their generalized Borda scores to recover the items with the highest partworths.

Original: Formally, we can characterize this class of IID-RUMs using a property we term Borda consistency .
Revised: Formally, we can characterize this class of IID-RUMs using a property we term Borda consistency .

Original: Deﬁnition 1.
Revised: Definition 1.

Original: An IID-RUM ρ satisﬁes Borda consistency if for any two items i, j and any menu size m ≥ 2 ,
Revised: An IID-RUM ρ satisfies Borda consistency if for any two items i, j and any menu size m ≥ 2 ,

Original: [Equation]
Revised: [Equation]

Original: The follow lemma establishes that many commonly used
Revised: The follow lemma establishes that many commonly used

Original: IID-RUMs such as the MNL (Gumbel distributed noise) and the Probit (Normal distributed noise) model satisfy Borda consistency.
Revised: IID-RUMs such as the MNL (Gumbel distributed noise) and the Probit (Normal distributed noise) model satisfy Borda consistency.

Original: Lemma 4.1.
Revised: Lemma 4.1.

Original: All IID-RUMs whose noise distribution has absolutely continuous density function and support on the real line satisfy Borda consistency.
Revised: All IID-RUMs whose noise distribution has absolutely continuous density function and support on the real line satisfy Borda consistency.

Original: In the supplementary materials, we will show that Borda consistency is satisﬁed by an even broader class of IID-RUMs that include other commonly used models such as the IIDRUM with exponentially distributed noise.
Revised: In the supplementary materials, we will show that Borda consistency is satisfied by an even broader class of IID-RUMs that include other commonly used models such as the IIDRUM with exponentially distributed noise.

Original: Intuitively, this stems from the property enjoyed by many IID-RUMs: for any two items i, j where U
Revised: Intuitively, this stems from the property enjoyed by many IID-RUMs: for any two items i, j where U

Original: i > U j , ρ ( i | S ) >
Revised: i > U j , ρ ( i | S ) >

Original: ρ ( j | S ) ∀ S ∈
Revised: ρ ( j | S ) ∀ S ∈

Original: C ( m ) : i, j ∈ S ; and ρ ( i | S ∪ { i } ) >
Revised: C ( m ) : i, j ∈ S ; and ρ ( i | S ∪ { i } ) >

Original: ρ ( j | S ∪ { j } ) ∀ S ∈
Revised: ρ ( j | S ∪ { j } ) ∀ S ∈

Original: C ( m − 1) : i, j / ∈ S .
Revised: C ( m − 1) : i, j / ∈ S .

Original: To the best of our knowledge, this fundamental property that holds across a very broad class of
Revised: To the best of our knowledge, this fundamental property that holds across a very broad class of

Original: IID-RUMs has not been previously decribed in the literature and may be useful to future works exploring the intersection of ranking and choice modeling.
Revised: IID-RUMs has not been previously decribed in the literature and may be useful to future works exploring the intersection of ranking and choice modeling.

Original: 4.3 EXACT TOP- K RECOVERY
Revised: 4.3 EXACT TOP- K RECOVERY

Original: Having established Borda consistency as a property enjoyed by many IID-RUMs, we will now present the ﬁnite sample guarantees of the choice-based Borda count algorithm for top- K recovery that holds for all choice models in this broad class of IID-RUMs.
Revised: Having established Borda consistency as a property enjoyed by many IID-RUMs, we will now present the finite sample guarantees of the choice-based Borda count algorithm for top- K recovery that holds for all choice models in this broad class of IID-RUMs.

Original: To show that the choice-based Borda count algorithm accurately identiﬁes all of the top K items with high probability, it sufﬁces to bound the probability that the algorithm mistakenly ranks an item j / ∈ S ∗ K higher than another item i ∈ S ∗ K .
Revised: To show that the choice-based Borda count algorithm accurately identifies all of the top K items with high probability, it suffices to bound the probability that the algorithm mistakenly ranks an item j / ∈ S ∗ K higher than another item i ∈ S ∗ K .

Original: Speciﬁcally, we want to bound the following probabilities.
Revised: Specifically, we want to bound the following probabilities.

Original: [Equation]
Revised: [Equation]

Original: where ˆ W i is deﬁned in Algorithm (1).
Revised: where ˆ W i is defined in Algorithm (1).

Original: Considering this, the fundamental hardness of top- K ranking lies in distinguishing between the K -th and K + 1 -th best item, and therefore depends on the gap between their generalized Borda scores:
Revised: Considering this, the fundamental hardness of top- K ranking lies in distinguishing between the K -th and K + 1 -th best item, and therefore depends on the gap between their generalized Borda scores:

Original: [Equation]
Revised: [Equation]

Original: The smaller this gap, the more data the algorithm requires in order to correctly separate between the top K and the bottom n − K items.
Revised: The smaller this gap, the more data the algorithm requires in order to correctly separate between the top K and the bottom n − K items.

Original: Building on this intuition and generalizing to any pair of items ( i, j ) where τ i > τ j , we obtain the following upper bound on P ( ˆ W j > ˆ W i ) .
Revised: Building on this intuition and generalizing to any pair of items ( i, j ) where τ i > τ j , we obtain the following upper bound on P ( ˆ W j > ˆ W i ) .

Original: Lemma 4.2.
Revised: Lemma 4.2.

Original: Consider an IID-RUM that satisﬁes Borda consistency per Deﬁnition 1.
Revised: Consider an IID-RUM that satisfies Borda consistency per Definition 1.

Original: Assume input choice data with menu size m is generated according to the sampling model described in Section 4.1.
Revised: Assume input choice data with menu size m is generated according to the sampling model described in Section 4.1.

Original: For any two items i and j where
Revised: For any two items i and j where

Original: [Equation]

Original: m )

Original: i > τ

Original: [Equation]
Revised: [Equation]

Original: , the choice-based Borda count algorithm satisﬁes
Revised: , the choice-based Borda count algorithm satisfies

Original: [Equation]
Revised: [Equation]

Original: The proof of Lemma 4.2 uses a standard concentration inequality argument based on Bernstein’s inequality (cf.
Revised: The proof of Lemma 4.2 uses a standard concentration inequality argument based on Bernstein’s inequality (cf.

Original: Theorem 2.8.4 Vershynin [2018]).
Revised: Theorem 2.8.4 Vershynin [2018]).

Original: The lemma itself states that, for each pair i ∈ S ∗ K , j / ∈ S ∗ K , if pR (cid:0) nm (cid:1) ≥ 8 n [Equation] τ 
Revised: The lemma itself states that, for each pair i ∈ S ∗ K , j / ∈ S ∗ K , if pR (cid:0) nm (cid:1) ≥ 8 n log n ( τ ( m ) i + τ (

Original: [Equation]
Revised: [Equation]

Original: + [Equation] − τ ( m ) j ) 2 , then P ( ˆ W j > ˆ W i ) =
Revised: ) j ) m ( τ ( m ) i − τ ( m ) j ) 2 , then P ( ˆ W j > ˆ W i ) =

Original: O ( 1 n 3 ) .
Revised: O ( 1 n 3 ) .

Original: We also have the following lemma which presents an upper bound on the item-dependent term τ ( m )i +
Revised: We also have the following lemma which presents an upper bound on the item-dependent term τ ( m )

Original: [Equation]

Original: j
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Lemma 4.3.
Revised: Lemma 4.3.

Original: Consider an IID-RUM that satisﬁes Borda consistency per Deﬁnition 1.
Revised: Consider an IID-RUM that satisfies Borda consistency per Definition 1.

Original: For any K , we have 
Revised: For any K , we have τ ( m ) K

Original: [Equation]
Revised: [Equation]

Original: By combining the two lemmas above and applying union bound over all pairs i ∈ S ∗ K , j / ∈ S ∗ K , we obtain the following sample complexity bound for exact top- K recovery:
Revised: By combining the two lemmas above and applying union bound over all pairs i ∈ S ∗ K , j / ∈ S ∗ K , we obtain the following sample complexity bound for exact top- K recovery:

Original: Theorem 4.4.
Revised: Theorem 4.4.

Original: Assume the conditions of lemma (4.2).
Revised: Assume the conditions of lemma (4.2).

Original: Given sufﬁciently large p, R such that pR (cid:0) n 
Revised: Given sufficiently large p, R such that pR (cid:0) n m

Original: [Equation]
Revised: [Equation]

Original: ≥

Original: [Equation] n [Equation] 2 τ ( m ) K +1 ) , the choice-based Borda count algorithm correctly identiﬁes all of the top K items with probability at least 1 − O
Revised: 2 τ ( m ) K +1 ) , the choice-based Borda count algorithm correctly identifies all of the top K items with probability at least 1 − O

Original: ( Kn 2 ) .
Revised: ( Kn 2 ) .

Original: The reader may also recognize that ∆ ( m ) K +2 τ ( m ) K +1 is simply τ ( m ) K + τ ( m ) K +1 .
Revised: The reader may also recognize that ∆ ( m ) K +2 τ ( m ) K +1 is simply τ ( m ) K + τ ( m ) K +1 .

Original: The former presentation is, however, useful in highlighting the main quantities that will also reappear in our matching lower bound.
Revised: The former presentation is, however, useful in highlighting the main quantities that will also reappear in our matching lower bound.

Original: In summary, the choice-based
Revised: In summary, the choice-based

Original: Borda count algorithm has the following sample complexity for exact top- K recovery:
Revised: Borda count algorithm has the following sample complexity for exact top- K recovery:

Original: [Equation]
Revised: [Equation]

Original: This shows that overall, we only need O ( n log n ) examples to recover the top K items from choice data with high accuracy.
Revised: This shows that overall, we only need O ( n log n ) examples to recover the top K items from choice data with high accuracy.

Original: Our upper bound (and matching lower bound to be shown) can be seen as both generalization and reﬁnement of Theorem 1 of Shah and Wainwright [2017].
Revised: Our upper bound (and matching lower bound to be shown) can be seen as both generalization and refinement of Theorem 1 of Shah and Wainwright [2017].

Original: Under the pairwise comparison setting ( m = 2 ), we can simply upper bound τ ( m ) K +1 ≤ 1 and recover the (optimal) sample complexity O (cid:0) n log n ∆ (2) K 2 (cid:1) of Borda count obtained by Shah and
Revised: Under the pairwise comparison setting ( m = 2 ), we can simply upper bound τ ( m ) K +1 ≤ 1 and recover the (optimal) sample complexity O (cid:0) n log n ∆ (2) K 2 (cid:1) of Borda count obtained by Shah and

Original: Wainwright [2017].
Revised: Wainwright [2017].

Original: The analysis approach there, however, is insufﬁcient to produce an optimal sample complexity bound in the discrete choice setting.
Revised: The analysis approach there, however, is insufficient to produce an optimal sample complexity bound in the discrete choice setting.

Original: Note also that there can be combinatorially many realizations of the data in the discrete choice setting as |C ( m ) |
Revised: Note also that there can be combinatorially many realizations of the data in the discrete choice setting as |C ( m ) |

Original: =
Revised: =

Original: (cid:0) nm (cid:1) .
Revised: (cid:0) nm (cid:1) .

Original: Our proof therefore requires considerably more effort.
Revised: Our proof therefore requires considerably more effort.

Original: Our bound also shows that the sample complexity depends not only on the gap ∆ ( m ) K between the K -th and K + 1 -th item, but also the relative ‘strength’ of the K + 1 -th item, as captured by the τ ( m ) K + ∆ ( m ) K term.
Revised: Our bound also shows that the sample complexity depends not only on the gap ∆ ( m ) K between the K -th and K + 1 -th item, but also the relative ‘strength’ of the K + 1 -th item, as captured by the τ ( m ) K + ∆ ( m ) K term.

Original: In general, the factors τ ( m ) K +1 and ∆ ( m ) K don’t admit closed form expressions because both are sums of (cid:0) n − 1 m − 1 (cid:1) terms.
Revised: In general, the factors τ ( m ) K +1 and ∆ ( m ) K don’t admit closed form expressions because both are sums of (cid:0) n − 1 m − 1 (cid:1) terms.

Original: The reader may also recognize that these parameters also depend on the menu size m , the partworth parameters and the noise distribution.
Revised: The reader may also recognize that these parameters also depend on the menu size m , the partworth parameters and the noise distribution.

Original: In the next section, we will show a matching lower bound in terms of the same parameters, establishing the optimality of the choice-based Borda count algorithm, and discuss why the exact relation between ∆ ( m ) K , τ ( m ) K +1 and the model parameters remains elusive.
Revised: In the next section, we will show a matching lower bound in terms of the same parameters, establishing the optimality of the choice-based Borda count algorithm, and discuss why the exact relation between ∆ ( m ) K , τ ( m ) K +1 and the model parameters remains elusive.

Original: Often in practice, we may tolerate some error for top- K ranking by allowing the algorithm to misidentify, up to a threshold, some number of items.
Revised: Often in practice, we may tolerate some error for top- K ranking by allowing the algorithm to misidentify, up to a threshold, some number of items.

Original: This is known as approximate top- K recovery .
Revised: This is known as approximate top- K recovery .

Original: We include detailed discussions of this problem in the supplementary materials and show that the choice-based Borda count algorithm also has optimal sample complexity for approximate top- K recovery under the broad class of IID-RUMs that satisfy Borda consistency.
Revised: We include detailed discussions of this problem in the supplementary materials and show that the choice-based Borda count algorithm also has optimal sample complexity for approximate top- K recovery under the broad class of IID-RUMs that satisfy Borda consistency.

Original:  INFORMATION-THEORETIC LOWER BOUND
Revised:  INFORMATION-THEORETIC LOWER BOUND

Original: In this section, we will show that the choice-based Borda count algorithm enjoys optimal sample complexity by furnishing a matching lower bound.
Revised: In this section, we will show that the choice-based Borda count algorithm enjoys optimal sample complexity by furnishing a matching lower bound.

Original: To show a lower bound, we will construct a special subclass of the MNL family where any estimator requires Ω( n log n ) examples in order to exactly recover the top K items.
Revised: To show a lower bound, we will construct a special subclass of the MNL family where any estimator requires Ω( n log n ) examples in order to exactly recover the top K items.

Original: We defer detailed descriptions of this model to the supplementary materials while stating the main results as follows.
Revised: We defer detailed descriptions of this model to the supplementary materials while stating the main results as follows.

Original: Theorem 5.1.
Revised: Theorem 5.1.

Original: Consider the sampling model described in
Revised: Consider the sampling model described in

Original: Section 4.1.
Revised: Section 4.1.

Original: There exists a class of MNL models such that for n ≥ 20 , if pR (cid:0) nm (cid:1) ≤ n log n
Revised: There exists a class of MNL models such that for n ≥ 20 , if pR (cid:0) nm (cid:1) ≤ n log n

Original: [Equation]
Revised: [Equation]

Original: then any estimator fails to correctly identify all of the top K items with probability at least 112 .
Revised: then any estimator fails to correctly identify all of the top K items with probability at least 112 .

Original: The proof of Theorem (5.1) ﬁrst reduces the problem of exact top- K recovery to a multiple hypothesis testing problem and then applies Fano’s lemma [Cover, 1999].
Revised: The proof of Theorem (5.1) first reduces the problem of exact top- K recovery to a multiple hypothesis testing problem and then applies Fano’s lemma [Cover, 1999].

Original: Each hypothesis in the testing problem corresponds to an MNL model.
Revised: Each hypothesis in the testing problem corresponds to an MNL model.

Original: Within each model, the set of the top K items always includes items 1 , . .
Revised: Within each model, the set of the top K items always includes items 1 , . .

Original: , K − 1 .
Revised: , K − 1 .

Original: However, the index of remaining item in the top- K set is different for each model (i.e., there are n − K + 1 different models).
Revised: However, the index of remaining item in the top- K set is different for each model (i.e., there are n − K + 1 different models).

Original: We make all of the top K items have the same partworths while the bottom n − K items have the same (and lower) partworths.
Revised: We make all of the top K items have the same partworths while the bottom n − K items have the same (and lower) partworths.

Original: The key challenge then is to obtain a tight upper bound on the
Revised: The key challenge then is to obtain a tight upper bound on the

Original: KL divergence between any two hypothesis models.
Revised: KL divergence between any two hypothesis models.

Original: In summary, Theorem (5.1) implies the following minimum sample complexity for any algorithm for top-
Revised: In summary, Theorem (5.1) implies the following minimum sample complexity for any algorithm for top-

Original: K recovery:
Revised: K recovery:

Original: [Equation]
Revised: [Equation]

Original: Comparing with the bound in Theorem 4.4, one can see that the sample complexity of Borda Count is optimal in terms of both m , n as well as the model dependent parameters ∆ ( m ) K and τ ( m ) K +1 .
Revised: Comparing with the bound in Theorem 4.4, one can see that the sample complexity of Borda Count is optimal in terms of both m , n as well as the model dependent parameters ∆ ( m ) K and τ ( m ) K +1 .

Original:  THE ROLE OF THE MENU SIZE m
Revised:  THE ROLE OF THE MENU SIZE m

Original: The effect of the menu size on the performance of top- K recovery algorithms is an aspect of both theoretical and practical importance.
Revised: The effect of the menu size on the performance of top- K recovery algorithms is an aspect of both theoretical and practical importance.

Original: In real life applications, the menu size could range from 2 to hundreds of items.
Revised: In real life applications, the menu size could range from 2 to hundreds of items.

Original: One may suspect that increasing the menu size means the data carries more information per data point, and thereby reduces the sample complexity for top- K recovery.
Revised: One may suspect that increasing the menu size means the data carries more information per data point, and thereby reduces the sample complexity for top- K recovery.

Original: However, to the best of our knowledge, such a relationship has not been theoretically established in the literature on choice modeling, even for the very commonly used MNL model.
Revised: However, to the best of our knowledge, such a relationship has not been theoretically established in the literature on choice modeling, even for the very commonly used MNL model.

Original: As seen in the matching lower and upper bound for the sample complexity of top- K recovery, the menu size enters in complex ways through the factors ∆ ( m ) K and τ ( m ) K +1 .
Revised: As seen in the matching lower and upper bound for the sample complexity of top- K recovery, the menu size enters in complex ways through the factors ∆ ( m ) K and τ ( m ) K +1 .

Original: Both factors can vary in subtle ways with m , depending on the underlying choice model.
Revised: Both factors can vary in subtle ways with m , depending on the underlying choice model.

Original: Even for the class of MNL models which admit closed form choice probabilities, these factors don’t seem to have a closed form expression as each of them is a sum of (cid:0) n − 1 m − 1 (cid:1) terms.
Revised: Even for the class of MNL models which admit closed form choice probabilities, these factors don’t seem to have a closed form expression as each of them is a sum of (cid:0) n − 1 m − 1 (cid:1) terms.

Original: To bypass the difﬁculty of exactly evaluating ∆ ( m ) K and τ ( m ) K +1 , we characterize the asymptotic dependency of 1 m ∆ ( m ) K and τ ( m ) K + ∆ ( m ) K on m under the MNL class of models and show that both of these factors monotonically decrease with m but at a decreasing rate .
Revised: To bypass the difficulty of exactly evaluating ∆ ( m ) K and τ ( m ) K +1 , we characterize the asymptotic dependency of 1 m ∆ ( m ) K and τ ( m ) K + ∆ ( m ) K on m under the MNL class of models and show that both of these factors monotonically decrease with m but at a decreasing rate .

Original: This implies that while there is an advantage to using choice data of larger menu sizes, there is a diminishing return to increasing the menu size.
Revised: This implies that while there is an advantage to using choice data of larger menu sizes, there is a diminishing return to increasing the menu size.

Original: Theorem 6.1.
Revised: Theorem 6.1.

Original: For any MNL model and a ﬁxed K ,
Revised: For any MNL model and a fixed K ,

Original: [Equation]
Revised: [Equation]

Revised: It can be seen in both 1 m

Revised: [Equation]

Revised: and

Original: It can be seen in both 1 m ∆ ( m ) K and τ ( m ) K + ∆ ( m ) K that the term which depends on m , 1 + 1 m − 1 , montonically decreases with m but at a diminishing rate.
Revised: [Equation] ∆ [Equation] that the term which depends on m , 1 + 1 m − 1 , montonically decreases with m but at a diminishing rate.

Original: Combining the above theorem and the matching sample complexity bounds obtained earlier, one can see that the optimal sample complexity for top-
Revised: Combining the above theorem and the matching sample complexity bounds obtained earlier, one can see that the optimal sample complexity for top-

Original: K recovery from choice data scales as θ (1 + 1 m ) .
Revised: K recovery from choice data scales as θ (1 + 1 m ) .

Original: Outside of the MNL family of models, we are not aware of any IID-RUM that admits a closed form expression for the choice probabilities.
Revised: Outside of the MNL family of models, we are not aware of any IID-RUM that admits a closed form expression for the choice probabilities.

Original: However, suppose that we know all of the partworths and the noise distribution, we can still approximate the choice probabilities via Monte Carlo sampling.
Revised: However, suppose that we know all of the partworths and the noise distribution, we can still approximate the choice probabilities via Monte Carlo sampling.

Original: Given these (approximated) choice probabilities, one can then evaluate
Revised: Given these (approximated) choice probabilities, one can then evaluate

Original: [Equation]
Revised: [Equation]

Original: As an example,
Revised: an example,

Original: Figure 1 shows how these quantities vary with m under a randomly generated MNL and Probit model (IID-RUM with standard normal noise)
Revised: Figure 1 shows how these quantities vary with m under a randomly generated MNL and Probit model (IID-RUM with standard normal noise)

Original: [Train, 2009] with n = 15 , K = 3 .
Revised: [Train, 2009] with n = 15 , K = 3 .

Original: The partworths were independently generated from a zeromean normal distribution which is also a commonly chosen prior in the literature [Parkes et al., 2012, Train, 2009].
Revised: The partworths were independently generated from a zeromean normal distribution which is also a commonly chosen prior in the literature [Parkes et al., 2012, Train, 2009].

Original: The curves for 
Revised: The curves for 1 m

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original:  decrease at a rate approximately similar to those of the MNL model as stated in Theorem 6.1.
Revised: K decrease at a rate approximately similar to those of the MNL model as stated in Theorem 6.1.

Original: Ranking from choice data under MNL model assumptions remains an active area of research [Agarwal et al., 2018, 2020] and to the best of our knowledge, our work presents the ﬁrst asymptotic characterization of the optimal sample complexity for top- K recovery in terms of the menu size m under this often used class of choice models.
Revised: Ranking from choice data under MNL model assumptions remains an active area of research [Agarwal et al., 2018, 2020] and to the best of our knowledge, our work presents the first asymptotic characterization of the optimal sample complexity for top- K recovery in terms of the menu size m under this often used class of choice models.

Original: [Figure]
Revised: [Figure]

Original: decrease with larger m under a randomly generated MNL and Probit model.
Revised: decrease with larger m under a randomly generated MNL and Probit model.

Original: This suggests that there is an advantage, albeit with dimnishing return, to using larger menu sizes.
Revised: This suggests that there is an advantage, albeit with dimnishing return, to using larger menu sizes.

Original:  CONNECTIONS TO COMMONLY USED RANKING ALGORITHMS
Revised:  CONNECTIONS TO COMMONLY USED RANKING ALGORITHMS

Original: In this section, we establish close connections among choicebased Borda count, the method of maximum likelihood estimate under MNL assumptions (MNL-MLE)
Revised: In this section, we establish close connections among choicebased Borda count, the method of maximum likelihood estimate under MNL assumptions (MNL-MLE)

Original: [Train, 2009] and Spectral Ranking [Negahban et al., 2017, Maystre and
Revised: [Train, 2009] and Spectral Ranking [Negahban et al., 2017, Maystre and

Original: Grossglauser, 2015, Agarwal et al., 2018] which will explain many experimental results we present in later sections.
Revised: Grossglauser, 2015, Agarwal et al., 2018] which will explain many experimental results we present in later sections.

Original: Firstly, one can prove that choice-based Borda count and
Revised: Firstly, one can prove that choice-based Borda count and

Original: MNL-MLE are ‘equivalent’ top-
Revised: MNL-MLE are ‘equivalent’ top-

Original: K recovery algorithms in the limit of inﬁnite data.
Revised: K recovery algorithms in the limit of infinite data.

Original: This connection is formalized as follows.
Revised: This connection is formalized as follows.

Original: Theorem 7.1.
Revised: Theorem 7.1.

Original: Consider the sampling model described in
Revised: Consider the sampling model described in

Original: Section 4.1, for any p > 0 , in the limit as R → ∞ , MNLMLE and choice-based Borda count will produce the same top- K estimate.
Revised: Section 4.1, for any p > 0 , in the limit as R → ∞ , MNLMLE and choice-based Borda count will produce the same top- K estimate.

Original: Moreover, this holds even if the data does not come from the MNL model or any IID-RUM.
Revised: Moreover, this holds even if the data does not come from the MNL model or any IID-RUM.

Original: A similar observation was made in Rajkumar and Agarwal [2014]: under the pairwise comparison setting the Borda count algorithm and MNL-MLE are both consistent for full ranking under a class of pairwise comparison models that is strictly more general than the BTL model 3 .
Revised: A similar observation was made in Rajkumar and Agarwal [2014]: under the pairwise comparison setting the Borda count algorithm and MNL-MLE are both consistent for full ranking under a class of pairwise comparison models that is strictly more general than the BTL model 3 .

Original: Our results generalize the relation between the two algorithms to the choice setting and show that in fact the Borda count algorithm and MNL-MLE produce the same estimate in the limit of inﬁnite data under any choice models.
Revised: Our results generalize the relation between the two algorithms to the choice setting and show that in fact the Borda count algorithm and MNL-MLE produce the same estimate in the limit of infinite data under any choice models.

Original: This connection between the two methods is reﬂected in our experiments where the performance of the choice-based
Revised: This connection between the two methods is reflected in our experiments where the performance of the choice-based

Original: Borda count algorithm is almost identical to that of MLE, when the sample size is large.
Revised: Borda count algorithm is almost identical to that of MLE, when the sample size is large.

Original: While performing similarly to MNL-MLE, the choice-based Borda count algorithm is several orders of magnitude faster thanks to its simplicity.
Revised: While performing similarly to MNL-MLE, the choice-based Borda count algorithm is several orders of magnitude faster thanks to its simplicity.

Original: This suggests that if the statistician is mostly concerned with recovering a small number of top items, the choicebased Borda count algorithm should be seriously considered due to its speed, simplicity and guaranteed optimal sample complexity.
Revised: This suggests that if the statistician is mostly concerned with recovering a small number of top items, the choicebased Borda count algorithm should be seriously considered due to its speed, simplicity and guaranteed optimal sample complexity.

Original: The above result also means that MNL-MLE is a consistent top- K ranking algorithm under the broad class of IIDRUMs, since the choice-based Borda count algorithm is consistent in recovering the top K items.
Revised: The above result also means that MNL-MLE is a consistent top- K ranking algorithm under the broad class of IIDRUMs, since the choice-based Borda count algorithm is consistent in recovering the top K items.

Original: This shows that
Revised: This shows that

Original: MNL-MLE may be used for ranking applications even when the data does not satisfy the MNL assumption.
Revised: MNL-MLE may be used for ranking applications even when the data does not satisfy the MNL assumption.

Original: Consistency of MLE under model misspeciﬁcation is an underexplored question and we leave the careful characterization of the sample complexity of MNL-MLE when the data comes from a non-MNL distribution as a subject of future studies.
Revised: Consistency of MLE under model misspecification is an underexplored question and we leave the careful characterization of the sample complexity of MNL-MLE when the data comes from a non-MNL distribution as a subject of future studies.

Original: On the other hand, Spectral Ranking does not in general produce the same top- K estimate as MNL-MLE/choicebased Borda count.
Revised: On the other hand, Spectral Ranking does not in general produce the same top- K estimate as MNL-MLE/choicebased Borda count.

Original: However, when the underlying choice model falls within a broad class of IID-RUMs which include many commonly used choice models such as the MNL and
Revised: However, when the underlying choice model falls within a broad class of IID-RUMs which include many commonly used choice models such as the MNL and

Original: Probit model, all three algorithms produce the same estimate given inﬁnite data.
Revised: Probit model, all three algorithms produce the same estimate given infinite data.

Original: Theorem 7.2.
Revised: Theorem 7.2.

Original: Consider the sampling model described in
Revised: Consider the sampling model described in

Original: Section 4.1.
Revised: Section 4.1.

Original: Assume that the underlying choice model generating the data is in the class of IID-RUMs whose noise distribution has absolutely continuous density function with support on the real line.
Revised: Assume that the underlying choice model generating the data is in the class of IID-RUMs whose noise distribution has absolutely continuous density function with support on the real line.

Original: For any p > 0 , in the limit as
Revised: For any p > 0 , in the limit as

Original: R → ∞ , then Spectral Ranking, MNL-MLE and choicebased Borda count produce the same top- K estimate.
Revised: R → ∞ , then Spectral Ranking, MNL-MLE and choicebased Borda count produce the same top- K estimate.

Original: On the other hand, there exists a choice model where in the limit as R → ∞ , the Spectral Ranking algorithm produces a different top- K estimate from MNL-MLE/Borda count.
Revised: On the other hand, there exists a choice model where in the limit as R → ∞ , the Spectral Ranking algorithm produces a different top- K estimate from MNL-MLE/Borda count.

Original:  EXPERIMENTS
Revised:  EXPERIMENTS

Original: In this section, we present experiment results on both synthetic and real datasets.
Revised: In this section, we present experiment results on both synthetic and real datasets.

Original: The main performance metric is top- K accuracy.
Revised: The main performance metric is top- K accuracy.

Original: More speciﬁcally, we measure top- K accuracy as the frequency at which the respective algorithm correctly identiﬁes all of the true top K items, overtrials .
Revised: More specifically, we measure top- K accuracy as the frequency at which the respective algorithm correctly identifies all of the true top K items, overtrials .

Original: 8.1 SYNTHETIC DATA
Revised: 8.1 SYNTHETIC DATA

Original: We verify, via synthetic experiments, the efﬁcacy of the choice-based Borda count algorithm and the effect of the menu size m on its performance.
Revised: We verify, via synthetic experiments, the efficacy of the choice-based Borda count algorithm and the effect of the menu size m on its performance.

Original: Let there be n = 50 items in the universe.
Revised: Let there be n = 50 items in the universe.

Original: We experiment with 3 different noise distributions: standard Normal noise (Probit), standard Gumbel noise (MNL) and standard Exponential noise.
Revised: We experiment with 3 different noise distributions: standard Normal noise (Probit), standard Gumbel noise (MNL) and standard Exponential noise.

Original: We vary the menu size m = 2 , 4 , 6 , 8 and K = 1 , 3 , 5 .
Revised: We vary the menu size m = 2 , 4 , 6 , 8 and K = 1 , 3 , 5 .

Original: Figure 2 shows top-
Revised: Figure 2 shows top-

Original: K accuracy against the sample size.
Revised: K accuracy against the sample size.

Original: In all experiments, choice-based Borda count successfully identiﬁes the top K items with high probability given sufﬁciently large sample size.
Revised: In all experiments, choice-based Borda count successfully identifies the top K items with high probability given sufficiently large sample size.

Original: Furthermore, using larger menu sizes improves the performance of Borda Count.
Revised: Furthermore, using larger menu sizes improves the performance of Borda Count.

Original: However, it can be seen that there is a diminishing return in performance gains from using larger menu sizes, agreeing with our theoretical analysis in
Revised: However, it can be seen that there is a diminishing return in performance gains from using larger menu sizes, agreeing with our theoretical analysis in

Original: Section 5.
Revised: Section 5.

Original: 8.2 REAL DATA
Revised: 8.2 REAL DATA

Original: Baseline algorithms: We compare choice-based Borda count against Accelerated Spectral Ranking (ASR)
Revised: Baseline algorithms: We compare choice-based Borda count against Accelerated Spectral Ranking (ASR)

Original: [Agarwal et al., 2018] and Maximum Likelihood Estimate (MLE) under MNL assumptions [Train, 2009] in terms of top K accuracy.
Revised: [Agarwal et al., 2018] and Maximum Likelihood Estimate (MLE) under MNL assumptions [Train, 2009] in terms of top K accuracy.

Original: We implement MLE using Scipy’s L-BFGS optimizer [Virtanen et al., 2020].
Revised: We implement MLE using Scipy’s L-BFGS optimizer [Virtanen et al., 2020].

Original: Data description: We follow standard procedures commonly used in previous works such as Rajkumar and Agarwal [2014], Agarwal et al.
Revised: Data description: We follow standard procedures commonly used in previous works such as Rajkumar and Agarwal [2014], Agarwal et al.

Original: Operating on full ranking datasets, we can estimate the choice probabilities for any menu S , i.e., choice probability ρ ( i | S ) is the proportion of rankings that ranks item i highest among all them items in
Revised: Operating on full ranking datasets, we can estimate the choice probabilities for any menu S , i.e., choice probability ρ ( i | S ) is the proportion of rankings that ranks item i highest among all them items in

Original: S .
Revised: S .

Original: Given these probabilities, we can simulate the sampling model as described in Section 4.1.
Revised: Given these probabilities, we can simulate the sampling model as described in Section 4.1.

Original: Our datasets include
Revised: Our datasets include

Original: SUSHI [Kamishima, 2003], APA election dataset [Diaconis, 1989], 3 Irish election datsets and F1 race dataset included in the library PrefLib
Revised: SUSHI [Kamishima, 2003], APA election dataset [Diaconis, 1989] [Table]

Original:  [Mattei and Walsh, 2013].
Revised: in the library PrefLib [Mattei and Walsh, 2013].

Original: Notably, the induced pairwise choice probabilities of these datasets all satisfy stochastic transitivity.
Revised: Notably, the induced pairwise choice probabilities of these datasets all satisfy stochastic transitivity.

Original: Therefore, there exists a universal ordering of the items which we can use as a true global ranking over the items.
Revised: Therefore, there exists a universal ordering of the items which we can use as a true global ranking over the items.

Original: Due to space constraint, we can only present a few representative experimental ﬁndings and leave additional results with detailed descriptions of data processing in the supplementary materials.
Revised: Due to space constraint, we can only present a few representative experimental findings and leave additional results with detailed descriptions of data processing in the supplementary materials.

Original: Speed advantage: Across all experiments, choice-based
Revised: Speed advantage: Across all experiments, choice-based

Original: Borda count is several orders of magnitude faster than
Revised: Borda count is several orders of magnitude faster than

Original: ASR and MLE.
Revised: ASR and MLE.

Original: This difference is especially pronounced in datasets with more items such as the F1 dataset, as shown in Figure 3.
Revised: This difference is especially pronounced in datasets with more items such as the F1 dataset, as shown in Figure 3.

Original: Competitive accuracy: Figure 4 show the performance of the algorithms under the Irish-Meath dataset and Figureshows the results for the Irish-West dataset.
Revised: Competitive accuracy: Figure 4 show the performance of the algorithms under the Irish-Meath dataset and Figureshows the results for the Irish-West dataset.

Original: Our theoretical analysis in Section 7 is reﬂected in our experimental ﬁndings: the performance of MNL-MLE and the Borda count algorithm are very similar given sufﬁciently large sample size.
Revised: Our theoretical analysis in Section 7 is reflected in our experimental findings: the performance of MNL-MLE and the Borda count algorithm are very similar given sufficiently large sample size.

Original: Spectral Ranking, on the other hand, may perform better or worse than MLE/Borda count depending on the dataset and the choice of m and K .
Revised: Spectral Ranking, on the other hand, may perform better or worse than MLE/Borda count depending on the dataset and the choice of m and K .

Original: For many combinations of m and K , we observe that the choice-based Borda count algorithm accurately recovers the top K items and is highly competitive with MNL-MLE and Spectral Ranking.
Revised: For many combinations of m and K , we observe that the choice-based Borda count algorithm accurately recovers the top K items and is highly competitive with MNL-MLE and Spectral Ranking.

Original: Notably, in most datasets, for smaller K and large m , ithe Borda count algorithm has considerable advantages thanks to its accuracy and faster running time.
Revised: Notably, in most datasets, for smaller K and large m , ithe Borda count algorithm has considerable advantages thanks to its accuracy and faster running time.

Original: In practice, this means choice-based Borda count is appropriate for applications where the statistician is interested in quickly determining a single (or a few) top candidate(s) from a large amount of data such as aggregating political surveys.
Revised: In practice, this means choice-based Borda count is appropriate for applications where the statistician is interested in quickly determining a single (or a few) top candidate(s) from a large amount of data such as aggregating political surveys.

Original:  CONCLUSION
Revised:  CONCLUSION

Original: Ranking under Random Utility Models is a promising area of research with many practical applications.
Revised: Ranking under Random Utility Models is a promising area of research with many practical applications.

Original: Our work shows how an efﬁcient algorithm can perform very well
Revised: Our work shows how an efficient algorithm can perform very well

Original: Meath dataset ( n = 14 ): Exact top- K accuracy against sample size.
Revised: - curacy against sample size.

Original: choice-based Borda count (blue) is competitive with baseline algorithms.
Revised: choice-based Borda count (blue) is competitive with baseline algorithms.

Original: Using larger menu sizes generally improves the performance of the algorithms.
Revised: Using larger menu sizes generally improves the performance of the algorithms.

Original: under a broad family of RUMs.
Revised: under a broad family of RUMs.

Original: That being said, the class of IID-RUMs constitutes only a subset of models within the class of general RUMs.
Revised: That being said, the class of IID-RUMs constitutes only a subset of models within the class of general RUMs.

Original: Beyond IID-RUMs, not much is known in terms of efﬁcient inference and ranking algorithms.
Revised: Beyond IID-RUMs, not much is known in terms of efficient inference and ranking algorithms.

Original: In the future we hope to see more ranking methods developed for more expressive RUMs which have non-identical noise distributions or dependent noise distributions
Revised: In the future we hope to see more ranking methods developed for more expressive RUMs which have non-identical noise distributions or dependent noise distributions.

Original: Acknowledgements
Revised: Acknowledgements

Original: The author thanks Shivani Agarwal, Prathamesh Patil and 
Revised: The author thanks Shivani Agarwal for suggesting the idea of generalizing the Borda count algorithm to the choice setting, proofreading earlier versions of this paper and for helpful discussions. The author also thanks Prathamesh Patil and William Zhang for proofreading this paper; and the anonymous reviewers for their comments. This work is supported in part by the National Science Foundation (NSF) under grant number 1717290 (awarded to Shivani Agarwal)

Original: William Zhang for proofreading this paper  and helpful discussions; and the anonymous reviewers for their comments.
Revised: Any opinions expressed in this paper are those of the author and do not necessarily reflect the views of the National Science Foundation.