From: auai.org/UAI/2022/Conference/Program_Chairs
title: Paper Decision
decision: Accept (Poster)
comment: Meta Review: This paper proposed a \ell_infty norm bound of the maximum likelihood estimator in the BTL Model under general comparison graphs. The result is a good addition to existing \ell_2 norm bounds. While one reviewer still has a reservation about the presentation of the paper, all the reviewers agree that the authors have addressed most of the questions raised by reviewers and the meta-reviewer very well. Thus, I recommend acceptance. Please prepare the camera ready carefully.

From: auai.org/UAI/2022/Conference/Paper649/Reviewer_CEag
Q1 Summary and contributions: In this paper, the authors study the model parameters in the ell_infinity loss of the maximum likelihood estimator in the Bradley-Terry-Luce model and prove various upper and lower bounds. 

Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 2: Fair: The paper contributes some new ideas.
Q2(2) Significance/Impact: 2: Fair: The paper is likely to have moderate impact within a subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 2: Fair: The experimental evaluation is weak: important baselines are missing, or the results do not adequately support the main claims.
Q2(5) Reproducibility: 4: Excellent: Key resources (e.g., proofs, code, data) are available and key details (e.g., proof sketches, experimental setup) are comprehensively described for competent researchers to confidently and easily reproduce the main results.
Q2(6) Clarity of writing: 3: Good: The paper is well organized but the presentation could be improved.
Q3 Main strengths: - Rigorous risk analysis of MLE over various graph topologies
- Bounds noticeably improve over Yan et al. [2012]
- Application to tournament design by noting subadditivity of ell_infinity loss in BTL

Q4 Main weakness: - UAI might not be the right venue for this submission
- Not clear exposition in the introduction

Q5 Detailed comments to the authors: - Page 7, right column: If you set \theta_i^* as you do, then how can you still enforce \theta_i^* to sum to zero (which is standard convention in the BTL model)? 
- The reader would benefit if you describe the maximum likelihood estimator in the introduction (for example see Section 2 of Yan et al. [2012])

Typos:
- Page 3: “Following the standard in the BTL literature” -> Following the standard assumption in the BTL literature
- Page 3: assume a that —> assume that


Q6 Overall score: 5: Borderline accept: Technically solid paper where reasons to accept, e.g., novelty, outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Q7 Justification for your score: It’s unclear if this submission is appropriate for UAI. For example, I cannot see how the contents of this paper can be presented in a talk/poster at the conference. Maybe a journal in statistics is a better venue? Also, I think the exposition in the introduction must be significantly improved. In the current form, it is not friendly to readers not familiar with MLE.
Post author response, while I'm not totally convinced about the aptness of the submission to UAI, I'm willing to increase my score
Q8 Confidence in your score: 2: Not very confident. I am able to defend my evaluation of some aspects of the paper, but it is quite likely that I missed or did not understand some key details, or can't be sure about the novelty of the work.
Q9 Complying with reviewing instructions: 1: Yes.

From: auai.org/UAI/2022/Conference/Paper649/Reviewer_cWWm
Q1 Summary and contributions: This paper furthers Yan's work in extending the comparison graph to more eneral and thus more practical settings.
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
Q2(2) Significance/Impact: 3: Good: The paper is likely to have high impact within a subfield of AI OR moderate impact across more than one subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(5) Reproducibility: 2: Fair: Key resources (e.g., proofs, code, data) are unavailable but key details (e.g., proof sketches, experimental setup) are sufficiently well-described for an expert to confidently reproduce the main results.
Q2(6) Clarity of writing: 3: Good: The paper is well organized but the presentation could be improved.
Q3 Main strengths: Fundamentally further Yan's work; gives many contributions to analyzing of the MLE for the BTL model.
Q4 Main weakness: The organization could be improved.
Q5 Detailed comments to the authors: The organization could be improved. Such as "Relevant and related literature" and 2.1 "Comparison to other work" should be merged. The reading flow is not smooth.

I didn't find where Table 1 's results come from.

I also wonder if section 5 is necessary for a theory paper. I prefer more on backgrounds.





Q6 Overall score: 7: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate to high impact on more than one area of AI, with good to excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations. 
Q7 Justification for your score: I have no objective for this work. And I am not so familiar with this field. I wish to see other reviewers' comments and to finalize my final judgment. 
Q8 Confidence in your score: 1: Not confident. My evaluation is an educated guess.
Q9 Complying with reviewing instructions: 1: Yes.

From: auai.org/UAI/2022/Conference/Paper649/Reviewer_FkM6
Q1 Summary and contributions: The paper addresses a gap in the BTL model literature, namely to analyze the performance of the MLE in the $l_\infty$ loss under general graph topologies (these represent the pairwise comparisons). Previous work considered either the $l_\infty$ loss (but only for restricted graph classes, such as dense graphs) or general graph topologies, but then not the $l_\infty$ loss.
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
Q2(2) Significance/Impact: 3: Good: The paper is likely to have high impact within a subfield of AI OR moderate impact across more than one subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q2(5) Reproducibility: 4: Excellent: Key resources (e.g., proofs, code, data) are available and key details (e.g., proof sketches, experimental setup) are comprehensively described for competent researchers to confidently and easily reproduce the main results.
Q2(6) Clarity of writing: 4: Excellent: The paper is well-organized and clearly written.
Q3 Main strengths: Originality/Novelty:
- The paper addresses an open problem in the BTL model literature and offers highly non-trivial results.
- This problem is dealt with thoroughly and convincingly


Significance/Impact:
- I would assume that the paper has impact mainly in the BTL literature and to some degree applications of the BTL model. (Disclaimer: I am personally not familiar with the rich literature in this area)


Correctness/Technical Quality:
- I did not check the proofs in detail as I'm far from an expert in this topic. In general, I judge the technical quality of this paper to be very high (see more below)


Clarity Of Writing/Reproducibility:
- The quality of writing is excellent. The main message of the paper is presented convincingly and the manuscript in general is very well prepared.
Q4 Main weakness: Generally, I don't see many weaknesses in this paper. While the scope of the paper could be described as rather narrow and it does not develop completely groundbreaking techniques, for me these are not major weaknesses in this case. On the contrary, I see it as a paper, which convincingly addresses a specific open problem.
Q5 Detailed comments to the authors: 1. In Sec. 4, the paper introduces a subadditivity property, which suggests that varying L for different item pairs might be useful. As far as I can see, this is only shown practically for the barbell example. I wonder whether some algorithmic approach is possible to find the best choices for L on general graphs.
2. The problem setting is compared in detail to related work in the introduction and in Sec. 2.1. Maybe one could make this even clearer by giving some table in the spirit of the ones in A.1 in the appendix. For example, let the columns be the norms and the rows the graph model. The fields could include the citations.
3. Writing:
- some commata are missing: [page 2] ranking loss, i.e., whether..., [page 2] However, a notable ..., [page 3] In Section 2, we ..., [page 6] In this section, we discuss...
- In Sec. 5 in the text you refer to Fig. 1 left/middle/right panel, but it is two upper and one lower panel.
Q6 Overall score: 7: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate to high impact on more than one area of AI, with good to excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations. 
Q7 Justification for your score: In my opinion the paper is a clear accept. It is well-written, on a high technical level and convincingly tackles an open problem in the BTL model literature. 
Q8 Confidence in your score: 3: Somewhat confident, but there's a chance I missed some aspects. I did not carefully check some of the details, e.g., novelty, proof of a theorem, experimental design, or statistical validity of conclusions.
Q9 Complying with reviewing instructions: 1: Yes.