Paper:
Is Approval Voting Optimal Given Approval Votes?
Some crowdsourcing platforms ask workers to express their opinions by approving a set of k good alternatives. It seems that the only reasonable way to aggregate these k-approval votes is the approval voting rule, which simply counts the number of times each alternative was approved. We challenge this assertion by proposing a probabilistic framework of noisy voting, and asking whether approval voting yields an alternative that is most likely to be the best alternative, given k-approval votes. While the answer is generally positive, our theoretical and empirical results call attention to situations where approval voting is suboptimal.

Review:
This paper studies the optimality of "k-approval" voting aggregation using the approval voting rule (i.e., picking the item(s) with highest votes). In particular it studies optimality of this principle when the data is generated under the Mallows model (for different distance functions). The paper's main contributions are theoretical results which indicate that for most distance functions not all alternatives selected under approval voting are MLE optimal. However under the Kendall-Tau distance function (easily the most commonly used distance function with the Mallows model) they are equivalent.Overall I found the paper to be an interesting read with intriguing results. That said, I'm uncertain if it reaches the high bar of acceptance for a NIPS submission. My main concerns are as follows:- (Significance) I am unclear of the impact and significance of this work. What should a practitioner take away from this? What about other researchers working on this topic? This remains unclear to me despite reading the paper closely. As the authors themselves state, approval voting isn't much worse than the MLE alternative in the situations where do they do not match. Furthermore the MLE estimator itself is infeasible to compute for large problems.The secondary set of results (as presented in Sec. 4) aren't very impactful in my opinion given that they discuss (and conjecture) about optimality in extreme regions of the space, which most real problems do not fall under.I would also have liked to see some discussion on why k-approval voting is preferable to approval voting as an feedback elicitation mechanism, given that the latter seems like the cognitively easier and more reliable of the two.- (Originality) I would have also wanted to see some the connection made to the numerous results of approximation quality of rank aggregation under the Mallows model (such as the works by Nir Ailon, Cynthia Dwork and other researchers in the field). Given the decades of work on this problem, there is a rich body of literature to draw from, which can actually explain some of these results.In general while this paper indicates non-optimality of some of these models, it does not indicate how suboptimal the approval voting solutions really are (which would have been a far more intriguing result).- (Quality) In general the empirical results indicate only small differences in performance of the two methods. Given this I would have expected some kind of significance testing or error bar reporting as is standard practice.- (Clarity) The paper was mostly clear. However there were notations and terminologies that were overloaded to the point of confusion. For instance the use of the d_{KT} notation on line 205 which is then followed by a completely different use of the same notation in the next two lines. I was also slightly confused at a few points in the paper with the use of the approval voting ruleterminology (which was designed for unconstrained approval feedback) was used under the k-approval voting problem.In general much of the core ideas and insights are lost behind the sea of notation and consequently aren't made clear to the reader. Saying that non-optimality is due to some information being excluded is fine, but I would like to further understand what this information is and what are some preliminary ideas on feasible alternates that can capture this information.A more minor comment: I also didn't understand the noise characterization for the datasets at the start of section 5. Just because differences are smaller does not make the underlying votes more noisy. Unless there were empirical results that indicated the same I wouldn't make that claim.