Paper:
In most machine learning tasks unambiguous ground truth labels can easily be acquired. However, this luxury is often not afforded to many high-stakes, realworld scenarios such as medical image interpretation, where even expert human annotators typically exhibit very high levels of disagreement with one another. While prior works have focused on overcoming noisy labels during training, the question of how to evaluate models when annotators disagree about ground truth has remained largely unexplored. To address this, we propose the discrepancy ratio: a novel, task-independent and principled framework for validating machine learning models in the presence of high label noise. Conceptually, our approach evaluates a model by comparing its predictions to those of human annotators, taking into account the degree to which annotators disagree with one another. While our approach is entirely general, we show that in the special case of binary classification, our proposed metric can be evaluated in terms of simple, closed-form expressions that depend only on aggregate statistics of the labels and not on any individual label. Finally, we demonstrate how this framework can be used effectively to validate machine learning models that we trained on two real-world tasks from medical imaging. The discrepancy ratio metric reveals what conventional metrics do not: that our models not only vastly exceed the average human performance, but even exceed the performance of the best human experts in our datasets.

Review:
Summary: This paper proposes a novel metric "discrepancy ratio" for evaluating the performance of a model where the ground truth for each data point comes from many expert-yet-imperfect annotators. The authors suggested that this problem has remained largely unexplored. The proposed metric is intuitive and is easy-to-use. The authors suggested that this metric can be used for many applications. For example, to evaluate if a model is better than the annotators on average, to evaluate the annotator, to compare between models. 
========================================================
Comments on clarity:
The writing is good overall and I am able to understand the contribution and the key idea of this paper.
Questions and comments on clarity:
1. While MPD is defined with an argument (Y, Y'), the definitions annotator discrepancy, model discrepancy and discrepancy ratio ignore the inputs of the function. It might be better to keep the arguments of the function for clarity. When people want to adopt the discrepancy ratio in their work and there are many ratios to discuss, then the proposed definition can be used conveniently. 
2. In Eq. (5) \sigma seems to be undefined. I believe it is a variance but it is kinder to readers to explicitly state it (I found the authors explicitly stated it in Appendix B).
3. In Eq. (8), is it intentional to have $i$ exists over m and y as written in the paper?
4. Does the discussion of the simplicity of the discrepancy ratio hold for other \delta, i.e., not the squared loss?
5. In Section 4.1 (experiment with MNIST), did the authors binarize the data or simply discuss the multi-class problem? Because this follows section 3.3 which dedicates the whole page to discuss binary classification. It may be a bit sudden to move to multi-class classification and an additional sentence to clarify this might help.
6. I could not understand what is "single annotator strictness in Figures 1d and 2d. It is not explained if I did not miss it.
7. I think we should not resize the figure/table caption. The text size of captions are much smaller than the main text.
8. Is PLAX binary classification? (I understood as yes from Figure (3a) but I think it's also nice to clarify it to avoid confusion (also for MNIST)).
9. What is the y-axis for figure 3a?
10. I think reporting the performance of the mean-per-class discrepancy ratio without defining it in the main text (it is defined in a natural language (English) in Appendix E) makes the main body of the paper not self-contained. I strongly suggest to either (1) remove mean-per-class discrepancy ratio entirely from the main body or (2) define it mathematically in the main body. Also, I don't see reporting mean-per-class discrepancy ratio adds values to a paper for current experiments because the reported mean-per-class ratio is very similar to that of discrepancy ratio (Figures 3b (almost overlap) and Table 2 (same trend and the values are very close to the discrepancy ratio for all cases)).
11. Can we use F1 to evaluate whether a model is better than the average human performance?
Minor points on clarity, which may just be a personal preference of a reviewer:
1. The authors may consider using \begin{definition} when defining three key values: the mean pairwise deviation (MPD), annotator discrepancy, and model discrepancy. So the readers can easily look up when getting lost in definitions. And it may help to make a definition more precise (e.g., given label Y, data points, MPD is defined as follows).
2. Eqs (1), (2), (3) may be easier to understand if we put a denominator out of the summation as much as we can. For example, for Eq. 2, N can be outside of the two sums, A^i(A^i-1) can be between the two sums. 
========================================================
Significance: 
The problem that this paper addresses is highly important. I like the idea and I am convinced that discrepancy ratio is potentially useful for the community. And it also opens many possibilities to analyze the discrepancy ratio and to find potential applications for it.  
========================================================
Other comments:
On the limitation:
I am satisfied that the authors clearly discuss the limitations of the proposed method and I agree with the discussion.
On the experiments: 
1. Why we use model 1.0, 0.95, 0.9 for Figures 1c, 1d, 2c, 2d but 1.0, 0.9, 0.8 for Figures 1a, 1b, 2a, 2b? Perhaps full results with many models should also be reported for completeness.
2. Isn't it more natural to show a graph that has x-axis as label swap probability in the same figure for comparison? For example, Figure 1 only report performance with respect to each measure (Naive Evaluation, Majority Vote, Relative F1, Discrepancy ratio) with respect to label swap probability. We can fix the number of annotators to 9 for relative F1 and discrepancy ratio, to validate whether it also becomes increasingly difficult to differentiate model improvements or not for them. Then after suggesting that Naive evaluation is not good and other metrics look fine. Majority vote cannot evaluate that the model is better than human average performance, which we want to know. Then Figure 2 highlights the contribution of the discrepancy ratio and shows that it outperforms relative F1 in some perspectives.
3. For the MNIST benchmark dataset, it is possible to show the performance on the test ground truth too. And I think it is interesting to compare different models with the same label noise rate and confirm that discrepancy ratio is successful to evaluate the best model with respect to the ground truth accuracy. It would be very impressive to see if the best model with respect to the discrepancy ratio can successfully be the best also on the ground truth test set.
On space usage:
I feel there are many important things to be included in the main body but it is instead in the appendix. On the other hand, I think there are several parts that can be shortened in the main body. And it seems that the binary classification example (3.3) also consumed a lot of space. The derivation may be omitted and make it a Theorem and then let's suggest the readers to read the appendix for the proof if they are interested. Then we can have more space to add important components in the main body of the paper.
========================================================
Decision:
Although there are issues on the clarity of the paper, I believe these issues are not too difficult to fix in the final version. The idea is interesting and potentially impactful. For these reasons, I vote a weak accept for this paper.
Update: I would like to thank the authors for the clarification. I have read the rebuttal.
I read through the updated version of the paper. I believe that although there is still room for improvement, the contribution of this paper is sufficient as an important step towards solving this difficult yet highly relevant problem. This paper also stimulates several future directions. The authors clearly stated the limitations of the paper, which is highly useful for researchers to improve this work or find a better solution.  The clarity of the paper is also improved. For these reasons, I increased my score from weak accept to accept.