Paper: OmniInput: A Model-centric Evaluation Framework through Output Distribution (ICLR-2024)

Reviewer comment (Reproducibility):
The authors didn't provide enough details for OmniInput (see questions).

Author response:
Estimate is provided in caption of Figure 4 with 40 bins, each containing at least 300 samples (at least 12000 samples). GWL has a gradient component to encourage the sampler to visit under-visited output values and it does not need many data points to ensure 50 samples.

Reviewer comment (Experiments):
The proposed OmniInput was not applied to more challenging / real world scenarios (large scale image classification, face recognition, etc.)

Author response:
(W2 and Q1.3) Yes, we believe our approach can be applied to problems with larger scales such as ImageNet classification directly. Please also refer to the general discussion at the beginning of the response "Common concerns on representative dataset and multi-class classification" about the scalability where we  discuss how to interpret our results in the experimental section for other datasets and how to extend our method to multi-class classification (applied for CIFAR100). Additional results of CIFAR100 are presented.

Reviewer comment (Presentation):
The scales of log(recall) on precision-recall graphs need to be provided.

Author response:
The scale of log(recall) in Figs. 2 and 4 is in natural log scale.

Reviewer comment (Experiments):
Can this approach be applied to problems with larger scales, i.e., ImageNet classification (1000 classes)?

Author response:
(W2 and Q1.3) Yes, we believe our approach can be applied to problems with larger scales such as ImageNet classification directly. Please also refer to the general discussion at the beginning of the response "Common concerns on representative dataset and multi-class classification" about the scalability where we  discuss how to interpret our results in the experimental section for other datasets and how to extend our method to multi-class classification (applied for CIFAR100). Additional results of CIFAR100 are presented.

Reviewer comment (Experiments):
Over-fitting Concern: The benchmark, based on binary MNIST or the initial two classes of CIFAR10, leverages low-resolution images, making them relatively easy to recognize. Consequently, models tend to over-fit on this training set, as evident from the near-perfect accuracy rates in Table 1. Advanced CNN architectures might face the over-fitting issue, potentially leading to a subpar performance on the proposed benchmark.

Author response:
(W1) Our setting does not assume we know how the model is trained. Thus whether the model simply memorizes the training set (overfitting) does affect our methodology. Overfitting only affects the results we are about to analyze to understand model performance.

Reviewer comment (Experiments):
Scalability Concerns: The authors may consider using a more diverse or challenging dataset to truly evaluate and validate model capabilities. However, the current approach may face challenges when extended to more complex, real-world datasets. The input space becomes considerably vast for such datasets, and there appears to be an absence of efficient sampling techniques in the current framework. It would be valuable to address how the proposed method plans to tackle these scalability issues.

Author response:
(W2, Q1) We discussed how to extend the framework to multi-category settings and more advanced methods of sampling in “Common concerns on representative dataset and multi-class classification” above in the general discussion. To verify the scalability of multi-classes and larger inputs (CIFAR10 and CIFAR100), we add a new Section I in the Appendix with a detailed discussion of the output distribution and representative samples.

Reviewer comment (Novelty):
The core component of the proposed framework, the sampler to estimate the output distribution over the entire input space, just simply follows the existing work [1], which makes the technique contribution not good enough. As mentioned by the authors in the paper, the proposed method is heavily relied on the sampler, while the sampler is simply borrowed from the existing works. To some extents, this paper can be viewed as an application of the sampler in the field of model evaluation. I realize the meaning of the proposed evaluation framework, but the technique contribution is not good enough to reach the bar of ICLR.

Author response:
(W1) The major technical contribution of this work is a novel approach to quantify model performance across the entire input space, as opposed to a novel sampler. This work is enabled by a good sampler, but the framework proposed in this work is completely new, and its implication transcends a mere application of the sampler. Specifically, the new contributions of this work include:  
  1. **Model-centric performance evaluation**: different from the conventional practice of hypothesizing potential model failures and subsequently conducting experiments for validation, our methodology revolves around the concept of model-centric evaluations and samples directly from the model under evaluation. The objective is to comprehend the intrinsic biases of the model self-sufficiently, eschewing the conventional approach of relying on human-probed datasets or other models to estimate model performance. This methodological renovation marks a substantial difference from prior works in the field of model evaluation.
  2. **Model performance available for the entire input space**: the utilization of the model’s output distribution represents an advancement of performance evaluation that extends from representative samples (only for generative models) to the entire input space (for both generative models and classifiers). This is the first time where model performance is estimated for the entire input space of a dataset.
  3. **Ability to quantify overconfident predictions**: the issue of overconfident predictions has been a longstanding concern in out-of-distribution (OOD) detection. Our contribution is that we are the first to quantitatively assess overconfident predictions by harnessing the output distribution. This novel approach allows for a quantitative comparison between two models, even when overconfident predictions are observed in both models. This is a significant advance in the context of OOD detection. Our framework not only provides a viable approach to quantify overconfident predictions; more importantly, it provides a comprehensive understanding of models’ performance even when overconfident predictions exist for both models.

Reviewer comment (Experiments):
This paper mainly focuses on a simple binary classification task to demonstrate the effectiveness of the proposed evaluation method.

Author response:
(W2) We extended the validation of our framework to CIFAR-100 to provide more evidence that our framework is generic to different applications. Please also refer to “Common concerns on representative dataset and multi-class classification.” To verify the scalability of multi-classes and larger inputs (CIFAR10 and CIFAR100), we add a new Section I in the Appendix with detailed discussion of the output distribution and representative samples.

Reviewer comment (Experiments):
How to generalize the framework to the state-of-the-art vision models remains a question. The paper only evaluates ResNet variants and should further involve ViT variants.

Author response:
(W2) The generalization to state-of-the-art vision models amounts to scaling to large models, which is linearly proportional to the number of parameters, as both forward and backward functions are called during the sampling process. The ViT model and its variants are transformer-based models, which we expect to have similarities to the transformer models we have tested in the NLP application. We believe our framework would be applicable to other models with similar architectures as we treat the middle layers as a complete black-box.