Paper: Dropout-Based Rashomon Set Exploration for Efficient Predictive Multiplicity Estimation (ICLR-2024)

Reviewer comment (Theory):
The biggest weakness of the approach concerns its limitations. As honestly discussed by the authors in Section 6 – Limitations, the fact that when the hypothesis space to explore is huge (that is when the predictor has many parameters to tune), the exploration that is done by the dropout approach is fairly limited. This is clearly related to Proposition 4, where an important value of $M$ is necessary for layers having hundreds of neurons and an important value of $k$ is necessary with complex neural networks.

Author response:
Indeed, when the hypothesis space is huge, exploring models in the Rashomon set, i.e., obtaining almost-equally-optimal models from the hypothesis space, is in general computationally infeasible. Despite dropout-based exploration is limited, it makes the exploration possible compared to other strategies such as re-training and AWP, which are much harder computational tasks as they requires repeated re-training and perturbation. We thank the reviewer for bringing this up, and will add a more thorough comparisons of the pros and cons among the three strategies, re-training, AWP, and the proposed one in Section 3 and 4.

Reviewer comment (Experiments):
Two scenarios could occur: the first one is that the hypothesis space to explore is relatively small. That is, it is fairly explored by the dropout method. But, even though a 30x to 5000x speedup over other approaches is seen, it is never defended that those other approaches are not scalable with small hypothesis space. Even though there is a huge speedup gain, if the other approaches are relatively fast (do not take hours to compute), then why favourising dropout? The second scenario is that the hypothesis space to explore is large. The time gain is then undermined by the limitation in the exploration. Plus, when it comes to large models, it is common to retrain only the classification head of the predictor, or to fix many layers; doing so could really fasten the retraining scheme, thus undermining the potential advantage of the dropout approach.

Author response:
We thank the reviewer for bringing up this practical concern. When the hypothesis space is relatively small, the re-training and AWP strategies could be fast; however, the dropout method could still be faster (despite that the speedup could be smaller) than re-training and AWP for the same reason—it does not require gradient computation and weight updating. Moreover, whether the proposed dropout method could explore the Rashomon set as effective as re-training or AWP when the hypothesis space is mall, is an interesting theoretical question and also requires numerical evidence.  
    
    The second scenario is quite a common practice when fine-tuning a pre-trained model. Indeed, re-training and AWP could be significantly faster when only applied on part of the weights (e.g., the classification head). In this sense, the dropout method could also be applied on part of the weights in order to accelerate. The comparison of runtime performance among re-training, AWP, and the dropout method when applied on partial weights (also briefly discussed in Section 6 Future Directions.) is an interesting and practical research direction. Due to needlessness of updating weights, the proposed dropout strategy could still be more computationally efficient with the speedup varying depending on different complexity of the hypothesis space.
    Finally, as we have discussed in Section 6 Future Direction, the proposed dropout method could also be applied as a compliment to re-training, in order to leverage between efficiency and effectiveness of Rashomon set exploration. We will address this concern by adding a paragraph in Section 4 or 5 to further discuss the impact of the cardinality of the hypothesis space with some numerical examples.

Reviewer comment (Experiments):
It seems to me that both the depiction in Figure 2 and the speedup reported in Table 1 are lacking important details. For example: How many different models are sought? How many reruns were done for retraining VS how many dropouts were computed? What was the total time for each individual method? To my understanding, many reruns are already needed in the first place, no matter the approach, in order to ensure that the reference model is an « empirical minimizer »; was that taken into account when comparing the time for building the empirical Rashomon sets in Table 1? What was the size of the predictor used on these different UCI tasks (this kind of information is necessary in the main article, not the supplementary material)?

Author response:
We appreciate the reviewer for pointing it out, all this information about implementation details are included in Section D and E in the Appendix, and are pointed to the readers in Section 4 and 5. We will move these information from the Appendix to a paragraph in the main text, and re-emphasize the pointers to the readers in the main text. In Figure 2, we first train an empirical minimizer $\mathbf{w}^\*$, and note that the runtime for obtaining this $\mathbf{w}^\*$ is not counted in Table 1. For each loss diff. $\epsilon$, we obtain 100 models for both re-training and the dropout method. For the re-training approach, we train a lot of models with different weight initializations, and collect the models that has test loss deviation against the empirical minimizer $\mathbf{w}^\*$ smaller than $\epsilon$. Similar the dropout approach we also collect 100 models. The predictors all have the same architecture, a neural network with a single hidden layer composed of 1k neurons. 
    
    We would like to emphasize that the empirical minimizer $\mathbf{w}^\*$ are given to all three strategies, therefore the time to rerun to obtain this empirical minimizer $\mathbf{w}^\*$ is not accounted in the speedup in Table 1. Moreover, the speedup in Table 1 is calculated per model, not the total time for each individual methods. We will improve this point by clarifying the speedup is compared per model in the caption.

Reviewer comment (Theory):
I feel like something is conceptually wrong with the comparison between retraining and the current dropout scheme. Retraining makes it such that the validation loss is the highest possible. Therefore, it makes sense that many runs are needed in order to find models close to the « empirical minimizer ». With the dropout scheme, an empirical minimizer is found, and then dropout is applied while making sure the training loss does not diminish too much. The two approaches do not have the same objectives.

Author response:
We would like to follow our answers to the previous question and clarify that an empirical minimizer $\mathbf{w}^\*$ is given to all the three strategies, re-training, AWP and dropout to explore the Rashomon set defined according to $\mathbf{w}^\*$. Indeed re-training and dropout are two very different strategies, with the same goal of exploring the Rashomon set. Re-training has been the most common method in existing literature such as [Long et al., 2023] and [Kulynych et al., 2023]. However, it also limits the scale of the experiments that could be presented in those papers due to time complexity. 
    
    Here we provide an additional numerical results on why re-training is time consuming. In Table E.7 in the Appendix, we obtain 500 models with re-training and the dropout methods, and report the percentage of the models from each strategies that are in the Rashomon set with a given empirical minimizer and varying $\epsilon$. It is shown that for most cases, the percentage of models in the Rashomon set obtained by re-training is significantly smaller than that by dropout. The reason is that it is very hard to control the loss fluctuations from randomized algorithms such as stochastic gradient descent. We will add a paragraph in Section 4 to summarize the discussions above in the final revision.

Reviewer comment (Theory):
The dropout leads to a scheme where each new model depends on the initial model. All of the models are thus dependent. Therefore, the estimation of the Rashomon metrics is biased. And while « not all estimators of predictive multiplicity metrics carry a theoretical analysis of its statistical properties such as consistency and sample complex », I feel like it is a property of interest. Indeed, one of the motivations of the work is the need for ethics and, more specifically, fairness. I see the goal in exploring the Rashomon sets to find many predictors giving different predictions to people for different reasons. But having all of the models interconnected makes it such that the reasons for the predictions are all linked and just a few are explored with the dropout scheme.

Author response:
We agree with the reviewer on the limitation of the dropout methods, and the need to study theoretical analysis of predictive multiplicity metric estimation for more general scenarios. We would like to emphasize that the goal of this paper is not using the dropout method to find models in the Rashomon set for other ethical needs, but provides an alternative method to estimate predictive multiplicity metrics more efficiently. How to explore in the Rashomon set to find models that can comply with other ethical needs (using the dropout method proposed here or not) is an interesting and open future direction. 

    Moreover, since the entire (true) Rashomon set is not computationally feasible, all estimations based on exploring the Rashomon set (re-training or the proposed dropout method) lead to under-estimation of multiplicity metrics. The best practice we could study in the future is to improve the under-estimation. Finally, despite that the proposed dropout method may only explore a small subset in the Rashomon set, it give an lower bound of the multiplicity metrics in a much more efficient manner.

Reviewer comment (Theory):
Proposition 5 aims at proving that the approach is not biased, but relies on the assumption that « the models around W∗ are uniformly distributed in a d-dimensional ball with center $\mathbf{W}^*$ and radius $\delta$, i.e., $B(\mathbf{W}^*, \delta)$. Accordingly, we may assume that the population means $\mu$ for a sample can be expressed as [...] ». The method explicitly does that (especially the Gaussian dropout), exploring around the « population mean », that is, the empirical minimizer. Therefore, assuming the uniform distribution of the Rashomon set around a center trivially leads to the unbiasedness of the dropout scheme, but is unreasonable.

Author response:
We thank the reviewer for reading Proposition 5 in the Appendix. Indeed, the assumption of the uniform distribution around $W\*$ may not reflect the true underlying distributions of models in the Rashomon set (e.g., when there are exponentially many local minima), but it is an ideal assumption that facilitates mathematical analysis that was also adopted in existing literature, e.g., [Kulynych et al., 2023]. How to characterize the distribution of models in the Rashomon set still remains an open and active challenge in the field.
    
    Additionally, please note that Gaussian distribution in a high dimensional space does not look like a bell (in our vanilla understanding, concentrated in the middle), but looks like a bubble where the mass is concentrated around $\sqrt{d}$, where $d$ is the dimension. This property contradicting to our intuition is stated in the Gaussian annulus theorem (Lemma A.3 in the Appendix). Therefore, we would like to argue that the derivation of Proposition 5 to bound the gap is not trivial. We will include a more detailed discussion on the uniform assumption in the Appendix, and add pointers in the main text.

Reviewer comment (Theory):
It is said that « not all estimators of predictive multiplicity metrics carry a theoretical analysis of its statistical properties such as consistency and sample complex » Could you provide some citation supporting this claim?

Author response:
We thank the reviewer for pointing this out and will add supporting discussions accordingly in the beginning of Section 4. There are some theoretical analyses of estimating predictive multiplicity metrics. For example, the exact estimation of ambiguity and discrepancy for linear classifiers are studied in [Marx et al., 2020]. Similarly, the theoretical analysis of estimating VPR with logistic regression is also provided in [Watson-Daniels et al., 2023]. Moreover, the estimation of disagreement is also discussed in [Kulynych et al., 2023]. However, the statistical properties of estimating other metrics such as the Rashomon Capacity, or with a general hypothesis space, still remain an open challenge.

Reviewer comment (Theory):
What justifies fixing a single Bernoulli or a Gaussian dropout parameter for all layers simultaneously? Shouldn’t the layers be treated independently?

Author response:
We thank the reviewer for raising this interesting direction. As an early study and for theoretical convenience, in this paper we fix the dropout parameters for all layers. However, as we have discussed in Section 6, different layers in a neural network may carry different levels of information, and therefore the dropout parameters could vary across the layers. We will re-emphasize this in the Future Direction in Section 6, and our theoretical analysis could serves as a first step in the “adaptive” dropout parameters to explore the Rashomon set.

Reviewer comment (Evaluation):
Concerning the quantification of predictive multiplicity, it is said that « [f]or example, Long et al. [2023], Cooper et al. [2023] and Watson-Daniels et al. [2023] quantify predictive multiplicity by the standard deviation, variance and the largest possible difference of the scores (termed viable prediction range (VPR) therein) respectively » So, what definition between those three is retained in the article?

Author response:
In this paper, we adopt the score variance from [Cooper et al., 2023] and the VPR from [Watson-Daniels et at., 2023] for score-based predictive multiplicity metrics, and report them in Section 4. We retain the definitions and implementations of them in the original article. Due to space limit, we summarize all the mathematical formulations and implementations in Appendix B. Please note that in this paper we did not propose a new predictive multiplicity metric, but a new strategy to efficiently estimate all existing predictive multiplicity metrics. We will re-emphasize this point by adding a sentence that summarizes the discussion above in Section 2.

Reviewer comment (Evaluation):
It is unclear what's the practical use of the propose method. It is fast, but it does not explore the Rashomon set well. For this reason, we can only mitigate predictive multiplicity *as estimated by Dropout* but not in general.

Author response:
We address the reviewer's concerns in Weaknesses 1 and Weaknesses 2 together here. The *good* exploration of the Rashomon set, i.e., obtaining very *diverse* models with almost-equally-optimal performance, comes with the cost of efficiency (cf. e.g. [Xin et al., 2023], [Watson-Daniels et al., 2023]). 
    With this in mind, the practical use of the proposed dropout method is to provide an alternative to the current re-training strategy that is not only more efficient but also bears with theoretical analysis (Proposition 1 to Proposition 4).
    Numerical results, e.g., in Figure 2, also show that the proposed method is comparable or outperforms the re-training strategy in terms of exploring the Rashomon set. 
    Note that both re-training and the dropout methods give a subset (defined in (2)) of the true Rashomon set, and will all lead to an *under-estimate* of predictive multiplicity metrics. Therefore, higher estimates of the multiplicity metrics indicate a better exploration of the Rashomon set. 
    To the best of our knowledge, it is the first time when the dropout method is connected and analyzed with notions of the Rashomon set and predictive multiplicity, making reporting predictive multiplicity for large-scale models possible. 
    
    Moreover, the methods to mitigate predictive multiplicity discussed in Section 5 are not limited to the dropout methods. In fact, the ensemble method shown in Figure 4(a) work for any strategies that could obtain models in the Rashomon set. Since more models in the the ensemble leads to smaller multiplicity metrics, the proposed dropout method serves as an efficient way to obtain a huge amount of models. 
    
    Mitigating predictive multiplicity, especially when selecting a model with the least predictive multiplicity from several given models, as shown in Figure 4(b), could also be achieved by the AWP. We will include an additional experiment in the Appendix of future revision. 
    Lastly, please note that the re-training strategy can be combined with the dropout method to order to leverage the efficiency and efficacy of Rashomon set exploration, as already discussed in Section 6 Future Directions. We will summarize the discussion above and add a paragraph accordingly in Section 2.

Reviewer comment (Experiments):
Following 1, it seems like additional experiments on whether the mitigation via Dropout also transfers to, say, AWP, is interesting.

Author response:
We address the reviewer's concerns in Weaknesses 1 and Weaknesses 2 together here. The *good* exploration of the Rashomon set, i.e., obtaining very *diverse* models with almost-equally-optimal performance, comes with the cost of efficiency (cf. e.g. [Xin et al., 2023], [Watson-Daniels et al., 2023]). 
    With this in mind, the practical use of the proposed dropout method is to provide an alternative to the current re-training strategy that is not only more efficient but also bears with theoretical analysis (Proposition 1 to Proposition 4).
    Numerical results, e.g., in Figure 2, also show that the proposed method is comparable or outperforms the re-training strategy in terms of exploring the Rashomon set. 
    Note that both re-training and the dropout methods give a subset (defined in (2)) of the true Rashomon set, and will all lead to an *under-estimate* of predictive multiplicity metrics. Therefore, higher estimates of the multiplicity metrics indicate a better exploration of the Rashomon set. 
    To the best of our knowledge, it is the first time when the dropout method is connected and analyzed with notions of the Rashomon set and predictive multiplicity, making reporting predictive multiplicity for large-scale models possible. 
    
    Moreover, the methods to mitigate predictive multiplicity discussed in Section 5 are not limited to the dropout methods. In fact, the ensemble method shown in Figure 4(a) work for any strategies that could obtain models in the Rashomon set. Since more models in the the ensemble leads to smaller multiplicity metrics, the proposed dropout method serves as an efficient way to obtain a huge amount of models. 
    
    Mitigating predictive multiplicity, especially when selecting a model with the least predictive multiplicity from several given models, as shown in Figure 4(b), could also be achieved by the AWP. We will include an additional experiment in the Appendix of future revision. 
    Lastly, please note that the re-training strategy can be combined with the dropout method to order to leverage the efficiency and efficacy of Rashomon set exploration, as already discussed in Section 6 Future Directions. We will summarize the discussion above and add a paragraph accordingly in Section 2.

Reviewer comment (Theory):
The bounds in Proposition 2 and 3 only converge to 1 when $d\to\infty$, which does not seem like useful. See Q3 as well.

Author response:
We thank the reviewer for bring this up, and will add explanations and citations to support this assumption. 
    The condition $d\to\infty$ is a regime commonly discussed in the asymptotic analysis of learning models such as over-parameterization [1], and the universal approximator theorem [2]. We would like to emphasize that the asymptotic behavior when $d\to\infty$ is based on the bound we derived in Eq. (5).

    [1] Cao, Y. and Gu, Q., 2020, April. Generalization error bounds of gradient descent for learning over-parameterized deep relu networks. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 3349-3356).
    
    [2] Hornik, K., Stinchcombe, M. and White, H., 1989. Multilayer feedforward networks are universal approximators. Neural networks, 2(5), pp.359-366.

Reviewer comment (Evaluation):
AWP is slower due to re-training, but the models are trained only once. Therefore, doesn't it run *faster* than Dropout (because it uses fewer samples/models to explore the Rashomon set) with a reasonably large test data?

Author response:
We thank the reviewer for raising this confusion, and will provide more details of the AWP algorithm to help future readers to better understand. Suppose we have a dataset with n samples and each sample could be classified into c candidate classes. AWP maximizes each entries of a **given** model $h\_w(x)$, i.e., $[h\_w(x)]\_i$ for each $i \in [c]$, under the constraint that the model after perturbation is still inside the Rashomon set. Therefore, the AWP requires $n\times c$ perturbations/re-training to estimate multiplicity metrics for an entire dataset. We can imagine that each of these $n\times c$ perturbations leads to a model in the Rashomon set that outputs the most inconsistent score for a given class per sample. On the other hand, the dropout procedure applies dropout on the given model $h\_w(x)$ without the need to re-train (gradient computation and weight updating) the model. We will add the discussion above in Section 2.

Reviewer comment (Theory):
Is $\epsilon$ and the $L$ in Eq.(5), (7) and (10) related to the "sum" of loss or the "mean" of loss? It seems like it's the sum? If so, by changing $\epsilon$ to some offset on the mean loss, we can probably get a convergence basing on the sample size, which is much more meaningful than dimension of the model's hidden layers.

Author response:
We thank the reviewer for bringing up this point that might cause potential confusion, and will add a paragraph in Section 3 to clarify. The Rashomon set in Eq. (2) is defined regarding the mean of the loss for a given dataset of a **fixed size $n$**.  In other words, the dataset is a given parameter, not a source of randomness in the definition of a Rashomon set. 
    Note that the randomness in the convergence in Eq. (7) and (10) is with respect to the dropout matrix, not like the vanilla concentration bound that consider drawing samples from a data distribution, it makes sense that those convergence results does not rely on the number of samples.
    In Eq. (5), the loss we used is the sum of the loss for each samples as we w.l.o.g. assume the data matrix $X$ is whitened. In Eq. (7) and (10) the losses are defined with the mean of the loss for each sample. We will make this point clear in future revision.

Reviewer comment (Evaluation):
Model augmented by Dropout could result in a fairly small search space of Rashomon set. I am not very convinced that this is a good idea in practice if our goal is to look for a better model that can address various reliability problem of predictive model. e.g. fairness etc. It maybe inspirational to see the movement of predictive multiplicity measurement, but I am wondering what is the practical meaning of it.

Author response:
We thank the reviewer to point out this confusion. Our goal is not finding models in the Rashomon set that comply with additional reliability problems such as fairness, but to efficiently estimate predictive multiplicity, despite that both involve exploring the Rashomon set, i.e., obtaining almost-equally-optimal models from the hypothesis space. As stated in the second paragraph in Section 1, our focus is estimating predictive multiplicity, a problem induced from the Rashomon effect, not the Rashomon effect stated in the first paragraph of Section 1. We agree with the reviewer that studying if the dropout models in the Rashomon set could lead to other reliability properties such as fairness is an interesting future direction. 

There is a trade-off between the efficiency and efficacy of exploring the Rashomon set and prediction multiplicity estimation. Note that re-training and our proposed dropout method all aim to approximate a subset of the entire (true) Rashomon set, and thus all estimates based on re-training or our method are under-estimation of the true multiplicity metrics. Despite that our method may only explore a small subset in the Rashomon set, it give an lower bound of the multiplicity metrics in an efficient manner, as clearly shown in Table 1. Moreover, the estimation using the dropout method also gives results comparable to re-training and AWP, as shown in the empirical evidence in Figure 2. Therefore, the proposed dropout strategy serves as yet another option (besides AWP and re-training) to explore the Rashomon set depending on the need of practitioners. 

We will summarize the discussion above and add it to the final revision.

Reviewer comment (Theory):
There is a descriptive gap in section 3.2 where transforming deviation between $L_{SSE}(\mathbf{w}_D^*)$ and $L_{SSE}(\mathbf{w})$ suddenly become deviation between $L_{SSE}(\mathbf{w}_D^*)$ and $L_{SSE}(\mathbf{w}')$. I don't quite see why they are aligned or if the model works correctly under $L_{SSE}(\mathbf{w}')$ if it is not trained with such dropout rate.

Author response:
Thank you for raising this point and we will address these two related questions together here. Proposition 1 proves that the expected deviation between the losses evaluated on dropout weights $\mathbf{w}_D^*$ and on the **diluted** weights $\mathbf{w}' = (1-p)\mathbf{w}^\*$.  The dilution factor $(1-p)$ comes from the expectation of the dropout matrix $D$, and in fact, dropout has been known as weak dilution in past literature [Hertz et al., 1991]. That’s why we did not introduce the notion of the Rashomon set in Proposition 1. Despite that in Proposition 1, the loss deviation does not directly work with the original optimal weight $\mathbf{w}^\*$, Proposition 1 serves as the inspiration to control loss deviation via dropout for the rest of the propositions presented in this paper. In the rest of the propositions, we use different proving techniques to make the loss deviation directly work with the original weight $\mathbf{w}^\*$ instead of the diluted weight $\mathbf{w}'^\*$. Note that the dropout here is applied in the inference/ test time, and therefore it does not depend how the model is trained (e.g., with or without dropout), i.e., the proposed dropout-based Rashomon set exploration is model-agnostic. We thank the reviewer again for helping us be aware of the potential confusion in the description of Proposition 1, and will clarify it by adding more detailed discussion accordingly.

Reviewer comment (Theory):
The proposition 1 uses deviation between $L_{SSE}(\mathbf{w}_D^*)$ and $L_{SSE}(\mathbf{w}')$ but not original model parameter $L_{SSE}(\mathbf{w}')$. How to make the connection?

Author response:
Thank you for raising this point and we will address these two related questions together here. Proposition 1 proves that the expected deviation between the losses evaluated on dropout weights $\mathbf{w}_D^*$ and on the **diluted** weights $\mathbf{w}' = (1-p)\mathbf{w}^\*$.  The dilution factor $(1-p)$ comes from the expectation of the dropout matrix $D$, and in fact, dropout has been known as weak dilution in past literature [Hertz et al., 1991]. That’s why we did not introduce the notion of the Rashomon set in Proposition 1. Despite that in Proposition 1, the loss deviation does not directly work with the original optimal weight $\mathbf{w}^\*$, Proposition 1 serves as the inspiration to control loss deviation via dropout for the rest of the propositions presented in this paper. In the rest of the propositions, we use different proving techniques to make the loss deviation directly work with the original weight $\mathbf{w}^\*$ instead of the diluted weight $\mathbf{w}'^\*$. Note that the dropout here is applied in the inference/ test time, and therefore it does not depend how the model is trained (e.g., with or without dropout), i.e., the proposed dropout-based Rashomon set exploration is model-agnostic. We thank the reviewer again for helping us be aware of the potential confusion in the description of Proposition 1, and will clarify it by adding more detailed discussion accordingly.

Reviewer comment (Reproducibility):
In the experiments, the authors mentioned that "On the other hand, AWP outperforms both dropouts and re-training, since it adversarially searches the models that mostly flip the decisions toward all possible classes for each sample." I may miss some details of the method part, how can the proposed method to adversarially search the models since the dropout is random?

Author response:
We thank the reviewer for pointing out this confusion. We would like to further clarify that AWP and re-training  are two existing baselines we compared against with our proposed dropout method. The AWP, re-training and ours are three different strategies to explore the Rashomon set, i.e., obtaining almost-equally-optimal models from the hypothesis space, and ours and AWP will not be applied at the same time. The AWP algorithm, as proposed in [Eq. 9, Hsu \& Calmon, 2022], will adversarially perturb the output scores of a given sample to each class. For example, if there are $c$ classes, AWP maximizes each entries of $h\_w(x)$, i.e., $[h\_w(x)]\_i$ for each $i \in [c]$, under the constraint that the model after perturbation is still inside the Rashomon set. Therefore, for one sample, AWP requires to perform one perturbations (adversarial training) via SGD for each class (i.e., $c$ adversarial training in total for one sample), and if there is a dataset with $n$ samples, we need $n\times c$ adversarial training to estimate multiplicity metrics for the entire dataset. We can imagine that each of these $n\times c$ perturbations leads to a model in the Rashomon set that outputs the most inconsistent score for a given class per sample. On the other hand, the dropout method does not require to iterate over all samples and does not require gradient computations and updating model weights. In this sense, AWP is very different from the proposed dropout strategy.
    
Due to space limit, we refer the readers to [Section 4, Hsu \& Calmon, 2022] for more details of the AWP algorithm in the initial draft. We will add pseudo codes for the AWP in Appendix B, and a brief discussion in Section 2 to clarify the difference of the three strategies in the final version of the paper.

Reviewer comment (Experiments):
It seems the proposed method is only evaluated on in-distribution scenarios. Can it be applied to out-of-distribution data?

Author response:
We thank the reviewer for bringing this up. Indeed, the loss, accuracy and predictive multiplicity metrics reported in this paper are all evaluated on test data, i.e., unseen but in-distribution samples. The proposed method can also be applied for out-of-distribution (OOD) data. However, without special treatment (e.g., OOD generalization), the pre-trained model would have much higher loss $L(h_{w*}, \mathcal{D}\_{OOD})$ when fed with OOD data, leading to a larger Rashomon set (with the same $\epsilon$) and potentially much higher values of predictive multiplicity metrics (since multiplicity metrics are optimized over the Rashomon set; please also refer to [Defn 1, Hsu et al., 2022]). We will add a brief paragraph to include the discussion above in Section 3.

Reviewer comment (Evaluation):
Are the uncertainty scores calibrated? In other words, are the confidence scores reliable?

Author response:
Thanks for the great question! We would like to further clarify that if a perfectly calibrated classifier assigns a 50\% score to a sample (e.g., in binary classification), it does not necessarily mean that this sample has high multiplicity. A perfectly calibrated classifier is one whose predicted classes matches the true classes **on average** across samples (e.g., samples predicted to be 50\% of one class have true outcomes matching that class 50\% of the time). However, this does not necessarily translate to a (in)consistent set of predictions for a **single target sample** across equally calibrated classifiers. It may be the case that **all** calibrated models drawn from the Rashomon set assign the same 50\% probability for that sample (no multiplicity). Conversely, some models may assign higher and lower confidence for that sample (high multiplicity) yet, on average, still be well-calibrated. Again, this happens because calibration (like accuracy and loss) is an average metric across all samples. We will add a brief discussion to include the response above in Section 4 in the final revision.