From: ICLR.cc/2023/Conference/Paper2398/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper2398/Authors
title: Response to the counterexamples
comment: First of all, thank you for pointing out the theoretical limitations of our claim about the relationship between LDM and uncertainty, or the connection between LDM-based and disagreement-based active learning algorithms, with appropriate counter examples.

However, it is regrettable that counterexample 1 is a somewhat unusual form of hypothesis. This paper deals with a practical algorithm for pool-based active learning in classification with usual deep networks, and we will clarify this point.

The counterexample 2 is a great example. We assume that small perturbations of $\hat{h}$ are in the version space based on the empirical observations on MNIST, however, the counterexample you provided confirms that this assumption is very strong. As you said, the theoretical basis for the sample complexity of LDM is not supported (thus, the sample complexity is not discussed in the paper), but the intuition, novelty, computing time, and performance show that our algorithm is a meaningful addition to the field of practical uncertainty-based active learning.

Lastly, we would like to have productive discussions on other points too.

From: ICLR.cc/2023/Conference/Paper2398/Reviewer_xLdz
title: Counter-examples to theory from reviewer
comment: I wish to focus on what I see as my highest priority and most objective concern: "no theory for LDM" (if resolved, I'm happy to comment on the other points). In particular, I would like to provide two counter examples against the claims in the above comment.

## Counter example 1

First, for "Let us clarify our claim."

Consider binary classification with $n$ datapoints $x \in \{1,2,\dots,n\}$ (for now, say, a uniform distribution over the datapoints). Let $w \in \mathbb{R}^n$ and $h_w(x) = \mathbf{1}[w_x \geq 0]$. Examine $\hat{w} = (1,2,3,\dots,n)$ and $\sigma^2 = 1$. Let $x_1 = 1$ and $x_2=2$. In this case, $L(\hat{h},x_1) = 1/n = L(\hat{h},x_2)$ (so the left hand side of the author's equation is not satisfied) but the right hand of the author's equation is satisfied. Thus, a contradiction.

## Counter example 2

Second, for the connection to disagreement-based active learning.

Consider we have a hypothesis class of noiseless binary linear classifiers in two dimensions (with a bias term) and an unlabeled distribution of an origin-centered Gaussian with identity covariance. Fix an $\epsilon>0$. Suppose we have already labeled

$x_1 = (\epsilon,\epsilon), y_1 = 1$

$x_2 = (\epsilon,-\epsilon), y_2 = 1$

$x_3 = (-\epsilon,\epsilon), y_3 = 0$

$x_4 = (-\epsilon,-\epsilon), y_4 = 0$

and we trained a max-margin classifier so that $\hat{h}(x) = \mathbf{1}[x \cdot (1,0) \geq 0]$.

Consider two unlabeled points: $x_5=(2\epsilon,0), x_6=(1-\epsilon,1)$. $x_5$ is not in the disagreement region while $x_6$ is in the disagreement region. However, $L(\hat{h},x_5) = O(\epsilon)$ while $L(\hat{h},x_6) \approx 1/4$. Thus, for small enought $\epsilon$, LDM prefers the point that is NOT in the disagreement region rather than the point that is in the disagreement region.

## Conclusion

From these two counter examples (and assuming I didn't make a mistake), I find that although LDM uses basic quantities (like $\rho$ and disagreement sets) that theoretical papers in the literature also use, I can't think of any good theoretical reason to use LDM.

From: ICLR.cc/2023/Conference/Paper2398/Authors
title: Response to Additional Comments for part 2
comment: *I understood the motivations for the choices. But the choices are not constrained by the motivations. Why the exponential decay? Why a threshold into two groups rather than a comprehensive function? The weighting doesn't seem to be derived from any principle.*

: The reason for using exponential decay is that exponential decay is a very common choice as a decay function in machine learning, e.g., EXP3 in bandits (Auer et al., 2002). Such decay function penalizes samples with large LDMs when sampling.

Dividing the pool data into two groups, $\mathcal{P}\_q$ and $\mathcal{P}\_c$, was done to balance the effect of choosing samples with the least LDMs and diversity. To avoid being biased towards either LDM or diversity regardless of the choice of $m$ and $q$, we balance the total weights of $\mathcal{P}\_q$ and $\mathcal{P}\_c$ to be equal. Here, $m$ and $q$ are the given pool size and query size, respectively.

All in all, to clarify the above points, we’ve replaced Eq.5 of the previous version with the equivalent Eq.5 in Section 3.1 of our revised manuscript. Note that with the current form of $\tilde{\gamma} (\boldsymbol{x})$, the weights of $\mathcal{P}\_q$ and $\mathcal{P}\_c$ are of the same form. Precisely, our weighting is
$$\gamma (\boldsymbol{x}) = \left\\{ \begin{array}{ll} \tilde{\gamma} (\boldsymbol{x}) / \sum\_{\boldsymbol{x}_j \in \mathcal{P}\_q} \tilde{\gamma} (\boldsymbol{x}_j) = \frac{1}{q}, & \boldsymbol{x} \in \mathcal{P}\_q \\\\ \tilde{\gamma} (\boldsymbol{x}) / \sum\_{\boldsymbol{x}_j \in \mathcal{P}\_c} \tilde{\gamma} (\boldsymbol{x}_j), & \boldsymbol{x} \in \mathcal{P}\_c \end{array} \right.$$

where 

$$\tilde{\gamma} (\boldsymbol{x}) = \exp \left(-\frac{(L\_{\boldsymbol{x}} - L_q)_+}{L_q} \right),$$

$(\cdot)_+ = \max\\{0, \cdot\\}$, and $L_q = \max\_{\boldsymbol{x} \in \mathcal{P}\_q} L\_{\boldsymbol{x}}$.

Again, we emphasize that our weighting scheme, as described above, is based on the principle of querying samples with the least LDMs while pursuing diversity.


*In total, this paper introduces a theoretical concept, an empirical approximation, a batching technique, and experiments. However, the theoretical concept doesn't seem to have any good active-learning-specific properties, the empirical approximation has no guarantees except for convergence with extremely large N, the batching technique is unprincipled with arbitrary choices, and the experiments show a rather marginal gain in terms of the reduction in required labeled data. As such, I'm afraid I can't support this paper.*

: All in all, as it can be clearly seen from our responses to all of your questions, we believe that our algorithm is indeed a meaningful addition to the field of uncertainty-based active learning. We urge the reviewer to read through our responses and consider re-evaluating the decision. If you have any other questions, feel free to let us know, and we will happily answer.

From: ICLR.cc/2023/Conference/Paper2398/Authors
title: Response to Additional Comments for part 1
comment: *For the convergence, are you referring to Figure 2.b? It doesn't seem to me that Figure 2.b. does not show convergence but that the lines continue to decrease.*

: We’ve done additional experiments by considering more stop conditions $s \in \{2000, 5000, 10000, 20000, 50000, 100000\}$, which we’ve included in Appendix F. In Figure 13, one can see that as $s$ increases, the curve clearly becomes flatter, and for most samples, flatten.


*I don't think the equation given makes sense. The left hand side of the equivalence is independent of a distribution over h, while the right hand side critically depends a distribution over h. Certainly, one can construct distributions such that the equivalence is false.*

: Let us clarify our claim. Precisely, we assume that hypothesis $h$ is parameterized by some parameter $\boldsymbol{w}$. Let $\hat{\boldsymbol{w}}$ and $\boldsymbol{w}$ be the parameter of $\hat{h}=h_{\hat{\boldsymbol{w}}}$ and $h=h_{\boldsymbol{w}}$, respectively. In this setting, we claim that when $\boldsymbol{w} \sim \mathcal{N}(\hat{\boldsymbol{w}}, \mathbf{I} \sigma^2)$,
$$L(\hat{h}, \boldsymbol{x}_1) < L(\hat{h}, \boldsymbol{x}_2) \Longleftrightarrow \mathbb{P}\_{\boldsymbol{w} \sim \mathcal{N}(\hat{\boldsymbol{w}}, \mathbf{I} \sigma^2)} [h(\boldsymbol{x}_1) \ne \hat{h}(\boldsymbol{x}_1)] > \mathbb{P}\_{\boldsymbol{w} \sim \mathcal{N}(\hat{\boldsymbol{w}}, \mathbf{I} \sigma^2)} [h(\boldsymbol{x}_2) \ne \hat{h}(\boldsymbol{x}_2)].$$
We emphasize that as our algorithm uses Gaussian distribution for the perturbation of parameters, no other distribution should be considered. We’ve accordingly revised our manuscript (specifically in Section 2.1).


*I was expecting a result showing that if LDM was computed exactly, then the resulting algorithm has nice sample complexity guarantees under some assumptions.*

: In separable case, Hanneke et al. theoretically show that selecting an unlabeled sample in the disagreement region of the version space, which is the set of hypotheses that are consistent with labeled samples, attains exponential improvement over random sampling in sample complexity. The most important assumption is that the small perturbations of $\hat{h}$ are in the version space, which is generally true. With this assumption, as our algorithm chooses samples in the disagreement region of the version space, our algorithm’s sample complexity is at least that of (Hanneke et al., 2014).


*How are the weights for the other uncertainty scores calculated? It seems that quite a bit of tuning was used to construct a weight based on LDM; was the same amount of tuning used for the other uncertainty scores?*

: First, there is no hyperparameter to be tuned for our proposed weight. For other algorithms, we replaced $\tilde{\gamma}$ by 
$$\tilde{\gamma} (\boldsymbol{x}) = \exp \left(-\frac{U_q - U_\boldsymbol{x}}{U_q} \right), \quad \text{for} \ \boldsymbol{x} \in \mathcal{P}\_c$$
where $U_{\boldsymbol{x}}$ is the corresponding uncertainty score of $\boldsymbol{x}$, and $U_q$ is the $q$-th largest uncertainty score. Thus, the comparison is fair.


*On all six datasets, although LDM-S performs the best, an existing method (BADGE or even ENS-VarR) with a bit more (<10%) data outperforms LDM-S. Thus, rather than implementing LDM-S which seems to require more tuning, one might use a simpler algorithm with just a bit more data. I'm unsure if the quite small gains justify the added complexity and hyperparameters.*

: Our algorithm has a single hyperparameter, which is the stop condition $s$, for evaluating the empirical LDM. We set $s=10$ by considering rank/Pearson correlations and computing time on MNIST. Even with that choice of $s$, we observed that our algorithm showed good performance. We expect that with the same choice of $s$ (i.e., without any further need for tuning of $s$) our algorithm will perform well for other datasets that are not considered in the paper.
The gain of our algorithm is obtained without intensive hyperparameter tuning, as we discuss above. Additionally, BADGE, although of similar complexity and computing time as ours, underperforms compared to ours, and does not have any sample complexity guarantee unlike our algorithm, which we’ve discussed above to be at least the sample complexity of (Hanneke et al., 2014); ENS-VarR requires enormous computing time for implementation.

From: ICLR.cc/2023/Conference/Paper2398/Reviewer_uDhr
title: further response
comment: _On computational complexity of approximating empirical LDM_

Thank you for appending experiment results showing empirical LDM can be approximated in practical with reasonble budget.

_On the corollary I mentioned_
I apologize for my ambiguous expression. I was thinking of the final hypothesis (or the final hypothesis set) returned by the whole batch active learning algorithm. This seems not quite relevent to the main point of the paper and can be ignored.

From: ICLR.cc/2023/Conference/Paper2398/Reviewer_xLdz
title: Reviewer response
comment: (5) "a bit hacky and arbitrary"

I understood the motivations for the choices. But the choices are not constrained by the motivations. Why the exponential decay? Why a threshold into two groups rather than a comprehensive function? The weighting doesn't seem to be derived from any principle.


In total, this paper introduces a theoretical concept, an empirical approximation, a batching technique, and experiments. However, the theoretical concept doesn't seem to have any good active-learning-specific properties, the empirical approximation has no guarantees except for convergence with extremely large N, the batching technique is unprincipled with arbitrary choices, and the experiments show a rather marginal gain in terms of the reduction in required labeled data. As such, I'm afraid I can't support this paper.

From: ICLR.cc/2023/Conference/Paper2398/Reviewer_xLdz
title: Reviewer response
comment: Thank you for your response to my review.

(1) "empirical LDM and LDM are quite different"

Thank you for the additional experiments. I agree that it appears that the ordering of the points remains approximately the same for a variety of (relatively small) N. For the convergence, are you referring to Figure 2.b? It doesn't seem to me that Figure 2.b. does not show convergence but that the lines continue to decrease.

(2) "no theory is given why this is the a good notion of uncertainty"

I don't think the equation given makes sense. The left hand side of the equivalence is independent of a distribution over h, while the right hand side critically depends a distribution over h. Certainly, one can construct distributions such that the equivalence is false.

I was expecting a result showing that if LDM was computed exactly, then the resulting algorithm has nice sample complexity guarantees under some assumptions.

(3) "due to batching or due to the superiority of the introduced uncertainty score"

Thank you for the additional experiments. How are the weights for the other uncertainty scores calculated? It seems that quite a bit of tuning was used to construct a weight based on LDM; was the same amount of tuning used for the other uncertainty scores?

(4) "The experimental gains are quite small"

On all six datasets, although LDM-S performs the best, an existing method (BADGE or even ENS-VarR) with a bit more (<10%) data outperforms LDM-S. Thus, rather than implementing LDM-S which seems to require more tuning, one might use a simpler algorithm with just a bit more data. I'm unsure if the quite small gains justify the added complexity and hyperparameters.

From: ICLR.cc/2023/Conference/Paper2398/Authors
title: Reply to Reviewer xLdz (part 1)
comment: *I am concerned that empirical LDM and LDM are quite different: I would guess that N would need to scale exponentially in the dimension (e.g., for linear classifiers) which is not computationally feasible for high-dimensional data. Using smaller N (or rather s, which I'm not sure is reported in the paper) like in the experiments isn't necessarily bad, but might be very different from the theoretical value. In fact, given the expressivity of neural networks, I would guess that the exact theoretical LDM of any point is very close to 0.*

: Indeed, as the reviewer pointed out, theoretically, for the empirical LDM to converge to the true LDM, $N$, the number of sampled hypotheses, would need to be very large in high dimensions. However, in Section 2.3, for MNIST, we show that using our hypotheses sampling scheme (Algorithm 1), the empirical LDM seems to converge to a strictly positive number, even with $N$ relatively small. (Of course for Algorithm 1, we vary the stopping condition $s$, which is equivalent to varying $N$; see Appendix D.1). Moreover, even in this regime of $N$, the rank/Pearson correlation is observed to be almost 1. We’ve included the above discussions in our revised manuscript.


*While the theoretical definition is quite intuitive, no theory is given why this is the a good notion of uncertainty. Is it possible to show that theoretical LDM has guarantees if computed exactly?*

: The sample’s uncertainty for a given predictor is the confidence in the predicted label, i.e., if the sample’s uncertainty is high, even a small perturbation of the predictor could alter the predicted label of the sample. From this intuition, we’ve motivated and defined the least disagree metric (LDM). Thus, it is expected that our proposed LDM has some connection with the intuition of uncertainty. Precisely, we claim that
$$L(\hat{h}, \boldsymbol{x}_1) < L(\hat{h}, \boldsymbol{x}_2) \Longleftrightarrow \mathbb{P} [h(\boldsymbol{x}_1) \ne \hat{h}(\boldsymbol{x}_1)] > \mathbb{P} [h(\boldsymbol{x}_2) \ne \hat{h}(\boldsymbol{x}_2)]$$
where $h$ is sampled with $w \sim \mathcal{N} (\hat{w}, \mathbf{I} \sigma^2)$.
In Appendix B.1, we rigorously proved this claim in two-dimensional binary classification with the linear classifier, where the theoretical LDM can be computed exactly as in the example in Figure 1.


*It is not clear if the gains of LDM-S over the standard uncertainty methods is due to batching or due to the superiority of the introduced uncertainty score. For an appropriate comparison, I think either batching should be done for the standard uncertainty methods, or no batching with LDM.*

: Thank you for your valuable suggestion. Indeed we missed your point in that our comparison wasn’t exactly fair as we did not apply seeding (batching) to the standard uncertainty methods (Entropy, MC-BALD, ENS-VarR). In Appendix D.3, we’ve included the additional results of running the aforementioned methods with seeding. Even in that case, it is shown that our LDM-S still outperforms all the other algorithms over all datasets, which implies that our introduced uncertainty score (LDM) does show some superiority over other methods. If the reviewer feels that this result should be included in the main text, we will gladly do so in our next revision.


*The experimental gains are quite small (compared to BADGE or BALD).*

: We believe that when comparing various active learning algorithms, one should consider the overall performance across the considered datasets, i.e., although showing that the algorithm works particularly well for a particular dataset is meaningful, we believe that showing that the algorithm works fairly well over a wide range of datasets is just as meaningful.
To do this, we’ve included the Dolan-More curve (Dolan & More, 2001), which is a cumulative plot of the fraction of cases where the performance gap between the considered algorithm and the best competitor is less than $\delta$, as $\delta$ increases. Such curve has been used in benchmarking active learning (Tsymbalov et al., 2018; 2019), optimization (Dolan & More, 2002), and even general deep learning tasks (Burnaev et al., 2015a;b). As shown in Figure 5 of Section 4.2 (newly added), our LDM-based active learning algorithms perform consistently better than the other algorithms, across the considered range of $\delta$. We’ve included the above discussions in our revised manuscript.

From: ICLR.cc/2023/Conference/Paper2398/Authors
title: Reply to Reviewer xLdz (part 2)
comment: *The batching procedure with k-means++ seems a bit hacky and arbitrary (e.g., equation (5)). Can this be justified?*

: In Section 3.1 of our manuscript, above Eqn. (5), we’ve provided detailed motivations on how we came to the form of Eqn. (5). To emphasize once more, we wanted 1) samples of $\mathcal{P}_q$ to be weighted the same, 2) samples of $\mathcal{P}_c$ to be weighted according to their LDMs, and 3) the sum of the weights of $\mathcal{P}_q$ and $\mathcal{P}_c$ are the same. 1) is to ensure that we choose samples with the least LDM (equivalently, the most uncertain samples); 2) is for injecting diversity while taking the remaining LDMs into account as well; 3) to balance the effect of 1) and 2). Based on this weighting, we’ve defined the sampling probability of each sample $\boldsymbol{x}$ as shown below Eqn. (5). This is largely inspired by k-means++-based active learning (Ash et al., 2020). We’ve included this discussion in our revised manuscript.

*There are some undefined notations like d_H(), which I assume is rho*

: $d_{\mathcal{H}}(\cdot, \cdot)$ is defined as the underlying metric for the hypothesis space H, which was clearly defined in Assumption 1 of Section 2.2.


*I think the algorithm would be clearer if variables were not re-used, like L_S(h, x)*

: Thank you for your suggestion. In the revised manuscript, we’ve modified the notations in the algorithm to make it clearer.

From: ICLR.cc/2023/Conference/Paper2398/Authors
title: Reply to Reviewer uDhr
comment: *As the expression "brute force" used by authors, how the comparison of computational complexity may influence the superiority of the proposed method against existing methods.*

: As the reviewer pointed out, theoretically, for the empirical LDM to converge to the true LDM, $N$, the number of sampled hypotheses, would need to be very large, and thus the computational complexity may be a bottleneck. However, in Section 2.3, we show that the empirical LDM of $s = 10$ is highly correlated with that of $s = 1000$ (we actually show that the rank/Pearson correlation is almost 1). Thus, in the actual implementation, we use $s = 10$, which incurs only a slight computational overhead than Entropy (3~6%); this is comparable to Coreset, MC-BALD, and BADGE, and much less computational overhead than ENS-VarR. The complete result can be seen in Appendix E.1. All in all, even though the very nature of our approach is “brute-force”, we’ve empirically shown that our approach has moderate computational complexity while being superior to the existing methods.


*A corollary of the generalization bound of the hypothesis returned by the proposed algorithm may be needed for the self-containedness of the paper.*

: We apologize, but we do not fully understand what you mean by “hypothesis returned by the proposed algorithm.” If you meant the sampled hypotheses (for computing empirical LDM), then we would like to clarify that the hypothesis $\hat{h}$ and the data point $\boldsymbol{x}$ w.r.t. which we intend to compute the empirical LDM are fixed beforehand. Algorithm 1 doesn’t return any hypothesis; it only returns the value of empirical LDM.

From: ICLR.cc/2023/Conference/Paper2398/Authors
title: Reply to Reviewer gsKX
comment: *A more complete explanation is missing. At the beginning of section 2.2 of the article, it was mentioned that LDM is not computable in most cases, but it was so brief that it didn’t explain why LDM is hard to compute. It jumped directly from the mathematical definition to the definition of empirical LDM.*

: There are two parts as to why LDM is not computable in general. One is that $\rho$, the disagree metric defined as $\mathbb{P}_{X \sim \mathcal{D}_\mathcal{X}} [h(X) \not= \hat{h}(X)]$, is generally intractable, especially when the data distribution of $\mathcal{X}$ and the form of $h$ are both complicated, e.g., neural networks over real-world image datasets. Another is that one needs to take an infimum over $\mathcal{H}$, which is usually an infinite set. In the revised manuscript, we’ve included such discussions at the beginning of Section 2.2.


*Mathematical formulas are missing definitions. Eg. P is not defined in Section 2.1 of the paper; It is suggested to be able to explane d_H (h^*,h) in the Equation 3 of Theorem 1 in Section 2.2 of the paper.*

: $\mathbb{P}_{X \sim \mathcal{D}_\mathcal{X}}$ in Section 2.1 is the probability measure on $\mathcal{X}$, induced by the distribution $\mathcal{D}_\mathcal{X}$ over $\mathcal{X}$. We’ve made it clear in our revised manuscript. Also, $d_\mathcal{H}(\cdot, \cdot)$ is defined as the underlying metric for the hypothesis space $\mathcal{H}$, which was clearly defined in Assumption 1 of Section 2.2.


*More arguments are missing. The uncertainty of samples and the diversity among samples are both key factors in active learning. Chapter 3 of the paper introduces the active learning sampling strategy based on LDM, which combines LDM with the diversity among samples. The sampling strategy adopted is to first query samples with low LDM and then query samples with high diversity. However, if the categories of the samples queried by LDM are very close, will the similarity among the selected samples still be high?*

: Yes, you’re right. In most cases, the features are quite different due to the different classes. $\mathbb{P}({\boldsymbol x})$, which is used for seeding (see Section 3.1), depends on the LDM and distance between features. Thus, when the classes are different, as the corresponding features are different, it must be that the $\mathbb{P}({\boldsymbol x})$ is large, and vice-versa.


*[Summary] However, the experimental results show that the proposed method has not achieved significant improvements than the other comparisons.*

: We believe that when comparing various active learning algorithms, one should consider the overall performance across the considered datasets, i.e., although showing that the algorithm works particularly well for a particular dataset is meaningful, we believe that showing that the algorithm works fairly well over a wide range of datasets is just as meaningful.
To do this, we’ve included the Dolan-More curve (Dolan & More, 2001), which is a cumulative plot of the fraction of cases where the performance gap between the considered algorithm and the best competitor is less than $\delta$, as $\delta$ increases. Such curve has been used in benchmarking active learning (Tsymbalov et al., 2018; 2019), optimization (Dolan & More, 2002), and even general deep learning tasks (Burnaev et al., 2015a;b). As shown in Figure 5 of Section 4.2 (newly added), our LDM-based active learning algorithms perform consistently better than the other algorithms, across the considered range of $\delta$. We’ve included the above discussions in our revised manuscript.

From: ICLR.cc/2023/Conference/Paper2398/Authors
title: Reply to Reviewer kkTt
comment: *The authors should implement a toy experiment to show the proposed algorithm can accurately estimate the LDM. In the theoretical proof, the empirical value of LDM converges to the mathematical definition as the sample size is infinity. However, in the application, it is impossible to get an infinity sample size, so the author should explore how large the estimation error is and whether it is acceptable or not. The author can take the binary classification in this paper as an example to plot a figure to show the estimation error between the estimated LDM and the ground truth.*

: Thank you for your valuable suggestion. Indeed our theoretical result for ensuring asymptotic consistency requires infinite sample sizes. In Section 2.3, we’ve included a toy experiment with two-dimensional binary classification with a set of linear hypotheses (see Figure 1 of Section 2.1) to see how close the empirical LDM (obtained using our Algorithm 1) is to the true LDM. We’ve varied $S \in \{1000, 10000, 100000\}$ and $s \in \{1, 2, 5, 10, 20, 50, 100, 200, 500, 1000\}$, where $S$ is the number of samples for estimation of rho, and $s$ is the stop condition of Algorithm 1. The results show that the empirical LDM estimated by our Algorithm 1 reaches the true LDM, even when $s \ge 20$, with an estimation error of ~$10^{-4}$. This trend remains largely unchanged over the considered values of $S$. We’ve included these discussions in our revised manuscript.


*The empirical result is not convincing. In figure 4, it shows the performance of different methods. I do not find LDM-S outperform the state-of-the-art methods significantly, especially in dataset SVHN and Food101. In addition, the authors also need to draw the error bar, which will convince the reader  the LDM-S is much better than others.*

: We believe that when comparing various active learning algorithms, one should consider the overall performance across the considered datasets, i.e., although showing that the algorithm works particularly well for a particular dataset is meaningful, we believe that showing that the algorithm works fairly well over a wide range of datasets is just as meaningful.
To do this, we’ve included the Dolan-More curve (Dolan & More, 2001), which is a cumulative plot of the fraction of cases where the performance gap between the considered algorithm and the best competitor is less than $\delta$, as $\delta$ increases. Such curve has been used in benchmarking active learning (Tsymbalov et al., 2018; 2019), optimization (Dolan & More, 2002), and even general deep learning tasks (Burnaev et al., 2015a;b). As shown in Figure 5 of Section 4.2 (newly added), our LDM-based active learning algorithms perform consistently better than the other algorithms, across the considered range of $\delta$. We’ve included the above discussions in our revised manuscript.
Lastly, we thank the reviewer for suggesting to include the error bar plot. We also thought that it was a very good idea, but the visibility of the error bar plot of our results was too low. Regretfully, we’ve replaced it with Table 2, which shows the mean and standard deviation of each experimental setting.


*The authors should report the computing time and how it is related to the number of sampled hypotheses. How many hypotheses do you choose in experiments?*

: Thank you for your valuable suggestion. In Appendix E.1, we’ve included a table showing the running time (in min), measured in wall-clock time, across our considered datasets and algorithms in Appendix ?. Other than MNIST, LDM-S has slightly more runtime overhead than Entropy (3~6%). As expected, we’ve observed that the runtime is proportional to the number of sampled hypotheses; this is empirically verified in Appendix ?. We’ve set the stop condition to $s = 10$. For instance, in MNIST, this amounts to about 5000 hypotheses being sampled. 

*Reproducibility The algorithm is clear in the main paper, but the authors do not provide how they set the parameters and the hyper parameters, such as learning rate, hidden dimensions, network layers, etc in appendix.*
  
: The details requested by the reviewer are clearly reported in Appendix C.

From: ICLR.cc/2023/Conference/Paper2398/Authors
title: Revision Summary
comment: We thank the reviewers for constructive comments and suggestions, and we’ve revised the paper accordingly. Some big changes include:

* Thanks to Reviewer kkTt for asking whether the proposed algorithm accurately estimates the true LDM. To address this issue, in Section 2.3, we’ve added toy experiments showing that empirical LDM converges to the true LDM. 
Also, we would like to thank Reviewer kkTk, uDhr, xLdz for pointing out the computational issue of empirical LDM. In Section 2.3, we’ve done some empirical analyses on MNIST and deep networks, and showed that using a moderate stop condition $s$ is sufficient since the rank order of the samples’ empirical LDMs is maintained moderate $s$ (e.g., $s = 10$). Moreover, we show in Appendix E.1 that with this choice of $s$, the computational overhead is at least comparable or better than other algorithms such as Entropy, MC-BALD, ENS-VarR.

* We would like to thank Reviewer kkTt, gsKX, xLdz for pointing out that our experimental results seemed a bit weak and for giving various suggestions. In Section 4.2, we’ve included Dolan-More curve (previously used in benchmarking active learning tasks; Tsymbalov et al., IJCAI’19), as well as relevant discussions, to clearly show that our algorithm outperforms other algorithms across the considered datasets; in Appendix E.1, we’ve included a comparison of the runtime of the algorithms; in Appendix D.3, we’ve added in additional experiments such that we compare ours with the standard uncertainty methods with weighted seeding to show that indeed the performance boost can be attributed to the superiority of our newly designed uncertainty measure LDM.

* We thank Reviewer xLdz for asking the fundamental question of why our LDM is indeed a good notion of uncertainty. In Section 2.1 we’ve provided additional discussions on why our LDM is a good measure; we’ve supplemented this with theoretical verifications in Appendix B.1. 

In our replies to comments from each reviewer, we’ve answered each individual question of each reviewer in detail. We will continually update this comment whenever a new revision comes up, and notify all the reviewers.

From: ICLR.cc/2023/Conference/Paper2398/Reviewer_kkTt
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This work proposed a new metric to quantify the uncertainty in active learning, named least degree metrics (LDM).
The key idea is to measure how robust a sample point is to a perturbation in prediction. 
The authors prove that the LDM can guarantee theoretical properties that LDM can converge in probability to zero as sample size goes to infinity and empirical estimation converges to mathematic definition. 
In the empirical example, it shows querying the smallest LDM does not have the best performance and the diversity in batch should be considered. After accounting for the batch diversity, the authors proposed LDM-S and compare it to many competing methods in real dataset.
strength_and_weaknesses: Strength:
1.	The authors propose a novel method to quantify the uncertainty
2.	The metric has many good theoretical properties. 

Weakness:
1.	The authors should implement a toy experiment to show the proposed algorithm can accurately estimate the LDM. In the theoretical proof, the empirical value of LDM converges to the mathematical definition as the sample size is infinity. However, in the application, it is impossible to get an infinity sample size, so the author should explore how large the estimation error is and whether it is acceptable or not. The author can take the binary classification in this paper as an example to plot a figure to show the estimation error between the estimated LDM and the ground truth. 
2.	The empirical result is not convincing. In figure 4, it shows the performance of different methods. I do not find LDM-S outperform the state-of-the-art methods significantly, especially in dataset SVHN and Food101. In addition, the authors also need to draw the error bar, which will convince the reader the LDM-S is much better than others. 
3.	The authors should report the computing time and how it is related to the number of sampled hypotheses. How many hypotheses do you choose in experiments?

clarity,_quality,_novelty_and_reproducibility: Clarity
The writing is clear and easy to follow. 

Quality
The idea is clear and novel. The metric has good theoretical properties, but the authors do not show the proposed algorithm can accurately estimate LDM and do not provide strong empirical evidence to convince reader the method outperform the other competing methods. (See weakness). Almost experiments result is mainly about LDM-S is the best compared to other LDM variants, but failing to demonstrate LDM-S is much more appealing than the other state-of-the-art methods.

Novelty
This paper proposes a novel metric to quantify the uncertainty in active learning. The idea is new. 

Reproducibility
The algorithm is clear in the main paper, but the authors do not provide how they set the parameters and the hyper parameters, such as learning rate, hidden dimensions, network layers, etc in appendix.

summary_of_the_review: This paper needs to address the above mentioned concerns.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2398/Reviewer_gsKX
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes a method to measure the uncertainty of a sample by perturbing the predictor to see how the predicted label is altered w.r.t. that perturbation, and incorporate it into active learning.  An active learning algorithm sampling unlabeled samples based on the empirical LDM is designed. The least disagree metric (LDM) is mathematically defined. In the proposed method,  an empirical LDM is incoporated into a brute-force algorithm to estimat the theoretical LDM of each sample.
strength_and_weaknesses: Strength: 
1. The experimental design and implementation are complete. For the comparison factors involved in the sample sampling of LDM, the author has carried out rigorous and complete experimental verification.
2. The article is logically rigorous and has a complete structure. In the mathematical definition of LDM and empirical LDM, the formula derivation process is complete and rigorous, and the overall structure of the article is relatively complete.

Weakness: 
1. A more complete explanation is missing. At the beginning of section 2.2 of the article, it was mentioned that LDM is not computable in most cases, but it was so brief that it didn’t explain why LDM is hard to compute. It jumped directly from the mathematical definition to the definition of empirical LDM. 
2. Mathematical formulas are missing definitions. Eg. P is not defined in Section 2.1 of the paper; It is suggested to be able to explane d_H (h^*,h) in the Equation 3 of Theorem 1 in Section 2.2 of the paper.
3．More arguments are missing. The uncertainty of samples and the diversity among samples are both key factors in active learning. Chapter 3 of the paper introduces the active learning sampling strategy based on LDM, which combines LDM with the diversity among samples. The sampling strategy adopted is to first query samples with low LDM and then query samples with high diversity. However, if the categories of the samples queried by LDM are very close, will the similarity among the selected samples still be high? 

clarity,_quality,_novelty_and_reproducibility: Clarity: The idea of the paper is clear, and the derivation process of the formula is rigorous and complete.
Quality: The starting point of the article is high, and I hope to propose a general method for quantifying sample uncertainty. The LDM algorithm proposed by the author also basically met expectations, so the quality of the article is high.
Novelty: The main innovation of this paper is to propose a method that can be universally used in various deep active learning tasks to quantify sample uncertainty, and has a smaller computational cost than other methods.

summary_of_the_review: This paper proposes a novel active learning criterion for deep learning based on least disagree metric. The theoretical anlysis show that the proposed method with solid foundation. However, the experimental results show that the proposed method has not achieved significant improvements than the other comparisons. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: None
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2398/Reviewer_uDhr
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes a new uncertainty evaluation method using the disagree metric defined in the active learning literature, and applies it to a pool-based active learning algorithm. A consistent estimator is also proposed, and the active learning algorithm is also improved by taking account the lack of diversity of each selected batch. Detailed empirical exploration and extensive comparisons with various existings pool-based active learning methods are conducted to show the superiority of the proposed method.
strength_and_weaknesses: Strength
- The paper is well-constructed and well-written, overall easy to follow.
- The new uncertainty evaluation metric is proposed followed by a consistent estimator, which shows meaningfulness both theoretically and empirically.
- The drawback of not considering diversity if illustrated using the MNIST example.
- Sufficient comparisons are conducted againts typical existing active learning methods.

Weakness
- As the expression "brute force" used by authors, how the comparison of computational complexity may influence the superioty of the proposed method against existing methods.
- A corollary of the generalization bound of the hypothesis returned by the proposed algorithm may be needed for the self-containedness of the paper.
clarity,_quality,_novelty_and_reproducibility: - The paper is well-organized and shows sufficient clarity.
- The paper has relatively high quality of contribution considering the theoretical self-containedness of the proposed uncertainty measurement.
- The idea has some novelty. Although it is based on a metric proposed in the literature, adapt it for uncertainty measurement and develop a practical active learning algorithm based on it requires sufficent work.
- The paper has high reproducibility as the detailed steps of algorithms are clearly described.

summary_of_the_review: I woule like to recommend to accept the paper considering its sufficent quality of devloping a new uncertainty measurement and a pratical active learning algorithm.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2398/Reviewer_xLdz
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes an uncertainty score referred to as "Least Disagree Metric" (LDM). Intuitively, this score measures the distance (in disagreement) to the nearest hypothesis that yields a different classification prediction. As this score is intractable to compute, the paper introduces the "Empirical LDM", which is a finite sample approximation that is consistent in the limit of many samples. This score is used with a batching technique (k-means++) as the empirical algorithm. Experimental results are shown on 6 datasets.
strength_and_weaknesses: Strengths:
 - The paper is quite clear and the story is easy to follow.
 - The proposed method is the highest performing in the experiments.
 - The proposed method has connections to other areas of active learning.

Weaknesses:
 - I am concerned that empirical LDM and LDM are quite different: I would guess that N would need to scale exponentially in the dimension (e.g., for linear classifiers) which is not computationally feasible for high-dimensional data. Using smaller N (or rather s, which I'm not sure is reported in the paper) like in the experiments isn't necessarily bad, but might be very different from the theoretical value. In fact, given the expressivity of neural networks, I would guess that  the exact theoretical LDM of any point is very close to 0.
 - While the theoretical definition is quite intuitive, no theory is given why this is the a good notion of uncertainty. Is it possible to show that theoretical LDM has guarantees if computed exactly?
 - It is not clear if the gains of LDM-S over the standard uncertainty methods is due to batching or due to the superiority of the introduced uncertainty score. For an appropriate comparison, I think either batching should be done for the standard uncertainty methods, or no batching with LDM. 
 - The experimental gains are quite small (compared to BADGE or BALD).
 - The batching procedure with k-means++ seems a bit hacky and arbitrary (e.g., equation (5)). Can this be justified?
clarity,_quality,_novelty_and_reproducibility:  - The paper is overall quite clear.
 - The ideas in the paper are quite novel.
 - There are some undefined notations like $d_\mathcal{H}(\cdot,\cdot)$, which I assume is $\rho$. 
 - I think the algorithm would be clearer if variables were not re-used, like $L_S(\hat{h},x)$.


summary_of_the_review: Overall, while novel and mostly clear, this paper defines a heuristic algorithm which performs only marginally better which may be at least partially due to the use of batching. I think the significance of this paper is low, especially given the large number of active learning algorithms in the literature that propose heuristics which only perform marginally better.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough