Paper: Uncertainty for Active Learning on Graphs (ICLR-2024)

Reviewer comment (Theory):
Theoretical contributions in this paper appear to be somewhat limited. The proposed uncertainty estimation is based on the posterior probability given the ground-truth label of the unobserved nodes. However, the essence of active learning lies in addressing this problem without access to ground-truth information, which remains inadequately addressed.

Author response:
We understand the concerns the reviewer has about the applicability of our results to a practical setting in which neither all labels nor the generative process is available. Our analysis, however, is purely theoretical: We show that ground-truth epistemic uncertainty is an optimal AL strategy. This result about the alignment of uncertainty estimation and AL also holds on real graphs, even though the proposed ground-truth uncertainties can not be computed exactly in practice. Our goal is to formally justify why we should expect US sampling to be an effective strategy in the first place. Furthermore, we can formally prove that disentanglement plays a crucial role in effective US, which again translates to real-world scenarios. Our work therefore provides a sound, theoretical basis on which novel US can be developed and applied to AL for interdependent data, which we believe to be a valuable contribution to the field.

We want to point out that it is not the access to ground-truth data that is central to the effectiveness of epistemic US: Measures of total, aleatoric, and epistemic uncertainty are defined in terms of ground-truth information (see Defintions 1-3) and exhibit very different efficacy to AL.

Nonetheless, we agree that our manuscript benefits from a more practical evaluation as well. To that end, we extended the theoretical framework to also allow a classifier to model the parameters of an underlying process in a learnable fashion, a framework that GNNs for example fit into. Based on this, we apply our theoretical insights to real-world data using GNNs as an approximative approach to our proposed, unknown ground-truth uncertainties. In this evaluation, we do not utilize any information about unavailable labels or the generative process. Nonetheless, this very off-the-shelf application of our findings elevates the uncertainty of a GNN classifier to outperform SOTA methods on multiple datasets. We hope to specifically also address weakness b with this additional empirical study, as we now propose an approximate estimator of epistemic uncertainty that aligns with our theory. We additionally discuss potential approximative errors of this simplistic approach in Appendix H.

Overall, we also adapted the manuscript to address the overall intention of our analysis more clearly: Our goal is not the proposal of a novel AL strategy, but rather to bridge the gap between principled US and AL on graphs. We theoretically prove that disentangled US and AL align in general. We reveal pitfalls in UQ and how to address them with an evaluation in an idealized setting on a CSBM. Lastly, we confirm the applicability to real-world scenarios by evaluating a simple approximative realization of our insights on real graphs and observing strong performance.

Reviewer comment (Experiments):
The experimental results provided are restricted to synthetic data, and the method's reliance on knowledge of the true data generation process probabilities pose practical challenges. How to approximately compute the estimation remains unclear.

Author response:
Nonetheless, we agree that our manuscript benefits from a more practical evaluation as well. To that end, we extended the theoretical framework to also allow a classifier to model the parameters of an underlying process in a learnable fashion, a framework that GNNs for example fit into. Based on this, we apply our theoretical insights to real-world data using GNNs as an approximative approach to our proposed, unknown ground-truth uncertainties. In this evaluation, we do not utilize any information about unavailable labels or the generative process. Nonetheless, this very off-the-shelf application of our findings elevates the uncertainty of a GNN classifier to outperform SOTA methods on multiple datasets. We hope to specifically also address weakness b with this additional empirical study, as we now propose an approximate estimator of epistemic uncertainty that aligns with our theory. We additionally discuss potential approximative errors of this simplistic approach in Appendix H.

**Addressing Question a:** Estimating the true generative process of graph data is indeed a challenging topic. The aforementioned approach utilizes a simple GNN to address this problem and exhibits already reasonable performance. Nonetheless, we believe that our work can guide US toward the development of more faithful estimators: For example, on CSBMs, we observed non-edges to play a crucial role in successful US. Contemporary GNNs do not explicitly utilize such information which we conjecture to be one interesting direction for future research.

Reviewer comment (Experiments):
In the empirical evaluation, the compared baselines are only random queries and other uncertainty-based methods, the state-of-the art methods are missing, e.g. SEAL[1] and IGP[2].

Author response:
**Addressing Weakness c:**
We thank the reviewer for the interesting pointer to these two methods. SEAL is a very interesting approach and we have run experiments using SEAL and included them in all benchmarking Figures and Tables in our updated manuscript. As one can see, it provides merit on some graphs while not outperforming random acquisition on others. However, tuning the hyperparameters of SEAL might improve performance even further which we would include in the camera-ready version of the paper. These results strengthen our point further that non-UQ-based AL methods can outperform random queries while US falls short of this so far.

The IGP approach deals with a very interesting setting. Assuming soft labeling is cheaper, they query the oracle with the predicted label and retrieve the answer as to whether or not the prediction was correct without acquiring its true label. We included a description of this setting in our revised related work section since we believe it is interesting to the readers. However, their setting is not directly applicable to an AL scenario with an oracle that returns the true label of an instance which is what most literature in AL on graphs discusses.

While those two approaches do not fundamentally change any of our results in the benchmark or analysis, we think including them in our paper improves the manuscript, and thus we are thankful to the reviewer for pointing them out.

Reviewer comment (Writing):
The paper's presentation could be improved. For instance, when introducing concepts like aleatoric and epistemic uncertainty, the authors provide limited explanations and intuitions, potentially causing readers unfamiliar with these terms to struggle to follow the paper.

Author response:
**Addressing Weakness d:** We allocated more explanation to the aleatoric and epistemic uncertainty notions to the background section 2, including an intuitive example to better help readers familiarize themselves with the concepts of the topic (see the updated version of our manuscript). If you have any concerns about the paper’s presentation, we are happy to address them as well.

Reviewer comment (Experiments):
The study primarily concentrates on a specific graph type, the CSBM, which might not fully represent the characteristics of all real-world graphs.

Author response:
We follow the reviewer’s suggestion to provide more evidence for applicability of our results to real-world problems. Nonetheless, the result that US is an optimal strategy for AL still holds, even if the true epistemic uncertainty can not easily be computed in practice. We both theoretically prove and empirically (on CSBMs) this optimality. Hence, we expect a strong, disentangled UQ method to perform well in AL. Our work motivates principled UQ and shows its relationship to AL formally and does not explicitly aim at proposing a novel improved AL strategy. We updated our paper to make this point more clear.

We also acknowledge the concerns regarding transferability to real-world settings. To that end, in our updated manuscript, we now introduce a method to apply our theory to real-world data in Section 6, paragraph “Real-World Data”. We detail a straightforward framework that models the epistemic uncertainty as the difference between total and aleatoric. This strategy is directly derived from our theoretic results and does not introduce any other components or tuning: The total uncertainty originates from a classifier trained solely on labeled data. We then apply the predictions from this classifier as pseudo labels to train a subsequent model on all labeled nodes (pseudo labels + real label) to model the aleatoric uncertainty. Our method outperforms many state-of-the-art US strategies on most datasets (all Figures can be seen in the updated version, Section 6 and Appendix H). We also provide a discussion of potential failure modes of this framework in Appendix H.

This experiment shows that our analysis translates to real-world graphs, where we explicitly do not use any unavailable data or knowledge about the generative process. Additionally, we adapted our theory to a setting where the Bayesian classifier has to learn the parameters of the underlying unknown generative process. GNNs that predict marginal probabilities for each node fit into this framework. Since our optimality guarantees still hold under these relaxed assumptions, we believe the applicability of our findings to be even more clearly relevant to practical scenarios.

We hope that with these results on real data, we can convince the reviewer that the theoretical insights on CSBMs apply to real-world graphs as the newly proposed framework is a simple straightforward implementation thereof. We want to mention that we still believe that, even though we outperform other SOTA AL strategies on multiple datasets, our theoretic analysis is the main contribution, as it bridges the gap between understanding the relationship between US and AL on graphs which has not been discussed in previous literature. As an example of how to incorporate our findings, one could try to explicitly tailor GNNs towards better modeling an assumed generative process, e.g. by considering the non-edges in a transformer-like fashion. Furthermore, as a consequence of our result that US and AL are well aligned, we strongly motivate using AL as a test of quality for newly devised uncertainty estimators (which are currently mostly evaluated on the detection of distribution shifts).

We also revised our manuscript to make the structure and value of our analysis clear: We first theoretically prove the alignment between epistemic US and AL and then proceed to support this with empirical evidence. First, we focus on a simplified CSBM setting and access to all labels. This analysis isolates the effects stemming from proper disentanglement and data modeling. We then proceed to a real-world setting, where we have to rely on various approximations to apply our theoretical insights. We observe our framework to show high efficacy nonetheless. This structure enables us to both propose theoretical soundness of the results we provide as well as high practicability at the same time.

Reviewer comment (Novelty):
Novelty concern: undoubtedly, this work offers an extensive exploration of uncertainty-based Active Learning (AL) within the context of graphs. However, it does not introduce any novel methods for active learning in the graph domain.

Author response:
We follow the reviewer’s suggestion to provide more evidence for applicability of our results to real-world problems. Nonetheless, the result that US is an optimal strategy for AL still holds, even if the true epistemic uncertainty can not easily be computed in practice. We both theoretically prove and empirically (on CSBMs) this optimality. Hence, we expect a strong, disentangled UQ method to perform well in AL. Our work motivates principled UQ and shows its relationship to AL formally and does not explicitly aim at proposing a novel improved AL strategy. We updated our paper to make this point more clear.

We also acknowledge the concerns regarding transferability to real-world settings. To that end, in our updated manuscript, we now introduce a method to apply our theory to real-world data in Section 6, paragraph “Real-World Data”. We detail a straightforward framework that models the epistemic uncertainty as the difference between total and aleatoric. This strategy is directly derived from our theoretic results and does not introduce any other components or tuning: The total uncertainty originates from a classifier trained solely on labeled data. We then apply the predictions from this classifier as pseudo labels to train a subsequent model on all labeled nodes (pseudo labels + real label) to model the aleatoric uncertainty. Our method outperforms many state-of-the-art US strategies on most datasets (all Figures can be seen in the updated version, Section 6 and Appendix H). We also provide a discussion of potential failure modes of this framework in Appendix H.

This experiment shows that our analysis translates to real-world graphs, where we explicitly do not use any unavailable data or knowledge about the generative process. Additionally, we adapted our theory to a setting where the Bayesian classifier has to learn the parameters of the underlying unknown generative process. GNNs that predict marginal probabilities for each node fit into this framework. Since our optimality guarantees still hold under these relaxed assumptions, we believe the applicability of our findings to be even more clearly relevant to practical scenarios.

We hope that with these results on real data, we can convince the reviewer that the theoretical insights on CSBMs apply to real-world graphs as the newly proposed framework is a simple straightforward implementation thereof. We want to mention that we still believe that, even though we outperform other SOTA AL strategies on multiple datasets, our theoretic analysis is the main contribution, as it bridges the gap between understanding the relationship between US and AL on graphs which has not been discussed in previous literature. As an example of how to incorporate our findings, one could try to explicitly tailor GNNs towards better modeling an assumed generative process, e.g. by considering the non-edges in a transformer-like fashion. Furthermore, as a consequence of our result that US and AL are well aligned, we strongly motivate using AL as a test of quality for newly devised uncertainty estimators (which are currently mostly evaluated on the detection of distribution shifts).

We also revised our manuscript to make the structure and value of our analysis clear: We first theoretically prove the alignment between epistemic US and AL and then proceed to support this with empirical evidence. First, we focus on a simplified CSBM setting and access to all labels. This analysis isolates the effects stemming from proper disentanglement and data modeling. We then proceed to a real-world setting, where we have to rely on various approximations to apply our theoretical insights. We observe our framework to show high efficacy nonetheless. This structure enables us to both propose theoretical soundness of the results we provide as well as high practicability at the same time.

Reviewer comment (Experiments):
In the experimental results, such as Figure 3, the curves depicting acquired labels versus accuracy exhibit significant fluctuations. Did the authors conduct repeated trials to mitigate these fluctuations in the model's performance?

Author response:
**Addressing Question 1**: As we use multiple dataset splits and model initializations, all curves are already average performances. We added plots for the requested experiments including the standard deviation to the Appendix (Figure 9), such that readers can see the high variance. Further, we added a description of the repeated trial setting to the main text in Section 4.

**B)** We agree that the variance is extremely high on CSBMs. We point out two potential reasons: First, AL settings can inherently be noisy and since the CSBM graphs are relatively small, differences in acquired nodes (especially when picked almost randomly) may be even more pronounced. This is due to the fact that the CSMBs we study have a relatively low signal-to-noise ratio (SNR), otherwise, the whole task would be very easy to solve with GNNs (achieving near-perfect accuracy after only one acquisition per class). In this SNR regime, however, we observe high variance. Second, we study not one graph but average results over five CSBMs sampled from the same distribution: Again, due to relatively small graph sizes, we observe very different graphs in practice. We chose to average over multiple graphs to avoid “cherry-picking” one CSBM graph, where our ground-truth uncertainty performs well, and instead drew 5 random graphs to give an overall more truthful impression of performance.

However, due to the new real-world graph experiments, where we show that our RHS method outperforms all US strategies, showing the superiority of ours compared to baselines on CSBMs might be obsolete. What still is interesting is the performance of our approach using various uncertainties but we can remove the other figure if the reviewer is unhappy with it.

Reviewer comment (Presentation):
In the Introduction section, the authors dedicated an extensive portion of the text to explain uncertainty sampling. This level of detail might be excessive as uncertainty sampling is a straightforward concept. It would be more beneficial to present the essential formulations and allocate additional space to elaborate on active learning in graph-related tasks.

Author response:
**Addressing Question 2**: We adapted the manuscript to give the background about US more concisely and added more explanation of epistemic and aleatoric effects to the background Section 2 at the same time. All additions are made visible with a blue font color in the new PDF version.

Reviewer comment (Theory):
The theoretical investigation, which is a major contribution of the article, not only considers a specific model (which is perfectly acceptable), but also assumes the full knowledge of the parameters underlying the model. As in practice the model parameters are rarely known and have to be estimated from data, their estimation error will contribute to the reducible uncertainty. Therefore defining the reducible uncertainty while assuming the model parameters to be pre-known seems to be problematic and needs at least to be discussed.

Author response:
We thank the reviewer for this comment and question as it is a valid point that we need to discuss in our work. We generalized our definitions to incorporate a parametrized classifier that opts to learn the true generative process: 

$f_\theta^*(A, X, y_O^{gt}) =  \text{argmax}_{c \in [C]^{|U|}} \textcolor{blue}{\int}\text{Pr}[{y_U= c \mid A, X, y_O = y_O^{gt}, \textcolor{blue}{\theta}}] \textcolor{blue}{p(\theta \mid A, X, y_O)d\theta}$

We updated and uploaded a revision of our paper where we highlight changes in blue. The Bayesian classifier averages its prediction over its parameters $\theta$ according to a posterior distribution $p(\theta | \mathbf{X}, \mathbf{A}, \mathbf{y})$. For example, these parameters $\theta$ could be estimates of the parameters of the underlying data-generating process in the CSBM setting or GNN parameters for real data. These changes can be found in Definitions 1-3 and the corresponding proofs.
We adapt definitions 2 and 3 accordingly. Importantly, the proposed optimality results still hold, as only the computation of the ground-truth uncertainty is affected by this change.

While we can also try to learn the parameters of the underlying CSBM, this would  introduce unwarranted noise into the evaluation. We have further clarified in our manuscript that the analysis on CSBMs opts to isolate the effect of disentangling and correctly modeling uncertainty (and therefore assumes no modeling errors).

In a new paragraph in Section 6, we detail an AL strategy that models the epistemic uncertainty as the difference between total and aleatoric (as in our theoretic result). Here, the total uncertainty originates from a classifier trained solely on labeled data. We then apply the predictions from this classifier as pseudo labels to train a subsequent model on all labeled nodes, with the exception of one (pseudo labels + real label) to model the aleatoric uncertainty for this one node.

Our method outperforms state-of-the-art US strategies on many datasets. All Figures can be seen in the updated version, Section 6 and Appendix H).

This experiment does not rely on explicit knowledge of either the generative process or unavailable labels but uses multiple approximations instead and thus could even be used as an AL strategy in practice. We, therefore, believe to provide even more valuable insights by closing the gap between an idealized synthetic setting and real scenarios. Looking ahead, future work could include improving the GNN approximators based on our other findings, such as modeling non-edges, for example in a transformer-like model.

We hope that with the adaptation of our theoretic results and the new experiments we properly addressed the parameter knowledge concern. If desired by the reviewer, we are open to including an additional experiment in the CSBM setting, conducted without parameter knowledge in the camera-ready version of our work.

We thank the reviewer for this pointer since framing and defining the problem in this more general parametrized way helps to better exemplify the applicability towards real-world scenarios which improves our manuscript substantially.

Reviewer comment (Writing):
It should be made clear earlier in the article (e.g. in the abstract or introduction) that the proposed uncertainty measure is not directly applicable in practice, and rather of theoretical interest.

Author response:
We believe that the most substantial contribution of our work lies in the analysis of the problem. This groundwork paves the way for the principled development of new UQ strategies and strongly motivates AL as a valuable evaluation benchmark. For example, future studies could explore whether their uniquely disentangled uncertainties, potentially designed for different tasks, are also effective in an AL context.

Therefore, while we now even showcase a straightforward and practical application of our findings, it is our intent to present it as an experimental validation of our theory's applicability to real-world graphs, rather than as the centerpiece of a methods-focused paper. Hence, we further emphasize in our updated manuscript that a new method is not the goal of our work in the introduction and hope that the reviewer agrees with our view.

Reviewer comment (Theory):
How will the reducible uncertainty change without the knowledge of model parameters?

Author response:
We thank the reviewer for this comment and question as it is a valid point that we need to discuss in our work. We generalized our definitions to incorporate a parametrized classifier that opts to learn the true generative process: 

$f_\theta^*(A, X, y_O^{gt}) =  \text{argmax}_{c \in [C]^{|U|}} \textcolor{blue}{\int}\text{Pr}[{y_U= c \mid A, X, y_O = y_O^{gt}, \textcolor{blue}{\theta}}] \textcolor{blue}{p(\theta \mid A, X, y_O)d\theta}$

We updated and uploaded a revision of our paper where we highlight changes in blue. The Bayesian classifier averages its prediction over its parameters $\theta$ according to a posterior distribution $p(\theta | \mathbf{X}, \mathbf{A}, \mathbf{y})$. For example, these parameters $\theta$ could be estimates of the parameters of the underlying data-generating process in the CSBM setting or GNN parameters for real data. These changes can be found in Definitions 1-3 and the corresponding proofs.
We adapt definitions 2 and 3 accordingly. Importantly, the proposed optimality results still hold, as only the computation of the ground-truth uncertainty is affected by this change.

While we can also try to learn the parameters of the underlying CSBM, this would  introduce unwarranted noise into the evaluation. We have further clarified in our manuscript that the analysis on CSBMs opts to isolate the effect of disentangling and correctly modeling uncertainty (and therefore assumes no modeling errors).

In a new paragraph in Section 6, we detail an AL strategy that models the epistemic uncertainty as the difference between total and aleatoric (as in our theoretic result). Here, the total uncertainty originates from a classifier trained solely on labeled data. We then apply the predictions from this classifier as pseudo labels to train a subsequent model on all labeled nodes, with the exception of one (pseudo labels + real label) to model the aleatoric uncertainty for this one node.

Our method outperforms state-of-the-art US strategies on many datasets. All Figures can be seen in the updated version, Section 6 and Appendix H).

This experiment does not rely on explicit knowledge of either the generative process or unavailable labels but uses multiple approximations instead and thus could even be used as an AL strategy in practice. We, therefore, believe to provide even more valuable insights by closing the gap between an idealized synthetic setting and real scenarios. Looking ahead, future work could include improving the GNN approximators based on our other findings, such as modeling non-edges, for example in a transformer-like model.

We hope that with the adaptation of our theoretic results and the new experiments we properly addressed the parameter knowledge concern. If desired by the reviewer, we are open to including an additional experiment in the CSBM setting, conducted without parameter knowledge in the camera-ready version of our work.

We thank the reviewer for this pointer since framing and defining the problem in this more general parametrized way helps to better exemplify the applicability towards real-world scenarios which improves our manuscript substantially.

Reviewer comment (Experiments):
Will the conclusion regarding the optimality of using the reducible uncertainty to guide AL stay the same?

Author response:
We thank the reviewer for this comment and question as it is a valid point that we need to discuss in our work. We generalized our definitions to incorporate a parametrized classifier that opts to learn the true generative process: 

$f_\theta^*(A, X, y_O^{gt}) =  \text{argmax}_{c \in [C]^{|U|}} \textcolor{blue}{\int}\text{Pr}[{y_U= c \mid A, X, y_O = y_O^{gt}, \textcolor{blue}{\theta}}] \textcolor{blue}{p(\theta \mid A, X, y_O)d\theta}$

We updated and uploaded a revision of our paper where we highlight changes in blue. The Bayesian classifier averages its prediction over its parameters $\theta$ according to a posterior distribution $p(\theta | \mathbf{X}, \mathbf{A}, \mathbf{y})$. For example, these parameters $\theta$ could be estimates of the parameters of the underlying data-generating process in the CSBM setting or GNN parameters for real data. These changes can be found in Definitions 1-3 and the corresponding proofs.
We adapt definitions 2 and 3 accordingly. Importantly, the proposed optimality results still hold, as only the computation of the ground-truth uncertainty is affected by this change.

While we can also try to learn the parameters of the underlying CSBM, this would  introduce unwarranted noise into the evaluation. We have further clarified in our manuscript that the analysis on CSBMs opts to isolate the effect of disentangling and correctly modeling uncertainty (and therefore assumes no modeling errors).

In a new paragraph in Section 6, we detail an AL strategy that models the epistemic uncertainty as the difference between total and aleatoric (as in our theoretic result). Here, the total uncertainty originates from a classifier trained solely on labeled data. We then apply the predictions from this classifier as pseudo labels to train a subsequent model on all labeled nodes, with the exception of one (pseudo labels + real label) to model the aleatoric uncertainty for this one node.

Our method outperforms state-of-the-art US strategies on many datasets. All Figures can be seen in the updated version, Section 6 and Appendix H).

This experiment does not rely on explicit knowledge of either the generative process or unavailable labels but uses multiple approximations instead and thus could even be used as an AL strategy in practice. We, therefore, believe to provide even more valuable insights by closing the gap between an idealized synthetic setting and real scenarios. Looking ahead, future work could include improving the GNN approximators based on our other findings, such as modeling non-edges, for example in a transformer-like model.

We hope that with the adaptation of our theoretic results and the new experiments we properly addressed the parameter knowledge concern. If desired by the reviewer, we are open to including an additional experiment in the CSBM setting, conducted without parameter knowledge in the camera-ready version of our work.

We thank the reviewer for this pointer since framing and defining the problem in this more general parametrized way helps to better exemplify the applicability towards real-world scenarios which improves our manuscript substantially.