Paper: Conformal Inductive Graph Neural Networks (ICLR-2024)

Reviewer comment (Novelty):
Missing literature review: The work does not have a standalone literature review section.

Author response:
(1) Our approach was to discuss the related works in context throughout the paper. For completeness, we now added a dedicated section in the appendix (section F) and provide a forward reference in the main paper (end of section 2).

Reviewer comment (Presentation):
Organization: the lengths between sections 4 and 5 are imbalanced. Specifically, section 4 contains the "method", theoretical analyses, and examples, where section 5 made reference to section 4 but the algorithm is delayed until the appendix. I think it makes more sense to first introduce the method, provide intuitions of why it works, and then give the theoretical analyses with additional remarks.

Author response:
(2) The suggestion to first introduce the method and then give the theoretical analyses is interesting and we agree that it can indeed improve readability. Since this is a major change that would disrupt the discussion with the reviewers (since sections will change) we defer this to after the rebuttal. 
Note, the algorithm which we added to the appendix for completeness is implicitly defined via the equations in section 4 since it essentially has only two steps: (1) computing (a potentially weighted) quantile line 3 or line 5, and (2) creating a set of scores above the quantile line 6. We also now updated it to clarify how $S_t$ is used.

Reviewer comment (Presentation):
The Algorithm in Appendix A is not well-written. For example, it is unclear how $\boldsymbol{S}_t$ in line 5 is used later. The $\tau^t$ in line 11 is also different from that in lines 7 and 9. Furthermore, it is unclear where the algorithm starts because the line numbers for inputs are intertwined with those for algorithmic steps.

Author response:
(2) The suggestion to first introduce the method and then give the theoretical analyses is interesting and we agree that it can indeed improve readability. Since this is a major change that would disrupt the discussion with the reviewers (since sections will change) we defer this to after the rebuttal. 
Note, the algorithm which we added to the appendix for completeness is implicitly defined via the equations in section 4 since it essentially has only two steps: (1) computing (a potentially weighted) quantile line 3 or line 5, and (2) creating a set of scores above the quantile line 6. We also now updated it to clarify how $S_t$ is used.

Reviewer comment (Experiments):
Experiments lack additional baselines for the setting where standard CP is expected not to work (e.g., in terms of reaching the coverage guarantees). I expect to see additional baselines for this setting that can be compared.

Author response:
(3) We are not aware of any baselines that are applicable expect standard CP and NAPS. In Appendix C we compared our approach to NAPS. On Figure 6 (left) we show the empirical coverage and we see that our approach matches the nominal value while NAPS consistently under/overcovers. Note, here we outperform NAPS even though the experimental setup favours NAPS since we can only evaluate the coverage on the subset of applicable nodes (nodes that have at least one calibration node in their neighbourhood). On Figure 6 (right) we further show the number of non-applicable nodes for NAPS which grows with time. All nodes are applicable for our approach.

Reviewer comment (Experiments):
Missing examples on larger graphs. For example, the Flickr and Reddit2 datasets in https://arxiv.org/pdf/2211.14555.pdf, which have considerably more nodes than the current settings.

Author response:
We added results on large graphs Flickr, and Reddit2 to the appendix (section D.5). Interestingly the results on node-exchangeable sampling is close for both standard and subgraph CP. However in the edge-exchangeable sampling the result is different and only weighted subgraph CP can provide a valid coverage guarantee.

Reviewer comment (Evaluation):
Missing standard deviation of results. From Figure 3 and 4, it seems the standard deviations can be high (I assume each line with light color indicates results from a specific run).

Author response:
As mentioned in section 2 and subsection 2.1, the coverage guarantee is probabilistic and can deviate from the specified value $1 - \alpha$. The concentration around $1 - \alpha$ (or the standard deviation) is a function of the calibration set size. The distribution (including the variance) is precisely characterized in Theorem 3. A realistic assumption for sparse graph is to have a calibration set not larger than the train set. Hence the small calibration set results in a relatively large variance. As we increase the calibration set size the variance quickly decreases (see figure 17).

Reviewer comment (Evaluation):
Misleading Table 1: it says "average deviation from guarantee", but it does not mean a higher deviation necessarily means worse performance. This is because if deviation is positive, this indicates over-coverage, which is still acceptable (one just needs to lower the significance level). In fact, this seems to be the case from looking at Figure 3.

Author response:
(4) As we explained in the paper, the coverage guarantee is bounded by upper- and lower-bounds. The coverage probability either follows a beta distribution, or a collection of hyper-geometric distributions which in both cases a valid CP results in a coverage close to $1 - \alpha$. An invalid guarantee can result in either over- or mis-coverage. There are two important notes: (i) an over-covering break of the guarantee results in larger set sizes and less singleton hits (see Figure 4) (ii) if the guarantee is invalid, one can not spot whether over- or mis-coverage happens before evaluating the result which makes it inapplicable. That is why in we reported the deviation from the guarantee showing that validity of the guarantee not the empirical coverage.

Reviewer comment (Theory):
What is the complexity of the method with respect to graph size, number of samples, number of classes, etc. Specifically, if the graph $\mathcal{G}_t$ is large with complicated definition of score $s$, I expect line 5 of Algorithm 1 to create some computational burden in real-time use.

Author response:
(1) We added a discussion on the computational complexity in Section A (appendix). Mainly the added computational cost in subgraph CP is that the quantile threshold and the calibration scores should be recomputed upon each update in the graph.

Reviewer comment (Theory):
How does the method work if two or more nodes/edges are added at each time step?

Author response:
(2) The guarantees hold for any number of added vertices or edges. Essentially, we we would then only recompute the scores/quantile after $k$ updates.

Reviewer comment (Reproducibility):
What exactly is the "standard CP" in the inductive context and how is it actually implemented? The idea of standard CP is mentioned in section 2, but that section is for general CP and transductive GNNs. I think the details should be discussed in the appendix with a reference in the main text, unless the baseline is taken directly from existing methods. Also, I think the abbreviation "std. CP" reads too much like standard deviation of CP, so I suggest changing the name.

Author response:
(1) By the standard CP we refer to the same procedure as applied in [1]. In the standard CP, we compute the calibration scores and the threshold once the calibration nodes are present. While new nodes appear, we only compute the conformity scores for them and compare it with the threshold computed prior to the updates. We added an explanation of standard CP in the appendix (Section A).

Reviewer comment (Experiments):
Does the method always work well on any GNN models being considered? It is helpful and important to discuss the generality and/or limitation of the method for different GNN baselines.

Author response:
(2) Any GNN or model that is permutation-equivariant maintains exchangeability. Also there are other methods with focus on increasing the efficiency leveraging the network structure. As an example [2] diffuses the conformity scores, and [3] trains a GNN for the same purpose. Both methods are applicable on top of subgraph CP. At the end of section 6 we extended the discussion on limitations of our approach. We also examined the performance of other models in Section D.3.

Reviewer comment (Theory):
Conformal prediction relies crucially on rigorous mathematical statements about how data is generated / how test data appears in the set, how score functions are defined, how the score is trained and applied. These are all important factors for the validity of CP. While the technical discussion in this paper appears correct, some crucial mathematical terminologies are missing, which forces the reader to re-check the statements mentally. Clarifications that would strengthen the paper include: (1) formal definition of node and edge exchangeability, (2) clearly stating the conditions on the score function and how the model is trained, and (3) formal definition and examples of the "evaluation mask".

Author response:
(1) We added a formal description of node-inductive and edge-inductive sequences and node-exchageability and edge-exchageability in section B of the appendix and modified sections 4.1 and 4.2. All changes are shown with blue color in the updated paper.

Reviewer comment (Theory):
Mathematical definition of node exchangeability and edge exchangeability — a formal definition should be added for node and edge exchangeability to avoid confusion. It would also be beneficial to mention important implications of such exchangeability, e.g., at each time \( t \), the existing nodes are exchangeable.

Author response:
(1) We added a formal description of node-inductive and edge-inductive sequences and node-exchageability and edge-exchageability in section B of the appendix and modified sections 4.1 and 4.2. All changes are shown with blue color in the updated paper.

Reviewer comment (Evaluation):
How is the evaluation set \( \mathcal{V}_{eval}^{(t)} \) decided? — there is confusion about how the evaluation set is determined at each time step. Is there a pre-defined rule (e.g., evaluating all newly arrived nodes immediately at each \( t \)) or is the set of indices that will be evaluated at each \( t \) predetermined? If \( \mathcal{V}_{eval}^{(t)} \) is data-dependent, it may require more justification for the approach/theory.

Author response:
(3) Which node is included at which timestep is a design choice. One can wait until the last timestep and evaluate all the nodes at once, or evaluate each node upon arrival, or $k$ steps after arrival, etc. This corresponds to looking at e.g. columns of the coverage matrix $\boldsymbol{C}$, or the different diagonals of matrix C. As long as the decision is made prior to observing the prediction sets (independent of the data) the coverage validity holds. 
Importantly, this implies that the guarantee does not hold for adversarially chosen evaluation sets. To illustrate this, we can pick the smallest prediction set for each node across all timesteps. This breaks the guarantee as shown in Figure 13 (Section D.4 in appendix).

Reviewer comment (Writing):
Definition of \( Cov(\mathbb{I}_T) \) — this is an important quantity in the theorem and should be formally defined.

Author response:
(4) Thank you for this suggestion. We now define $\mathrm{Cov}(\cdot)$ in section 2.1.

Reviewer comment (Writing):
Section 4, line 3 — “callibration node” should be corrected to “calibration node”.

Author response:
We also thank the reviewer for mentioning the typos. We have resolved them in the modified file.

Reviewer comment (Writing):
Lemma 1 — the phrase “defined on a subsets of \( \mathcal{X} \)” contains an error and should be corrected.

Author response:
We also thank the reviewer for mentioning the typos. We have resolved them in the modified file.

Reviewer comment (Writing):
I found the paper difficult to follow. For instance, the setting of transductive node-classification of Section 2.1 or the inductive one of Section 3 are not properly defined. Another example is Fig.1 which is, in my opinion, not very clear and not explained properly. The definition of a graph is never given etc. Globally, the paper lacks explanation.

Author response:
Thank you for these valuable suggestions. We added the definition of the graph, and the assumptions on the trained model in section 2.1. We also added formal definitions of node-inductive and edge-inductive sequences, as well as node- and edge-exchangeability. We updated the caption of Fig. 1 to improve clarity and included a forward pointer to the appendix (D.5) where the experiment is explained in full detail.

Reviewer comment (Writing):
There is a misuse of several words/expressions to discuss known concepts, which ultimately confuses the paper. What is the added value of a phrase such as "calibration budget" to mean that we only use the first N nodes or "no longer congruent with test scores" to mean that it is no longer exchangeable?

Author response:
We agree that the term "calibration budget" does not bring additional value so we removed it from the paper. We also agree that the term "no longer congruent" is not necessary and have replaced it with "no longer exchangeable".

Reviewer comment (Writing):
There is no "limitation" in the conclusion.

Author response:
We identified three main limitations. First, as with most other CP methods, the guarantee is marginal. Second, real-world graph may not satisfy node- or edge-exchagability. This can be partially mitigated by the beyond exchangeability framework. Finally, the guarantee does not hold for adversarially chosen evaluation sets $\mathcal{V}_\mathrm{eval}$. In other words, the choice of which nodes and at which timesteps are included in the evaluation set must be done prior to observing the prediction set. In the initial submission we only mentioned the first limitation. We now added a discussion of all three (last paragraph of section 6).

Reviewer comment (Writing):
The presentation of this paper leaves room for improvement, as some notations are either omitted or unclear, making it misleading for readers. For instance, in Equation (3), the graph is not explicitly noted, and on page 5, in the linear message passing example, the meanings of A, X, and W are not immediately clear. Although these aspects become clearer upon reading the entire paper, the initial lack of clarity can impede readers from following the logic in their first encounter.

Author response:
(1) We added the explicit definition of the graph in section 2.1. We also added the assumptions for the trained model (e.g. GNN), and the notation of graph in equation (3) follows the same definition. The graph $\mathcal{G}_t$ here shows the timestep in which the scores are computed (it refers to the present nodes and edges at timestep $t$). 
For the definition of simple message passing we added the definition of each matrix. Here $\boldsymbol{A}$ refers to the adjacency matrix , and $\boldsymbol{X}$ is the feature matrix (defined in section 2.1) and $\boldsymbol{W}$ refers to the weight matrix  which is defined immediately after.  We thank the reviewer for mentioning that and in case the notation is still not clear we ask the reviewer to kindly point it out.

Reviewer comment (Evaluation):
For CP methods, it's essential to consider both validity and efficiency. While the experimental results focus on "coverage," which assesses validity, it's equally crucial to assess efficiency. A predictive set that is overly large may not be practically useful. Therefore, it is recommended to include at least one experiment that demonstrates the efficiency of the subgraph CP method.

Author response:
(3) We draw the reviewer’s attention to Figures 4, 7 (middle, right), 9 (right), 10 (right), 11 (right), 12 (right). We report two metrics regarding efficiency: average set size, and singleton hit ratio which captures what proportion of test nodes are predicted with a singleton set the includes the true label. On all figures and all metrics our approach is superior and has lower average set size and higher singleton hit ratio.