Paper: Fishnets: Information-Optimal, Scalable Aggregation for Sets and Graphs (ICLR-2024)

Reviewer comment (Writing):
The logic of the introduction part is illogical and too short. The reader does not understand what problem this article is going to solve and what the shortcomings of existing methods are. The authors do not emphasize the main contribution of this article and what are the differences from existing methods.

Author response:
Thank you for your thorough response and questions ! 


**Updates to the paper:** We have revised the introduction in Section 2 to clarify the rigorous theoretical foundation, based on the likelihood principle, upon which we base our claim of information optimality. We include a detailed treatment of the information inequality and the Cramer-Rao bound (with new proof in the appendix), and then proceed to set-based likelihoods and Fishnets embedding. We additionally provide a new, more in-depth empirical analysis of Fishnets aggregation on three OGB graph datasets (for graph- and node-level prediction). Replacing their aggregators with Fishnets we are able to reproduce or exceed the SOTA benchmarks of Li et al’s DeeperGCN paper but with far fewer GNN layers and learnable parameters. Finally, we improved the organisation and flow of the paper from theory to Bayesian information saturation to GNN prediction.

**Response to comments:** We have revised our introduction section and explicitly defined information optimality and the Fisher information matrix. We also include an Appendix in which we prove information saturation with knowledge of the score function.

Reviewer comment (Writing):
"Fisher" is mentioned frequently in the paper. The author should add a "preliminary" section to introduce what Fisher are to make it easier for readers with different backgrounds.

Author response:
Thank you for your thorough response and questions ! 


**Updates to the paper:** We have revised the introduction in Section 2 to clarify the rigorous theoretical foundation, based on the likelihood principle, upon which we base our claim of information optimality. We include a detailed treatment of the information inequality and the Cramer-Rao bound (with new proof in the appendix), and then proceed to set-based likelihoods and Fishnets embedding. We additionally provide a new, more in-depth empirical analysis of Fishnets aggregation on three OGB graph datasets (for graph- and node-level prediction). Replacing their aggregators with Fishnets we are able to reproduce or exceed the SOTA benchmarks of Li et al’s DeeperGCN paper but with far fewer GNN layers and learnable parameters. Finally, we improved the organisation and flow of the paper from theory to Bayesian information saturation to GNN prediction.

**Response to comments:** We have revised our introduction section and explicitly defined information optimality and the Fisher information matrix. We also include an Appendix in which we prove information saturation with knowledge of the score function.

Reviewer comment (Writing):
There exist a large number of grammatical errors and spelling errors in the article, such as:  
- In "GNNs can acheive state-of-the-art", "acheive" should be "achieve".  
- In "On one hand, frequentist analyses", it's better to use "On the one hand" for clarity.  
- In "In a deep learning context graph neural networks (GNNs) rely", a comma is needed after "context".  
- In "for predictive or regression tasks", consider using "for either predictive or regression tasks" for clarity.  
- In "Up until now, graph aggregation schemes", the comma after "now" is unnecessary.  
- In "This paper is organised as follows:", "organised" might be considered a British spelling. You might want to use "organized", the American spelling.  
- In "In this paper we built up", a comma is needed after "paper".

Author response:
Thank you for your thorough response and questions ! 


**Updates to the paper:** We have revised the introduction in Section 2 to clarify the rigorous theoretical foundation, based on the likelihood principle, upon which we base our claim of information optimality. We include a detailed treatment of the information inequality and the Cramer-Rao bound (with new proof in the appendix), and then proceed to set-based likelihoods and Fishnets embedding. We additionally provide a new, more in-depth empirical analysis of Fishnets aggregation on three OGB graph datasets (for graph- and node-level prediction). Replacing their aggregators with Fishnets we are able to reproduce or exceed the SOTA benchmarks of Li et al’s DeeperGCN paper but with far fewer GNN layers and learnable parameters. Finally, we improved the organisation and flow of the paper from theory to Bayesian information saturation to GNN prediction.

**Response to comments:** Grammatical and spelling errors have been addressed.

Reviewer comment (Experiments):
The experiments are not very convincing. While I do appreciate the simple examples, I think that additional comparisons on real world benchmarks should be conducted to truly show the effectiveness of the proposed aggregation scheme, also given that the authors say the proposed mechanism can be implemented as a 'drop in' aggregation function. Therefore it should be shown on multiple datasets and multiple GNN backbones.

Author response:
Thank you for your thorough response and questions ! 

**Updates to the paper:** We have revised the introduction in Section 2 to clarify the rigorous theoretical foundation, based on the likelihood principle, upon which we base our claim of information optimality. We include a detailed treatment of the information inequality and the Cramer-Rao bound (with new proof in the appendix), and then proceed to set-based likelihoods and Fishnets embedding. We additionally provide a new, more in-depth empirical analysis of Fishnets aggregation on three OGB graph datasets (for graph- and node-level prediction). Replacing their aggregators with Fishnets we are able to reproduce or exceed the SOTA benchmarks of Li et al’s DeeperGCN paper but with far fewer GNN layers and learnable parameters. Finally, we improved the organisation and flow of the paper from theory to Bayesian information saturation to GNN prediction.

**Response to comments:** We have incorporated more real-world benchmark datasets, which show improvement in performance with fewer learnable parameters (see new Section 5.1).

Reviewer comment (Experiments):
In the experimental results on the ogb protein dataset, the authors should compare with more methods. The results of other methods are significantly better than just using GCN.

Author response:
We wanted to keep the study limited to highlight the advantage of Fishnets aggregation compared to Li et al’s learned softmax aggregation. We do not compare to other architectures since we wish to highlight the improvement of the aggregation scheme with other hyperparameters held fixed.

Reviewer comment (Theory):
The authors say both in title and in abstract that the method is optimal, but it is not proven anywhere, and again the experimental results are lacking.

Author response:
**Updates to the paper:** We have revised the introduction in Section 2 to clarify the rigorous theoretical foundation, based on the likelihood principle, upon which we base our claim of information optimality. We include a detailed treatment of the information inequality and the Cramer-Rao bound (with new proof in the appendix), and then proceed to set-based likelihoods and Fishnets embedding. We additionally provide a new, more in-depth empirical analysis of Fishnets aggregation on three OGB graph datasets (for graph- and node-level prediction). Replacing their aggregators with Fishnets we are able to reproduce or exceed the SOTA benchmarks of Li et al’s DeeperGCN paper but with far fewer GNN layers and learnable parameters. Finally, we improved the organisation and flow of the paper from theory to Bayesian information saturation to GNN prediction.

Optimality is now discussed in depth in Section 2 and Appendix A.

Reviewer comment (Theory):
"**optimality**" is mentioned a few times. However, it is not clear from the text: "**optimal, from what sense?**". There needs to be a clearly stated (i) objective function or (ii) inequality, and showing that the method either recovers the (i) (unique) optimal or (ii) satisfies the inequality with equality. Currently, this discussion is skipped.

Author response:
W2: Optimality is now described in Section 2 (proof in Appendix A)

Reviewer comment (Experiments):
Experiments: The only real dataset used is ogbn-protein. If the authors already integrated with PyG and have ogbn training setup, wouldn't it be straightforward to rerun the code while changing only the dataset name?

Author response:
W3: We have now produced more OGB experiments, with similar levels of improvements once Fishnets is adopted.

Reviewer comment (Writing):
The paper should define $\theta_{fid}$.

Author response:
Q1: $\theta_{\rm fid}$ has been renamed $\theta_*$ and defined in the update

Reviewer comment (Writing):
The term "(iteratively)" mentioned before Equation 3 should be clarified — what is meant by "iteration" in this context?

Author response:
Q3: this has been clarified in Section 2

Reviewer comment (Writing):
In Equation (9), the parameters with respect to which the objective function is optimized should be listed (e.g., $\mathcal{L}(w_t, w_F)$). It is unclear if $\theta$ is a constant, and whether $\theta_{fid}$ is always fixed. If not fixed, list it on the LHS of Eq. 9.

Author response:
Q2: Please see Section 2 and Appendix A

Reviewer comment (Writing):
Clarify if in “(iteratively)” before Equation 3, the authors imply that $\theta_{fid}$ is the previous iteration value and $\hat{\theta}^{MLE}$ is the next iteration.

Author response:
Q3: this has been clarified in Section 2

Reviewer comment (Writing):
Clarify what $\theta$ refers to in Eq. 9.

Author response:
Q2: Please see Section 2 and Appendix A

Reviewer comment (Writing):
Specify the space/dimensionality of $d$ (the input examples). Specifically, do Equations 11 and 12 have a range equal to the space of $d$, or do they output scalars?

Author response:
Q4: $\textbf{d}_i \in \mathbb{R}^N$; $\textbf{t}_i \in \mathbb{R}^{n_p}$

Reviewer comment (Writing):
For Equations 11 and 12, clarify the type of the input argument — is $\beta$ a vector or scalar?

Author response:
Q5: Eqs 11&12: the input to the softmax aggregation is either a data vector $\textbf{d}_i \in \mathbb{R}^N$,  

or some neural network embedding $ q_i \in \mathbb{R}^{n_{\rm h}} $. This has been clarified in both cases.

Reviewer comment (Experiments):
The Bayesian Information saturation experiments are detached from graph neural network aggregation experiments. Why they should be separately discussed? And why the GNN aggregation experiments cannot share the same analysis from the Bayesian Information saturation experiments?

Author response:
**Response to Comments:** The Bayesian Information saturation section is detached from the graph neural network aggregation experiments because in the latter we wished to compare to a ground truth inference, e.g. the HMC sampler, which is not possible for graph aggregation tasks because we do not have access to sampling distributions for the ogbn-proteins dataset features. The noisy edge protein analysis demonstrates how GNN point estimators can be made robust to noise.

Reviewer comment (Experiments):
Only experimenting on OGBN-Proteins seems to be too limited. Experiments on more graph datasets are needed.

Author response:
We have incorporated more OGB dataset comparisons (Section 5.1). We show that we obtain same- or better results with shallow Fishnets GCNs in these benchmark cases.

Reviewer comment (Experiments):
Only using deep GNNs is a weird setting. Why not experiment with shallow GNNs? The paper should at least include results regarding shallow GNNs for comparison.

Author response:
We have incorporated more OGB dataset comparisons (Section 5.1). We show that we obtain same- or better results with shallow Fishnets GCNs in these benchmark cases.

Reviewer comment (Evaluation):
Why the GNN aggregation experiments cannot share the analysis from the Bayesian Information saturation experiments?

Author response:
**Response to Comments:** The Bayesian Information saturation section is detached from the graph neural network aggregation experiments because in the latter we wished to compare to a ground truth inference, e.g. the HMC sampler, which is not possible for graph aggregation tasks because we do not have access to sampling distributions for the ogbn-proteins dataset features. The noisy edge protein analysis demonstrates how GNN point estimators can be made robust to noise.

Reviewer comment (Experiments):
The paper does not include many empirical comparisons.

Author response:
Thank you for your response and questions ! 

**Updates to paper:** We have revised the introduction in Section 2 to clarify the rigorous theoretical foundation, based on the likelihood principle, upon which we base our claim of information optimality. We include a detailed treatment of the information inequality and the Cramer-Rao bound (with new proof in the appendix), and then proceed to set-based likelihoods and Fishnets embedding. We additionally provide a new, more in-depth empirical analysis of Fishnets aggregation on three OGB graph datasets (for graph- and node-level prediction).  Replacing their aggregators with Fishnets we are able to reproduce or exceed the SOTA benchmarks of Li et al’s DeeperGCN paper but with far fewer GNN layers and learnable parameters. Finally, we improved the organisation and flow of the paper from theory to Bayesian information saturation to GNN prediction.

Reviewer comment (Presentation):
There is a slight decline in organization as the paper progresses; for example, there could be better organization between sections 4 and 5, perhaps by differentiating them more or integrating discussion of the experiments in the introduction.

Author response:
Thank you for your response and questions ! 

**Updates to paper:** We have revised the introduction in Section 2 to clarify the rigorous theoretical foundation, based on the likelihood principle, upon which we base our claim of information optimality. We include a detailed treatment of the information inequality and the Cramer-Rao bound (with new proof in the appendix), and then proceed to set-based likelihoods and Fishnets embedding. We additionally provide a new, more in-depth empirical analysis of Fishnets aggregation on three OGB graph datasets (for graph- and node-level prediction).  Replacing their aggregators with Fishnets we are able to reproduce or exceed the SOTA benchmarks of Li et al’s DeeperGCN paper but with far fewer GNN layers and learnable parameters. Finally, we improved the organisation and flow of the paper from theory to Bayesian information saturation to GNN prediction.

Reviewer comment (Evaluation):
The authors mention faster training time in multiple occurrences, but data on computation time is not included in the paper. Clarification is requested on the extent of the improvement.

Author response:
Q1: Faster training time here refers to fewer learning (gradient descent) epochs needed to achieve convergence of the GNN model. We indicate this improvement by showing test ROC-AUC curves in the ogbn-proteins focus section. We additionally indicate improved benchmark comparisons (see Section 5.1) with drastically fewer learnable parameters than Li et al’s best GCN models.

Reviewer comment (Experiments):
The rationale for choosing the proteins dataset over other potential graph benchmarks is unclear.

Author response:
Q2: We initially chose to work with the ogbn-proteins dataset because the prediction task consisted of aggregating weighted edges (association scores) between proteins (nodes) in the graph. These association scores could be modelled with noise akin to the linear regression and BHM models in previous sections to demonstrate the robustness of the Fishnets aggregation to changing noise because the Fisher weighting scheme explicitly re-weights embeddings in a heterogeneous manner as a function of data (and in the linear regression and noisy proteins case, the $\sigma$ and $N$ noise amplitudes).

Reviewer comment (Writing):
The presentation of the method could be improved. In particular, the key concepts of “information optimal” and “information saturation” are not clearly defined. Section 2 first explains that the Fisher scoring method can be used to “(iteratively) form a maximum likelihood estimator” and then transitions to a “pseudo-MLE”. Alsing & Wandelt (2018) provide a considerably more complete and clearer discussion. This paper should aim to be more precise. For example, it’s not clear what “(iteratively)” means here – Alsing & Wandelt spell it out that it converges in the limit and explain that after k iterations it provides a (clearly defined) quasi maximum-likelihood estimator. The paper doesn’t explain the transition to a “pseudo-MLE” later in the section. It doesn’t explain the concept of “saturating” the Fisher information; it doesn’t provide the Cramer-Rao bound. One could argue that some of this might be considered necessary background knowledge, but I would suggest that the paper should be accessible to practitioners who are interested in learning about a new aggregation method and perhaps do not have an information theoretic background.

Author response:
We have revised the introduction in Section 2 to clarify the rigorous theoretical foundation, based on the likelihood principle, upon which we base our claim of information optimality. We include a detailed treatment of the information inequality and the Cramer-Rao bound (with new proof in the appendix), and then proceed to set-based likelihoods and Fishnets embedding. Finally, we improved the organization and flow of the paper from theory to Bayesian information saturation to GNN prediction.

Reviewer comment (Experiments):
In the GNN aggregation experiment, rather than providing a systematic and complete study over multiple datasets and comparing against multiple baselines to demonstrate how the proposed method acts as the best aggregator (in some sense), the provided experiments are for a single dataset, with essentially two variants of a single baseline as comparison points. The selected baseline does not achieve close to the state-of-the-art performance – as asked below, it’s not even clear that the selected baseline is configured to achieve its best performance. These experiments therefore only allow the conclusion that, for one dataset, the proposed method outperforms one questionably-configured baseline that is itself well behind the state-of-the-art (although one could argue that the leaders above it are not “pure” GNNs and achieve higher performance via processing steps unrelated to graph learning or aggregation). The experiments do not establish that the proposed aggregator can be reliably combined with multiple different GNN architectures and employed confidently for a new dataset.

Author response:
We additionally provide a new, more in-depth empirical analysis of Fishnets aggregation on three OGB graph datasets (for graph- and node-level prediction).  Replacing their aggregators with Fishnets we are able to reproduce or exceed the SOTA benchmarks of Li et al’s DeeperGCN paper but with far fewer GNN layers and learnable parameters. Q1: We have updated the codebase to compare Li et al’s best-performing models to our aggregation technique. We first do a drop-in replacement study on three OGB datasets with Li et al’s DeeperGCN code and achieve similar results with their baseline models before proceeding with our Fishnets modifications. We revised our initial ogbn-proteins study into a focus study. We used a stripped-down version of Li et al’s codebase initially in order to modify the raw data during training in the noisy edge case. We provide our benchmark models within this training and preprocessing scheme to provide a benchmark against which to compare the noisy setting.

Reviewer comment (Experiments):
While it is not essential for the paper to provide evidence for every possible application of the proposed method, the paper would be much stronger if it gave compelling evidence for at least one of the suggested applications. The paper does not make a strong theoretical contribution, although the proposed technique is well-founded theoretically. As such, the bar for experimentation is higher; convincing empirical evidence is expected to support the claims.

Author response:
We additionally provide a new, more in-depth empirical analysis of Fishnets aggregation on three OGB graph datasets (for graph- and node-level prediction).  Replacing their aggregators with Fishnets we are able to reproduce or exceed the SOTA benchmarks of Li et al’s DeeperGCN paper but with far fewer GNN layers and learnable parameters.

Reviewer comment (Experiments):
Why does the AUC-ROC performance for the baseline differ so much from the value reported in Li et al. 2020 (where the best-performing model is 0.855 for a 28-layer network)? Do the experiments avoid residual connections for some reason? (The description “out-of-the-box 28-layer GCN” is unclear.) Li et al. explicitly state “residual connections significantly improve the performance of deep GCN models. PlainGCN without skip connections does not gain any improvement from increasing depth.”

Author response:
Q1: We have updated the codebase to compare Li et al’s best-performing models to our aggregation technique. We first do a drop-in replacement study on three OGB datasets with Li et’s DeeperGCN code and achieve similar results with their baseline models before proceeding with our Fishnets modifications.  We revised our initial ogbn-proteins study into a focus study. We used a stripped-down version of Li et al’s codebase initially in order to modify the raw data during training in the noisy edge case. We provide our benchmark models within this training and preprocessing scheme to provide a benchmark against which to compare the noisy setting.

Reviewer comment (Theory):
Regarding the statement “Explicitly learning the $F^{−1}$ weights in addition to the score allows us to achieve 1) asymptotic optimality” – can the paper concretely define the nature of the optimality? Is there a simple proof? Or is it considered obvious? Presumably there is reliance on a universal function approximation result for the neural networks learning the score and the Fisher. Does this combine obviously with the asymptotics of the Fisher method?

Author response:
Q2: This optimality guarantee (under the likelihood principle) has been explicitly provided in Section 2 with proof in Appendix A. Thank you for your follow-up response ! What should be emphasised is that the *form* of the aggregation is optimal under the likelihood principle. 

Here we model members of a set (be it a set of data or graph node neighbourhood) as members of an *implicit* probability distribution (likelihood) $p( \\{ d_i \\} | \theta)$, where $\theta$ is some quantity of interest, be it a parameter of a statistical model or some latent representation within a graph network. The optimality with respect to a *true* score or Fisher can be demonstrated empirically against a ground-truth in analytically-known cases, like the linear regression validation case. However, for implicit (intractable) likelihoods, we rely on the form of the aggregation (score embedding weighted by a the inverse-Fisher weight matrix) to guide our neural network architecture design to improve aggregated information capture. 

We agree that the wording "sufficiently well" might be misleading and imply that there is some guarantee that the true score and Fisher are being learned in every case. We will consider changing this to "descriptive neural embedding" for clarity.

Reviewer comment (Theory):
What makes the Fishnets robust to noise?

Author response:
To respond to your questions, we refer you to our revised Section 2. Fishnets is robust to heterogeneous noise because we explicitly parameterise the inverse-Fisher weighting scheme as a learned function of the data (which in the linear regression and noisy proteins cases includes estimates of the noise variance).

Reviewer comment (Experiments):
In section 5, why does the experiments only perform on deep GNNs? Edge aggregation can be also performed on shallow GNNs, such as GAT [1].

Author response:
We additionally revised our GNN section to include far more examples of the Fishnets aggregation. We elected to use the GCN framework due to many readily-available examples, as well as to show that the Fishnets aggregation could improve performance and reduce learnable parameters as a drop-in replacement.