Paper: OLGA: One-cLass Graph Autoencoder (ICLR-2024)

Reviewer comment (Theory):
Many claims in the article have not been reasonably explained. The rationale behind the benefits of low-dimensional node representations within the OCL context for enhancing model interpretability is not adequately elucidated.

Author response:
1. Thank you for pointing out this issue. We support the issue of interpretability in relation to the dimension with more facts and references. Basically, without the help of other methods, such as t-SNE, OCL methods for graphs naturally do not offer interpretability.

"Interpretability has gained increasing attention, driven by the demand for model transparency (Carvalho et al., 2019; Gilpin et al., 2018). Generally, interpretability can be categorized into distinct types, including feature-based, model-based, and latent space-based methods (Das et al., 2020; Guidotti et al., 2018; Gilpin et al., 2018). With the rise of representation learning models, there has been a growing interest in interpretability centered around the latent space (Gilpin et al., 2018; Chen et al., 2016). Latent space-based methods delve into the internal representations learned by models. Many studies explore t-SNE (Van der Maaten & Hinton, 2008) for embedding visualization (Wang et al.,2021) to visualize these representations in lower dimensions and provide interpretability. While these methods are invaluable, they are not inherently integrated into the representation learning process. However, methods that inherently yield low-dimensional representations while preserving classification performance enable mapping these representations to two or three dimensions, offering users an intuitive means to interact with and comprehend the model's decisions."

See these references in the PXa1 reviewer response.

Reviewer comment (Theory):
Many claims in the article have not been reasonably explained. The reason behind the integration of the autoencoder framework into one-class classification models is not well illustrated. In page 5, authors wrote "If we only use the L1, all instances will converge towards the center, regardless of whether they are interest nodes, as the GNN aggregates representations at each iteration. Therefore, we propose multi-task learning with additional loss functions to assist our main task solved by L1, combining the loss functions." Such explanation is too high-level, and the underlying mechanism why autoencoder is beneficial to one-class classification is still unclear.

Author response:
2. Thanks for pointing out this gap. The main motivation for using the GAE reconstruction loss function is to create a constraint so that the hypersphere loss function does not completely bias learning since this total bias can leave all instances within the hypersphere. We added more information about this in the introduction.

“(...) The first is based on the graph autoencoder loss, which aims to reconstruct and preserve the graph topology by mapping nodes into a new latent space. The second is a newly proposed hypersphere loss that leverages instances from the interest class to enhance one-class-oriented representation learning and classification. The GAE reconstruction loss function works as a constraint so that the hypersphere loss function does not map all inputs inside the hypersphere. (...)”.

Reviewer comment (Theory):
Many claims in the article have not been reasonably explained. The paper presents the challenge of combining GAEs and hypersphere loss functions, which it appears to resolve by simply combining these functions without a detailed explanation of the challenges involved.

Author response:
3. Thanks for pointing out this issue. The challenge is related to the scales of the loss functions. We modified the section where we highlighted the challenge to be clearer. Furthermore, at the end of our proposal section, we discuss the scales of the loss functions and what we did to maintain the same scales.

“(...) However, GAEs were not exploited in OCGNNs that use the hypersphere loss function. One reason is the challenge of combining the loss function of the GAEs and hyperspheres since we have different scales for the loss functions. In this sense, in the next section, we present OLGA, the One-cLass Graph Autoencoder, a method to mitigate these gaps.”

“However, when combining two different loss functions, one potential issue is their scales, where one loss could have a stronger influence on the network's learning, causing the other task to be ignored entirely. However, using hyperbolic tangents serves as a solution to this problem. (...)”

Reviewer comment (Theory):
Many claims in the article have not been reasonably explained. The paper attempts to address the hypersphere collapse problem in one-class classification using low-dimensional representations, but this paper does not sufficiently discuss the advantages of lower dimensions compared with existing solutions. Furthermore, the explanation of why lower dimensions enhance interpretability remains unclear.

Author response:
4. We show in the appendices that using our proposal with high dimensionality generates the collapse of the hypersphere (all points converge to 0). Furthermore, when using low dimensions, we can provide interpretability (now, this is clearer in the introduction) since the representations can be seen at each learning epoch. We agree that the work needs more foundation for this part of interpretability. Therefore, we provide more information and explanation with a new paragraph on why low-dimensional representations provide more interpretability, as presented in item 1.

Finally, in 2021, when Wang proposed OCGNN, it obtained state-of-the-art results for one-class learning on graphs. Therefore, we mainly compared this method and not with others proposed before 2021 that obtained worse results than OCGNN. We highlight a scarcity of end-to-end one-class learning methods proposed after 2021 with reproducible code.

Reviewer comment (Novelty):
The choice of comparison algorithms in the paper is not sufficiently novel. For example, the OCSAGE method, which uses GraphSAGE, was introduced in 2018, and deep one-class classification was also proposed in the same year. This lack of innovative comparison algorithms diminishes the paper's contribution.

Author response:
Finally, in 2021, when Wang proposed OCGNN, it obtained state-of-the-art results for one-class learning on graphs. Therefore, we mainly compared this method and not with others proposed before 2021 that obtained worse results than OCGNN. We highlight a scarcity of end-to-end one-class learning methods proposed after 2021 with reproducible code.

Reviewer comment (Presentation):
Adding a diagram in the introduction to illustrate the significance and importance of one-class learning on graphs would be beneficial.

Author response:
Thanks for the suggestion. We added an illustration in the appendices because of limited space reasons.

Reviewer comment (Novelty):
The lack of innovation is a concern, as the author has primarily combined two conventional and commonly used loss functions.

Author response:
We would like to point out that the proposed hypersphere loss function is new. The way of combining loss functions using the same scales is also new. To the best of our knowledge, no other work has proposed a hypersphere loss function encouraging examples to go to the center when they are inside the hypersphere, nor has it combined hypersphere and GAE loss functions, maintaining their scales.

Reviewer comment (Experiments):
I suggest the author provide detailed statistics about the dataset.

Author response:
The statistics were in the paper. We moved them to the appendix for space reasons.

Reviewer comment (Reproducibility):
The authors could provide an anonymous GitHub link to ensure the reproducibility of the paper.

Author response:
The work has GitHub and will be released as soon as it is published. We avoid using links to avoid breaking the double-blind since some reverse engineering cases are being applied to anonymous GitHub to find the original creators.

Reviewer comment (Experiments):
Due to the popularity of large language models, it is advisable for the authors to consider using Large Language Models (LLMs) or other encoders to initialize text representations for comparison.

Author response:
We appreciate your suggestion and agree that we can make improvements by modifying the initial representation. Unfortunately, we do not have the time to run all the experiments again, but we will use this strategy in future work. However, the study focuses on the proposed method for one-class learning, agnostic to the data. Thus, seeking more robust initializations could improve the results and expand the work's scope.

Reviewer comment (Experiments):
In the introduction, the author claims, "Existing methods often assume high-dimensional latent spaces, which can hamper interpretability." However, this statement lacks a basis in terms of theoretical or experimental analysis. As a result, the subsequent experimental results do not provide evidence that high-dimensional features are significantly worse or even better than low-dimensional features, such as OCGAN, OCGAT, and OCSA. This raises doubts about the accuracy of the author's description.

Author response:
Thanks for pointing out this issue. We removed this statement from the introduction as suggested.

Reviewer comment (Experiments):
There is a scarcity of comparative methods in the paper, and recent research is underrepresented. Out of the six comparative methods, three are derived from the same paper.

Author response:
In 2021, when Wang proposed OCGNN, it obtained state-of-the-art results for one-class learning on graphs. Therefore, we mainly compared this method and not with others proposed before 2021 that obtained worse results than OCGNN. We highlight a scarcity of end-to-end one-class learning methods proposed after 2021 with reproducible code.

Reviewer comment (Theory):
The third motivation point mentions "the methods' lack of interpretability and visualization." Nevertheless, the paper itself lacks corresponding theoretical explanations.

Author response:
Thank you for pointing out this issue. We support the issue of interpretability in relation to the dimension with more facts and references. Basically, without the help of other methods, such as t-SNE, OCL methods for graphs naturally do not offer interpretability.

"Interpretability has gained increasing attention, driven by the demand for model transparency (Carvalho et al., 2019; Gilpin et al., 2018). Generally, interpretability can be categorized into distinct types, including feature-based, model-based, and latent space-based methods (Das et al., 2020; Guidotti et al., 2018; Gilpin et al., 2018). With the rise of representation learning models, there has been a growing interest in interpretability centered around the latent space (Gilpin et al., 2018; Chen et al., 2016). Latent space-based methods delve into the internal representations learned by models. Many studies explore t-SNE (Van der Maaten & Hinton, 2008) for embedding visualization (Wang et al.,2021) to visualize these representations in lower dimensions and provide interpretability. While these methods are invaluable, they are not inherently integrated into the representation learning process. However, methods that inherently yield low-dimensional representations while preserving classification performance enable mapping these representations to two or three dimensions, offering users an intuitive means to interact with and comprehend the model's decisions."

Diogo V Carvalho, Eduardo M Pereira, and Jaime S Cardoso. Machine learning interpretability: A survey on methods and metrics. Electronics, 8(8):832, 2019.

Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pp. 80–89. IEEE, 2018.

Saikat Das, Namita Agarwal, Deepak Venugopal, Frederick T Sheldon, and Sajjan Shiva. Taxonomy and survey of interpretable machine learning method. In Proceedings of the Symposium Series on Computational Intelligence, pp. 670–677. IEEE, 2020.

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 51(5):1–42, 2018.

Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.

Reviewer comment (Theory):
The paper lacks a complexity analysis of the proposed method, both theoretically and experimentally.

Author response:
We appreciate the comment, but due to time constraints, we did not carry out experiments comparing algorithm execution times. Furthermore, we emphasize that other OCL studies have already carried out this analysis, and we developed an algorithm with the same strategy, i.e., we don't have a significant difference in time compared to the OCGNN.

Reviewer comment (Experiments):
In Formula 8, there are three hyperparameters. Unfortunately, the paper does not include a sensitivity analysis for these parameters, making it difficult to intuitively gauge the effectiveness of the design module.

Author response:
We decided to vary the other parameters and keep these three fixed. We chose these fixed values to activate and deactivate the loss functions during learning, as this characterizes the learning of OLGA. Therefore, we only use the configuration reported in the appendices of these three parameters, both in validation and testing. It is worth mentioning that this does not prevent other works from exploring intermediate values as extensions (values between 0 and 1).

Reviewer comment (Novelty):
The innovation of the author's method appears limited, as it seems to be based on GAE and two MSE losses.

Author response:
The innovations are: (i) In the new hypersphere loss function ($\mathcal{L_1}$); (ii) The combination obeying the scales of our new hypersphere loss function and the GAE reconstruction function that resulted in the OLGA approach; and (iii) a new interpretable approach for one-class learning in graphs.

Reviewer comment (Theory):
The motivation for using GAE is not well explained. Also, how simply introducing the graph autoencoder loss into one-class learning can help to solve the three gaps introduced in the abstract. A more detailed discussion should be added.

Author response:
1. Thanks for pointing out this gap. The main motivation for using the GAE reconstruction loss function is to create a constraint so that the hypersphere loss function does not completely bias learning since this total bias can leave all instances within the hypersphere. We added more information about this in the introduction.

“(...)  The first is based on the graph autoencoder loss, which aims to reconstruct and preserve the graph topology by mapping nodes into a new latent space. The second is a newly proposed hypersphere loss that leverages instances from the interest class to enhance one-class-oriented representation learning and classification. The GAE reconstruction loss function works as a constraint so that the hypersphere loss function does not map all inputs inside the hypersphere. (...)”.

Reviewer comment (Theory):
It is unclear how the function in Eq.5 is derived.

Author response:
2. We added the derivative of equation 5 to the paper.

$f'(d_i) = 1, ~~ \text{if} ~~ d_i > 0 ~~~  e^{d_i}, ~~ \text{otherwise.} $

Reviewer comment (Theory):
For the two reconstruction losses, A contains topology information of unlabeled nodes, setting a reconstruction loss on A^u could be repetitive and meaningless. The authors should give more explanation.

Author response:
3. The idea of separating into two reconstruction loss functions is so that only the reconstruction of unlabeled nodes is used when using the hypersphere loss function. This is clear in the definition of the alpha, beta, and gamma parameters in the appendices, where L2 and L3 are not used together during learning. We agree that this may confuse the reader. Therefore, we change L2 to reconstruct only the labeled nodes of interest. Therefore, we will not have a repeat. Furthermore, we can use the new L2 and L3 together when reconstructing the entire graph.

$\mathcal{L}_{2}(\mathbf{{W}}) = mse(\mathbf{A^{in}}, \hat{\mathbf{A}}^{\text{in}})$,

$\mathcal{L}_{3}(\mathbf{{W}}) = mse(\mathbf{A^{u}}, \hat{\mathbf{A}}^{\text{u}})$,

"in which $\mathbf{A}^{\text{u}}$ is the adjacency matrix of the unlabeled nodes in the graph, $\hat{\mathbf{A}}^{\text{u}}$ is the reconstruction of this matrix generated by OLGA, $\mathbf{A}^{\text{in}}$ is the adjacency matrix of the labeled nodes, and $\hat{\mathbf{A}}^{\text{in}}$ is the reconstruction of the $\mathbf{A}^{\text{in}}$."

Reviewer comment (Writing):
The writing should be improved. Many discussions are hard to understand.

Author response:
4. We carried out an extensive reading of the paper to improve writing and discussions.

Reviewer comment (Theory):
A mathematical problem formulation would be helpful.

Author response:
5. We insert a formulation of the problem.

"We define OCL for graphs as a function $ocl: \mathcal{V} \rightarrow \mathbf{y}$ that maps a node $v_i \in \mathcal{V}$ to a value $y_i$ that indicates the confidence of a node $v_i$ belonging to the class of interest training only with positive labels of the interest node set $\mathcal{V}^{in}$. Thus, classification through OCL aims to learn a function $ocl^*$, which approximates the unknown mapping function $ocl$."