Paper: Extending Power of Nature from Binary to Real-Valued Graph Learning in Real World (ICLR-2024)

Reviewer comment (Writing):
Over-exaggerated descriptions in the main context. The paper seems to overclaim some parts, like mentioning, “Regrettably, despite their perceived potential, most nature-powered ML methods are still predominantly theoretical, outperforming NNs only in toy problems under highly idealized conditions” while the Ising machine is also not quite partial and only applicable for some problems.

Author response:
We sincerely thank the reviewer for this comment. We have revised the sentence pointed out by the reviewer and will comprehensively revise the paper and adjust the tone. Again, we thank the reviewer for pointing out this.

Reviewer comment (Experiments):
Out-dated baselines. In comparison, the authors chose three “SOTA” spatial-temporal GNNs, while the earliest was published in 2020. The author should compare with more recent advances.

Author response:
The reviewer's suggestion is very much appreciated. We have added 2 more recent work, including the work the reviewer suggests in our revision. See table 2. To enrich the evaluation, we also added more latency results and accuracy results of NP-GL equipped with different loss functions for better comparison. The corresponding modifications are highlighted with label "Reviewer-qast"

Reviewer comment (Experiments):
Could the authors provide more comparison with recent SOTA GNN spatial-temporal GNNs? I found one recent paper, Jiang, Renhe, et al. "Spatio-temporal meta-graph learning for traffic forecasting." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 7. 2023.

Author response:
The reviewer's suggestion is very much appreciated. We have added 2 more recent work, including the work the reviewer suggests in our revision. See table 2. To enrich the evaluation, we also added more latency results and accuracy results of NP-GL equipped with different loss functions for better comparison. The corresponding modifications are highlighted with label "Reviewer-qast"

Reviewer comment (Theory):
What’s the physical meaning of asymmetric weight decomposition? As it is a natural process (Hamitonian), is this symmetric weight decomposition meaningful for the physical system besides the efficiency consideration?

Author response:
The symmetric weight decomposition acts as a natural constraint such that the spin correlation matrix (the "J'" matrix in context) becomes symmetric. In analogy to Newton's 3rd Law, the forces applied to the objects in a pair are the same in magnitude. In the real-world, most of dynamic systems have pair-wise connection among nodes. Such symmetry is very common in physics, including Ising model. To explain in the language of math, after taking the first derivative of the Hamiltonian, the symmetric term (J+J transpose) appears, indicating the symmetric nature of the correlation matrix.

Reviewer comment (Presentation):
Minor: Add one-liner for the future insights summary to main text from Appendix.

Author response:
We have added this discussion of weakness in the Appendix of the paper due to page limit. See highlight labeled as "Reviwer-w2U6" in the "Conclusion and Future Insights" section and Appendix A.5.

Reviewer comment (Reproducibility):
Could you elaborate on the adaptation for NP-GL on top of SOTA Ising machine? The hardware implementation, challenges, and potential optimizations could provide valuable insights for readers interested in the practical aspects of the work.

Author response:
1. **Area:** Although it is much smaller than many other types of nature-based computer, e.g. oscillator-based Ising machines, the major limitation of the proposed method is still the area of machine. The chip is around 5mm^2, much smaller than GPUs and CPUs. However, as a chip with less than 1W power, the area is large. The reason is that the chip is predominantly based on analog circuits that are naturally larger than digital chips with the same power. If the chip works alone, the area is not an issue. However, if the proposed chip is to be integrated with other computers, the area problem is not negligible. For example, integrating the proposed nature-based hardware within GPUs (like tensorcores as a co-processor of CUDA cores) will equip GPUs with huge extra computational capability, but the chip area can be too big to fit in for now.

2. **Formulation:** Currently, due to the complexity of the coupling network, the NP-GL hardware is only capable of representing linear and quadratic terms within the Hamiltonian function. Considering the practicality in chip manufacturing, the quadratic limitation is untapped now. This may be a limitation on accuracy, although quadratic terms perform well with most of the applications we investigated (probably due to dynamic systems in the real-world being mostly pairwise). We are studying how to realistically add complexity to the current Hamiltonian.

We have added this discussion of weakness in the Appendix of the paper due to page limit. See highlight labeled as "Reviwer-w2U6" in the "Conclusion and Future Insights" section and Appendix A.5.

Reviewer comment (Experiments):
To confirm I assume the the engergy and latency are based on the simulation in CAD software?

Author response:
Yes. We will try to provide results based on the actual chip (being taped out) in the final version.

Reviewer comment (Experiments):
Would this machine be able to generalize beyond GNN?

Author response:
Yes. Here we list a few examples to demonstrate the deeper influences of our framework. Actually, NP-GL can generate unknown data based on a small amount of observed data. It reads training samples to understand the data distribution of a certain problem and generates data based on probabilities of the distribution. For graph prediction as an example, the observed data are historical time frames, and the unknown data are to be predicted. As NP-GL can impute unknown data as described, it can reasonably do more. Followed are some examples that we have largely validated, though not the focus of the paper.

1. **Data compression.** This framework can complete data based on the partial data stored in memory. This will save bandwidth and storage space, leading to higher performance. For example, this framework can impute a gradient map of deep learning training with 20% known gradient data points rather accurately. Therefore, with this framework, the storage and bandwidth requirement of deep learning training will be reduced by 5x.

2. **3D synthesis.** We already have some preliminary results showing this framework can be used for 3D synthesis and, although so far with lower quality than NeRF, construct pictures with micro-second latency. This is especially useful in AR/VR development, as real-time rendering is highly demanded. Here, the polar and azimuthal angles and some seed pixels are used as the observed data, with the remaining unknown pixels to be completed based on the observed data.

Even within the scope of GNN, this framework can be used to improve GNN. For example, given a graph, we can use GNN to perform inference for only 10% nodes, while letting NP-GL complete the remaining 90% nodes based on the prior 10% results. In addition to speedup and reduction in storage and communication, if the 10% nodes are carefully chosen, e.g., the nodes of the highest accuracy in GNN, the overall accuracy can be improved.

Reviewer comment (Experiments):
How it performs as the size of the graph increases and any potential strategies for handling large-scale graph learning problems?

Author response:
If the graph is too large, we generally have two solutions: (1) temporal co-annealing – different nodes are mapped onto the chip for annealing at different time, such that the originally continuous annealing process is broken into several iterative steps. In this scenario, the strongly connected nodes should anneal at the same time for better convergence. (2) multi-chip integration - the system can be scaled up with multiple chips connected, with the strongly connected nodes mapped onto the same chip. For both solutions, there may be slowdowns. Nevertheless, for real-world graphs with community structure, the slowdown is acceptable if the nodes in the same community can be annealed simultaneously or on the same chip (less communication for synchronization).

Reviewer comment (Novelty):
To me, the most significant problem of this work is insufficient review for the prior work. Section 2.1 and 2.2 is a good background introduction, but not too many prior works on the variation of Ising model are discussed, making the contribution of this work hard to justify.

Author response:
Thank you sincerely for pointing this weakness out. We have drafted a new related work section in the paper (section 2). Please see the highlighted text with label "Reviewer-UYh4". In addition, we provide more information below about the development stage of using Ising model and Ising machine in solving real-world problems.

We believe this is a good time now to study novel learning methods based on the Ising model due to the recent emergence of efficient Ising machines. Although the Ising model has been famous and studied for decades, especially in statistical physics (widely used to describe complex dynamic systems), its related research largely remains on the theory level and rarely can be adopted in practice. One of the major reasons is the extreme complexity in solving an Ising model (find the lowest-energy states of a dynamic system) with a large variable space. From the perspective of computational time, solving the Ising model with a digital processor can be over 1000x slower than GNN inference. Fortunately, this situation has changed recently - along with the approaching death of Moore's law, computer architects are inventing new types of computers with diverse computing powers (other than digital processors). Among these computers, a computer that can swiftly solve Ising models at low cost (nanoseconds and milliwatts -- 10^6 faster than being solved on digital computers) has recently emerged. The computer is essentially a CMOS-based Ising machine, which benefits from mature semiconductor manufacturing processes (guaranteeing its easy manufacturing and integration) and attracts much attention to the Ising model in recent 2 years. It has already enabled a wide discussion of using methods rooted from the Ising model and the Ising machine to solve different types of problems. Despite great promise and increasing attention, this type of research is still in its early stage, like quantum computing, waiting for its killer applications, while unlike quantum computing, holding short-term promise of unleashing the power of nature in the real world. In the current stage of the development of this method, the binary limitation is believed to be one of the most important challenges to this adoption in the real world. The essence of this challenge is that the vanilla Ising formulation is already complex enough to be realized and implemented as a CMOS chip – any augmentation to the Ising formulation/Hamiltonian that complicates the circuit implementation should be very cautious. This paper tries to break this limitation by finding a new Ising formulation with slight augmentation that both enables real-value support and avoids complicating the Ising hardware.

Reviewer comment (Novelty):
Technically, this paper simply propose a modified version of Ising model by using a pure quadratic term to replace the linear term in Ising Hamiltonian.

Author response:
As detailed above, the biggest challenge in extending potentials of Ising method from binary to real value is that the Hamiltonian formulation needs to remain simple, such that the Ising machine can be realistically manufactured – note that the Ising machine to support vanilla is already very challenging. To this end, our design goal is to make the change as slight as possible into the Hamiltonian while enabling the support of real value (enabling non-polarized voltages in the machine).