Paper: Tensor methods to learn the Green's function to solve high-dimensional PDE (ICLR-2024)

Reviewer comment (Experiments):
According to Table 1, it seems there is no reduction in the number of samples required. With the same amount of samples and the same neural network architecture for the Green function, how does DecGreenNet-TT-C compare to the original algorithm DecGreenNet? Does the structure of the samples enforced by the TT framework affect the performance?

Author response:
For the first experiment, we used same number of Monte-Carlo sample for all models, hence, Table 1 shows same sample size for DecGreenNet and DecGreenNet-TT. If you look at the Table 2 and Table 3, you can see that DecGreenNet-TT can learn with a large number of Monte-Carlo samples with higher accuracy and lower training times for higher-order PDE compared to DecGreenNet. This is possible since DecGreenNet-TT learns from partitions of dimensions with tensor-products via tensor-train decomposition instead of storing a large number of Monte-Carlo samples in as input for DecGreenNet.

Reviewer comment (Theory):
On Page 6, I don’t see why (12) can be used to express functions like $\log(x_1 + x_2)$ and $\sqrt{x_1^2 + x_2^2}$. Also, can you explain the intuition why the Green function has a low-rank structure so that the TT can be adopted? And if not, what is the benefit of this method compared to MOD-Net [2]?

Author response:
Yes, it is possible to approximate $\log(x_1 + x_2)$ and $\sqrt{x_1 + x_2}$ by a polynomial of $x_1$ and $x_2$. However, such expansions will require many neural networks to construct the tensor-train structured Green’s functions as in Theorem 1. Such approach can be computationally expensive and may lead to low accuracy. The directly separable functions as given in the paper can be easily integrated with the proposed model.  

We do not have a theoretical analysis for the tensor-structured low-rankness for Green’s function. Establishing tensor low-rankness is a future work and in this paper we have made an assumption that it exists. We point out that robust learning of PDEs in experiments justifies our claim that low-rankness exists for the Green’s function.

MOD-net is computationally expensive to train for 2D PDE problems due to approximation of the Monte-Carlo integral approximation as demonstrated in [1]. The low-rank decomposition of the Green’s function has shown to improve the training times with DecGreenNet. In this paper we further improve on the DecGreenNet with tensor-structured low-rank decomposition and tensor-product grids (inspired by sparse grid and Smolyak grid [2])

Reviewer comment (Presentation):
In Table 2 in the experiment section, the "Segments" for "N=4" should possibly be $200^4$ instead of $200^2$.

Author response:
For N=4, we have used DecGreenNet-TT-C with combinations of 2 dimensions and from each combination we made 200 random partitions, hence, the total number of simulated samples are 200^2. All the other orders we used DecGreenNet-TT.

Reviewer comment (Presentation):
In Table 2 in the experiment section, the caption says "results ... using DecGreenNet-TT and DecGreenNet-TT-C", but there is only one column "error" — it is unclear whether this result is from DecGreenNet-TT or DecGreenNet-TT-C.

Author response:
In the Table 2, for N=2,3 we have used DecGreenNet-TT since we do not combine any dimensions. This can be identified by the number of $T_{*}$ networks. For N=4, we combined dimensions by pairs, so we only have 2 networks of  $T_{1}$ and $T_{2}$, hence, we used the DecGreenNet-TT-C.

Reviewer comment (Experiments):
Since the motivation targets high-dimensional PDEs, setting the maximum dimension to 4 with only the Poisson equation family is not convincing enough; explorations with higher-order settings and more kinds of PDEs are encouraged.

Author response:
We extended existing Poisson problems to higher-orders in our experiments since we could not find a suitable higher-order PDE problem in the existing machine learning literature. Does the reviewer has any suggestions?

Reviewer comment (Experiments):
While the trick of the Combined-Tensor-Product-Grid lacks a theoretical guarantee, more numerical results should be provided with different partition settings using this trick. This would help to demonstrate exactly how this trick helps improve efficiency, and how it balances efficiency and model complexity.

Author response:
We conducted further experiments with fixed equal spaced grid partitions for homogeneous Poisson problems with N=2 and N=3 using DecGreenNet-TT with similar networks structures used in Table 2. It shows that our proposed method performs well with different grid constructions.

N |  Error    | Time (sec)

2  |  0.006     |    51.97

3  |  0.205     | 146.21

Reviewer comment (Theory):
The paper seems to refer to [Bebendorf and Hackbusch, 2003] to justify the claim that Green's matrices are approximable with low-rank. However, that paper and much other literature on approximation of kernel/Green's matrices is using *block* low rank structure of the Green's matrices. The low-rank structure arises as a result of e.g., interaction of far away particles, which is described by a block given by a subset of rows and columns of the Green's matrix. The overall Green's matrix is invertible. This is why H-matrices, HSS-matrices, and the butterfly decomposition all try to apply low-rank approximation on blocks and not the overall the matrix. The use of tensor train decomposition here makes sense, it is to an extent capturing the block low-rank structure. But the paper does not present things in this way and I believe claims that Green's matrices generally admit good low rank approximations to be false/misleading.

Author response:
Method such as H-matrix and butterfly decomposition  are indeed used in solving PDE problems with Green’s function or Kernels [1]. In our understanding, for Green’s function formulation as ours can be formulated with Galerkin methods or collocation methods  (Page 117 [1]) and solved as a linear system where matrix inverse approaches can be employed as suggested by the reviewer.  If we parameterize the Green’s function with a neural network and apply Galerkin methods or collocation methods, then such model will not be equivalent to  the PINNs framework. We are not certain, however, such models may be more computationally expensive to train neural networks using back-propagation due to recursive multiplications used in methods such as H-matrix. 
We agree that our proposed methods of tensor-structured decomposition of the Green’s function does not stem from existing theory. Our objective in this paper is to propose a model using tensor-train based decomposition with combination of sparse grid and Smolyak grid [3] to parameterize the Green’s function and support it with experiments. We think that analytical methods in [2] can be extended to tensor structured decomposition of the Green’s function, which we propose as a future research.

Reviewer comment (Experiments):
The experimental evaluation seems to consider only very simple PDE model problems, without noise. These types of methods would be of interest to apply to experimental data to infer complex dynamics.

Author response:
Our referenced PDE models [4] and [5] have not considered the noise in PDE experiments. Hence, we also did not consider noise based PDE problems.

Reviewer comment (Writing):
The paper uses python notation instead of mathematical notion (eq 10), and contains frequent minor wording errors.

Author response:
Regarding notations and writing, we will update the writing in the revised version.