Paper: Koopman-based generalization bound: New aspect for full-rank weights (ICLR-2024)

Reviewer comment (Theory):
Proposition 5 includes factors of $\|K_{\sigma}\|$. According to the authors, such factors can be controlled by Proposition 1, which justifies absorbing those factors into the $O$ notation in equation (1). However, the proof is considered sloppy: each term in the sum over multi-indices (with components in a set of size equal to the layer width) is bounded individually, which cannot be directly used to achieve the order claimed in equation (1), because this would introduce an additional dependency on the network width. This might be avoidable with a component-wise activation function, but such an assumption is neither clearly stated nor used, and there are many missing details to make the results applicable to concrete neural networks.

Author response:
**Factor $\Vert K_{\sigma_j}\Vert$**

To derive a bound of $\Vert K_{\sigma_j}\Vert$, 
we bound $\sum_{\vert \alpha\vert\le s}c_{\alpha,s,d}\Vert \partial^{\alpha}(h\circ\sigma)\Vert^2$ by $\sum_{\vert \alpha\vert\le s}c_{\alpha,s,d}\Vert\partial^{\alpha}h\Vert^2$.
As the proof of the boundedness of $\Vert K_{\sigma}\Vert$, one strategy is using the Faa di Bruno formula.
However, as we added explanations in Appendix B, this strategy does not give us a tight bound since it involves a summation of derivatives, and the bound depends on the number of terms in the sum.
Moreover, estimating the number of terms in the sum is difficult.

Our main goal of this paper is to investigate how the property of the weight matrices affects the generalization property.
Since the norm of $K_{\sigma}$ does not depend on the weight matrices, if we assume the structure of the network is given, the property of the weight matrices does not affect the norm of $K_{\sigma}$.
As we stated in Conclusion, investigating $K_{\sigma}$ and deriving a tighter bound is future work.

In addition, the dependency of $\Vert K_{\sigma}\Vert$ on $d_j$ and $s_j$ can be alleviated to $d_{j-1}$ and $s_{j-1}$ by the factor $G_j$.
Please see Remark 5 and Appendix C for more details.
We admit that if the network is deep and if the width is sequentially increased, then the factor $G_j$ does not remove the dependency on the width of the intermediate layers.
In this work, we considered the Koopman operator for each transformation.
However, combining several layers that have similar roles together and considering the Koopman operator for the combined transformation may alleviate the dependency on the width of the intermediate layers.
In our future work, we will try to derive a tight bound by combining several layers together.

Reviewer comment (Theory):
The argument in Appendix B is vague and appears to introduce dimensional dependence.

Author response:
**Clarification of Appendix B (Appendix E in the current version)**

We updated Appendix E and added the explanation of the derivative of $\psi$.
Assume $\delta_L=1$, $\psi(x)=1$ for $x\in\Omega$, and $\Omega$ is an interval (e.g., $\psi$ is the bump function defined in Appendix D).
Then we can create a new function $\tilde{\psi}$ that satisfies $\Vert \tilde{\psi}^{(i)}\Vert_{L^2(\mathbb{R})}=\Vert \psi^{(i)}\Vert_{L^2(\mathbb{R})}$ for any $i=1,2,\ldots$, from $\psi$.
Here, $\psi^{(i)}$ is the $i$th derivative of $\psi$.

Reviewer comment (Theory):
The equivalence of Sobolev norms (as sums over derivatives and as the RKHS norm defined on page 3) should be explained in greater detail.

Author response:
**About Example 1**

We can show the representation of the Sobolev norm with the derivatives by using the properties of the Fourier transform.
We added the explanation in Appendix K.
(To keep the section number consistent with the response to other reviewers, we put the explanation in the last part of the appendix.
However, we will put it in an appropriate place in the camera-ready version.)

Reviewer comment (Writing):
Minor typos and language issues:
- Introduction, fourth line: "a large number of parameters make the complexity" → "a large number of parameters makes the complexity"
- Remark 1: "Let $g$ be a smooth function which doesn't decay at infinity..." is better phrased as suggested.
- Section 4.3: "as a result $h\circ W_j$ does not contained" → "is not contained".
- Section 4.3.2: "not whole $W_j$" → "not the whole of $W_j$".
- Section 4.4: "set of all functions which has" → "which have".
- Page 9: "we constructed a network and learned it" → "trained".
- Appendix B, top of Page 18: remove unnecessary capitalisation in "then By".

Author response:
**Minor comments**

Thank you for your comments.
We revised the paper based on your comments.

Reviewer comment (Theory):
The authors mainly consider the neural networks with dense layers. I wonder whether these theoretical results can generalize well to neural networks with other structures such as convolution. A simple explanation is recommended.

Author response:
**Generalization to other structures**

We can generalize our results to convolutional layers by regarding the convolution as the action of a matrix.
For the convolution $\sum_{i=1}^n\sum_{j=1}^mf_{k-i,l-j}x_{i,j}$ with a convolutional filter $F=[f_{i,j}]$, we can construct a tensor $W_{i,j,k,l}=f_{k-i,l-j}$.
If $i$ or $j$ is out of the bound of the index of the filter, then we set $f_{i,j}=0$.
We can combine the indices $(i,j)$ and $(k,l)$ and obtain a matrix that represents the convolution.
We added the explanation of this in Appendix J.4.
Please see Appendix J.4 for more details.
Generalizing our results to other structures, such as pooling layer, is future work.

Reviewer comment (Experiments):
The experimental results on MNIST validate the effectiveness of the induced regularization term. Can it boost model performance on datasets with larger scales such as CIFAR?

Author response:
**Experiment with CIFAR-10**

We conducted an additional experiment with AlexNet and CIFAR-10 to observe the generalization property with a regularization based on our result.
Please see Appendix J.4 for more details.
We observed that without the regularization, although the train loss becomes small, the test loss becomes large as the iteration proceeds.
On the other hand, with the regularization, the train loss becomes small, and the test loss does not become so large as the case without the regularization.

Reviewer comment (Theory):
Looking at the entire proof, the conclusion of this paper seems to be highly related to the hypothesis of RKHS. My intuitive feeling is that the conclusion of this paper mainly comes from the RKHS assumption, and whether the neural network is abstracted into Koopman operators has little relevance. I hope the author can explain the relationship between Koopman operators, RKHS and the final conclusion and give an idea of what kind of these techniques/assumptions play a role in the proofs.

Author response:
**Relationship among Koopman operators, RKHS, and final conclusion**

The derivation of our bound depends on the assumption of RKHS, but it also strongly depends on the application of the Koopman operators.
By the assumption of RKHS, evaluating the Rademacher complexity is reduced to evaluating $\Vert f\Vert$.
As we also stated above, by virtue of the Koopman operators, we can bound $\Vert f\Vert$ using the norm of Koopman operators.
An important feature of our bound is that it involves the determinant of the weight matrix in the denominator.
This determinant factor appears by bounding the norm of the Koopman operator.
We didn't have this type of factor in existing results, which means the norm of the Koopman operator gives us a new perspective on the generalization of neural networks.
By the effect of the determinant factor, our bound becomes small if the smallest singular value of the weight matrix is large.
The direction of this result is completely different from that of existing results that focus on low-rank weight matrices or weight matrices with small singular values. 
Our result reveals why networks with full or high-rank weight matrices generalize well.