Paper: A path-norm toolkit for modern networks: consequences, promises and challenges (ICLR-2024)

Reviewer comment (Theory):
The paper establishes generalization bounds with respect to $L^1$ path-norms (as opposed to general $L^p$ path-norms).

Author response:
Thank you for your comment. Establishing generalization bounds based on general $L^p$-norms would be very interesting given their magnitude order reported in the experiments but it is still a major open question in the literature. Apart from the bounds that derive from the $L^1$-bound using basic comparisons between $L^p$-norms, we believe that non-trivial bounds can only be obtained if the activations are taken more carefully into account, the latter being a non-trivial matter. We plan to investigate that in the future.

Reviewer comment (Evaluation):
In the experiments, $L^1$ path-norms tend to be extremely large (at least $10^{30}$) which makes the generalization bounds vacuous.

Author response:
Regarding the size of the $L^1$ path-norm, the main point of the document is precisely to assess and report these *existing* weaknesses *of current theory* for the first time on modern neural networks.

Reviewer comment (Theory):
I think Setting 1 of Lemma C.1 can be obtained by a simple application of the well-known scalar contraction inequality (Ledoux & Talagrand, 1991). Given $v\in V$ and $t\in T^v$, we can embed it into a higher dimensional space $\mathbb{R}^{I\times T}$ by defining $\tilde t(v,t) = [0, 0 ,\ldots, v,\ldots, 0]$ as a $I\times T$ matrix. We further define $\tilde f_{i,v}(x) = f_i(x)$ for all $x\in\mathbb{R}$. Therefore we have the following which is exactly the conclusion of Setting 1 of Lemma C.1 

$$\mathbb{E} \max_{v \in V} \sup_{t \in T^v} G \left( \sum_{i \in I} \varepsilon_{i,v} f_i(t_i) \right) $$

$$=\mathbb{E} \max_{v \in V} \sup_{t \in T^v} G \left( \sum_{i \in I} \varepsilon_{i,v} f_i([\tilde t(v,t)] _{i,v}) \right)$$

$$=\mathbb{E} \max_{v \in V} \sup_{t \in T^v} G \left( \sum_{i \in I, v' \in V} \varepsilon_{i,v'} \tilde f_{i,v'}([\tilde t(v,t)] _{i,v'}) \right)$$ 

$$= \mathbb{E} \max_{v \in V} \sup_{t \in T^v} G \left( \sum_{i \in I, v' \in V} \varepsilon_{i,v'} [\tilde t(v,t)] _{i,v'} \right) $$

$$=\mathbb{E} \max_{v \in V} \sup_{t \in T^v} G \left( \sum_{i \in I} \varepsilon_{i,v} [\tilde t(v,t)] _{i,v} \right)$$

$$=\mathbb{E} \max_{v \in V} \sup_{t \in T^v} G \left( \sum_{i \in I} \varepsilon_{i,v} t_i \right) $$

Author response:
Thank you very much for your comment, one can indeed reduce to (Ledoux and Talagrand, 1991) in the more elegant and easier way you describe. However, this proves only the case where the $t_i$ are scalar because (Ledoux and Talagrand, 1991) does not apply to the general vector-valued case. Thus, it still remains to prove a similar contraction lemma in the case of max-pooling neurons. In this situation, the same argument would basically correspond to the part of the proof that reduces to the case $|V|=1$ and then, it still remains a significant part of work. All of this will be mentioned in the final version.

Reviewer comment (Theory):
measurable condition in Lemma D.1 seems unnecessary?

Author response:
Thank you for your remark. What is actually needed is that the expectations are well-defined. For that, we will rather assume in the final version of the document that the suprema are measurable (not only $G$). Everything being non-negative, the expectations will make sense in $[0,\infty]$.

Reviewer comment (Theory):
Why 2-norm of $\Phi(\theta)$ is relevant? Isn't it small just because the number of paths is huge? It doesn't seem to be a valid generalization bound even for a very simple 2-layer linear network. Is there any real evidence that 2-norm of $\Phi(\theta)$ could be useful for indicating generalization?

Author response:
This is an interesting and open question that we plan to investigate in the future. The experiments report the values of $L^p$-path-norms for $p\neq 1$ for the simple reason that it is the first time they are computed on modern neural networks. However, the relevance of general $L^p$-path-norm for theory and practice is not clear for now. What is known is that: 1) in practice, $L^2$-path-norm is found to empirically correlate better with the generalization error than the $L^1$-path-norm (Jiang et al, 2020), and 2) because of basic comparisons between $L^p$-norms, any $L^1$-path-norm bound yield $L^p$-path-norm bounds ($p>1$) but with large dimensional constants. We think that these dimensional dependencies could be avoided by carefully taking into account the activations (for now, the bound corresponds to the worst case where all paths are activated), see also the discussion in the conclusion.

Reviewer comment (Writing):
Some minor typos: (1). $\overline N_{in}$ seems not defined in appendix A; (2). The third row of the first equation on page 29 seems missing a factor of $5^d$

Author response:
Thank you, this will be corrected in the final version.

Reviewer comment (Theory):
I wonder if the authors can just ignore all the linear activations by linking the incoming nodes of the current nodes to the outgoing nodes of the current node. This should not change the path norm. If this is true, then we should not count the number of Conv layers in ResNet, but the max number of ReLU and max pooling along the path, which is smaller than the former for ResNets.

Author response:
You are right, thank you. This will be added in the final version. For experiments, this allows us to reduce by one the depth $D$ by incorporating the average-pooling layer into the next fully connected layer of ResNets (we still count one for each Conv Layer as each such layer ends up with a ReLU). The numerical results will be updated accordingly (it will not change the associated message). We will still keep the definition of the model as it is (with identity neurons) as we believe it to be clearer for implementing operations such as average-pooling.

Reviewer comment (Writing):
Though the writing quality is already great, I still feel like the paper can benefit from first providing a result without any bias, which would greatly simplify the notations in the theoretical statement and the proof. Then the authors can point out that the results can be generalized to networks with bias by appending a constant 1 dimension in the input (as it is done now), adding an edge between the constant 1 input dimension to the neuron, and then calling the main result. Writing the theorem in this way would clearly explain to the readers where the plus one in $d_{in}+1$ and the $\max(n,\ldots)$ in $\sigma$ comes from. (Theorem 3.1)

Author response:
Thank you for your suggestion. The final version will contain the statement without biases, and we will mention how to adapt the result in the case with biases. The proof will still be given in the general case directly with biases.

Reviewer comment (Theory):
The statement of Lemma D.2 is a little bit confusing because $\Theta$ is not arbitrary, but the one preprocessed by Algorithm 1. Maybe it is more clear to just define $r$ as the sup of 1-norm of the last layer of $\Theta$? A similar issue occurs for Lemma D.3, where the authors can just say the 1-norm of incoming weights of non-output node in $\Theta$ are bounded. In this way, Lemma D.2 and Lemma D.2 will together motivate Algorithm 1 and the path norm as a generalization metric.

Author response:
We agree with your suggestion and this will be taken into account in the final version.

Reviewer comment (Experiments):
The numerical experiments were partial, and I would expect to see how the theoretical bound differs from an estimation of the generalization using some validation dataset.

Author response:
In such a situation, the validation error is very close to the empirical test error so that the generalization error is well predicted using a validation set (with an error typically of the order of one or less for the top-1 accuracy), as it is usually done in practice.

Reviewer comment (Theory):
"Proof of the Lipschitz Property" in page 14 ends with an incorrect statement (IMHO). The boundaries of the constant regions must be examined, and the proof holds because there is a finite number of such regions.

Author response:
Thank you for your remark. The final version will contain more details that were indeed missing here: this remains true on the borders by continuity, and it extends to the whole domain because between any pair of points, one changes of region only a finite number of times.