Paper: Scalable Lipschitz Estimation for CNNs (ICLR-2024)

Reviewer comment (Experiments):
I think the title "Scalable Lipschitz Estimation for CNNs" is overclaiming, while the proposed approach is more scalable than previous ones, the approach hardly scales to medium size convolutional neural networks (CIFAR10, TinyImagenet, ImageNet). Furthermore, to improve the scalability, the authors rely on the bounds of Eq. (3) and Eq. (4), which are known to be very loose.

Author response:
3.**On Equations (3, 4) (Weakness)**: We would like to point out that Eq. (3) has been used by the LipSDP paper [Fazlyab2019](https://proceedings.neurips.cc/paper_files/paper/2019/file/95e1533eb1b20a97777749fb94fdb944-Paper.pdf) to accelerate Lipschitz estimation for deep, fully-connected neural networks. While our work uses Eqs. (3) and (4) to accelerate deep and wide CNNs. Both their experiments and ours show the obtained estimations are sufficiently good in practice. However, a rigorous analysis of the gaps between the bounds resulting from applying Eqs. (3) and (4) and the exact Lipschitz constant, as well as developing and proving tighter bounds, are interesting future research directions.

Reviewer comment (Novelty):
The authors seem to have missed an important related work [1], which proposes a Lipschitz estimation for the CNN that is independent of the image size and only dependent on the channel size.

Author response:
2.**Comparison with [[Gramlich2023]](https://arxiv.org/pdf/2303.03042.pdf), MNIST (Question and Weakness)**:
As suggested, we have included [Gramlich2023](https://arxiv.org/pdf/2303.03042.pdf) in related work and attempted to compare with it. Reviewer G8PH suggested the same comparison, and we repeat the same response here:

We conducted the CNN1 experiment trained on MNIST images, hoping to reproduce Example 3 in [Gramlich2023](https://arxiv.org/pdf/2303.03042.pdf) as closely as possible. We used an identical architecture to Example 3 apart from padding. Their CNN size is $28\times 28\times 1 \rightarrow 24\times 24\times 1 \rightarrow 16\times 16\times 1 \rightarrow 50 \rightarrow 10$, while CNN1 has a larger size $20\times 20\times 1$ in the second convolutional layer. To the best of our knowledge, the code for [Gramlich2023](https://arxiv.org/pdf/2303.03042.pdf) is not publicly available, and it is challenging for us to implement their approach from scratch within the time allowed by discussion period. Moreover, they do not report computing environment, so we cannot compare results directly. However, we  summarise the results for reference; For CNN1, the matrix norm product bound computed a Lipschitz estimation of 91.31 , and our method computed an estimation of $65.84$ in 
$864$ seconds. In comparison, [Gramlich2023](https://arxiv.org/pdf/2303.03042.pdf) reported an estimation of $18.9$ computed in $1550$ seconds.

Reviewer comment (Experiments):
The experiments are performed on very small input sizes, more than twice smaller than MNIST: "[...] by constructing a convolutional block with an input size of 10 × 10 × 1 [...]". I understand that the authors want to evaluate their approach with different channel sizes: "We increased the width by varying c from 1 to 14", but an image size of 10 × 10 × 1 is too small to demonstrate scalability.

Author response:
1.**Larger Trained Networks (Weakness)**: First of all, we would like to mention that, in our submitted draft, we did experiment with several neural networks with a larger input size than $10 \times 10\times 1$. The largest of which is a 10-layer convolutional block with an input size of $64 \times 64$ and a maximum hidden-layer width of 3600 when flattened, and is used to analyse the behaviour of the proposed DCP approach, resulting in Figure 2(c). Now we have added another experiment testing with an even larger network trained on CIFAR10 images, referred to as CNN2 in the draft. CNN2 has 5 convolutional layers of dimension: $30 \times 30 \times 5, 28 \times 28 \times 5, 26 \times 26 \times 5, 24 \times 24 \times 5$ and $22 \times 22 \times 5$, corresponding to a maximum hidden-layer width of 4500 when flattened. This was followed by 2 fully-connected layers of size 50 and 100 respectively.

Reviewer comment (Presentation):
I have reviewed the paper on a printed version and all the figures are unreadable, the authors should definitely increase the font of the figures.

Author response:
We thank the reviewer for their constructive comments. As suggested, we have added new experiments for larger neural networks and attempted to compare with [Gramlich2023](https://arxiv.org/pdf/2303.03042.pdf). Please see the new results added to Section 5.1 in the revised draft and our response below. We have also increased the font size of the figures.

Reviewer comment (Reproducibility):
The link given in the paper to access the code does not work for me.

Author response:
**Code Link (Weakness)**: The link works when clicked on as opposed to copy and pasting. We apologise for not noticing this earlier and have fixed it in the revised draft.

Reviewer comment (Writing):
Section 4 is difficult to follow and, in my opinion, lacks many details. See some comments in the Questions below.

Author response:
**Section 4 (Weakness and Question)**: We have clarified how to construct $f_{ij}$ and that the domains $X_{ij}^{0}$ are not disjoint in general, in Section 4.1.1 of the revised manuscript. Regarding the impact of the potential overlap between the layers, if we understand the reviewer correctly, then we remark that the layer-wise cutting is applied in such a way that the output layer of one subnetwork is the input layer of the proceeding one. Furthermore, while the receptive fields of the smaller convolutional blocks overlap in general, the corresponding weights do not. This independence between convolutional blocks is a key component of our method as it permits the computation of the Lipschitz constants in parallel.

Reviewer comment (Experiments):
The experimental section is weak. In my opinion, it lacks experiments using trained networks for which the behaviour is expected to be really different from randomly generated ones, in particular for its Lipschitz constant.

Author response:
**Bigger and Trained Networks (Weakness, Question on Experiments)**: To strengthen the experiments, we have tested against larger CNNs trained on MNIST and CIFAR10 images where Lip-SDP would fail, and compared our method with the naive estimation. Please see our revised Section 5.1 for details. When designing the experiments, our focus was on assessing the scalability to wide layers resulting from flattening the convolutional operators, as opposed to LipSDP [Fazlyab2019](https://proceedings.neurips.cc/paper_files/paper/2019/file/95e1533eb1b20a97777749fb94fdb944-Paper.pdf), who  focused on fully-connected layers and scaled to deeper networks through layer-wise cutting. As our approach also incorporates layer-wise cutting, experiment emphasis was placed on scalability to wider layers under the SDP-based framework.  For instance, [Fazlyab2019](https://proceedings.neurips.cc/paper_files/paper/2019/file/95e1533eb1b20a97777749fb94fdb944-Paper.pdf) experimented with a  maximum network width of $100$ neurons, while we experimented with maximum widths ranging from $3600$ to $4500$ neurons.

Reviewer comment (Experiments):
It is not clear if LipSDP profits from the 20-cores CPU machine and hence if the experimental benchmark is fair.

Author response:
**LipSDP/20-core CPU (Weakness)**: Both LipSDP and our method DCP-LipSDP have access to the 20 cores, so the benchmark is fair. We have clarified this at the start of revised version of Section 5.

Reviewer comment (Presentation):
Bibliography: many references are incomplete; please refer to peer-reviewed articles instead of arXiv when possible.

Author response:
**Bibliography (Weakness)**: We have fixed this in the revised draft, replacing all the arXiv papers with the official venues where applicable.

Reviewer comment (Writing):
In Section 4.1.1, it is not clear what are precisely the functions \( f_{i, j} \), and how to practically construct them.

Author response:
**Section 4 (Weakness and Question)**: We have clarified how to construct $f_{ij}$ and that the domains $X_{ij}^{0}$ are not disjoint in general, in Section 4.1.1 of the revised manuscript. Regarding the impact of the potential overlap between the layers, if we understand the reviewer correctly, then we remark that the layer-wise cutting is applied in such a way that the output layer of one subnetwork is the input layer of the proceeding one. Furthermore, while the receptive fields of the smaller convolutional blocks overlap in general, the corresponding weights do not. This independence between convolutional blocks is a key component of our method as it permits the computation of the Lipschitz constants in parallel.

Reviewer comment (Theory):
In Section 4.1.1, it seems that the domain of \( f_{i, j} \) should not be disjoint as it is stated in Eq. (10). If \( \ell \times k > m_0, n_0 \) where \( k \) is the size of the kernel of these convolutions, then all \( f_{i,j} \) should have domain \( X^0 \).

Author response:
**Section 4 (Weakness and Question)**: We have clarified how to construct $f_{ij}$ and that the domains $X_{ij}^{0}$ are not disjoint in general, in Section 4.1.1 of the revised manuscript. Regarding the impact of the potential overlap between the layers, if we understand the reviewer correctly, then we remark that the layer-wise cutting is applied in such a way that the output layer of one subnetwork is the input layer of the proceeding one. Furthermore, while the receptive fields of the smaller convolutional blocks overlap in general, the corresponding weights do not. This independence between convolutional blocks is a key component of our method as it permits the computation of the Lipschitz constants in parallel.

Reviewer comment (Theory):
The possible inaccuracy regarding the domains of \( f_{i, j} \) appears to be repeated in the entire paper (e.g., in Section 4.1.2 Eq. (14)), though this idea may be exploited to better constrain the search space for the partitions.

Author response:
**Section 4 (Weakness and Question)**: We have clarified how to construct $f_{ij}$ and that the domains $X_{ij}^{0}$ are not disjoint in general, in Section 4.1.1 of the revised manuscript. Regarding the impact of the potential overlap between the layers, if we understand the reviewer correctly, then we remark that the layer-wise cutting is applied in such a way that the output layer of one subnetwork is the input layer of the proceeding one. Furthermore, while the receptive fields of the smaller convolutional blocks overlap in general, the corresponding weights do not. This independence between convolutional blocks is a key component of our method as it permits the computation of the Lipschitz constants in parallel.

Reviewer comment (Theory):
A discussion should clarify the concerns about the domains of \( f_{i, j} \) and the impact of potential overlap between the layers on the computation time.

Author response:
**Section 4 (Weakness and Question)**: We have clarified how to construct $f_{ij}$ and that the domains $X_{ij}^{0}$ are not disjoint in general, in Section 4.1.1 of the revised manuscript. Regarding the impact of the potential overlap between the layers, if we understand the reviewer correctly, then we remark that the layer-wise cutting is applied in such a way that the output layer of one subnetwork is the input layer of the proceeding one. Furthermore, while the receptive fields of the smaller convolutional blocks overlap in general, the corresponding weights do not. This independence between convolutional blocks is a key component of our method as it permits the computation of the Lipschitz constants in parallel.

Reviewer comment (Experiments):
All neural networks considered have weights initialized with the Kaiming distribution, aiming at having gradients of magnitude \( O(1) \), which is most likely very different from a trained neural network. The paper would be seriously improved if the author were to reproduce the experiments of Lip-SDP and benchmark them with the proposed DCP approach.

Author response:
**Bigger and Trained Networks (Weakness, Question on Experiments)**: To strengthen the experiments, we have tested against larger CNNs trained on MNIST and CIFAR10 images where Lip-SDP would fail, and compared our method with the naive estimation. Please see our revised Section 5.1 for details. When designing the experiments, our focus was on assessing the scalability to wide layers resulting from flattening the convolutional operators, as opposed to LipSDP [Fazlyab2019](https://proceedings.neurips.cc/paper_files/paper/2019/file/95e1533eb1b20a97777749fb94fdb944-Paper.pdf), who  focused on fully-connected layers and scaled to deeper networks through layer-wise cutting. As our approach also incorporates layer-wise cutting, experiment emphasis was placed on scalability to wider layers under the SDP-based framework.  For instance, [Fazlyab2019](https://proceedings.neurips.cc/paper_files/paper/2019/file/95e1533eb1b20a97777749fb94fdb944-Paper.pdf) experimented with a  maximum network width of $100$ neurons, while we experimented with maximum widths ranging from $3600$ to $4500$ neurons.

Reviewer comment (Experiments):
Figure 2(c) shows a 5-fold time reduction that only decreases the upper bound by 20%, which is surprising given the behaviour expected from Eq. (18). It would be interesting to see if the proposed approach could tackle much bigger neural networks where vanilla Lip-SDP would fail.

Author response:
**Bigger and Trained Networks (Weakness, Question on Experiments)**: To strengthen the experiments, we have tested against larger CNNs trained on MNIST and CIFAR10 images where Lip-SDP would fail, and compared our method with the naive estimation. Please see our revised Section 5.1 for details. When designing the experiments, our focus was on assessing the scalability to wide layers resulting from flattening the convolutional operators, as opposed to LipSDP [Fazlyab2019](https://proceedings.neurips.cc/paper_files/paper/2019/file/95e1533eb1b20a97777749fb94fdb944-Paper.pdf), who  focused on fully-connected layers and scaled to deeper networks through layer-wise cutting. As our approach also incorporates layer-wise cutting, experiment emphasis was placed on scalability to wider layers under the SDP-based framework.  For instance, [Fazlyab2019](https://proceedings.neurips.cc/paper_files/paper/2019/file/95e1533eb1b20a97777749fb94fdb944-Paper.pdf) experimented with a  maximum network width of $100$ neurons, while we experimented with maximum widths ranging from $3600$ to $4500$ neurons.

Reviewer comment (Theory):
It is suggested to investigate whether the curves in Figure 2(c) could be derived analytically directly from Eq. (18) under the assumption that the weights follow Kaiming initialization.

Author response:
**Other Questions on Experiments**:  Regarding to an analytical derivation of the type of curves in Figure 2(c), this is an interesting research question posed by the reviewer and we hope to pursue it rigorously in the future.

Reviewer comment (Experiments):
The method, while novel, is not scalable to modern large convolutional neural networks. However, the improvements on top of the existing methods are impressive.

Author response:
We appreciate the reviewer's acknowledgement of the novelty, theoretical foundation and empirical validation of our work. We agree that we should strengthen our experiments, e.g. testing with larger networks, for which we have added new experiments. Please see our response below.

1.**Weakness**: We have added new experiments testing against larger neural networks trained on MNIST and CIFAR10 images. Please see our revised Section 5.1 for further details. We have also added the two suggested works [Sedghi](https://openreview.net/pdf?id=rJevYoA9Fm)  and [Singla2021a](https://openreview.net/pdf?id=JCRblSgs34Z) to Section 2, in the revised draft.

Reviewer comment (Novelty):
The paper is missing references to important works on estimating the Lipschitz constants of convolutional layers, specifically: "Singular Values of Convolution Layers" and "Fantastic Four: Differentiable and Efficient Bounds on Singular Values of Convolution Layers".

Author response:
We appreciate the reviewer's acknowledgement of the novelty, theoretical foundation and empirical validation of our work. We agree that we should strengthen our experiments, e.g. testing with larger networks, for which we have added new experiments. Please see our response below.

1.**Weakness**: We have added new experiments testing against larger neural networks trained on MNIST and CIFAR10 images. Please see our revised Section 5.1 for further details. We have also added the two suggested works [Sedghi](https://openreview.net/pdf?id=rJevYoA9Fm)  and [Singla2021a](https://openreview.net/pdf?id=JCRblSgs34Z) to Section 2, in the revised draft.

Reviewer comment (Experiments):
The experiments are still on toy examples. Is the proposed approach working for larger networks trained on CIFAR10/100? At least, the proposed method should be better than the matrix product bound for CIFAR?

Author response:
We thank the reviewer for their appreciation of the proposed method and their constructive comments on improving experiments, for which we have added a series of new experiments testing for larger networks trained on MNIST and CIFAR10 images (see our revised Section 5.1). Please see our detailed response below.
1. **Weakness 1 and Question 1**: We have compared the proposed approach with the matrix product bound (referred to as naive estimation in our paper) for two larger networks (CNN1 and CNN2). CNN1 has size: $28\times 28\times 1 \rightarrow 24\times 24\times 1 \rightarrow 20\times 20\times 1 \rightarrow 50 \rightarrow 10$, trained on MNIST, and CNN2 has size: $32 \times 32 \times 3 \rightarrow 30 \times 30 \times 5 \rightarrow  28 \times 28 \times 5 \rightarrow  26 \times 26 \times 5 \rightarrow  24 \times 24 \times 5 \rightarrow  22 \times 22 \times 5 \rightarrow 50 \rightarrow 100 \rightarrow 100$, trained on CIFAR10. Our approach  outperforms the matrix product bound in both cases. Please see Trained Networks and Table 1 in our revised Section 5.1 for further details.

Reviewer comment (Novelty):
The authors missed several important related work on LipSDP. Comparisons and discussions of the following works are missing:

Author response:
2. **Weakness 2, [Gramlich2023](https://arxiv.org/pdf/2303.03042.pdf) and Question 2**: We have conducted the CNN1 experiment trained on MNIST images, aiming to reproduce Example 3 in [Gramlich2023](https://arxiv.org/pdf/2303.03042.pdf) as closely as possible, given the details provided. Our CNN1 used an identical CNN setting to Example 3 apart from padding. Their CNN size is $28\times 28\times 1 \rightarrow 24\times 24\times 5 \rightarrow 16\times 16\times 5 \rightarrow 50 \rightarrow 10$, while our CNN1 has a slightly larger size $20\times 20\times 5$ in the second convolutional layer. To the best of our knowledge, the code for [Gramlich2023](https://arxiv.org/pdf/2303.03042.pdf) is not publicly available, and it is challenging for us to implement their approach from scratch within the time allowed by the discussion period. Moreover, they do not report their computing environment, so we cannot compare results directly. However, we  summarise the results for reference; for CNN1, the matrix norm product bound computed a Lipschitz estimation of 91.31, and our method computed an estimation of $65.84$ in 
 $864$ seconds. [Gramlich2023](https://arxiv.org/pdf/2303.03042.pdf) reported an estimation of $18.9$ computed in $1550$ seconds.

Reviewer comment (Experiments):
[Gramlich2023] Convolutional neural networks as 2-D systems (arXiv) — The above paper extends LipSDP for 2D convolutional networks. Is the proposed approach more scalable than the above paper?

Author response:
2. **Weakness 2, [Gramlich2023](https://arxiv.org/pdf/2303.03042.pdf) and Question 2**: We have conducted the CNN1 experiment trained on MNIST images, aiming to reproduce Example 3 in [Gramlich2023](https://arxiv.org/pdf/2303.03042.pdf) as closely as possible, given the details provided. Our CNN1 used an identical CNN setting to Example 3 apart from padding. Their CNN size is $28\times 28\times 1 \rightarrow 24\times 24\times 5 \rightarrow 16\times 16\times 5 \rightarrow 50 \rightarrow 10$, while our CNN1 has a slightly larger size $20\times 20\times 5$ in the second convolutional layer. To the best of our knowledge, the code for [Gramlich2023](https://arxiv.org/pdf/2303.03042.pdf) is not publicly available, and it is challenging for us to implement their approach from scratch within the time allowed by the discussion period. Moreover, they do not report their computing environment, so we cannot compare results directly. However, we  summarise the results for reference; for CNN1, the matrix norm product bound computed a Lipschitz estimation of 91.31, and our method computed an estimation of $65.84$ in 
 $864$ seconds. [Gramlich2023](https://arxiv.org/pdf/2303.03042.pdf) reported an estimation of $18.9$ computed in $1550$ seconds.

Reviewer comment (Theory):
[Araujo2023] A unified algebraic perspective on Lipschitz neural networks (ICLR) — Theorem 4 of the above paper extends LipSDP for residual networks. Is the proposed approach directly applicable to the residual network considered in the above paper? Some remarks are needed.

Author response:
3. **Weakness 2 of [Araujo2023](https://openreview.net/pdf?id=k71IGLC8cfc), [Revay2020](https://arxiv.org/pdf/2010.01732.pdf) and Questions 3, 4**: As suggested, we have added a discussion of potential applications of the proposed approach to other types of neural networks, including the discussion based on the two papers [Araujo2023](https://openreview.net/pdf?id=k71IGLC8cfc) and [Revay2020](https://arxiv.org/pdf/2010.01732.pdf). This can be found in the revised version of Section 6.

Reviewer comment (Theory):
[Revay2020] Lipschitz bounded equilibrium networks (arXiv) — Theorem 2 of the above paper extends LipSDP for equilibrium networks. Is the proposed approach directly applicable to the equilibrium network considered in the above paper? Some remarks are needed.

Author response:
3. **Weakness 2 of [Araujo2023](https://openreview.net/pdf?id=k71IGLC8cfc), [Revay2020](https://arxiv.org/pdf/2010.01732.pdf) and Questions 3, 4**: As suggested, we have added a discussion of potential applications of the proposed approach to other types of neural networks, including the discussion based on the two papers [Araujo2023](https://openreview.net/pdf?id=k71IGLC8cfc) and [Revay2020](https://arxiv.org/pdf/2010.01732.pdf). This can be found in the revised version of Section 6.

Reviewer comment (Experiments):
Is the proposed approach working for larger networks trained on CIFAR10/100 and can at least outperform the matrix norm product for those tasks?

Author response:
We thank the reviewer for their appreciation of the proposed method and their constructive comments on improving experiments, for which we have added a series of new experiments testing for larger networks trained on MNIST and CIFAR10 images (see our revised Section 5.1). Please see our detailed response below.
1. **Weakness 1 and Question 1**: We have compared the proposed approach with the matrix product bound (referred to as naive estimation in our paper) for two larger networks (CNN1 and CNN2). CNN1 has size: $28\times 28\times 1 \rightarrow 24\times 24\times 1 \rightarrow 20\times 20\times 1 \rightarrow 50 \rightarrow 10$, trained on MNIST, and CNN2 has size: $32 \times 32 \times 3 \rightarrow 30 \times 30 \times 5 \rightarrow  28 \times 28 \times 5 \rightarrow  26 \times 26 \times 5 \rightarrow  24 \times 24 \times 5 \rightarrow  22 \times 22 \times 5 \rightarrow 50 \rightarrow 100 \rightarrow 100$, trained on CIFAR10. Our approach  outperforms the matrix product bound in both cases. Please see Trained Networks and Table 1 in our revised Section 5.1 for further details.

Reviewer comment (Theory):
[Araujo2023] A unified algebraic perspective on Lipschitz neural networks (ICLR) — Theorem 4 of the above paper extends LipSDP for residual networks. Is the proposed approach directly applicable to the residual network considered in the above paper? Some remarks are needed.

Author response:
3. **Weakness 2 of [Araujo2023](https://openreview.net/pdf?id=k71IGLC8cfc), [Revay2020](https://arxiv.org/pdf/2010.01732.pdf) and Questions 3, 4**: As suggested, we have added a discussion of potential applications of the proposed approach to other types of neural networks, including the discussion based on the two papers [Araujo2023](https://openreview.net/pdf?id=k71IGLC8cfc) and [Revay2020](https://arxiv.org/pdf/2010.01732.pdf). This can be found in the revised version of Section 6.

Reviewer comment (Novelty):
[Revay2020] Lipschitz bounded equilibrium networks (arXiv) — Theorem 2 of the above paper extends LipSDP for equilibrium networks. Is the proposed approach directly applicable to the equilibrium network considered in the above paper? Some remarks are needed.

Author response:
3. **Weakness 2 of [Araujo2023](https://openreview.net/pdf?id=k71IGLC8cfc), [Revay2020](https://arxiv.org/pdf/2010.01732.pdf) and Questions 3, 4**: As suggested, we have added a discussion of potential applications of the proposed approach to other types of neural networks, including the discussion based on the two papers [Araujo2023](https://openreview.net/pdf?id=k71IGLC8cfc) and [Revay2020](https://arxiv.org/pdf/2010.01732.pdf). This can be found in the revised version of Section 6.