Paper: Novel Quadratic Constraints for Extending LipSDP beyond Slope-Restricted Activations (ICLR-2024)

Reviewer comment (Novelty):
The main concern I have is that this paper seems to be an extension of two works Fazlyab'19 and Wang'22 to the case of having sum-preserving activations like GroupSort, Maxmin, and Householder, which seems incremental.

Author response:
We think that our contribution in deriving new quadratic constraints for GroupSort, MaxMin, and Householder are novel and significant, complementing the existing works such as Fazlyab'19 and Wang'22. We provide a detailed response in our general response (Part 1). Let us restate some of our key points here. We also included Appendix B to discuss the issue in our revised paper. The underlying idea in the quadratic constraint approach is to abstract "challenging", e.g., nonlinear, uncertain or time-varying elements using quadratic constraints. This abstraction allows us to formulate an SDP that can be solved effectively. Since the introduction of the quadratic constraint framework in the 1960s, many papers appeared that are only devoted to deriving new quadratic constraints. For example, Kao'07 derives quadratic constraints for a delay-difference operator to describe varying time delays, Pfifer'15 uses a geometric interpretation to derive new quadratic constraints for delayed nonlinear and parameter-varying systems, and Carrasco'16 summarizes the development of Zames-Falb multipliers based on frequency domain arguments for slope-restricted nonlinearites. In the context of NNs, so far only quadratic constraints for slope-restricted nonlinearities were utilized. By formulating quadratic constraints for GroupSort/Householder activations, we are the first ones to study multivariate activations, to formulate quadratic constraints for NNs that go beyond slope-restriction or Lipschitz continuity and in general the first to present quadratic constraints for sum-preserving elements. Finding novel quadratic constraints is by no means trivial. It requires to study and understand the characteristics of the underlying nonlinearity and then to capture the identified property using a quadratic form. There is no recipe for such a derivation and finding a quadratic constraint does not guarantee that it improves the analysis over other methods. The tighter the description of the nonlinearity by quadratic constraints, the better the analysis in terms of conservatism. The idea we used to derive these quadratic constraints is creative in the sense that it is very different from all the existing quadratic constraint derivations, hence offering unique new insights and complementing the large body of existing arguments in deriving quadratic constraints.  To understand the technical details of our derivation, we refer to the proofs in Appendices C.2 and C.3.

Reviewer comment (Novelty):
How frequently are GroupSort, Maxmin, and Householder being used in practice? If they are not so popular, why we are studying LipSDP for them?

Author response:
Please see our general response (Part 2). GroupSort, MaxMin, and Householder activations are gradient norm preserving and have been of particular interest when fitting Lipschitz functions and for the design of certifiably robust neural networks (NNs). There are many works (e.g. Leino'21, Singla'21, Trockman'21, Huang'21, Singla'22, Prach'22, Hu'23) that use MaxMin activations and provide state-of-the-art certified robust accuracy (please see our general response, Part 2 for a detailed list of the references). To compute the certified robust accuracy, these works rely on information of Lipschitz bounds and prediction margins. Hence LipSDP for MaxMin networks is of great interests. Another interesting application of MaxMin NNs is in learning-based control (Yion'23), where the information of the Lipschitz constant of a neural network in the loop can be useful to verify safety and robustness. Please note that we included a section on applications of MaxMin NNs as Appendix A.

Reviewer comment (Novelty):
The major limitation of this work is the restriction to Householder and GroupSort activations. The utility of extending the LipSDP framework to these activations depends directly on how interesting the problem of Lipschitz estimation is for neural networks using these architectures. While the authors state that such activations are becoming popular for the design of Lipschitz neural networks, no concrete examples are provided.

Author response:
> Additional details on why the GroupSort and Householder activations are of particular interest for Lipschitz constant estimation?

 Please see our general response (Part 2). GroupSort and Householder activations are gradient norm preserving and have been of particular interest when fitting Lipschitz functions and for the design of certifiably robust neural networks (NNs). There are many works published in top machine learning conferences (e.g. Leino'21, Singla'21, Trockman'21, Huang'21, Singla'22, Prach'22, Hu'23) that use MaxMin activations and provide state-of-the-art certified robust accuracy (we provide details in our general response, Part 2). To compute the certified robust accuracy, these works rely on information of Lipschitz bounds and prediction margins of the trained networks. Hence LipSDP for MaxMin networks is of great interests. Another interesting application of MaxMin NNs is in learning-based control (Yion'23), where the information of the Lipschitz constant of a neural network in the loop can be useful to verify safety and robustness. Please note that we included a section on applications of MaxMin NNs as Appendix A.

Reviewer comment (Evaluation):
The authors do not provide the computation time for the naive baseline method for approximating Lipschitz constants based on operator norms (MP), so it is not clear what the trade-off between computation and accuracy is for the proposed method.

Author response:
> I would also appreciate it if the authors could provide running times for the naive estimation strategies.

We are happy to provide this additional information. We recalculated the computation times for all our naive bounds. The computation times of the matrix product bound are for all our networks below 10s, so they are basically instantaneous. This means there definitely is a price for the tighter bounds using our method. The computation times for the lower bounds from sampling based on 200k samples range between 12 and 23 s. The FGL bound takes 4.54s for the 2FC-16 NN and 81.92s for the 2FC-32 NN. For larger NNs it however becomes intractable (we stopped our code for 2FC-64 after 4 hours).

Find the running times of all naive l2 bounds for our networks in Table 1 in the table below:
| Architecture  | Sample   | FGL | MP |
| -- |:--:| --:|:--:| 
| 2FC-16        | 12.02 | 4.54 | 6.16 |
| 2FC-32        | 13.08 | 81.92| 5.33 |
| 2FC-64        | 12.84 |      | 3.95 |
| 2FC-128       | 12.61 |      | 3.6  |
| 5FC-32        | 13.18 |      | 5.43 |
| 5FC-64        | 14.83 |      | 5.68 |
| 8FC-32        | 14.14 |      | 5.57 |
| 8FC-64        | 14.27 |      | 5.6  |
| 8FC-128       | 14.15 |      | 9.85 |
| 8FC-256       | 13.74 |      | 9.3  |
| 18FC-32       | 23.28 |      | 7.64 |
| 18FC-64       | 21.09 |      | 9.22 |
| 18FC-128      | 19.48 |      | 4.9  |

Reviewer comment (Experiments):
The result seems to be weak since the quadratic constraints only applied to 2 specific activations. It would be more interesting if it can be applied to a class of activaitons.

Author response:
We would consider GroupSort and HouseHolder as two classes of activations. Both generalize MaxMin which is widely used for robust networks due to the ease of implementation. Our constructions of quadratic constraints are also general in the sense that our GroupSort quadratic constraint also holds for the class of all sum-preserving activations. We now clearly state this in introduction of the revised version of the paper.

Reviewer comment (Experiments):
How does the new formaulation compare to the original LipSDP for MaxMin networks in terms of computational efficiency/runtime?

Author response:
Thank you for this question. To investigate this question, we used the description of the MaxMin acitvation using a residual ReLU network (4), as shown in our motivating example in Section 3. For this equivalent formulation we then implemented an SDP for the residual ReLU equivalent form of MaxMin (the multi-layer variants of (5) under either $\ell_2$ or $\ell_\infty$ setting) using quadratic constraints for slope-restricted activations to compute Lipschitz bounds. We denote the SDP on the residual ReLU formulation by LipSDP-RR. We added our results in the paper and for convenience also state the new results in the table below, where we compare the bounds and the computation times. We observe that our LipSDP-NSR provides better bounds in most cases, with two exceptions for 2-layer networks under the $\ell_\infty$ setting, i.e. $\ell_\infty$ Lipschitz bounds for 2FC-64 and 2FC-128. For 5- and 8-layer networks, our LipSDP-NSR is always better than LipSDP-RR, and there even is a significant gap.  Notice that LipSDP-NSR outperforms LipSDP-RR in all cases under the $\ell_2$ setting. This is consistent with the intuition provided by our motivating example in Section 3 of our paper. Specifically, our motivating example in Section 3 implies that  the quadratic constraints derived from slope-restricted properties of ReLU have severe fundamental limitations in addressing the $\ell_2$ case of MaxMin Lipschitz analysis, i.e. it cannot recover the basic 1-Lipschitz property of the MaxMin activation under the $\ell_2$ setting. Our new quadratic constraints derived for MaxMin fix that, and our numerical results support our claim. In the $\ell_\infty$ setting, the output is always a scalar and the 1-Lipschitzness of MaxMin activation is not explicitly needed for SDP-based analysis. Hence our motivating example does not indicate much for the $\ell_\infty$ setting. However, our LipSDP-NSR still outperforms LipSDP-RR in most cases under the $\ell_\infty$ setting. Finally, we want to comment on the computation efficiency. We want to point out that advantages in computational efficiency of the MaxMin quadratic constraint formulation become apparent only for larger networks. It is interesting to notice that for small neural networks, LipSDP-RR can sometimes be faster, and this can be explained by the different implementations of the two problems. The computation time depends on the number of decision variables and on the conditioning of the problem. While the number of decision variables are the same in both problems, the multiple constraints for LipSDP-NSR are much smaller than the one constraint in LipSDP-RR. For networks larger than 8FC-64 we even run into memory problems when solving the SDP for the residual ReLU network formulation.

| Architecture  | L-l2 (Time) NSR | L-l2 (Time) RR  | L-linfty (Time) NSR | L-linfty (Time) RR | 
| ------------- |:------------:| ------------:|:------------:| -----------:|
| 2FC-16        | 22.59 (54)   | 23.92 (18)   | 203 (94)     | 207.3 (34)  |
| 2FC-32        | 38.55 (86)   | 40.28 (24)   | 368.7 (128)  | 404.8 (44)  |
| 2FC-64        | 73.77 (160)  | 74.31 (23)   | 885.9 (147)  | 870.8 (38)  |
| 2FC-128       | 70.86 (182)  | 72.18 (55)   | 1058.8 (236) | 1033.1 (77) |
| 5FC-32        | 224.6 (53)   | 388.0 (69)   | 2969.5 (96)  | 4283.1 (137)|
| 5FC-64        | 386.8 (111)  | 623.4 (228)  | 6379.8 (190) | 8321.3 (416)|
| 8FC-32        | 401.0 (58)   | 1078.5 (178) | 4303.9 (120) | 8131.1 (374)|
| 8FC-64        | 722.3 (116)  | 1683.7 (1426)| 9632.3 (188) | 1.886E+04 (2495)|