Paper: Expressivity of ReLU-Networks under Convex Relaxations (ICLR-2024)

Reviewer comment (Theory):
The main weakness is the restricted univariate assumption. All the positive results for the expressive power of convex relaxation methods hold for univariate functions, which is restricted in two ways: (1) in practice, almost all functions of interest are multivariate, and (2) in theory, the case of univariate functions is too special, which often avoids the general difficulty in high dimensions and is thereby hard to generalize.

Author response:
**Q1: Can you discuss the relevance of your results in the heavily restricted univariate domain? What are the implications of your negative results for the $\Delta$-relaxation in the multivariate setting?**  

First, we want to clarify that the $\Delta$-relaxation is strictly more precise than the DeepPoly and IBP relaxations. Thus, our negative result for the $\Delta$-relaxation in the multivariate setting directly implies negative results for the other, less precise relaxations (DeepPoly-0, DeepPoly-1, and IBP), even for extremely simple functions. We believe this result to be of great importance as the majority of neural network verifiers are upper bounded in their analysis precision by the so-called “convex relaxation barrier”, realized by the $\Delta$-relaxation [1]. Only a few, very recent methods use multi-neuron-relaxations (at great computational cost) to break this barrier [2,3]. This highlights that even extremely simple functions can not be encoded as ReLU networks such that these simpler verifiers can analyze them precisely and agrees well with the fact that the latter methods have recently dominated the international neural network verification competition [4].

However, as even simple multivariate functions can not be encoded such that their analysis is precise using any single neuron domain $D$ ($D$ is IBP, DeepPoly-0, DeepPoly-1 or $\Delta$), we can not separate them with respect to their expressivity in this setting. To do so and investigate if the more complex domains improve expressivity, we study the univariate case, and indeed show positive results and separation between methods, both in terms of expressivity and size of resulting solution spaces.

Reviewer comment (Experiments):
The focus is exclusively on ReLU networks, not covering other activation functions commonly used like sigmoid or tanh.

Author response:
**Q1: Can you discuss why you focus exclusively on ReLU networks instead of considering sigmoid or tanh activations?**  
We decided to focus on ReLU networks for two key reasons. 
Firstly, they are much more widespread and thus relevant in the certified robustness community, e.g., the international verification of neural networks competition last year considered ReLU activations in all 12/12 benchmark classes, sigmoids in only one, and tanh in none at all [1].
Secondly, finite sigmoid and tanh networks can not exactly encode (to the best of our knowledge) any commonly used function class, making the question of exact analysis less relevant and interesting. Given the limited space and already “dense and mathematical” writing of our work, we decided to focus on the most relevant class of networks.

Reviewer comment (Experiments):
Only fully-connected feedforward networks are considered, not convolutional or residual architectures widely used in practice.

Author response:
**Q2: You seem to only consider fully connected feedforward networks. Can you discuss the extensibility of your results to other architectures?**  
We already consider arbitrary ReLU networks with any architecture including arbitrary skip connections, which include both CNNs and ResNets (see the first paragraph in Section 2 and Theorem 17). We have made this more clear.

Reviewer comment (Theory):
The analysis is limited to deterministic networks, not touching on stochastic networks.

Author response:
**Q3: Why do you only consider deterministic neural networks and not stochastic networks?**  
As our work is motivated by (deterministic) neural network verification (where all state-of-the-art methods are based on convex relaxations), we focus on the general type of network typically considered in the field, i.e., deterministic networks. In particular, we are not aware of any such method considering stochastic networks. Perhaps the reviewer can point to a specific class of stochastic networks that would be relevant in this setting.

Reviewer comment (Experiments):
Only standardized datasets and perturbation sets are studied; results may not generalize to other domains.

Author response:
**Q4: You only seem to study standardized datasets and perturbation sets. Can you discuss to what extent your results generalize beyond that?**  
As we do not study any specific data(set) anywhere in the paper but argue about general function classes, our results will generalize to any applicable dataset.
For univariate functions, we show a positive result for the general case of continuous bounded perturbation sets, subsuming all $\ell_p$-norm bounded perturbations, typically studied in the field. In the multivariate case, we show a negative result even for the more restrictive (but very commonly used) $\ell_\infty$-norm bounded perturbations. That is, even extremely simple functions can not be expressed for box input regions. This entails that these functions can also not be expressed for more general classes of perturbation sets. We note that Theorem 19 and thus 20 can be trivially extended to other $\ell_p$-norms and have added a corresponding Theorem (Thm. 24) to Appendix A.

Reviewer comment (Experiments):
While hypotheses are provided for certified training, no experiments are conducted to validate the conjectured benefits.

Author response:
**Q5: While hypotheses are provided for certified training, no experiments are conducted to validate the conjectured benefits.**  
First, we want to highlight that our general results agree well with multiple recent empirical works as discussed in “Implications of our Results for Certified Training” (Section 1). 
While validating our conjecture of more precise relaxations being beneficial for certified training is an interesting item for future work, it is beyond the scope of this theoretical work, as no current certified training method supports more precise relaxations, as current approaches are non-differentiable.

Reviewer comment (Writing):
The writing is quite dense and mathematical, which could make it less accessible to a general AI audience.

Author response:
**Q6: The writing seems to be quite dense and mathematical - how accessible is it to a general AI audience?**  
As work on certified adversarial robustness [2,3,4], in general, and closely related prior theoretical work [5,6], specifically, is frequently published at general AI venues (including ICLR), we believe our work to also be accessible to the audience of these venues, given that multiple reviewers highlighted the good presentation.

Reviewer comment (Novelty):
Aside from certified training, implications for other applications of neural network analysis are not discussed much.

Author response:
**Q7: Aside from certified training, can you discuss the implications of your results for other applications of neural network analysis?**  
Generally, convex relaxations of neural networks are most frequently used in certified training and neural network certification, which are both very active research areas. Beyond the implications for certified training, which we discuss in detail in Section 1, our results directly imply the need for precise certification methods to obtain exact analysis results even for trivial multi-variate functions.

Reviewer comment (Theory):
Can the authors elaborate on how convex relaxations are helping with formal robustness guarantees? This was quickly glossed over, but I would hope to get a better explanation.

Author response:
We thank the reviewer for their candid admission that they are not an expert in the field and are happy to provide a significantly extended version of the background, which we believe addresses all the reviewer's questions. We have incorporated some of these extensions (including a better intuition for Definition 3) into Section 2 in an effort to make our work more accessible.  

## Extended Background  
**Problem Setting**  
For clarity of exposition, we focus on the most common case of adversarially robust classification. 
We call a classifier $h: \mathcal{X} \to \mathcal{Y}$ locally robust around some input $x$ if it predicts the same, correct class for all similar inputs $\mathcal{B}^\epsilon_p(x) := \\{ x' \in \mathcal{X} \mid \| x - x' \|_p \leq \epsilon \\}$. Thus, to prove that a classifier is locally robust, we have to show  $\forall x' \in \mathcal{B},  h(x') = h(x) = y$.
For a neural network that predicts the class $ h(x) \:= argmax_i f(x)_i $, we can equivalently show that the logit of the target class is always greater than that of all other classes, by solving the following optimization problem:  

$0 < \min_{x' \in \mathcal{B},  i \neq y}  f(x')_y - f(x')_i $  

**Convex Relaxations for Efficient Solution**  
Unfortunately, solving this problem exactly is NP-complete [1] as the neural network $f$ is highly non-linear and non-convex. Thus, state-of-the-art neural network verifiers use convex relaxations (combined with a branch-and-bound approach), to relax the above non-convex optimization problem to a (convex) linearly constrained linear program which can be solved much more efficiently [2]. More concretely, the non-linear activation layers (ReLUs), which can be seen as non-linear constraints in the original problem, are replaced with a set of linear constraints (the element-wise inequalities) in their input-output space. Intuitively, we relax the non-convex graph of the ReLU function with the convex over-approximations illustrated in Figures 1-4. However, as these convex-relaxations introduce an (over)-approximation error, these relaxed problems might be too imprecise to prove robustness even when the network is robust. In this work, we study for which classes of functions there exist ReLU networks such that this analysis does not lose precision depending on the relaxation method used.

**Relevance of a $D$-Analysis**  
Solving the above optimization problem can geometrically be interpreted as linearly projecting the graph of the network (set of input-output tuples) for inputs in $\mathcal{B}$ to the 1d-line $f_y - f_i$ corresponding to the output difference for every $i \neq y$. While this projection is computationally efficient, computing the exact graph is still NP-complete. Thus, we can instead use convex relaxations to compute and then project (often in one step) an over-approximation of this graph and then check whether all points in this over-approximation are classified correctly. We call this over-approximate graph the $D$-analysis depending on the convex relaxation $D$ that was used. As the analysis with relaxation $D$ is precise if and only if the 1d projection of the exact graph and the $D$-analysis agree, this is a very natural and intuitive perspective on the problem.

We can now reformulate our above question as “For which function classes exist ReLU networks such that the 1d projection of their $D$-analysis (over-approximation of the graph) is precise, i.e., agrees with the projection of the exact graph?”. For “expressivity under convex relaxation” we thus do not only require that a given function can be encoded by a network (the default notion of expressivity and already shown for CPWL functions and ReLU networks by prior work [3]) but additionally require the analysis of the resulting network using the $D$ relaxation to be precise. Thus, $D$-expressivity depends on both the relaxation $D$ and the function class $F$.

For a more in-depth explanation of (certified) adversarial robustness including examples and visualizations, we recommend the excellent blog post by Eric Wong: https://locuslab.github.io/2019-03-12-provable/

**Notation**  
All inequalities between vectors are element-wise. We have added a corresponding note to the Notation paragraph. Thanks for the pointer.

We are looking forward to the reviewer’s reply and are happy to provide further explanations or references and answer any follow-up questions.

**References:**  
[1] Katz et al. "Reluplex: An efficient SMT solver for verifying deep neural networks." CAV 2017  
[2] Salman, et al. "A convex relaxation barrier to tight robustness verification of neural networks." NeurIPS 2019  
[3] He, et al. "ReLU deep neural networks and linear finite elements." JCM 2020

Reviewer comment (Theory):
Can the authors explain exactly what is being relaxed into a convex function? Section 2.1 reads quite confusingly for me, as all I can discern are inequalities, and I do not see which parts are being relaxed.

Author response:
We thank the reviewer for their candid admission that they are not an expert in the field and are happy to provide a significantly extended version of the background, which we believe addresses all the reviewer's questions. We have incorporated some of these extensions (including a better intuition for Definition 3) into Section 2 in an effort to make our work more accessible.  

## Extended Background  
**Problem Setting**  
For clarity of exposition, we focus on the most common case of adversarially robust classification. 
We call a classifier $h: \mathcal{X} \to \mathcal{Y}$ locally robust around some input $x$ if it predicts the same, correct class for all similar inputs $\mathcal{B}^\epsilon_p(x) := \\{ x' \in \mathcal{X} \mid \| x - x' \|_p \leq \epsilon \\}$. Thus, to prove that a classifier is locally robust, we have to show  $\forall x' \in \mathcal{B},  h(x') = h(x) = y$.
For a neural network that predicts the class $ h(x) \:= argmax_i f(x)_i $, we can equivalently show that the logit of the target class is always greater than that of all other classes, by solving the following optimization problem:  

$0 < \min_{x' \in \mathcal{B},  i \neq y}  f(x')_y - f(x')_i $  

**Convex Relaxations for Efficient Solution**  
Unfortunately, solving this problem exactly is NP-complete [1] as the neural network $f$ is highly non-linear and non-convex. Thus, state-of-the-art neural network verifiers use convex relaxations (combined with a branch-and-bound approach), to relax the above non-convex optimization problem to a (convex) linearly constrained linear program which can be solved much more efficiently [2]. More concretely, the non-linear activation layers (ReLUs), which can be seen as non-linear constraints in the original problem, are replaced with a set of linear constraints (the element-wise inequalities) in their input-output space. Intuitively, we relax the non-convex graph of the ReLU function with the convex over-approximations illustrated in Figures 1-4. However, as these convex-relaxations introduce an (over)-approximation error, these relaxed problems might be too imprecise to prove robustness even when the network is robust. In this work, we study for which classes of functions there exist ReLU networks such that this analysis does not lose precision depending on the relaxation method used.

**Relevance of a $D$-Analysis**  
Solving the above optimization problem can geometrically be interpreted as linearly projecting the graph of the network (set of input-output tuples) for inputs in $\mathcal{B}$ to the 1d-line $f_y - f_i$ corresponding to the output difference for every $i \neq y$. While this projection is computationally efficient, computing the exact graph is still NP-complete. Thus, we can instead use convex relaxations to compute and then project (often in one step) an over-approximation of this graph and then check whether all points in this over-approximation are classified correctly. We call this over-approximate graph the $D$-analysis depending on the convex relaxation $D$ that was used. As the analysis with relaxation $D$ is precise if and only if the 1d projection of the exact graph and the $D$-analysis agree, this is a very natural and intuitive perspective on the problem.

We can now reformulate our above question as “For which function classes exist ReLU networks such that the 1d projection of their $D$-analysis (over-approximation of the graph) is precise, i.e., agrees with the projection of the exact graph?”. For “expressivity under convex relaxation” we thus do not only require that a given function can be encoded by a network (the default notion of expressivity and already shown for CPWL functions and ReLU networks by prior work [3]) but additionally require the analysis of the resulting network using the $D$ relaxation to be precise. Thus, $D$-expressivity depends on both the relaxation $D$ and the function class $F$.

For a more in-depth explanation of (certified) adversarial robustness including examples and visualizations, we recommend the excellent blog post by Eric Wong: https://locuslab.github.io/2019-03-12-provable/

**Notation**  
All inequalities between vectors are element-wise. We have added a corresponding note to the Notation paragraph. Thanks for the pointer.

We are looking forward to the reviewer’s reply and are happy to provide further explanations or references and answer any follow-up questions.

**References:**  
[1] Katz et al. "Reluplex: An efficient SMT solver for verifying deep neural networks." CAV 2017  
[2] Salman, et al. "A convex relaxation barrier to tight robustness verification of neural networks." NeurIPS 2019  
[3] He, et al. "ReLU deep neural networks and linear finite elements." JCM 2020

Reviewer comment (Writing):
When the authors write a vector is less than or equal to another vector, is this inequality entrywise?

Author response:
**Notation**  
All inequalities between vectors are element-wise. We have added a corresponding note to the Notation paragraph. Thanks for the pointer.

Reviewer comment (Theory):
There are a lot of definitions in section 2.2, can the authors provide a simpler explanation of these definitions and what they are trying to capture? In particular, why is it that we care about $D$-analysis in the definition of expressivity? I would have expected expressivity of a network architecture to be about function approximation.

Author response:
We thank the reviewer for their candid admission that they are not an expert in the field and are happy to provide a significantly extended version of the background, which we believe addresses all the reviewer's questions. We have incorporated some of these extensions (including a better intuition for Definition 3) into Section 2 in an effort to make our work more accessible.  

## Extended Background  
**Problem Setting**  
For clarity of exposition, we focus on the most common case of adversarially robust classification. 
We call a classifier $h: \mathcal{X} \to \mathcal{Y}$ locally robust around some input $x$ if it predicts the same, correct class for all similar inputs $\mathcal{B}^\epsilon_p(x) := \\{ x' \in \mathcal{X} \mid \| x - x' \|_p \leq \epsilon \\}$. Thus, to prove that a classifier is locally robust, we have to show  $\forall x' \in \mathcal{B},  h(x') = h(x) = y$.
For a neural network that predicts the class $ h(x) \:= argmax_i f(x)_i $, we can equivalently show that the logit of the target class is always greater than that of all other classes, by solving the following optimization problem:  

$0 < \min_{x' \in \mathcal{B},  i \neq y}  f(x')_y - f(x')_i $  

**Convex Relaxations for Efficient Solution**  
Unfortunately, solving this problem exactly is NP-complete [1] as the neural network $f$ is highly non-linear and non-convex. Thus, state-of-the-art neural network verifiers use convex relaxations (combined with a branch-and-bound approach), to relax the above non-convex optimization problem to a (convex) linearly constrained linear program which can be solved much more efficiently [2]. More concretely, the non-linear activation layers (ReLUs), which can be seen as non-linear constraints in the original problem, are replaced with a set of linear constraints (the element-wise inequalities) in their input-output space. Intuitively, we relax the non-convex graph of the ReLU function with the convex over-approximations illustrated in Figures 1-4. However, as these convex-relaxations introduce an (over)-approximation error, these relaxed problems might be too imprecise to prove robustness even when the network is robust. In this work, we study for which classes of functions there exist ReLU networks such that this analysis does not lose precision depending on the relaxation method used.

**Relevance of a $D$-Analysis**  
Solving the above optimization problem can geometrically be interpreted as linearly projecting the graph of the network (set of input-output tuples) for inputs in $\mathcal{B}$ to the 1d-line $f_y - f_i$ corresponding to the output difference for every $i \neq y$. While this projection is computationally efficient, computing the exact graph is still NP-complete. Thus, we can instead use convex relaxations to compute and then project (often in one step) an over-approximation of this graph and then check whether all points in this over-approximation are classified correctly. We call this over-approximate graph the $D$-analysis depending on the convex relaxation $D$ that was used. As the analysis with relaxation $D$ is precise if and only if the 1d projection of the exact graph and the $D$-analysis agree, this is a very natural and intuitive perspective on the problem.

We can now reformulate our above question as “For which function classes exist ReLU networks such that the 1d projection of their $D$-analysis (over-approximation of the graph) is precise, i.e., agrees with the projection of the exact graph?”. For “expressivity under convex relaxation” we thus do not only require that a given function can be encoded by a network (the default notion of expressivity and already shown for CPWL functions and ReLU networks by prior work [3]) but additionally require the analysis of the resulting network using the $D$ relaxation to be precise. Thus, $D$-expressivity depends on both the relaxation $D$ and the function class $F$.

For a more in-depth explanation of (certified) adversarial robustness including examples and visualizations, we recommend the excellent blog post by Eric Wong: https://locuslab.github.io/2019-03-12-provable/

**Notation**  
All inequalities between vectors are element-wise. We have added a corresponding note to the Notation paragraph. Thanks for the pointer.

We are looking forward to the reviewer’s reply and are happy to provide further explanations or references and answer any follow-up questions.

**References:**  
[1] Katz et al. "Reluplex: An efficient SMT solver for verifying deep neural networks." CAV 2017  
[2] Salman, et al. "A convex relaxation barrier to tight robustness verification of neural networks." NeurIPS 2019  
[3] He, et al. "ReLU deep neural networks and linear finite elements." JCM 2020

Reviewer comment (Novelty):
At the end, it seems like the authors are investigating whether or not ReLU networks can express certain functions. I thought this was already an answered question with respect to universal approximation, but perhaps I am missing something. Can the authors explain why we need to analyze the expressivity results for this certain class of functions?

Author response:
We thank the reviewer for their prompt reply and are happy to hear that we were able to address all their questions and that they now believe our results to be interesting. Regarding the follow-up questions:

**Q1: Can ReLU networks express the function $f(x,y) = \max(x,y)$ precisely?**  
Yes, ReLU networks can express (we call this ``encode’’ in the non-robust setting as per Definition 2) the $\max$ function.

In fact, He et al. [1] show that all (multivariate) continuous piecewise linear (CPWL) functions and thus all function classes we consider (including the $max$ function) can be expressed precisely by ReLU networks (first paragraph of the related work, Section 3).

The max function can even be expressed precisely using just a single neuron (see the proof to Theorem 19 in Appendix A):
Consider the network $ h(x,y) = y + \text{ReLU}(x-y) $, then $h$ is equal to $\max$ for all $x, y \in \mathbb{R}$: 
* Consider the case $x \geq y$, then $\text{ReLU}(x-y) = x-y$ as $x - y \geq 0$. Hence $h(x,y) = y + \text{ReLU}(x-y) = y + x - y = x = \max(x,y)$. 
* Consider the case $x < y$, then $\text{ReLU}(x - y) = 0$ as $x - y < 0$. Hence $h(x,y) = y + \text{ReLU}(x-y) = y + 0 = y = \max(x,y)$
* Hence, we have $h(x,y) = \max(x,y)$. 

While other constructions are possible as well, we first show that the above yields the most precise $\Delta$-abstraction of *all* ReLU networks (Theorem 17 and 18), and still it can not precisely $\Delta$-express the max function (Theorem 19).

To summarize, we only consider function classes that can be encoded precisely by ReLU networks and then show that, in the multivariate setting, even under severe restrictions (monotonicity and convexity of the function) they can not be $\Delta$-expressed, i.e., their analysis with the most precise single-neuron relaxation is imprecise for *any* ReLU network. Thus, we show that this is a limitation of convex relaxations.

**Q2: What are the takeaways for a practitioner aiming to train a (certifiably) adversarially robust network?**  
Great question: Current certified training methods are based on the imprecise IBP relaxation [2,3] as the more precise relaxations lead to harder optimization problems that current optimization strategies can not solve efficiently [4]. So far, it was an open question whether these more precise relaxations would yield any advantages if we could address the optimization issues. Our results now suggest that (at least in the univariate case), more precise relaxations allow larger function classes to be $D$-expressed precisely and lead to larger solution spaces for the same function class. Thus, our results directly suggest that working on these optimization issues is a promising path forward to improve certified training and obtain networks with higher certified robust accuracy.
We provide a high-level discussion of this question in the last paragraph of the first Section ``Implications of our Results for Certified Training’’. 

**References**  
[1] He, et al. "ReLU deep neural networks and linear finite elements." JCM 2020   
[2] Müller et al. Certified Training: Small Boxes are All You Need, ICLR 2023   
[3] Shi et al. Fast Certified Robust Training with Short Warmup, NeurIPS 2021