Paper: Minimum width for universal approximation using ReLU networks on compact domain (ICLR-2024)

Reviewer comment (Theory):
Can the authors please add more details and rigor in the construction in Lemma 6's proof. I know a lot of it is drawn from another paper's lemmata but it would be better to have it explicit and self contained. Right now it is not even a proof but a proof sketch/recipe.

Author response:
We thank the reviewer for pointing this out. Following the reviewer’s comment, we have added details to the proof of Lemma 6 in the revised draft (pages 14-16 in Appendix B.4).

Reviewer comment (Theory):
Fix a compact subset \(X\) of \(\mathbb{R}^{d_X}\), non-empty. Suppose that we are looking for universal approximators in \(C(X,\mathbb{R}^{d_Y})\) implementable by MLPs with ReLU-Like activation function and of bounded widths. How do you expect that the metric entropy/capacity of \(X\) will impact the minimum width? For instance, if \(X=\{x_0\}\) is a singleton and \(d_Y=1\), then clearly the width \(\min\{d_X,d_Y,2\}\) is suboptimal since the class
\[
\{ x\mapsto a\operatorname{ReLU}(1\cdot (x+b)):\, a\in \mathbb{R} ,\, b:= -x_0 + 1\}
\]
is universal in \(C(X,\mathbb{R})\). So I guess there is room for improvement for general \(X\). (The same question applies to the case where \(d_X=0\) and \(d_Y=1\), in which case the minimum width is
\[
1 < \max\{d_X,d_Y,2\}=\max\{0,1,2\}=2.
\])

Author response:
We are not sure if we correctly understand the definition of the metric entropy that you were considering, but we try to answer with the following definition, which appears in the covering argument in statistics: $\log N(X,\delta)$ where $N(X,\delta)$ denotes the minimum number of $\delta$-balls that can cover $X$ (i.e., the logarithm of the covering number). We consider $L^p$ approximation in this answer. If this problem setup is not what the reviewer was considering, please let us know.

As the reviewer pointed out, any map from a single point (of the zero metric entropy) to a real number can be exactly represented by a ReLU network of width one. We think this observation extends to any domain of zero Lebesgue measure (including any finite sets) since we can ignore such sets in $L^p$ approximation; ignoring them incurs zero $L^p$ error. Under this observation, one interesting question we think is that “what if our error use a different measure, other than the Lebesgue one?” For example, the Cantor set $\mathcal C$ has zero Lebesgue measure but can have non-zero $d$-dimensional Hausdorff measure for $d<\log(2)/\log(3)$. In such a case, for universal approximation, we expect that ReLU networks of width one are insufficient since they can only represent monotone functions. Here, since the value $\log(2)/\log(3)$ is the Minkowski dimension of the Cantor set defined as $d_M(\mathcal C):=\lim_{\delta\to0^+} (\log N(\mathcal C,\delta))/(\log(1/\delta))$, one may find a connection between the metric entropy and the minimum width for universal approximation under a Hausdorff measure. We believe investigating the universal approximation property of neural networks under various measures is an interesting future research direction.

Reviewer comment (Theory):
What's your intuition on how the metric entropy of \(X\) appears into the estimate? I thought about the case where \(X=\{-1,1\}\) but minimal width seems to apply there also. What am I missing?

Author response:
We are not sure if we correctly understand the definition of the metric entropy that you were considering, but we try to answer with the following definition, which appears in the covering argument in statistics: $\log N(X,\delta)$ where $N(X,\delta)$ denotes the minimum number of $\delta$-balls that can cover $X$ (i.e., the logarithm of the covering number). We consider $L^p$ approximation in this answer. If this problem setup is not what the reviewer was considering, please let us know.

As the reviewer pointed out, any map from a single point (of the zero metric entropy) to a real number can be exactly represented by a ReLU network of width one. We think this observation extends to any domain of zero Lebesgue measure (including any finite sets) since we can ignore such sets in $L^p$ approximation; ignoring them incurs zero $L^p$ error. Under this observation, one interesting question we think is that “what if our error use a different measure, other than the Lebesgue one?” For example, the Cantor set $\mathcal C$ has zero Lebesgue measure but can have non-zero $d$-dimensional Hausdorff measure for $d<\log(2)/\log(3)$. In such a case, for universal approximation, we expect that ReLU networks of width one are insufficient since they can only represent monotone functions. Here, since the value $\log(2)/\log(3)$ is the Minkowski dimension of the Cantor set defined as $d_M(\mathcal C):=\lim_{\delta\to0^+} (\log N(\mathcal C,\delta))/(\log(1/\delta))$, one may find a connection between the metric entropy and the minimum width for universal approximation under a Hausdorff measure. We believe investigating the universal approximation property of neural networks under various measures is an interesting future research direction.

Reviewer comment (Theory):
Perhaps I missed it, but it could be worth noting that your results also imply the minimal widths for universality/density in \(C(\mathbb{R}^{d_X},\mathbb{R}^{d_Y})\) in the topology of uniform convergence on compact sets. This is because of the extension and normalization arguments as in the proof of Proposition 3.10 [1] or in the proof of Proposition 53 [2], which allows one to reduce the problem of universality in \(C([0,1]^{d_X},\mathbb{R}^{d_Y})\). I.e. using either of the Tietze or McShane extension theorems.

Author response:
We thank the reviewer for this suggestion. As the reviewer pointed out, all of our results hold for an arbitrary compact domain in the Euclidean space. Using the extension lemmas provided by the reviewer, we have made this point clearer in the revised draft (page 5 in Section 3).

Reviewer comment (Theory):
In [5], the authors just recently showed that most MLPs with standard and continuous activation functions can approximately implement an MLP with ReLU activation function using roughly the same depth, width, and number of parameters. I was wondering, unless I am missing something, why not use their results to sharpen your statement for general continuous activation functions?

Author response:
We appreciate the reviewer for this suggestion. [Zhang et al., 2023] approximate ReLU networks using a network using a class of activation functions by scaling the width and depth of a network with multiplicative factors 3 and 2. Applying this result to our Theorem 1 gives us an upper bound $3\max\\{d_x,d_y,2\\}$ on the minimum width. However, we found that this bound exceeds the known upper bound $\max\\{d_x+2,d_y+1\\}$ in Theorem 4 by Park et al. (2021).

On the other hand, our results easily extend to other network architectures: recurrent neural networks (RNNs) and bidirectional recurrent neural networks (BRNNs). For these networks, we consider universally approximating the space of $L^p$ functions that maps a length $T$ sequence of $d_x$-dimensional vectors to a length $T$ sequence of $d_y$-dimensional vectors, denoted by $L^p([0,1]^{d_x\times T},\mathbb R^{d_y\times T})$. Since the $t$-th output of RNN is always a function of the first to $t$-th inputs, for RNNs, we consider universally approximating the space of such functions, called *past-dependent* $L^p([0,1]^{d_x\times T},\mathbb R^{d_y\times T})$, while we consider $L^p([0,1]^{d_x\times T},\mathbb R^{d_y\times T})$ for BRNNs.

We recently showed that the minimum width of RNNs to be dense in past-dependent $L^p([0,1]^{d_x\times T},\mathbb R^{d_y\times T})$ is exactly $\max\\{d_x,d_y,2\\}$ if the activation function is one of ReLU, SOFTPLUS, Leaky-RELU, ELU, CELU, SELU, GELU, SILU, and MISH; here GELU, SILU, and MISH require $d_x+d_y\ge3$. In addition, we also show that the minimum width of BRNNs to be dense in $L^p([0,1]^{d_x\times T},\mathbb R^{d_y\times T})$ is upper bounded by $\max\\{d_x,d_y,2\\}$ if the activation function is ReLU or in ReLU-Like. We have added these results with formal problem setups and proofs in the revised draft (pages 27-34 in Appendix F).

[Zhang et al., 2023] Deep Network Approximation: Beyond ReLU to Diverse Activation Functions, arXiv preprint arXiv:2307.06555, 2023