Paper: Batch normalization is sufficient for universal function approximation in CNNs (ICLR-2024)

Reviewer comment (Writing):
In the introduction, I'm a bit confused about the normalization being explored in this paper. The title mentions batch normalization, but the statement "we delve into the role of layer normalization" suggests layer normalization. Is this a typo?

Author response:
Indeed, we have used layer normalization when we meant normalization layers. We have revised the manuscript accordingly.

Reviewer comment (Theory):
I'm puzzled by the assertion in Section 2 regarding the existence of $f_t$, which the author claims is due to the universal function approximation. Perhaps the author meant to refer to the Universal Approximation Theorem. However, such approximations in neural networks are typically conditional. Has the author considered these conditions? Generally, these conditions are not mild and are idealized. Does this affect the theory presented in this paper? The author should provide an explanation and discussion on this.

Author response:
Universal Approximation Theorems often have mild assumptions on the activation functions (e.g. that they are monotonously increasing and continuous) and the smoothness of the data generating function (usually that it is continuous) [1].
Depending on the considered architecture (e.g. a three layer neural network or a width-constrained perceptron with ReLUs, the approximation quality of the neural network is derived dependent on the number variable network parameters (e.g. the width of the 3-layer neural network or the depth of the width constrained perceptron) [2].

    * By formulating theorems with respect to a given target network that solves a task of interest, we are flexible in which universal function approximation theorem could be invoked. Our results would inherit the specific assumptions.
    * Also irrespective of the specific universal function approximation theorem, we can cover typical application cases, for which target networks have been obtained by training the networks in a classical way. We have clarified this argument on page 3.

Reviewer comment (Theory):
I'm not entirely sure why throughout the paper, normalization is reduced to just a linear transformation and shift, i.e., $\gamma \mathbf{h} + \beta$. This includes batch norm, layer norm, instance norm, etc. They all have this form, so is their significance solely in the learnable parameters $\gamma$ and $\beta$? Of course, I understand that $[(x - \mu) / \sigma] \times \gamma + \beta$ can be equivalent to $\gamma \mathbf{h} + \beta$" in the end, but why emphasize batch normalization in the title? Where does "batch" come into play?

Author response:
We sincerely thank the reviewer for their response and motivate below why the analysis of $\gamma \mathbf{h} + \beta$ is sufficient for our purpose. To understand this, we need to carefully distinguish between the role of normalization parameters for the trainability and the expressiveness of a neural network.

**Trainability**
Normalization influences several aspects of the training dynamics. For instance, an important role is that it ensures that signal can propagate through neurons at least for some samples even when the network changes strongly due to high learning rates. To achieve this, the more intricate standardization operation is relevant. Effectively, it ensures reasonable values of $gamma$ and $\beta$ in $\gamma \mathbf{h} + \beta$, when the network is not well trained so that we do not have a lot of information how good values of $\gamma$ and $\beta$ could look like. Yet, the mean $\mu$ and $\sigma$ do not influence the actual expressiveness of the network, as they could just be integrated into $\gamma$ and $beta$.

**Expressiveness**
In this work, we focus on the contribution of the normalization to the expressiveness of the network. We thus derive the values of the effective normalization parameters that would allow a network with random weights to approximate a target network. We could also separately define the mean $\mu$ and standard deviation $\sigma$ based on training data batches and the target network and adjust $\gamma$ and $\beta$ accordingly, but this would not change the principle and make the exposition more complicated without any need. 

Actually, we find the fact intriguing that affine linear transformations on the neuron level are already sufficient to achieve highly expressive neural network models. In fact, if we have linear activation functions, only the parameters $\gamma$ in the intermediary layer would be enough, as they can implement linear combinations of the features that are randomly transformed by the network weights and biases. The bias term $\beta$ in the intermediary source layer is primarily needed to shift the input into a linear region of the activation function. The bias $\beta$ of the second layer assumes the role of the original target bias (and compensates for shifts in the intermediary layer). 

Does this explanation address the source of the reviewer's puzzlement? We would be happy to further elaborate on unclear aspects.

Reviewer comment (Theory):
The work describes BN as a subset of a Layer Normalization (LN). They are both normalization layers, but BN is NOT a type of LN. This issue is severe enough that the paper needs to be rewritten based on this.

Author response:
* According the reviewers suggestions, we have revised the manuscript and replaced our misnomer layer normalization by normalization or normalization layer.

Reviewer comment (Presentation):
References are broken, with issues such as "Theorem ??" appearing in the paper.

Author response:
* We have also addressed the broken Theorem link in the experiment section.

Reviewer comment (Experiments):
The paper's experiments are very small scale for modern deep learning, leaving the reader wondering if they will scale.

Author response:
* One of our main contributions is that we have derived the precise scaling requirements. While our experiments primarily serve the validation of our theoretical claims, note that our insights have allowed us to achieve much better performance than previous empirical work [1]. 
    - However, as we discuss on page 7 in Section 3.1 and on page 8, we have to acknowledge that the precision of numerical solvers is limited in solving large scale linear systems of equations. Note that the matrix $M$ in our experiments has already a dimension of up to $90000 \times 90000$ roughly.
    - In fact, our theoretical results imply that training only BN parameters for computational savings is not a practically reasonable approach if we want to maintain high expressiveness. Only inductive bias in the weight distributions (e.g. as obtained by foundation models) could enable training success despite a low number of degrees of freedom (see Lemma 2.1).

Reviewer comment (Evaluation):
Unclear practical relevance.

Author response:
We thank the reviewer for the time and effort they put into their review. To address their concern, in the following, we highlight the practical relevance and theoretical significance of our contributions.

**Practical relevance:**

- The practical relevance of our work is based on insights that can guide the design of neural network architectures, in which BN parameters contribute significantly to the expressiveness of the neural networks (like in WideResNets).

- Concretely, we have derived architectural constraints and make specific proposals how training only BN paramters can achieve the same performance as training a target network. 
This way, we could outperform previous experiments with architectures, where only BN parameters were trained [1].

- We have proven rigorously that a higher depth can compensate for a too narrow width. This is a relevant insight, in particular, since the random networks that were analyzed empirically were not wide enough.

- Our results imply that training only BN parameters for computational savings is not practically relevant if we want to maintain high expressiveness. Only inductive bias in the weight distributions (e.g. as obtained by foundation models) could enable training success despite a low number of degrees of freedom (see Lemma 2.1).
Note that this insight was not well supported even for fully-connected networks because not all trainable parameters were actually utilized in the construction [2].

Reviewer comment (Novelty):
The contribution is marginal compared to known results about approximation with models with random fixed parameters.

Author response:
**Significance of theoretical contributions:**

- Convolutional layers are an important building block of many contemporary neural network architectures. Therefore, we have to understand their intricacies if we want to gain theoretical insights that are of practical relevance and, up to our knowledge, we are the first to do so in the context of training only training only normalization parameters. 
(Note that our insights actually allow us to outperform previous purely empirical work on that matter and explain their findings.) 
- Convolutional architectures are challenging to analyze theoretically in our context, as the composition of multiple filters induces non-trivial equations that are quadratic (and not linear) in the trainable BN parameters. 
Furthermore, the composition of convolutional source filters has to overlap with a target filter, which induces additional constraints that do not occur in fully-connected layers.
- Even in comparison to previous work on fully-connected layers [2], we present an improved bound on the source width, which utilizes all trainable parameters (which requires solving a quadratic instead of a linear system of equations).
Note that this improved bound is crucial for an important insight into the limitations of the overall approach. 
If we want to maintain full expressiveness, we cannot use fewer parameters.
This insight is not fully supported by previous work [2], which does not utilize all trainable parameters.
- Furthermore, we cover almost arbitrary activation functions and not only linear ones or ReLUs.
- Conceptually, most challenging and novel is the proof of Theorem 2.5 that solves the depth versus width trade-off in the source network.