Paper: Role of Locality and Weight Sharing in Image-Based Tasks: A Sample Complexity Separation between CNNs, LCNs, and FCNs (ICLR-2024)

Reviewer comment (Theory):
Although the authors consider optimization in the upper bound analysis for LCNs and CNNs, the chosen algorithm, which has just two training iterations, seems quite unconventional and perhaps too simplistic. I'm uncertain whether this presents an improvement relative to the ERM analysis.

Author response:
*3. Benefits of gradient descent analysis over ERM*

We acknowledge the reviewer's observation regarding the simplicity of our gradient descent analysis on a one-hidden layer neural network with two iterations. However, our choice of gradient descent over an ERM-style argument was deliberate because of two key theoretical reasons:

(1) The gradient descent demonstrates a sample complexity separation between FCNs, LCNs, and CNNs for computationally-efficient (polynomial time) equivariant algorithms. This distinction is crucial because while a separation may exist for computationally inefficient algorithms, the separation might disappear under constraints of computational efficiency. Given that models in practice are trained using efficient algorithms, we believe establishing a separation for efficient algorithms is theoretically important.

(2) It is important to ensure that both the upper and lower bounds are derived for equivariant algorithms. This is because having lower bounds for equivariant algorithms and upper bounds for non-equivariant ones does not lead to a valid sample complexity separation as non-equivariant algorithms could potentially be more sample-efficient than their equivariant counterparts. Since, the equivariance of ERM is not clearly established and Wang et al. do not discuss this aspect in their work, we believe a gradient descent analysis is more appropriate.

Reviewer comment (Novelty):
Regarding the lower bound section, if I interpret it correctly, does the novelty lie mostly in the technical novelty? It appears that the primary objective is to determine the size of the enlarged function class, when the original task is enlarged by the global/local rotation group. I question the significance of the specific method used for this calculation.

Author response:
To address the above challenge, we developed a novel reduction technique that leveraged input-independent randomization and equivariance to show that both FCNs and LCNs must learn each patch independently from every other patch. This technique not only aids in computation, but also provides a theoretical justification for the intuition for the '$k$ cost' associated with learning each patch independently. Despite the fact that our reduced problem is relatively simpler, it still does not enjoy the semi-metricness on the $l_2$ loss function. To overcome this issue, we introduce a novel variant of Fano's Lemma for input-independent randomized algorithms. This allows to derive the lower bounds without requiring semi-metricness to hold across the entire space.

We respectfully disagree with the reviewer's perspective on the significance of our methods. First, our reduction technique through randomization alongwith the tailored variant of Fano's Lemma have broad applicability, and we believe it should be of independent mathematical interest. Additionally, given the widespread use of Gaussian Mixture Models (GMMs), our techniques could offer valuable tools for researchers in this area. Lastly, our techniques could serve as a foundation for future extensions of our work, such as incorporating the aforementioned interaction effects.

Reviewer comment (Novelty):
It would be beneficial if the paper could elaborate more on the contributions of Wang & Wu (2023) and comparisons with Wang & Wu (2023). For instance, when stating on page 2, "Wang & Wu (2023) extended this line of work to show a separation between FCNs, LCNs and CNNs. However, that work suffer from the same drawbacks as Li et al. (2021).", a more explicit discussion on the advancements and remaining problems in Wang & Wu (2023) would be useful.

Author response:
We sincerely thank the reviewer for their quick response and we would definitely include a more detailed comparison over Wang & Wu and Li et al.

Reviewer comment (Theory):
Regarding the model definitions, what is the reason for limiting the norm of $w_i$ to less than one? This seems implausible.

Author response:
**Comment 3: Why is the norm of $\mathbf{w}_i \le 1$?**

Answer:
We imposed the constraint $\mathbf{w}_i \le 1$ primarily to simplify our analysis. We use it to prove the technical requirement of our Fano's Lemma which is that the semimetric property should hold when two out of three functions belong to the set of 'hard instances'. We particularly selected the constant $1$ to enable the filters to capture the unit norm signal $\mathbf{w}^\star$. While this choice might appear somewhat artificial, it does not impact the main takeways of our theory.

Reviewer comment (Theory):
Concerning the proof outline for theorem 6.1, in its first step, could there be an inclusion of a more comprehensive technical elaboration? While some readers might be able to grasp its logic intuitively, offering a detailed technical justification, especially given its novel approach, would be beneficial.

Author response:
**Comment 4: Can the proof outline for step 1 of theorem 6.1 be elaborated more technically?**

Answer:
We appreciate the reviewer's request and we will definitely include a more technical explanation in our revised draft. We now outline step 1 of the proof, focusing on the key aspects while avoiding overly intricate details for better understanding.

We denote the DSD distrbution by $P$ and its marginal for the $i^{\text{th}}$ patch as $Q_i$. We denote the mean of the positive marginal of each $Q_i$ as $\mathbf{\mu}_i$. The proof begins by noting that the risk of the algorithm, $\bar{\theta}_n$, on $\mathbf{U} \circ P$ is given by the average of the risk of $\bar{\theta}_n$ over each $\mathbf{U} \circ Q_i$. We then use equivariance to show that the risk of $\bar{\theta}_n$ over each $\mathbf{U} \circ Q_i$ is exactly the same,

$$\mathbb{E}\_{S^n \sim (\mathbf{U} \circ P)^n} \left[
        R\left(
            \bar{\theta}\_n,
            \mathbf{U} \circ Q\_i
        \right)
    \right]
    =
    \mathbb{E}\_{S^n \sim (\mathbf{U} \circ P)^n}
    \left[
        R\left(
            \bar{\theta}\_n,
            \mathbf{U} \circ Q\_j
        \right)
    \right].
$$

This implies that even though we are "training" the model using $\mathbf{U} \circ P$, it is enough to "test" the model on $\mathbf{U} \circ Q_1$. Since this holds for all $\mathbf{U}$ and $\bar{\theta}_n$,

$$\mathbb{E}\_{S^n \sim P^n} \left[
        R\left(
            \bar{\theta}\_n,
            P
        \right)
    \right]
    \ge
    \inf\_{\bar{\theta}\_n}
    \sup\_{\mathbf{U}}
    \mathbb{E}\_{S^n}
    \left[
        R\left(
            \bar{\theta}\_n,
            \mathbf{U} \circ Q\_1
        \right)
    \right].
$$

We denote $R(
            \bar{\theta}_n,
            \mathbf{U} \circ Q_1
        ) = R_1(
            \bar{\theta}_n
        )$ for brevity. We now begin a series of reductions, first observe that the minimax risk can only decrease if the algorithm was instead supplied with patch separated input data $\{S^{n_i} \sim \mathbf{U} \circ Q_i\}$. This is because the algorithm can simply ignore the patch-id, which is strictly extra information. Please note that with high probability $n_i \approx n/k$,   

$$
    \ge
    \inf\_{\bar{\theta}\_n}
    \sup\_{\mathbf{U}}
    \mathbb{E}\_{\{S^{n\_i}\}}
    [
        R_1(
            \bar{\theta}\_n
        )
    ].
$$

Next, we provide the algorithm with $S^{n_1}$ and all the required quantities so that it can internally construct $S^{n_2},..S^{n_k}$. It is easy to see that this can only reduce the minimax risk. These quantities include the means $\mathbf{U}\mathbf{\mu}_2, .., \mathbf{U}\mathbf{\mu}_k$ and presampled gaussian noise vector, label vector, and $n_2,..,n_k$. We refer to the presampled quantities by $\xi$.

$$
    \ge
    \inf\_{\bar{\theta}\_n}
    \sup\_{\mathbf{U}}
    \mathbb{E}\_{\{S^{n\_1}, \xi\}}
    [
        R\_1(
            \bar{\theta}\_n(\{\mathbf{U}\mathbf{\mu}_2, .., \mathbf{U}\mathbf{\mu}_k\})
        )
    ].
$$

The final step is to note that since $\mathbf{U}\mathbf{\mu}_1, .., \mathbf{U}\mathbf{\mu}_k$ are mutually orthogonal, the only information about $\mathbf{U}\mathbf{\mu}_1$ we get from $\mathbf{U}\mathbf{\mu}_1, .., \mathbf{U}\mathbf{\mu}_k$ is that the former lies in a subspace of size $kd-k+1 \approx kd$. This information is too little and can effectively be ignored,

$$
    \ge
    \inf\_{\bar{\theta}\_n}
    \sup\_{\mathbf{U}}
    \mathbb{E}\_{S^{n_1}}
    [
        R\_1(
            \bar{\theta}\_n
        )
    ].
$$

We have now reduced learning $\mathbf{U} \circ P$ with $n$ samples to learning $\mathbf{U} \circ Q_1$ with $n/k$ samples. This marks the end of Step 1. In step 2, we show that $n/k \approx \sigma^2kd$, which proves the result.

Reviewer comment (Experiments):
The only downside I see to this work is the fact that there is no empirical result which supports the theoretic result. I myself cannot find any issue or criticism with the theory that was presented, but the complexity and abstractness of the setting and the numerous variables involved makes it difficult to find any error in the derivation if one exists.

Author response:
**Comment 1: Can experiments be added that validate the analysis?**

Answer:
We completely agree with the reviewer's suggestion that empirical validation would strengthen the credibility of our theoretical analysis. We request the reviewer to refer to the newly added supplementary material, which includes the detailed experimental protocol and results. We also plan to incorporate them in the forthcoming revision.

Reviewer comment (Experiments):
For example, for a fixed amount of $\sigma$ and $d$, each linear model can be trained on an increasing number of classes to show the empirical observation of the $k$ term. Similarly, this can also be shown with a fixed $\sigma$ and $k$. This is just the first example of what came to mind, but the authors may indeed be able to come up with an even better experiment given their study of the problem.

Author response:
**Comment 1: Can experiments be added that validate the analysis?**

Answer:
We completely agree with the reviewer's suggestion that empirical validation would strengthen the credibility of our theoretical analysis. We request the reviewer to refer to the newly added supplementary material, which includes the detailed experimental protocol and results. We also plan to incorporate them in the forthcoming revision.

Reviewer comment (Presentation):
After the conclusion, there is a QED symbol which is likely a typo and should have appeared after the previous section.

Author response:
**Comment 3: Misplaced QED symbol after the conclusion**

Answer:
We thank the reviewer for pointing out the typo, and we will fix in the next revision.

Reviewer comment (Theory):
I am curious if transformers can be described by FCN or LCN, or if they need a totally different treatment. I think this would be an interesting discussion point to add if there is any insight because the sample complexity for transformers is a relevant and interesting topic.

Author response:
**Comment 2: Can our framework also provide bounds for transformers, given their prevalance in machine learning?**

Answer:
Regarding the potential application of our framework to transformers, we can consider a transformer architecture where each input patch corresponds to an input token. If we assume that the keys $W_K$, queries $W_Q$, values $W_V$ matrices are initialized using standard isotropic gaussian distribution, then this transformer, trained with SGD, is equivariant to the application of the same $U \in \mathcal{O}(d)$ to each patch of the input. Ignoring the issues of semi-metricness, this leads to a $\Omega(\sigma^2d)$ lower bound. The upper bound, however, may be a little challenging as it requires gradient descent analysis for the softmax-based attention mechanism. Nonetheless, we fully agree with the reviewer that this is an important direction for future work.

Reviewer comment (Novelty):
The Local Signal Adaptivity (LSA) activation function is quite different compared to popular used activation function, like ReLU or more recent activation function. A better justification of using LSA could be provided.

Author response:
We note that the LSA activation function can be written as the sum of two ReLU activation functions as $\phi_b(x) = \text{ReLU}(x-b) - \text{ReLU}(-x-b)$. This decomposition allows us to adapt our framework for ReLU through the following architectural modifications:
1. We augment an additional hidden node for each patch in LCNs and CNNs and additional $k$ hidden nodes for FNNs.
2. We include a bias term for each patch.
3. We fix the second layer to $\{-1,1,..,-1,1\} \in \mathbb{R}^{2d}$

The idea is that the two hidden nodes corresponding to each patch learn $\mathbf{w}^\star$ and $-\mathbf{w}^\star$ respectively. Together with the bias and the fixed $\pm 1$ values from the second layer, we can simulate the thresholding effect of the LSA activation function. 

We also note that the LSA activation function, also known as the "soft-thresholding function", is extensively used in high-dimensional sparse recovery problems (Section 18.2 [1]). Since our task involves recovering the sparse mean vector $\mathbf{w}^\star$, it justifies our use of the LSA activation function.

Reviewer comment (Theory):
Can the current framework support ReLU like activation functions?

Author response:
We note that the LSA activation function can be written as the sum of two ReLU activation functions as $\phi_b(x) = \text{ReLU}(x-b) - \text{ReLU}(-x-b)$. This decomposition allows us to adapt our framework for ReLU through the following architectural modifications:
1. We augment an additional hidden node for each patch in LCNs and CNNs and additional $k$ hidden nodes for FNNs.
2. We include a bias term for each patch.
3. We fix the second layer to $\{-1,1,..,-1,1\} \in \mathbb{R}^{2d}$

The idea is that the two hidden nodes corresponding to each patch learn $\mathbf{w}^\star$ and $-\mathbf{w}^\star$ respectively. Together with the bias and the fixed $\pm 1$ values from the second layer, we can simulate the thresholding effect of the LSA activation function. 

We also note that the LSA activation function, also known as the "soft-thresholding function", is extensively used in high-dimensional sparse recovery problems (Section 18.2 [1]). Since our task involves recovering the sparse mean vector $\mathbf{w}^\star$, it justifies our use of the LSA activation function.