Paper: DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS (ICLR-2024)

Reviewer comment (Presentation):
The stddev in table 2 are a bit odd to me, either a higher numerical precision is needed, or it needs to be explained why the are many cases of zero std.

Author response:
Indeed most of the 0.00 standard deviations were simply because the results in the table were rounded to 2 decimal places. We have amended Table 2 to include higher precision to make the results clearer. 0.00 standard deviation was only accurate to higher precision as well in the cases when all of the seeds completely fail to train at all. Thank you for this suggestion.

Reviewer comment (Presentation):
Results on ST are only shown in the appendix. They should be shown and discussed in the main paper, given that they are an essential part in the rest of the manuscript.

Author response:
Thank you for this suggestion. Putting $ReLU_\tau$, $ST_\tau$, and $CST_{\tau, m}$ experimental results in the Appendix was originally done purely due to space constraints, but we acknowledge and agree that it is important for all key experimental results to appear in the main paper. In order to make space, we simplified and combined Figures 2 and 3, and so we have now included all the important experimental results in Table 2.

Reviewer comment (Reproducibility):
No source Code provided, implementation details on experiments only in the appendix.

Author response:
The implementation details were left to the appendix due to space constraints. We ask that this please not be considered a weakness of our paper.  We are working to make the source code available and will alert you if this is possible before the discussion period concludes.

Reviewer comment (Evaluation):
There could have been a study of the computational efficiency since that is the main motivation of the work.

Author response:
Unfortunately it was not possible for us to perform an empirical study of the computational efficiency gains due to sparser activations, because the sparse operations necessary to leverage sparse activations are not yet well supported on the accelerator hardware on which we are running our experiments. However, better support  for efficient sparse operations is a high priority and ongoing research area for both deep learning hardware and software developers, given the potential performance gains. The  number of flops in each matrix-vector multiplication $AB$ for $A \in \mathbb{R}^{m \times n}$, $B \in \mathbb{R}^{n \times d}$  in the forward pass could in theory shrink from $\mathcal{O}(mnd)$ to $\mathcal{O}(msd)$.

Reviewer comment (Novelty):
Is this the first time that the variance map equations are being derived for these non-linearities?

Author response:
Yes, to the best of our knowledge this is the first time they have been derived.

Reviewer comment (Presentation):
Plots of Figure 2 are missing axis labels and the plot legends are not readable on printed paper.

Author response:
Thank you for highlighting this. We have added axis labels and enlarged the legend font.

Reviewer comment (Theory):
What is the criterion on $V_\phi$ for successful training? For example, let's see Figure 2. Which of the shapes are good and expected to train well vs which are the ones that are expected to fail? I see that for $\tau=1$ the curves have higher curvature. In particular, the blue curve intersects the line $x=y$ at one point where the derivative is non-zero but the curvature is positive. Is this expected to fail because the derivative is non-zero? I found the explanations in the text somehow repetitive and hard to parse. Can the authors explain the criterion on $V_\phi$ in words just from Figure 2?

Author response:
According to the theory, two things are required for stable training: 

1. we require that $\chi_1 = 1$, which guards against vanishing and exploding gradients as well as instability to input perturbations at initialisation.

2. we require that the variance map $V(q)$ is sufficiently stable around $q^*$ such that $q^l$ does in practice converge to and remain at $q^*$. This is a necessary requirement for the EOC initialisation in practice since $q^*$ is used in the calculation of the values of $\sigma_w$ and $\sigma_b$ in order to ensure $\chi_1 = 1$. If in practice $q^l$ does not stably converge to  $q^*$, and instead grows larger, then we no longer have $\chi_1=1$ and experience the associated training failure modes of exploding or vanishing gradients.

The problem we have shown with $ReLU_\tau$ and $ST_\tau$ is precisely that it is impossible for them to satisfy both criteria $\chi_1=1$ and $q^l$ converging stably to $q^*$. 

In light of the above, we agree that the original Figures 2 and 3 could be misleading, since they included variance maps for $ReLU_\tau$ and $ST_\tau$ which did not correspond to EOC initialisation. To address your question and to improve clarity on this point we have combined Figures 2 and 3 into a single figure, now showing only the variance maps for both activation functions corresponding to $\chi_1=1$. We have also edited Section 2 of the paper to make this point more clearly (in particular see paragraphs 2, 3, 4, and 5 of Section 2 in the updated version). We think it reads better now, so thank you for this helpful feedback.

Reviewer comment (Experiments):
Can the authors please provide experiments with CReLU $m=0$ as well and also for ReLU in Table 2? Why is there only one row for ReLU? As the table stands now, I am not convinced that sparse activation functions without magnitude clipping fail to train. Is the 'unstable training dynamics' reported in the paper for very large $s$ and small $m$ as claimed in the conclusion?

Author response:
Indeed Table 2 is not intended to provide evidence that $ReLU_\tau$ and $ST_\tau$ fail to train at high sparsities. Due to space constraints, the experiments showing that $ReLU_\tau$ and $ST_\tau$ networks do indeed fail to train at higher sparsities were included only in the appendix in Table 3, top of page 17, in the original version. We fully agree that the pointer to these experimental results was not sufficiently clear (it was made on page 4 at the end of the following sentence: ``To make matters worse, ...  and consequently prove effectively impossible to train for large sparsity; see App. C."). Having made some additional space available by merging Figures 2 and 3 as described above, we have now included *all* experimental results in Table 2. For ease of reference, these are the experimental results for $ReLU_\tau$ and $ST_\tau$:

|            | $s$  | $\tau$ | $m$ | $V'_\phi$ at q* | $V''_\phi$ at q*  | DNN accuracy mean | DNN accuracy std   | DNN sparsity mean | DNN sparsity std   |   CNN accuracy   |  CNN sparsity     |
|------------|------|--------|-----|----------------|-----------------|------|-------|------|-------|------|------|
| $ReLU_\tau$ | 0.50 | 0.00   | N/A | 1.0            | 0.0             | 0.94 | 0.002 | 0.50 | 0.001 | 0.70 | 0.52 |
|            | 0.60 | 0.25   | N/A | 1.0            | 0.12            | 0.76 | 0.37  | 0.49 | 0.27  | 0.68 | 0.6  |
|            | 0.70 | 0.52   | N/A | 1.0            | 0.3             | 0.10 | 0.00  | 0.00 | 0.00  | 0.1  | 0.0  |
| $ST_\tau$      | 0.5  | 0.67   | N/A | 1.0            | 0.43            | 0.10 | 0.00  | 0.00 | 0.00  | 0.1  | 0.0  |
|            | 0.6  | 0.84   | N/A | 1.0            | 0.59            | 0.10 | 0.00  | 0.00 | 0.00  | 0.1  | 0.0  |
|            | 0.7  | 1.04   | N/A | 1.0            | 0.81            | 0.10 | 0.00  | 0.00 | 0.00  | 0.1  | 0.0  |



The results clearly show the failure of the $ReLU_\tau$ networks to train once sparsity reaches 70\%, and the failure of $ST_\tau$ at even 50\% sparsity. 

While $CReLU_{\tau,m}$ with $m=0$ will be just the 0 function for any $\tau$, and so would not work as a candidate activation function; potentially we are misunderstanding the request "experiments with CReLU m = 0."

Reviewer comment (Writing):
I do not understand the heuristics given in Section 3.1 for how to choose $m$. I understand the dependence of $V'(q^*)$ and $V''(q^*)$ on the magnitude value $m$ is non-trivial from Figure 4. Still, the curves follow regular shapes so maybe it is possible to give simple heuristics based on Figure 4?

Author response:
As you say, the dependence of the shape of $V_\phi$ on $s$ and $m$ is non-trivial, and we are hesitant to suggest a simple and clean heuristic, since as far as we know none can be derived analytically. Our experiments in Table to 2 suggest that a decent starting point for fully connected networks would be to maximise $m$ for a given sparsity subject to ensuring that $V''(q^*) \lessapprox 0.2$, while allowing $V''(q^*) \lessapprox 0.4$ appears to work in the CNN case.