Paper: SAS: Structured Activation Sparsification (ICLR-2024)

Reviewer comment (Theory):
It is inaccurate for the author to state that the computation of SWS and SAS are the same. I can agree that the computation of SAS and the original dense network are the same, but the computation of SWS is obviously higher than that of SAS.

Author response:
We really appreciate the insightful comments by *N3vE*.
Especially, *N3vE*'s comments helped us to clarify the potential inaccuracy and confusion about the experimental configuration. 
We consider that the following first three concerns kindly raised by *N3vE* come from essentially the same source of inaccuracies in the original manuscript regarding the comparison configuration of dense, SAS, and SWS: compare accuracy by increasing network width without increasing $\texttt{mult}$  count or comparing wall-clock time by changing sparse pattern while keeping the same matrix size.
We revised the manuscript as follows, and we are now confident that the revised manuscript and the following answer clarify the concerns.

>It is inaccurate for the author to state that the computation of SWS and SAS are the same. I can agree that the computation of SAS and the original dense network are the same, but the computation of SWS is obviously higher than that of SAS.

>The author doesn't express the dimensions of the weights corresponding to SWS and SAS. 

>Although it intuitively looks like SAS has a clear acceleration in the graph, I think this is unreasonable because the author also says the number of $\texttt{mult}$ count in SAS is the same as the base dense/narrow network.

In this paper, we propose SAS to increase the network capacity to improve the accuracy without increasing the actual $\texttt{mult}$ count.
Therefore, we compare the SWS in the same scenario (keeping the $\texttt{mult}$ count constant while increasing the sparsity).
Specifically, we consider a base layer consisting of $\texttt{matmul}$ between activation $\mathbf{X}$ and weight $\mathbf{W}$, where $\mathbf{X}\in \mathbb{R}^{\bar{C}_i\times {HW}}$ and $\mathbf{W}\in \mathbb{R}^{ {C}_o \times\bar{C}_i}$ (eq. (1)).
As *N3vE* explains, using the proposed SAS could increase the network width for $M$ times while maintaining the same $\texttt{mult}$ count as the base layer by utilizing the 1:$M$ sparsity pattern.
The sparsified matrix shape is, $\mathbf{X}\in \mathbb{R}^{M\bar{C}_i\times {HW}}$ and $\mathbf{W}\in {C}_o\times \mathbb{R}^{M\bar{C}_i}$ (eq. (2)).
SAS does not change the output channel dimension ${C}_o$. 

On the other hand, in the case of SWS, if one increases the network width for the $M$ times and uses the 1:$M$ sparse pattern on weight, the $\texttt{mult}$ count of the resultant network increases by about $M\times M/M=M$ because both input and output channel is $M$ times wider.
We consider *N3vE*’ kindly point out this issue.

In the case of SWS, by using $\sqrt{M}$ times wider network for the 1:$M$ sparsity pattern, we can construct the SWS network, which has roughly the same $\texttt{mult}$ count as the base dense network and SAS network (section 2.3).
Thanks to the questions raised by *N3vE*, we find the original manuscript was unclear from three perspectives: 1) the explanation in the original manuscript "$\sqrt{M}$ times wider input/output channel" was insufficient and needs more detail, 2) Although we adopt the opposite benchmarking configuration for the speed benchmarking in fig. 3 (we keep the same matrix dimension while changing the sparse pattern), we failed to explain this difference clearly, which may confuse the reader about the interpretation of the speed benchmarking result (and also the accuracy benchmarking result), 3) The reason why we evaluate the speed using the  $\texttt{matmul}$  of the same shaped matrix was not clearly stated.
We've updated the manuscript to clarify the above three issues as follows;

###  Issue 1) Configuration of SWS network
The original manuscript lacks the following detailed discussion.
When the network is $\sqrt{M}$ times (instead of $M$ times) wider with 1:$M$ sparsity pattern in weight, the $\texttt{mult}$ count will be the same as the base network.
But, the network width needs to be an integer value, and it also needs to be multiple of $M$.
Therefore, we use the weight having the shape of  $\lfloor\sqrt{M}\bar{C}_i^{(l)} \rceil\times \lfloor\sqrt{M}{C}_o^{(l)} \rceil$ for the 1:$M$ SWS network to get approximately the same $\texttt{mult}$ count as the base network, where $\lfloor\cdot\rceil$ is a rounding operator,  $\bar{C}_i^{(l)}$, ${C}_o^{(l)}$ is the input/output channel dimension of the base dense network. 
For the last chunk, when it does not equal to $M$, we use $(\lfloor\sqrt{M}\bar{C}_i^{(l)} \rceil  \lfloor\sqrt{M}{C}_o^{(l)} \rceil \mod M)$:1 sparse pattern.
In this way, we construct the SWS network having almost the same $\texttt{mult}$ count as possible.

Reviewer comment (Presentation):
Let's continue considering this point. Figure 3 raises a big question for me. The author doesn't express the dimensions of the weights corresponding to SWS and SAS. Although it intuitively looks like SAS has a clear acceleration in the graph, I think this is unreasonable because the author also says the number of mult count in SAS is the same as the base dense/narrow network. This greatly reduces my enthusiasm for this paper.

Author response:
Let's consider a specific example, 
the $l$-th layer of the base network consisting of convolution with $\bar{C}_i$ = 288 ($C_i$ is 32, kernel size $k$ is 3) and $C_o$ = 64.
When $M$=8, then  We have $\lfloor\sqrt{M}\bar{C}_i^{(l)} \rceil$=815 and $\lfloor\sqrt{M}{C}_o^{(l)} \rceil$=181.
Then $\texttt{mult}$ count of the original dense layer (for single pixel) is 288$\times$64=18432, the $\texttt{mult}$ count of the weight sparse layer (SWS) is 18440 (815$\times$181/8=18439 with modulo 3, we use M=3 for the last chunk).
The comparison of the layer shape in this example case is summarized as follows:

|    |  input $\bar{C}_i\rightarrow \tilde{C}_i$ |  output ${C}_o$ |  FLOPS | weight shape  | activation shape  |\\\
| -------- | ------- |------- |------- |------- |------- |------- |
| base (dense)    | 288 | 64  | 18432  | 64$\times$288 |  288$\times$HW |
 |SAS ($M$=8)   | 288 $\rightarrow$ 2304 | 64  | 18432 | 64$\times$2304  |  2304$\times$HW (1:8 sparse)|
| SWS ($M$=8)   | 815 | 181  | 18440 | 181$\times$815 (1:8 sparse) |  815$\times$HW|

### Issue 2 Configuration for speed benchmarking
On the wall-clock speed benchmarking reported in fig. 3, we use the opposite configuration, changing the sparsity pattern while keeping the matrix dimension unchanged to report more concrete comparisons.
The original manuscript did not emphasize this significant configuration difference, leading to a misunderstanding about the interpretation of the results. 
Furthermore, symbols used to describe the matrix shape ($\mathrm{M}, \mathrm{N}$) were confusing with the ones used for specifying the sparsity pattern ($N, M$).  

In the wall-clock speed benchmarking in fig. 3, we consider the $\texttt{mult}$  $\mathbf{W}\mathbf{X}$ where $\mathbf{X}\in \mathbb{R}^{\mathrm{\gamma} \times \mathrm{\alpha}}$ and $\mathbf{W}\in \mathbb{R}^{\mathrm{\beta}\times \mathrm{\gamma}}$  **which is identical for dense $\texttt{mult}$, SAS $\texttt{mult}$, and SWS $\texttt{mult}$**.
The $\texttt{mult}$ count of the three variants is the same $(\mathrm{\gamma}\times \mathrm{\alpha}\times \mathrm{\beta})$ when one does not take the sparsity into account.
By utilizing the 1:2 fine-grained (semi-structured) sparsity on *Sparse Tensor Core*, the $\texttt{mult}$count becomes half for SAS and SWS, i.e., $(\mathrm{\gamma}\times \mathrm{\alpha}\times \mathrm{\beta})/2$.
We show the number of dimensions of the weight and activation for wall-clock time benchmarking in the graph (we use fixed $\mathrm{\gamma}$=10240, changing $\mathrm{\alpha}=\mathrm{\beta}$ from 10240 to 20480), which is the same for dense $\texttt{matmul}$ SAS $\texttt{matmul}$, and SWS $\texttt{matmul}$.

Again, we want to emphasize that the scenario in this speed benchmarking (keep the same matrix dimension) is different from the scenario for neural network (keep the (almost) same $\texttt{mult}$ count). 
The statement "the number of $\texttt{mult}$ count in SAS is the same as the base dense/narrow network" is not meant for this speed evaluation experiment but for our main application scenario of the SAS for neural networks (section 3, section 4).

### Issue 3 Reason for the speed benchmarking configuration
As discussed above, we adopt the different configurations to evaluate the speed in a fair setting between SWS and SAS.
We can construct an SWS network having approximately the same  $\texttt{mult}$ count with base dense network and SAS network by adjusting the network width (as discussed in issue 1); however, it is not precisely the same, and the shape of the matrices will be different (please refer to the table in issue-1), which makes it hard to evaluate the overhead specific to SAS which we are interested in (index related operation).
While we can precisely evaluate this by measuring the time of the $\operatorname{matmul}$ of the same-sized matrix (section 2.3, supp. B). 

To clarify the above point, we've made the following changes.
* Added the details about constructing the SWS network with the similar $\texttt{mult}$ count as discussed above (section 2.3). We've also clarified in the experimental section (section 3, section 4) that the $\texttt{mult}$ count of the SWS network is not exactly the same as the dense or SAS network, but it is approximately the same.
* Moved the speed benchmarking to a separate paragraph and clarified the distinction between the two benchmark setups as discussed above (section 2.3).
* Changed the symbols  of the matrix size from $\mathrm{K}, \mathrm{M}, \mathrm{N}$ to $\mathrm{\gamma}, \mathrm{\alpha}, \mathrm{\beta}$ to avoid the notational clutter with the sparsity pattern $M, N$ (section 2.3, fig. 3).

Reviewer comment (Writing):
The terminology of structured sparsity is misleading. We typically refer the N:M sparsity as "semi structured sparsity" to distinguish it from standard structured sparsity including disjoint group sparsity, overlapping group sparsity and hierarchical sparsity.

Author response:
Thank you for the suggestion.
We agree with *W2qU* that the term "semi-structured" well describes the $N$:$M$ sparsity.
We used the term "Fine-grained structured" sparsity following [NVidia's official whitepaper](https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf) and use structured afterward for brevity (section 1).
Thanks to the *W2qU*'s comments, we find the original manuscript was unclear on this point.
We clarified the terminology in section 1 and added the note that this is also called "semi-structured" to give a clear distinction from the standard structured sparsity.

Reviewer comment (Evaluation):
The realistic benefits of structured sparse activation is not clear. Although the topic is interesting, I am not sure what is the actual speedup gain of such sparse activation that can deliver to the community. The paper seems equipping without numerical results regarding speedup as well.

Author response:
We report the wall-clock speed benchmarking in fig. 3, demonstrating that SAS-based multiplication can speed up the $\texttt{matmul}$  on actual hardware.
The overhead of SAS over SWS is the online computation of the index (eq. 3), reorder (supp. E), and the memory transfer of the index.
Our experiment shows that it is only a faction (1.5\%) of the entire  $\texttt{matmul}$ in terms of the wall-clock time.
However, we've evaluated the time for a single  $\texttt{matmul}$ (with varying sizes), and we have not benchmarked the speed as a whole neural network.
There are two reasons why we evaluate the wall-clock time using  $\texttt{matmul}$ of the same-sized matrix instead of the entire network.
1.  It is hard to exactly match the $\operatorname{mult}$ count between SWS and SAS network, and the matrix shape differs largely between the two, making the evaluation of SAS-specific computation (online index computation) time difficult. In contrast, we can clearly evaluate this by measuring the time of the $\operatorname{matmul}$ of the same-sized matrix (section 2.3, supp. B). This discussion also relates to the *N3vE*'s question; please refer to the response for the specific example of why this is the case. 
1. Due to the limited high-level API to utilize the *SparseTensorCore* (e.g., *cuSPARSELt*). We need to develop a custom CUDA kernel using a low-level library (e.g., [Cutlass](https://github.com/NVIDIA/cutlass) to deal with the specific $\texttt{mult}$ operation such as convolution or depth-wise convolution to evaluate the wall clock time as a whole network.

We are now working on the development of the custom CUDA kernel and will publish the source code.
We expect slightly better results (in terms of the percentage of SAS-specific overhead) than the result in fig. 3 will be achieved with the custom CUDA kernel (section 2.3, section 6.1) because we can integrate index computation, reorder, and memory copy.

Thanks to the *W2qU*'s comment, we find that the original manuscript failed to discuss this important motivation about the wall-clock benchmarking.
We've added the discussion in section 2.3 and section 6.1.

Reviewer comment (Evaluation):
As mentioned in Section 2.3, while the computational load in terms of FLOPS remains the same for a given level of activation sparsity M, the memory requirements for the Sparse Activation Sparsification (SAS) network increase linearly with M. This is in contrast to the Sparse Weight Storage (SWS) network, which maintains a constant storage requirement for weights at inference time regardless of M. This could become a significant limitation for devices with limited memory or when scaling to very large networks.

Author response:
As *C5D8* points out, in contrast to SWS, the memory requirement of SAS increases linearly with $M$ (section 2.3). 
Actually, this is the main drawback of the proposed method  (section 2.3, section 4.2).
We expect this could be mitigated by using quantized weight (section 5), which we left for future work.
From a practical perspective, commercially available hardware (at the time of submission) supports 50\% sparsity; doubling the memory might be a practical choice, especially when the FLOPS is critical, considering the significant accuracy gain from M=0 (dense) to $M$=2 (50\% sparse).
We clarified this point in (section 2.3 and section 5).

Reviewer comment (Theory):
In Section 3, the paper discusses the expressive power of SAS networks by comparing trajectory lengths in a specific constructed neural network with 2-dimensional input and output. However, this analysis might not generalize to all network architectures or datasets, which could limit the understanding of the practical implications of SAS's increased expressive power.

Author response:
We agree with *C5D8* about the interpretation of the trajectory length analysis.
Still, we consider the analysis gives important implications about the expressive power of the model, which does not depend on data, model, or several training details that affect the accuracy (It often happens, for example, model A performs better than B when using a specific optimizer, learning rate, and learning epochs; but model B beats model A when adopting different training scheme).
The trajectory length analysis does not depend on these settings.
In section 4, we present results using a practical model and dataset, and we hypothesize the increased accuracy by adopting the SAS is attributed to the increased expressive power, which conforms to the trajectory length analysis results.
Thanks to the *C5D8*'s comments, we find the above motivation for utilizing the trajectory length analysis was missing in the original manuscript (we only mentioned "more generic settings"); we've added the above discussion at the beginning of the trajectory length analysis (section 3).

Reviewer comment (Presentation):
In general, the listings (code pieces) in the paper is informative enough. However, I would suggest the authors to replace the source code with some high-level pseudo code. This is more readable and more accessible to some readers.

Author response:
We present the source code for inference to demonstrate that the SAS operation is realizable on actual commercial hardware (listing-1).
We also show the source code for training to demonstrate the conversion from the existing dense layer to the SAS layer for training is simple and trivial (listing-2).
We agree with *C5D8*'s suggestion for the importance of high-level explanation.
Following the suggestion, we further clarified the method during the training, and now we consider listing-2 to be somewhat redundant; therefore, we moved it to supplement for interested readers (and added more important discussion regarding the valuable comments from all the reviewers). 
Regarding listing-1 (Code for inference on actual hardware), we keep it in the main text to showcase that the SAS could actually be compatible with commercial hardware.
However, thanks to the *C5D8*'s comments, we noticed some details could be more precise to follow; therefore, we've clarified the high-level explanation in sections 2.1-2.3 and supp. E along with the visual description.