Paper: Differentiable Learning of Generalized Structured Matrices for Efficient Deep Neural Networks (ICLR-2024)

Reviewer comment (Theory):
Can you please provide the exact form of the $\ell_1$ penalty used in equation 8? How is FLOPs/parameter count being represented as an F$\ell_1$ penalty?

Author response:
**A2:** The FLOPs / parameter count is directly represented by the sum of the width parameters as in Eq. 4 (restated below):
$$
\mathrm{FLOPs}=\sum_{k=1}^K (w_k^R + w_k^C),
$$
which is the $\ell_1$ norm of the vector containing width parameters $\boldsymbol{w}=\{w_1^R, w_1^C,\ldots,w_K^R,w_K^C\}$.
Hence, minimizing the $\ell_1$ norm of this width parameter vector $\boldsymbol{w}$ of a GBLR matrix reduces the FLOPs for matrix-vector product.

Reviewer comment (Reproducibility):
What is the value of $\lambda$ used in experiments? How should it be chosen properly?

Author response:
**A3:** We used $\lambda=0.02-0.04$ for the ViTs, $\lambda=0.1-0.2$ for the MLP-Mixers, and $\lambda=0.001-0.005$ for ResNet-18. The range of the optimal choice for the $\lambda$ hyperparameter depends on the model, learning rate, and dataset. We used a grid search enumerating the search space to find the $\lambda$ hyperparameter in our experiments. Automatically selecting an optimal $\lambda$ from the budget is left as a topic for future work.

Reviewer comment (Writing):
Why does section 3.3 discuss the application on the basis of a two-layer MLP “for ease of understanding”? Wouldn’t an example on a single layer be much simpler? Also, there is a typo in this section: “multi-layer perception” should be “multi-layer perceptron.”

Author response:
**A6:** In order to illustrate our proposed method, we chose a 'multi'-layer perceptron. We opted for a two-layer MLP, as the simplest form of a neural network. Although a single-layer model is much simpler, we thought it might potentially confuse readers as it is essentially a linear model, not a neural network. Furthermore, a single-layer model with a scalar output does not effectively capture our method because, in such a model, the weight is a vector and not a matrix.

Reviewer comment (Writing):
In Algorithm 1, what’s AdamW(.), as well as clip?

Author response:
**Q1:** In Algorithm 1, what’s AdamW(.), as well as clip?   

**A1:** $\mathrm{AdamW}(\cdot )$ stands for the AdamW optimizer [1]. Line 5 in Algorithm 1 ($p \gets p - \eta \cdot \mathrm{AdamW}(\nabla \ell)$) corresponds to the conventional parameter update to minimize the loss $\ell$ by using the AdamW optimizer. 

The function $\mathrm{clip}(\mathbf{w}, 0, n)$ clamps each element of $\mathbf{w}$ to the interval $[0, n]$, which is the domain of the width parameter. We included this operation in Algorithm 1 to ensure that the width parameter stays within $[0,n]$ after the gradient descent. 

We have added a detailed explanation of Algorithm 1 in Section 3.3.

Reviewer comment (Writing):
In Eq. (8), what’s the variable, w or \theta?

Author response:
**Q2:** In Eq. (8), what’s the variable, w or $\theta$?   

**A2:** In Eq. (8), the variable $\theta$ denotes the set of the structural and content parameters, namely, $\theta=(\boldsymbol{\phi}, \boldsymbol{U}, \boldsymbol{V}, \sigma)$, as stated above Eq. (7).
The variable $\boldsymbol{w}$ denotes the width parameters $\boldsymbol{w}=\{w_1^R, w_1^C,\ldots,w_K^R,w_K^C\}$ of $\boldsymbol{W}^{\boldsymbol{\theta}}$. We updated Section 3.3 to clarify the definition of $\boldsymbol{w}$.

Reviewer comment (Novelty):
The advantage of the proposed method against existing methods is not clear.

Author response:
**Q3:** The advantage of the proposed method against existing methods is not clear.   

**A3:** The key advantage of our method lies in its ability to *learn the structural parameters of weight matrices* by stochastic gradient descent, overcoming the non-differentiable nature of typical matrix structures --e.g., non-zero locations of the sparse matrix or the rank of the low-rank matrix. Prior approaches used hand-crafted structures because of this issue. Our approach allows treating structural parameters (widths and locations of the masks in Eq. 2) as additional trainable parameters to automatically identify/learn efficient structures in practice. 
This enables the end-to-end, layer-wise learning of the optimal structured matrix for each weight matrix given the computational budget for each layer.

Most existing rank-based compression methods are confined to either a single type of structured matrix (e.g., low-rank or block low-rank) [2,3] relying on a heuristic rule for budget allocation across layers [3,4], or segregating the structural parameter optimization from the training process of the other parameters [2,4].
This is primarily because, in those prior approaches, the structural formats are defined in a non-differentiable and combinatorial manner.

Using the learned structures and parameters from our approach generally provides higher accuracy for the same complexity (or comparable accuracy with lower complexity) compared to the model in prior works.

Reviewer comment (Experiments):
The experimental results are not convincing. The authors should compare the performance of the proposed algorithm and more methods on more models and datasets.

Author response:
**Q5:** The authors should compare the performance of the proposed algorithm and more methods on more models and datasets. 


**A5:** We have extended our experimental results in two ways:
1. Additional evaluation of a new model (CNN) against a stronger baseline (ALDS [2]),
2. Additional evaluation on a language model using a text generation task on an additional dataset (WikiText-103) using GPT2.
We have included additional discussions and results to the manuscript as summarized below:


**CNN Results.**
We compare the ImageNet accuracy of ResNet18 with GBLR weight matrices to the Automatic Layer-wise Decomposition Selector (ALDS) [2], a similar rank-based layer-wise compression method.
Following [2], we consider the $(d, c, k_1, k_2)$-sized weight tensor of a convolution layer as $d \times (c k_1 k_2)$-sized matrix, where $d, c, k_1, k_2$ are the number of output channels, input channels, and the widths and heights of the kernels, respectively.
In this manner, we converted the weights of the convolution layers of the ResNet18 model to GBLR matrices, and retrained the model. 
Table 1 shows the accuracy and the relative FLOPs of ResNet18 models with dense weights, low-rank weights found by ALDS [2], and GBLR weights found by our method.
Even without any supervision on the structural properties of the convolution kernels, GBLR moderately outperforms ALDS low-rank weights in terms of both accuracy and complexity. 


**Table 1.**  
| ResNet18 Weight Matrix | Accuracy (\%) | Relative FLOPs (\%) |
|------------------------|---------------|---------------------|
| Dense                  |         69.62 |                 100 |
| ALDS [2]               |         69.22 |               56.49 |
| GBLR (ours)            |     **69.31** |           **55.62** |

**Language Model Results.** 
In this experiment, we fine-tuned the pre-trained GPT2 model using the WikiText-103 dataset. Table 2 presents the perplexity (the lower the better) of the GPT2 model with Dense, Low-Rank, Monarch [4], and Gaudi-GBLR matrices. Our proposed method achieves the lowest perplexity, outperforming the dense baseline and utilizing the smallest number of FLOPs. We generated more data points from different relative FLOPs (35\% and 52.5\%) and included them in the manuscript alongside the result presented in Table 2.


**Table 2.**  
| GPT2 Weight Matrix | Perplexity ($\downarrow$) | Relative FLOPs |
|--------------------|--------------------------:|---------------:|
| Dense              |                     19.36 |          100\% |
| Low Rank           |                     19.48 |        43.75\% |
| Monarch [4]        |                     20.56 |        43.75\% |
| Gaudi-GBLR (ours)  |                 **19.24** |     **43.7\%** |