Paper: Gradient-free Proxy for Efficient Language Model Search (ICLR-2024)

Reviewer comment (Novelty):
The novelty may be limited, as the proposed method has not provided a comprehensive and robust architectural design guideline or led to the discovery of an efficient and universally applicable architecture.

Author response:
We conducted additional experiments by applying the zero-shot proxy mentioned in the related work to our search space and discussed the impact of the search space on the experimental results. The experimental results have been added to Tables 2 and 3. Additionally, relevant discussions have been included in Section 6.3.

Reviewer comment (Evaluation):
Tables 2 and 3 demonstrate only modest performance improvements with W-PCA in comparison to baseline methods, with exceptions on a few datasets. This might raise questions about the overall effectiveness of the proposed method.

Author response:
As mentioned above, we have supplemented the relevant experimental content in Tables 2 and 3.

Reviewer comment (Evaluation):
The method struggled in performance on the GLUE & SQuAD datasets (e.g. BERT-base* performed the best, even if it is by a 2-point margin). Do you have any idea why?

Author response:
In fact, BERT-base is the teacher model, while our student model is based on knowledge distillation training, so the results generally do not surpass the teacher model.

Reviewer comment (Experiments):
Not related to the PCA necessarily but to the idea of reduction - have you tried or explored working in latent space?

Author response:
In fact, the discussion on KD loss in our current Appendix C should involve relevant categories, as the output of each layer can also be understood as a representation of a latent space.

Reviewer comment (Evaluation):
Table 5 results are close to each other. Thereby showing that the weight-weighted version might not improve the results too much as compared to Vanilla. A deeper qualitative analysis is required to establish the usage which is missing in the current version of this work.

Author response:
In fact, due to the nature of the search space, any zero-shot proxy has a baseline score. We have applied the zero-shot proxy mentioned in the related work to the search space illustrated in Figure 3. The relevant results have been added to Tables 2 and 3, and the discussion in Section 6.3 has been expanded.

Reviewer comment (Writing):
The first two lines of the abstract are not well connected. Kindly reframe the abstract to highlight the motivation and contribution better.

Author response:
Thank you very much for your correction.  We have reframed the abstract to better highlight the motivation and contribution of our work.

Reviewer comment (Writing):
Abstract: Unbiased assessment of what?

Author response:
The term "unbiased assessment" refers to the comprehensive evaluation of the language model's performance using the parameter count and PCA value. We have clarified this in the modified abstract.

Reviewer comment (Writing):
Section 1. 3rd last paragraph - “potentiallly overlooking important characteristics”. What are these important characteristics?

Author response:
These characteristics include some features of neural networks extracted in previous zero-shot NAS, such as gradients, parameter sizes, and parameter quantities discussed in Section 2.3.

Reviewer comment (Writing):
Section 2.2 mentions primitive operators - What are these?

Author response:
These include unary (such as negation, softmax) and binary (such as addition, multiplication) mathematical operations, which we have explained in the original text.

Reviewer comment (Presentation):
Section 2.3: Include year with the citations.

Author response:
Thank you for the correction. We have made the necessary amendments to the relevant citations.

Reviewer comment (Writing):
Section 2.3: Inconsistent notations throughout this section - N is parameters, as well as batch size in this section. This section needs to be written better to make the reader understand the motivation for using each of the metrics, instead of just writing the formulation.

Author response:
Thank you for your correction. We have replaced "n" with "N" as the parameter and added a description of the relevant motivation in the corresponding zero-shot proxy.

Reviewer comment (Theory):
Why is the scaling of hidden_dim needed?

Author response:
Thank you for helping us reconsider the role of hidden_dim. After repeated experiments, we found that removing this scaling would lead to better correlation in the dataset rankings. We have updated the relevant experimental results.

Reviewer comment (Writing):
Equation 4: Notation writeup should be improved; currently it looks like PCA(X) subtracted from W.

Author response:
We have updated the expression of the relevant formula, and now the "-" in the formula no longer looks like a minus sign.

Reviewer comment (Reproducibility):
Section 5.1: Kindly include the details of the appendix section of Serianni & Kalita. Each paper should be a standalone read.

Author response:
Thank you for your correction. We have added the relevant content to Appendix A.

Reviewer comment (Reproducibility):
Section 6.2.1: Details of the genetic algorithm being used is missing here. Kindly include it here or the appendix.

Author response:
We employ a genetic algorithm with a population size of 50 and a generation count of 40 to identify the combination of blocks that yields the highest PCA value. The crossover probability is set to 1, the mutation probability to 0.1, and the upper limit for the model parameters to 15.7M in order to obtain the W-PCA-Small model. By further reducing the upper limit for the model parameters to 10M and halving the number of layers ($m$), we obtain the W-PCA-Tiny model.

This part of the content has been written in Section 6.2.1.

Reviewer comment (Writing):
Section 6.2.3: KD Loss - $\mathcal{L}^i_{attn} = \text{MSE}(\mathbb{A}^{S}*i\mathbb{W}a, \mathbb{A}^T*j).

Author response:
Thank you for your correction. We have made the necessary revisions. We have moved all the sections related to KD loss to Appendix C.

Reviewer comment (Reproducibility):
Section 6.2.3: KD Loss - “jth teacher model layer corresponds to ith student model layer” - How do we get this correspondence?

Author response:
For our fixed teacher model, BERT-base, which comprises 12 layers, a one-to-one sequential correspondence exists between the layers of the student and teacher models when both models have 12 layers. However, in the case of a student model with only 6 layers, the correspondence remains one-to-one, but with a 2-layer interval. This implies that the first layer of the student model corresponds to the second layer of the teacher model, and so forth, until the sixth layer of the student model aligns with the twelfth layer of the teacher model.

We have incorporated this part of the content into the explanation of the formula.

Reviewer comment (Evaluation):
Conclusion: Kindly mention some quantitative metrics here. In the current version it looks like a paraphrase of the abstract.

Author response:
We have revised the Conclusion section according to your suggestions.

Reviewer comment (Experiments):
The paper primarily focuses on experiments with the BERT model, and it would be beneficial to conduct more experiments on other types of language models, such as generative models or larger-sized models, to ascertain the method's applicability.

Author response:
We conducted experiments using a larger-sized model by increasing the size of the search space, as described in Section 6.2.1. Specifically, we doubled the hidden\_size and the hidden\_dimension of $n$ candidate dimensions. Additionally, we raised the parameter limit in the genetic algorithm to 67M, resulting in our W-PCA-Large model. As shown in the Table, despite having a slightly lower parameter count, our model outperforms TinyBERT-6 [1] and EfficientBERT [2] models of similar scale in terms of average GLUE score. This indicates that our proposed zero-shot proxy also demonstrates good adaptability in larger search spaces.

|  Model   | #Params  |  QNLI   | MRPC  |  SST-2   | CoLA |  STS-B   | MNLI-m/mm  | RTE  |  QQP   | AVG  |
|  ----  | ----  |  ----  | ----  |  ----  | ----  |  ----  | ----  |  ----  |  ----  | ----  |
| TinyBERT-6 [1]  |  67.0M  | 89.8 | 89.0 | 92.0 | 38.8 | 83.1 | 83.8/83.2 | 65.8 | 71.4 |77.4 |
| EfficientBERT [2]  | 70.1M  | 90.4 | 89.0 | 92.6 | 46.2 | 83.7 | 84.1/83.2 | 67.7 | 74.4 | 78.7 |
| W-PCA-Large  | 66.9M  | 90.9 | 88.7 | 93.0 | 40.0 | 87.5 | 84.6/83.3 | 75.6 | 71.5 | 79.5 |

We have included this part of the content in Appendix E.1 of the paper (Page 13).

Reviewer comment (Experiments):
A more in-depth discussion of the proposed method is needed. It would be beneficial to conduct ablation studies to assess the impact of each loss term, considering that multiple loss terms are included in the fine-tuning process.

Author response:
We utilized the components of W-PCA, as described in Equation (4) where the first component is the number of parameters (\#Params) and the second component is the V-PCA value (defined in Equation (2)), as fitness values for the genetic algorithm to explore the optimal network structure in Section 6.2.1. We then compared the performance of the discovered network structures with W-PCA.

The results, presented in Table 5, demonstrate that by multiplying the number of parameters with the V-PCA value and using W-PCA as the zero-shot evaluation metric, the performance of the searched networks significantly improves compared to using either \#Params or V-PCA alone as the evaluation metric.

We have added this explanation to Section 6.5 of the original text.

Reviewer comment (Experiments):
The differences in training datasets and objectives compared to previous works make it unclear whether the improvement stems from the searched architecture or other factors.

Author response:
We have applied the zero-shot proxy from related work to the search space we designed (Figure 3), and the results from GLUE test set are as follows.

|  Model   |   Type | #Params   |  Time  |  QNLI   | MRPC  |  SST-2   | CoLA |  STS-B   | MNLI-m/mm  | RTE  |  QQP   | AVG  |
|  ----  | ----  | ----  | ----  | ----  | ----  |  ----  | ----  |  ----  |  ----  |  ----  | ----  |
| Synaptic Saliency [3] |zero-shot |15.7M | 58ms | 89.4 | 88.1 | 91.0| 33.6 | 83.1 | 82.6/81.1 | 70.6|  70.3 | 76.6 |
| Activation Distance [4] | zero-shot |15.6M | 60ms | 88.9 | 87.6 | 91.2 | 30.7 | 82.9 | 81.1/80.4 | 70.4| 70.1 |75.9 |
| Synaptic Diversity [5] | zero-shot |15.6M | 57ms | 88.3 | 88.1 | 91.5 | 25.8 | 84.7 | 81.3/80.2 | 70.6| 70.3 |75.6  |
| Head Confidence [6] |  zero-shot |15.6M | 63ms | 89.5 | 88.3 | 92.4 | 31.7 | 85.7 | 82.8/81.9 | 74.0 | 70.9 |77.5  |
| Softmax Confidence[6] |  zero-shot |15.6M | 61ms | 88.4 | 87.5 | 90.8 | 32.5 | 83.5 | 81.2/80.5 | 70.3| 69.9 |76.1  |

The results from GLUE dev set are as follows.

|  Model   |   #Params  |  Time  |  QNLI   | MRPC  |  SST-2   | CoLA |  STS-B   | MNLI-m/mm  | RTE  |  QQP   | AVG  |
|  ----  | ----  | ----  | ----  |  ----  | ----  |  ----  | ----  | ----  |  ----  | ----  | ----  |
| Synaptic Diversity [5] | 15.6M | 0.7 d | 88.9 | 87.6 | 91.4 | 32.0 | 84.1 | 81.0 | 73.4| 88.2 |78.3  |
| Head Confidence [6] |  15.6M | 0.5 d | 90.1 | 89.7 | 92.4 | 37.5 | 84.1 | 82.5 | 75.9| 89.1 |80.2  |
| Softmax Confidence[6] |  15.6M | 0.5 d | 89.4 | 88.3 | 92.0 | 32.6 | 84.7 | 81.6 | 73.9| 88.9 |78.9  |

We have added the comparative results of this part in the same search space to Table 2 and Table 3, and discussed the relevant content in the main text.

Reviewer comment (Evaluation):
The improvement observed in GLUE and SQuAD is described as marginal.

Author response:
We have expanded the relevant discussions in Sections 6.3 and 6.4.

Reviewer comment (Presentation):
This article requires significant improvement in writing and formatting. The misplaced tables contribute to a lack of clarity in the paper's presentation, such as in section 6.3.

Author response:
We have reorganized the relevant tables and improved multiple expressions in the original text. You can also refer to the revisions made based on the guidance of Reviewer pvZH.

Reviewer comment (Experiments):
Could you provide information regarding the zero-shot performance of the searched model? Additionally, I'm interested in learning about the performance of architectures discovered using W-PCA with different parameters.

Author response:
We conducted supplementary experiments on different initialization parameters, which can be seen in the newly added Appendix B. Additionally, the discussion on model visualization has been included in Appendix D.

Reviewer comment (Experiments):
Have you conducted experiments with multiple shots and employed an iterative searching strategy for multiple shots searching?

Author response:
There is limited research on n-shots (n > 1) NAS in NLP tasks. The only one we are aware of is [7], as it requires significant computational resources. We have conducted supplementary experiments for the n = 1 case, and the results are as follows:

|  Model   |   Type | #Params   |  Time  |  QNLI   | MRPC  |  SST-2   | CoLA |  STS-B   | MNLI-m/mm  | RTE  |  QQP   | AVG  |
|  ----  | ----  | ----  | ----  | ----  | ----  |  ----  | ----  |  ----  |  ----  |  ----  | ----  |
| W-PCA-Tiny |zero-shot | 9.6M | 0.4 d | 88.7 | 87.6 | 91.9 | 27.4 | 84.8 | 81.1/79.8 | 71.1| 70.3 |75.9 |
|  | one-shot |9.7M | 24 d | 89.2 | 87.5 | 92.3 | 28.9 | 83.7 | 81.4/80.5 | 71.4 | 70.5 |76.2 | 
| W-PCA-Small | zero-shot |15.6M | 0.5 d | 90.3 | 88.7 | 91.5 | 38.4 | 86.4 | 82.8/82.2 | 73.8 | 70.8 |78.3  |
| |  one-shot |15.6M | 28 d | 90.3 | 88.9 | 92.5 | 36.1 | 86.7 | 83.7/82.5 | 74.4 | 70.6 |78.4 |

Despite investing a significant number of GPU days in the one-shot NAS search, the performance improvement on various-sized models of W-PCA is not significant. Zero-shot NAS remains the most cost-effective search solution.

The relevant content has been included in Appendix E.2 （Page 14）