Paper: LLM Performance Predictors are good initializers for Architecture Search (ICLR-2024)

Reviewer comment (Experiments):
Fixed Downstream Tasks, Unknown Effects on Other Tasks: The paper primarily focuses on performance prediction for specific downstream tasks. However, it does not explore the potential impact or applicability of LLM-PP or LLM-Distill-PP models on a broader range of tasks. This limits the generalizability of the approach and its potential in different contexts.

Author response:
* **Weakness 2:** Our evaluation setup of using machine translation benchmarks is borrowed from the NAS for NLP literature [Wang ACL'20, Jawahar ACL'23], which also uses only machine translation tasks. Our contribution is still non-trivial and significant, which as you’d pointed out is “innovative”, “valuable approach to reduce the cost of using LLMs for performance prediction”, “can be a substantial advantage for practitioners looking to optimize their deep learning models.”  Exploring other NLP tasks, computer vision, speech recognition tasks is orthogonal to our main research questions, but will be an important future direction.

Reviewer comment (Reproducibility):
Is the prediction result provided by LLM repeatable? How is it handled if the results given each time are different?

Author response:
* **"Is the prediction result provided by LLM repeatable?"** We compute the predictions for 8500 randomly sampled architectures using LLM-PP GPT-4 three times and compute the standard deviation of the three predictions for each architecture. The mean of the standard deviation for 8500 architectures is very low: 0.21, 0.27, 0.27 BLEU for WMT’14 En-De, WMT’14 En-Fr, and WMT’19 En-De respectively. Thus, LLM-PP provides fairly robust performance predictions. For all our search experiments, we use a single estimate from LLM-PP.

Reviewer comment (Experiments):
Since the final search should use the model distilled from LLM-PP instead of LLM-PP itself, why not use a PP that performs better than LLM-PP for distillation?

Author response:
* **"why not use a PP that performs better than LLM-PP for distillation?..."** LLM-PP achieves the best performance (SOTA MAE, marginally worse Kendall-Tau but doesn’t have a big negative impact for the NAS task, as detailed in Section 5.2, second paragraph) compared to the supernet based performance predictors (see Table 1). Hence, LLM-PP-Distill is built by distilling the performance predictions from LLM-PP. In A.6.4, we show that LLM-PP-Distill that distills from HAT (marginally better Kendall-Tau than LLM-PP) leads to search hours reduction of at least 40% compared to HAT, but at the expense of big increase in latency and GFLOPs (9.7%, 32% respectively). This motivates why we distilled from LLM-PP.

Reviewer comment (Evaluation):
The effectiveness of the proposed method largely depends on how much information GPT-4 has "memorized." Since GPT-4 is a language model, its impressive prediction performance on WMT'14 (or WMT'19), transformer-base, translation direction, and BLEU is primarily because these elements are commonly used for machine translation. The authors need to recognize the limitations when dealing with less conventional models, datasets, translation directions, metrics, and other tasks and discuss these in the paper.

Author response:
* **"The effectiveness of the proposed method largely depends on how much information GPT-4 has "memorized...."** Thank you for the questions. While WMT’14 and WMT’19 are popular MT benchmarks, it’s highly unlikely that the high performance of LLM-PP can be largely attributed to “memorization” of the underlying LLM for the following reasons:
  - Since the test set is created by randomly sampling from the search space, the probability of a test architecture to be part of the LLM training data is extremely low (e.g., 1 / 6000000 as the WMT’14 En-De search space has 6 million architectures). Note that we need a large number of resources (e.g.,  training a single architecture on 4 NVIDIA V100 GPUs takes 2 days) to train even a few 100s of architectures from scratch to convergence to get their true performance. Hence, it’s infeasible to have non-negligible coverage of architectures (along with its corresponding true performance) in the LLM training data. 
  - If the performance of LLM-PP is largely due to “memorization”, LLM-PP should achieve competitive performance even when the number of demonstrations in the prompt is minimal. Consider the table below. 

| Dataset | WMT’14 En-Fr | WMT’14 En-Fr | WMT’19 En-De | WMT’19 En-De |
| - | -------- | -------- | -------- | -------- | 
| Perf. Predictor | MAE | Kendall-Tau | MAE | Kendall-Tau |
| Neuron-wise MoS | 1.18 | 0.87 | 0.87 | 0.67 |
| LLM-PP ChatGPT (June)  1 demonstration | 2.81 | 0.55 | 1.72 | 0.56 |
| LLM-PP ChatGPT (June)  3 demonstrations |  2.28 | 0.67 | 1.12 | 0.57 |
| LLM-PP ChatGPT (June) 10 demonstrations | 0.82 | 0.61 | 0.72 | 0.56 |

From this table, it is clear that LLM-PP did not achieve competitive performance compared to the SOTA supernet baseline when the number of demonstrations in the prompt is minimal (e.g., less than or equal to 3 demonstrations). This result highlights that the performance of LLM-PP might not be largely due to “memorization”, and is likely due to other LLM capabilities such as “understanding” and “reasoning”. Due to lack of time in the rebuttal phase, it might not be possible to experiment with other suggested datasets (we will send an update response if we could experiment with other datasets within the timeline). 
* **"It is very possible that the author is testing the model that it had been trained on the test dataset"** As the training data of LLMs like ChatGPT and GPT-4 are not public, it is impossible to conclusively say that the training data do not contain any of our test architectures. Despite this, we have shown strong evidence on why the performance of LLM-PP is highly unlikely to largely come from “memorization”.

We hope we have answered your questions to your satisfaction and hope you are willing to raise the score. Please let us know if there are any further questions that we can address during the interactive rebuttal period.

Reviewer comment (Experiments):
What would occur if training and testing were done on WMT'22 data?

Author response:
Thanks for your response. 

We disagree with the reviewer’s point that **”GPT-4 only works on the dataset (architecture information and its results) that it has seen before”**. Compared to SOTA performance predictors, LLM-PP GPT-4 works well for recent datasets (e.g., machine translation benchmark introduced in the 2023 year), low-resource/indigenous languages (e.g., Bribri, Chatino), and uncommon evaluation metrics (e.g., Comet), as discussed below.

From the recent shared task: “AmericasNLP 2023 Shared Task on Machine Translation into Indigenous Languages” [AmericasNLP ACL’23], we take three machine translation benchmarks: Bribri to Spanish, Chatino to Spanish, and Spanish to Bribri. Compared to WMT 2014, WMT 2019 benchmarks (experimented in the paper), these three benchmarks are very recent (2023 year) and one of the languages in each translation direction is an low-resource/indigenous language (Bribri, Chatino). We compare LLM-PP GPT-4 against SOTA performance (BLEU) predictors on these benchmarks in terms of quality (MAE, Kendall-Tau). Consider the table below (performance averaged across two seeds),

| | Bribri to Spanish | Bribri to Spanish | Chatino to Spanish | Chatino to Spanish | Spanish to Bribri | Spanish to Bribri|
|-|-|-|-|-|-|-|
|Performance Predictor | MAE | Kendall | MAE | Kendall | MAE | Kendall|
| HAT | 0.28 | 0.15 | 1.55 | **0.16** | 0.72  | 0.02 |
| Layer-wise MoS | 0.33 | -0.13 | 2.42 | -0.17 | 0.63 | -0.14 |
| Neuron-wise MoS | 0.29 | -0.35 | 2.94 | -0.06 | 0.43 | 0.09  | 
| LLM-PP GPT-4 | **0.16** | **0.29** | **1.21** | 0.08 | **0.32** | **0.20** | 

It is clear that **LLM-PP achieves the SOTA MAE score across these benchmarks**, which is consistent with the trends in WMT 2014, WMT 2019 benchmarks (as shown in Table 1). Impressively, on two of these benchmarks, **LLM-PP also achieves the SOTA Kendall-Tau score**. Put together, these results clearly showcase that **LLM-PP generalizes well to recent datasets and low-resource languages.**

Further, we show that LLM-PP also generalizes well to uncommon evaluation metrics. Based on the reviewer's suggestion, we build performance predictors that predict the Crosslingual Optimized Metric for Evaluation of Translation (COMET) [Comet HF] (Unbabel/wmt22-comet-da), which is relatively newer than the BLEU metric. Consider the table below (performance averaged across two seeds),

| | Bribri to Spanish | Bribri to Spanish | Chatino to Spanish | Chatino to Spanish |
|-|-|-|-|-|
|Performance Predictor | MAE | Kendall | MAE | Kendall |
| HAT | 0.03 | 0.24 | 0.02 | -0.15 |
| Layer-wise MoS | 0.02 | -0.05 | 0.02 | 0.26 |
| Neuron-wise MoS | 0.02 | **0.32** | **0.01** | 0.34 | 
| LLM-PP GPT-4 | **0.01** | **0.32** | **0.01** | **0.54** | 

On Bribri to Spanish task and Chatino to Spanish task, LLM-PP achieves the SOTA MAE and SOTA Kendall Tau performance compared to SOTA performance predictors. These results show that LLM-PP generalizes well to uncommon evaluation metrics like COMET. Note that we exclude Spanish to BriBri task, since COMET does not support Bribri. 

We will add these new results in the revision. Now, we will respond to the recent quoted comments below:
* **“'GPT-4 only works on the dataset ... that it has seen before.”** **”I strongly disagree with the claim in the paper that 'LLMs have a ‘general understanding’ of DNN architectures.'”** As shown above, we disagree with these claims as LLM-PP GPT-4 indeed generalizes to recent datasets, low-resource/indigenous language, and uncommon evaluation metrics, which are highly unlikely to be seen before by GPT-4,. These results can be largely attributed to LLM’s general understanding of DNN architectures.
* **”It predicts the model's performance based on what is most likely, with the restriction only on commonly used data (like WMT'14).”** **”This method, under my understanding, is difficult to generalize and only applies to very commonly used data.”** We disagree with these claims as LLM-PP GPT-4 generalizes to uncommon datasets such as Bribri to Spanish, Chatino to Spanish, and Spanish to Bribri, as shown above.
*  **”Without showing the limitations on the uncommon data (or, say, generalization), this study should be a clear reject.”** Thanks for the suggestions. In the current response, we showed that LLM-PP generalizes to machine translation datasets introduced in the 2023 year, low-resource/indigenous languages (Bribri, Chatino), and uncommon evaluation metrics (Comet). In the previous response, we already showed multiple pieces of evidence on why “memorization” cannot be a key factor for the superior performance of LLM-PP in general. We hope we have answered your questions to your satisfaction and hope you are willing to raise the score. 

Pls. let us know if there are any further questions.

References
* [AmericasNLP ACL’23] https://turing.iimas.unam.mx/americasnlp/2023_st.html
* [Comet HF] https://huggingface.co/spaces/evaluate-metric/comet

Reviewer comment (Experiments):
What if the testing were on a low-resource language, say, Wolof?

Author response:
Thanks for your response. 

We disagree with the reviewer’s point that **”GPT-4 only works on the dataset (architecture information and its results) that it has seen before”**. Compared to SOTA performance predictors, LLM-PP GPT-4 works well for recent datasets (e.g., machine translation benchmark introduced in the 2023 year), low-resource/indigenous languages (e.g., Bribri, Chatino), and uncommon evaluation metrics (e.g., Comet), as discussed below.

From the recent shared task: “AmericasNLP 2023 Shared Task on Machine Translation into Indigenous Languages” [AmericasNLP ACL’23], we take three machine translation benchmarks: Bribri to Spanish, Chatino to Spanish, and Spanish to Bribri. Compared to WMT 2014, WMT 2019 benchmarks (experimented in the paper), these three benchmarks are very recent (2023 year) and one of the languages in each translation direction is an low-resource/indigenous language (Bribri, Chatino). We compare LLM-PP GPT-4 against SOTA performance (BLEU) predictors on these benchmarks in terms of quality (MAE, Kendall-Tau). Consider the table below (performance averaged across two seeds),

| | Bribri to Spanish | Bribri to Spanish | Chatino to Spanish | Chatino to Spanish | Spanish to Bribri | Spanish to Bribri|
|-|-|-|-|-|-|-|
|Performance Predictor | MAE | Kendall | MAE | Kendall | MAE | Kendall|
| HAT | 0.28 | 0.15 | 1.55 | **0.16** | 0.72  | 0.02 |
| Layer-wise MoS | 0.33 | -0.13 | 2.42 | -0.17 | 0.63 | -0.14 |
| Neuron-wise MoS | 0.29 | -0.35 | 2.94 | -0.06 | 0.43 | 0.09  | 
| LLM-PP GPT-4 | **0.16** | **0.29** | **1.21** | 0.08 | **0.32** | **0.20** | 

It is clear that **LLM-PP achieves the SOTA MAE score across these benchmarks**, which is consistent with the trends in WMT 2014, WMT 2019 benchmarks (as shown in Table 1). Impressively, on two of these benchmarks, **LLM-PP also achieves the SOTA Kendall-Tau score**. Put together, these results clearly showcase that **LLM-PP generalizes well to recent datasets and low-resource languages.**

Further, we show that LLM-PP also generalizes well to uncommon evaluation metrics. Based on the reviewer's suggestion, we build performance predictors that predict the Crosslingual Optimized Metric for Evaluation of Translation (COMET) [Comet HF] (Unbabel/wmt22-comet-da), which is relatively newer than the BLEU metric. Consider the table below (performance averaged across two seeds),

| | Bribri to Spanish | Bribri to Spanish | Chatino to Spanish | Chatino to Spanish |
|-|-|-|-|-|
|Performance Predictor | MAE | Kendall | MAE | Kendall |
| HAT | 0.03 | 0.24 | 0.02 | -0.15 |
| Layer-wise MoS | 0.02 | -0.05 | 0.02 | 0.26 |
| Neuron-wise MoS | 0.02 | **0.32** | **0.01** | 0.34 | 
| LLM-PP GPT-4 | **0.01** | **0.32** | **0.01** | **0.54** | 

On Bribri to Spanish task and Chatino to Spanish task, LLM-PP achieves the SOTA MAE and SOTA Kendall Tau performance compared to SOTA performance predictors. These results show that LLM-PP generalizes well to uncommon evaluation metrics like COMET. Note that we exclude Spanish to BriBri task, since COMET does not support Bribri. 

We will add these new results in the revision. Now, we will respond to the recent quoted comments below:
* **“'GPT-4 only works on the dataset ... that it has seen before.”** **”I strongly disagree with the claim in the paper that 'LLMs have a ‘general understanding’ of DNN architectures.'”** As shown above, we disagree with these claims as LLM-PP GPT-4 indeed generalizes to recent datasets, low-resource/indigenous language, and uncommon evaluation metrics, which are highly unlikely to be seen before by GPT-4,. These results can be largely attributed to LLM’s general understanding of DNN architectures.
* **”It predicts the model's performance based on what is most likely, with the restriction only on commonly used data (like WMT'14).”** **”This method, under my understanding, is difficult to generalize and only applies to very commonly used data.”** We disagree with these claims as LLM-PP GPT-4 generalizes to uncommon datasets such as Bribri to Spanish, Chatino to Spanish, and Spanish to Bribri, as shown above.
*  **”Without showing the limitations on the uncommon data (or, say, generalization), this study should be a clear reject.”** Thanks for the suggestions. In the current response, we showed that LLM-PP generalizes to machine translation datasets introduced in the 2023 year, low-resource/indigenous languages (Bribri, Chatino), and uncommon evaluation metrics (Comet). In the previous response, we already showed multiple pieces of evidence on why “memorization” cannot be a key factor for the superior performance of LLM-PP in general. We hope we have answered your questions to your satisfaction and hope you are willing to raise the score. 

Pls. let us know if there are any further questions.

References
* [AmericasNLP ACL’23] https://turing.iimas.unam.mx/americasnlp/2023_st.html
* [Comet HF] https://huggingface.co/spaces/evaluate-metric/comet

Reviewer comment (Experiments):
It is very likely that GPT has already encountered the architecture selection and results for your model, data, and metric settings since they have been prevalent in recent years. It is very possible that the author is testing the model that it had been trained on the test dataset. However, the author didn't explicitly address the performance in less conventional settings in the paper, rendering the study meaningless.

Author response:
* **"It is very possible that the author is testing the model that it had been trained on the test dataset"** As the training data of LLMs like ChatGPT and GPT-4 are not public, it is impossible to conclusively say that the training data do not contain any of our test architectures. Despite this, we have shown strong evidence on why the performance of LLM-PP is highly unlikely to largely come from “memorization”.

Reviewer comment (Experiments):
The experimental methodology would be stronger if the authors compared to performance prediction methods beyond just supernetworks, such as those in these references:
- https://arxiv.org/abs/2008.03064  
- https://proceedings.neurips.cc/paper_files/paper/2021/file/2130eb640e0a272898a51da41363542d-Paper.pdf  
- https://proceedings.mlr.press/v188/laube22a/laube22a.pdf  
- https://arxiv.org/abs/2101.08134

Author response:
* **”It is great that you added “number of parameters” as a baseline. I agree that it is tough to add during the short author response period, but I still think that comparing to other methods would improve the evaluation, ….”** Thanks for the suggestions. We add comparison to **five** non-supernet baselines: #Params, #FLOPs, grad_norm, snip, and snyflow (see [Zero-cost ICLR’22 Blog] for details). Consider the table below,

| Kendall | WMT’14 En-De | WMT’14 En-Fr | WMT’19 En-De |
| - | - | - | - |
| #Params | 0.42 | 0.51| 0.54 |
| #FLOPs | 0.43 | 0.53 | 0.54 |
| grad_norm | -0.42 | -0.42 | -0.52 |
| snip | -0.42 | -0.27 | -0.3 |
|synflow | -0.31 | -0.47 | -0.49 |
|LLM-PP GPT-4 | **0.65** | **0.75** | **0.65** |

It is clear that LLM-PP GPT-4 achieves a high Kendall Tau, outperforming all the non-supernet baselines. These results along with Table 1 from the paper showcases the superior performance of LLM-PP across a wide range of baselines.

Reviewer comment (Experiments):
The simple baseline “number of parameters” has recently been found to be a surprisingly strong baseline for performance prediction, and it would be beneficial to include it as an additional baseline in Table 1.

Author response:
Based on your suggestion, we experiment with the simple baseline “number of parameters”. The kendall-tau for this simple baseline is 0.42, 0.51, 0.54 for WMT’14 En-De, WMT’14 En-Fr, and WMT’19 En-De respectively, which is clearly worse than LLM-PP GPT-4 (0.65, 0.75, 0.65, as shown in Table 1 of the paper).

Reviewer comment (Experiments):
The authors only test their performance predictor on a single NAS framework (HAT). There are other NAS frameworks, such as Bayesian optimization (https://arxiv.org/abs/2110.10423) or BOHB, that could be included for a more comprehensive evaluation.

Author response:
* **".. the authors only test their performance predictor on a single NAS framework; the one from HAT. . There are other NAS frameworks too, for example, Bayesian optimization or BOHB."** We show the efficacy of HS-NAS in two SOTA NAS for MT frameworks: HAT and Mixture-of-Supernet (MoS) NAS framework. Due to lack of space, HS-NAS HAT results are in the main paper, while HS-NAS MoS results can be seen in Section A.6.3. Generalizing HS-NAS to other search frameworks such as bayesian optimization is orthogonal to our main research questions, but will be an important future direction.

Reviewer comment (Reproducibility):
The authors mention that they share their code, but it could not be found. They should provide a link to the code, for example via https://anonymous.4open.science/.

Author response:
* **"The authors mention that they share their code, but I couldn't find it. Can the authors share their code, e.g. with https://anonymous.4open.science/?"** Thanks for the suggestion. Please find the code in this link: https://anonymous.4open.science/r/llmpp-code-FF36

Reviewer comment (Experiments):
How does the LLM-Distill-PP model's efficiency and accuracy compare to other existing models?

Author response:
**"How does the LLM-Distill-PP model's efficiency and accuracy compare to other existing models?"** The following table shows the efficiency (time taken to predict performance for 10 architectures) and accuracy (MAE, Kendall-Tau, as shown in Table 1) for supernet-based PP (HAT, Layer-wise MoS, Neuron-wise MoS), LLM-PP (GPT-4), and LLM-Distill-PP (GPT-4) for the WMT'14 En-De task.  
| Perf. Predictor (PP)| MAE | Kendall-Tau | Prediction Time (seconds) |  
| -- | -- | -- | -- |  
| HAT | 1.14 | 0.71 | 10.5 |  
| Layer-wise MoS | 1.05 | 0.81 | 13.9 |  
| Neuron-wise MoS | 0.97  | 0.56 | 13.3 |  
| LLM-PP GPT-4 | 0.28 | 0.65 | 11.9 |  
| LLM-Distill-PP GPT-4  | 0.22 | 0.64 | 0.01|  

It is clear that LLM-Distill-PP provides the best efficiency-accuracy tradeoff with on par (if not better) accuracy as LLM-PP but significantly faster prediction time (0.01s vs. 11.9s).

Reviewer comment (Experiments):
Could the authors elaborate on the rationale and design process behind the PP prompts used for LLMs?

Author response:
**"Could the authors elaborate on the rationale and design process behind the PP prompts used for LLMs?"** The goal of creating PP prompts is to precisely communicate the performance prediction task to the LLM. We selected machine translation benchmarks due to their extensive study in the NAS for NLP literature. The design process began by examining crucial elements of the machine translation task, commonly used model architectures, and relevant efficiency metrics. Initially, we presented only demonstrations, borrowing hyperparameter wording from HAT’s configuration file. Subsequently, we added the role and definition of each hyperparameter, using wording from HAT’s helper description. Moving forward, our aim was to craft instructions enabling the LLM to grasp essential tasks, architecture, and metric details. Most instructions are prefixed with 'You should' to encourage strict adherence. Five instructions were incorporated. The first specifies the dataset, translation direction, and quality metric. The second provides examples randomly sampled from the training set, presented with generic prefixes ('Input:' for source sentence, 'Output:' for target sentence). The third outlines the architecture, citing the 'Attention Is All You Need' paper, assuming the LLM is familiar with this popular work. Standard settings and optimization algorithms are noted for training the architectures. The fourth identifies the efficiency metric in the demonstrations. The final instruction aims to summarize the relationships the LLM should learn to solve the task effectively.