Paper: HART: Efficient Adaptation via Regularized Autoregressive Parameter Generation (ICLR-2024)

Reviewer comment (Experiments):
Due to the use of the T5 model's weights to initialize the hypernetwork, the structure of the hypernetwork is subject to certain constraints and cannot arbitrarily change the parameter sizes. At the same time, the initialization parameters of the hypernetwork (such as from T5-Large/XL) might significantly impact its performance. I am curious to see what the results would be if training a completely new hypernetwork from scratch.

Author response:
**Due to the use of the T5 model's weights to initialize the hypernetwork, the structure of the hypernetwork is subject to certain constraints and cannot arbitrarily change the parameter sizes.**

Using architectures and weights other than T5 to initialize the hypernetwork is flexible. We adopt T5 mainly because it is a widely used encoder-decoder model family for text-conditioned generation, offering multiple scale options. 

Furthermore, even sticking with a specific model, we can still change its parameter size. For example, in our experiments, we only use the first 8 layers (out of 24) in the T5 decoder for training efficiency. 

**The initialization parameters of the hypernetwork (such as from T5-Large/XL) might significantly impact its performance. I am curious to see what the results would be if training a completely new hypernetwork from scratch.**

We agree with the reviewer that the hypernetwork initialization will impact its performance, but we emphasize that all methods in this paper are evaluated under the same initialization scheme, ensuring fair comparisons.

Technically, we are able to train a hypernetwork from scratch but it will require much more resources than training from pre-trained weights. The pre-trained weights are valuable as they can provide the hypernetwork with the capability to understand the input demonstrations. So it makes sense to take advantage of pre-trained weights whenever they are available.

Reviewer comment (Experiments):
The large number of trainable parameters in the transformer-based hypernetwork contradicts the original intention of PEFT. Compared to the typical PEFT method, it may require more computational overhead.

Author response:
**The large number of trainable parameters in the transformer-based hypernetwork contradicts the original intention of PEFT. Compared to the typical PEFT method, it may require more computational overhead.**

We agree with the reviewer that training a hypernetwork is costly. However, for any new task, a trained hypernetwork can directly generate PEFT weights within one forward pass, which takes less than a second to finish. In contrast, PEFT requires iterations of forward and backward passes that take hours to finish. Note that the backward pass is not only slower than the forward pass but also requires more memory.

Considering the case when the number of tasks scales up to $m$. The hypernetwork approach only requires $m$ forward passes, while PEFT requires $m \times n$ forward and backward passes, where $n$ is the number of training iterations for each task. As a result, the hypernetwork approach is more efficient than PEFT.

Reviewer comment (Experiments):
It's unclear to me (and somewhat counterintuitive) why using Fusion-in-Decoder performs better than the continual pre-training (the first and third row in Table 6), and have the authors explored the performance of HART + pre-training?

Author response:
**It's unclear to me (and somewhat counterintuitive) why using Fusion-in-Decoder performs better than the continual pre-training (the first and third row in Table 6).**

Fusion-in-Decoder (FiD) has demonstrated superior performance compared to the hypernetwork pre-training also in previous studies such as the HINT paper (Table 4, [[1]](https://arxiv.org/pdf/2212.10315.pdf)). The reason for its better performance might be attributed to the following:

In FiD, we employ a direct concatenation of the output representation of the hypernetwork's encoder (referred to as "context") with the output representation of the main model's encoder (referred to as "query"). This fusion enables the main model's decoder to generate responses directly conditioned on the context. In contrast, in pre-training, the hypernetwork needs to learn the dependency between the context and query via learning a set of context-conditioned weights.

Essentially, FiD simplifies the task for the model since ***it fuses context information into the main model more explicitly***, reducing the need for the model to learn context-query dependencies from scratch, resulting in improved performance.

In our paper, we primarily adopt FiD because it offers comparable performance while incurring significantly lower computational overhead compared to pre-training. However, it's worth noting that both methods are complementary, as demonstrated in the HINT paper (Table 4, [[1]](https://arxiv.org/pdf/2212.10315.pdf)). If budget constraints were not an issue, we would employ both approaches to harness their combined benefits. 

[1] Ivison, Hamish, et al. "Hint: Hypernetwork instruction tuning for efficient zero-and few-shot generalisation." 

**Have the authors explored the performance of HART + pre-training?**

We did not prioritize pre-training due to limited computing resources, as it is not the primary focus of this work. We plan to explore pre-training in future works on more advanced models than T5.

We anticipate that pre-training will enhance the performance of HART. Given that HART relies on input dependencies between the hypernetwork and the main model, learning these dependencies in advance is expected to be beneficial.

Reviewer comment (Experiments):
I am not fully convinced by the experiments that the methods are evaluated on the "unseen" tasks. In Tables 2 to 5, what is the separation between the training tasks and evaluation tasks? Are the tasks that appeared in training, not in the evaluation tasks? For example, question-answering is only used in training but not in evaluation. I found the paper didn't describe the dataset split in detail, but I think it is important in understanding the evaluation approach.

Author response:
**I am not fully convinced by the experiments that the methods are evaluated on the "unseen" tasks. In Tables 2 to 5, what is the separation between the training tasks and evaluation tasks? Are the tasks that appeared in training, not in the evaluation tasks?**

We strictly adhere to the training and evaluation subsets used in HyperTuning for both P3 and S-NI. Our evaluation tasks are entirely unseen during the training process.

For P3, we use the T0 model's training subset, specifically selecting tasks with an average input length of fewer than 320 tokens following HyperTuning. We also use the T0 model's evaluation subset after excluding StoryCloze, as StoryCloze is not publicly distributed. The task taxonomy for both training and evaluation subsets can be found in Figure 2, Section 3 of the T0 paper [[2]](https://arxiv.org/pdf/2110.08207.pdf), which explicitly states, "To test zero-shot generalization, we hold out all constituent datasets... We also verify that data for those tasks is not leaked through the pretraining corpus." You can find the complete list of training tasks in Figure 6 of the HyperTuning paper [[3]](https://arxiv.org/pdf/2211.12485.pdf).

For S-NI, we utilize the training and evaluation split from the Super-NaturalInstructions V2 dataset, excluding non-English tasks. The task taxonomy details are located in Section 5.1 and Appendix G of the S-NI paper [[4]](https://arxiv.org/pdf/2204.07705.pdf), where it states, "We split... into two subsets: one for evaluation and the other for supervision... To avoid data leakage, we exclude tasks from the training set if they are sourced from the same dataset as any test task."

[2] Sanh, Victor, et al. "Multitask prompted training enables zero-shot task generalization." 

[3] Phang, Jason, et al. "Hypertuning: Toward adapting large language models without back-propagation." 

[4] Wang, Yizhong, et al. "Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks."

Reviewer comment (Experiments):
In Table 3, do you have the accuracy of using FlanT5 to initialize the hypernetwork?

Author response:
**In Table 3, do you have the accuracy of using FlanT5 to initialize the hypernetwork?**

We do not have the accuracy of using FlanT5-XL to initialize the hypernetwork due to limited computing resources at this time. We hope to include this result in future for comprehensiveness. However, the gain on FlanT5-Large, which is ~6\%, should be sufficient to demonstrate the superiority of this initialization.