Paper: READ: Recurrent Adaptation of Large Transformers (ICLR-2024)

Reviewer comment (Writing):
The writing quality should be improved. For example, Appendix?? and Table ?? in Sec 4.

Author response:
Thank you for spotting these errors. We will revised the paper to correct these mistakes.

Reviewer comment (Experiments):
The experiments are quite restricted, only using T5 on GLUE benchmarks. It would be better to evaluate more model architectures like GPT-style (if LLaMA is too expensive for the hardware setup, maybe evaluate smaller ones like GPT-2) and more tasks.

Author response:
Thank you for your suggestion, we will run this experiment in the revised version.

Reviewer comment (Experiments):
The power statistics from NVIDIA's smi are highly unstable. Have you found the calculation to be stable based on a minute-level sampling?

Author response:
Thank you for highlighting the significance of a reliable energy consumption comparison. To ensure a fair comparison, we controlled for key factors like training code, optimization algorithm, and hardware across all experiments. Lastly, it's worth noting that the methodology we employ for measuring energy consumption using NVIDIA System Management Interface (i.e. nvidia-smi) is not new, but rather widely recognized and adopted. For example, one can refer to the following papers

[1] Eva García-Martín, Crefeda Faviola Rodrigues, Graham Riley, Håkan Grahn, Estimation of energy consumption in machine learning

[2] Kiran Kasichayanula; Dan Terpstra; Piotr Luszczek; Stan Tomov; Shirley Moore; Gregory D. Peterson, Power Aware Computing on GPUs

[3] Gingfung Yeung, Damian Borowiec, Adrian Friday, Richard Harper, and Peter Garraghan, Towards GPU Utilization Prediction for Cloud Deep Learning

[4] Joseph McDonald, Baolin Li, Nathan Frey, Devesh Tiwari, Vijay Gadepally, Siddharth Samsi, Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Models

[5] ROBERT A. BRIDGES, NEENA IMAM, and TIFFANY M. MINTZ, Understanding GPU Power: A Survey of Profiling, Modeling, and Simulation Methods

Reviewer comment (Experiments):
A drawback in figure 2 is that it’s hard to justify the advantage of the proposed READ method by just a single data point. Ideally, it’d be helpful to have multiple data points for READ, under different energy/memory consumptions, in order to build a “Pareto curve”. In this way, it’d be much easier to tell whether READ advances the whole Pareto frontier.

Author response:
(2) Thank you for your suggestion, we will run this experiment in the revised version.

Reviewer comment (Experiments):
In Table 1, it is unfair to only try the proposed method on the larger T5-large. You need to also try other methods + larger T5 in order to prove your points.

Author response:
(3) Given the same memory constraint, one can use read to fine-tune a larger model. We show the comparison between the base models and READ on T5 large to show the benefit of READ on the same memory constraint.

Reviewer comment (Experiments):
As the authors also mentioned in limitation section, it’d be great to add evaluation on GPT-style models and corresponding downstream tasks.

Author response:
(4)  Thank you for your suggestion, we will run this experiment in the revised version

Reviewer comment (Writing):
I find the notion of recurrence in this paper somewhat tricky. Typically, Recurrent neural networks process the input sequentially by taking one token as the input at a time and updating the hidden state. However, in this paper, recurrence does not occur in the sequence level. Instead, it occurs at the model layer level, which seems non-standard for me. I highly recommend the authors to clarify this to avoid confusions.

Author response:
We believe that using the recurency at the model hidden state level is the key behind our method effectiveness. By operating at the model hidden state level, READ does not grow the number of trainable parameters as model sizes increase like other methods, lowering inference cost and fine-tune cost compared to other PEFT methods. The recurrence nature of RNN allows information to flow from one layer to the next, which is why READ outperforms other PETL methods. Last, our
fine-tuning is transformer-free as only RNN and Feed-Forward Network (FNN) structures are used in READ and require no transformer or attention mechanism.

Reviewer comment (Experiments):
The scope of the method and experiment seems narrow. It only considers the encoder-decoder Transformers on a single natural language understanding benchmark. I believe the paper will be strengthened significantly by introducing more Transformer settings (e.g., decoder only) and/or more benchmark datasets (e.g., other NLP/vision tasks).

Author response:
We appreciate the reviewer's suggestions on extending READ to more tasks. In response to the reviewer’s valuable insights, we have extended our experimentation to include the MSCOCO dataset using T5-clip model. Notably, our findings remain consistent: READ consistently outperforms most PEFT baselines, while nearly matching the performance of the Adapter, with 68% reduction in training memory consumption. This is highlighted in the provided table below

|               | Full  | Adapter | LoRA  | Prompt | BitFit | READ  |
|---------------|-------|---------|-------|--------|--------|-------|
| CIDEr         | 91.9  | 83.4    | 69.5  | 65.2   | 33.6   | 81.7  |
| Train memory  | 21.20 | 13.38   | 8.16  | 8.39   | 15.77  | 3.88  |

Reviewer comment (Evaluation):
Training time is not reported in this experiment. I believe it is important to add this Information for a comprehensive comparison among different methods.

Author response:
To provide a comprehensive understanding, we include an analysis below to show the mean and standard deviation for the sums of GPU energy, epochs to convergence, and training time across all GLUE tasks. While this analysis does reveal some variations in energy/time levels, they are not significantly substantial to alter the general trend, as READ continues to stand out as the most energy-efficient approach, with faster convergence than most baselines except for full-tuning. We believe this further supports the merits of our proposed method, and we will include this detailed analysis and clarification in our revised paper.

|              | Full     | Adapter | LoRA   | Prompt | BitFit | LST     | READ  |
|--------------|----------|---------|--------|--------|--------|---------|-------|
| Energy (kWh) | 12.52| 6.99 | 10.58| 6.45| 7.68| 10.59 | 2.06|
| Time (mins)  | 472.74| 485.25| 409.41| 315.35| 291.89| 984.55| 155.11|

Reviewer comment (Presentation):
What does "normalized" mean in the caption of Fig. 1 mean? What are the original results and how are they normalized?

Author response:
Normalized here means to the normalized energy consumption relative to full-tuning on GLUE tasks. With full tuning as 1.0 and the PEFT methods as a fraction of full tuning energy consumption.