Paper: Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors (ICLR-2024)

Reviewer comment (Evaluation):
It is stated in Section 3.2 that sub-par LRA performance is often cited as a prime motivating factor for new methods, but it seems efficiency is often just as important a reason new methods are proposed.

Author response:
```It is stated in Section 3.2 that sub-par LRA performance is often cited as a prime motivating factor for new methods, but it seems efficiency is often just as important a reason new methods are proposed```: We agree that SSMs and transformer variants augmented with SSMs are typically more efficient in terms of compute compared to vanilla transformers and **have stated this point on Page 5, para 4**.

Reviewer comment (Evaluation):
Even when trained with SPT, it seems from the results that methods such as S4 with structured biases consistently outperform methods with less structured biases across almost every task (Text and Retrieval in Table 2 are possible exceptions). The reviewer agrees these differences are not as drastic as it seems when using the traditional procedure, but it still seems the structured biases are helping and the traditional procedure is somewhat predictive of the ordering. Or is this just an artifact of the experiments? A potential discussion on this point could be useful.

Author response:
```...it still seems the structured biases are helping and the traditional procedure is somewhat predictive of the ordering. Or is this just an artifact of the experiments? A potential discussion on this point could be useful.```: We agree with the reviewer that those indeed help and SPT does not render them redundant. Our goal in section 3.3 was to address this exact issue in a controlled setting, evaluating the structured biases in S4, since a comparison directly with Transformers is difficult due to the large differences between models. In the revised version, we included possible implications of different inductive bias between S4 and Transformers as a possible explanation for the remaining gap, mentioned in the beginning of section 3.3, 1st paragraph of page 6.

Reviewer comment (Experiments):
Table 1 lists many efficient attention methods that were originally evaluated on LRA. It would have been interesting to see a couple of these methods also trained with SPT to confirm empirically that they also do not perform as poorly when using SPT. (The reviewer suspects this is the case, but currently has to guess since no result is provided).

Author response:
```Table 1 lists many efficient attention methods that were originally evaluated on LRA. It would have been interesting to see a couple of these methods also trained with SPT to confirm empirically that they also do not perform as poorly when using SPT...```: The purpose of our work was to re-examine the common evaluation pipeline for popular long range benchmarks & therefore we chose to focus on widely adopted models and how performance gaps between them change when SPT is incorporated. While we agree this is an interesting point, efficient attention methods are not as widely-used as vanilla attention and hence we leave their evaluation as future work.

Reviewer comment (Experiments):
The tasks considered in this paper are standard, but the addition of less synthetic or larger scale tasks and experiments would make the paper more compelling to a broader audience.

Author response:
```The tasks considered in this paper are standard, but nonetheless the addition of less synthetic or larger scale tasks and experiments would also make the paper more compelling to a broader audience```: We agree that to prove the efficiency of SPT in general (rather than only on long sequence tasks) a larger study is in order. In the introduction, 1st paragraph page 2, we cite Krishna et al [1], El-Nouby et al [2] that provide such an analysis on standard NLP and Vision tasks respectively, with further details in the related work section. Along with our work, a large number of tasks and modalities demonstrate the efficacy of SPT. Since we observed training from scratch is widespread when evaluating long range sequence models, we focused on these types of tasks and provide a thorough investigation of SPT and its benefits in that context, and how it leads to a more modern evaluation scheme.

Reviewer comment (Reproducibility):
No code appears to be included (it seems it will be made available based on the anonymized link, but it would have been nice to explore during the review).

Author response:
```No code appears to be included ...```: We **have added the link to the codebase** made available through the anonymized link at the end of the introduction in page 3.

Reviewer comment (Presentation):
In Table 2, was the Transformer + Rotary embeddings trained without SPT? This seems unlikely. Perhaps there is a typo in the description of the Transformer methods in this table.

Author response:
```In Table 2, was the Transformer + Rotary embeddings trained without SPT? This seems unlikely. Perhaps there is a typo in the description of the Transformer methods in this table?```: In Table 2 the first line (Transformer + Rotary) is indeed trained from scratch, the difference between this model and in Table 1 is the use of Rotary embeddings, as well as the larger model size for several tasks.

Reviewer comment (Presentation):
In Table 2, the X "denotes computationally infeasible or unreported results". These are 2 drastically different things and 2 separate symbols should probably be used.

Author response:
```In Table 2, the X "denotes computationally infeasible or unreported results. These are 2 drastically different things and 2 separate symbols should probably be used.```: As per your suggestion we **have fixed** this in the revised version.

Reviewer comment (Experiments):
The methods evaluated for Figure 2 still seem to use complex valued parameterizations even though they are randomly initialized. Is this still necessary when using SPT? Since complex values can be problematic when scaling large scale systems, it would be interesting if SPT also removed the need for this in SSMs/linear RNNs.

Author response:
```The methods evaluated for Figure 2 still seem to use complex valued parameterizations even though they are randomly initialized. Is this still necessary when using SPT? Since complex values can be problematic when scaling large scale systems, it would be interesting if SPT also removed the need for this in SSMs/linear RNNs.```: We agree with the reviewer this is an interesting experiment, but leave it for future work.

Reviewer comment (Experiments):
Self-training may be effective not only across the data scale but also the model size. Have you tried the same experiments with different model sizes? It would be interesting to see the phenomenon.

Author response:
1. ```Self-training may be effective not only across the data scale but also the model size```: This observation indeed aligns with our reported results. Our experiments show that original model sizes used in LRA visual tasks (Image, Pathfinder, PathX) are too small for high performance (Table 1) which led us to scale up the model sizes (Table 2), as mentioned in the 1st paragraph of page 5, section 3.2. To further investigate this, **we plan on adding a scaling experiment as suggested in the next revision**, comparing performance across model sizes for SPT and trained from scratch models.

Reviewer comment (Experiments):
Regarding the experiment about pretraining on text corpora: I appreciate the idea of comparing results with a pretrained model on a large language dataset. However, the current results are meaningless since the downstream task datasets are very different from Pythia 70M. I'm not sure we are able to find the right dataset to cover all the tasks for Long Range Arena. What about separating this experiment from Long Range Arena and showing the comparison for another downstream task on the language domain?

Author response:
3. ```The current results are meaningless since the downstream task datasets are very different from Pythia 70M. I'm not sure we are able to find the right dataset to cover all the tasks for Long Range Arena. What about separating this experiment from Long Range Arena and showing the comparison for another downstream task on the language domain?```: The purpose of Pythia experiment is to see if pretraining on text is helpful on LRA, as has been observed on various modalities such as molecular data, tabular data, code and speech [1,2,3]. The issue of finding a unifying dataset that can be used for pretraining across modalities seen in LRA is indeed difficult, as mentioned by the reviewer, and we view SPT as a remedy for it.  For text-based tasks specifically, the question of evaluating the effectiveness of SPT vs. pretraining on large text corpora is discussed thoroughly in Krishna et. al. [4], showing that in many cases SPT rivals PT on large corpora as mentioned both in the 1st paragraph of page 2, and the related work section. Due to their comprehensive analysis, we focused on SPT in the context of evaluating models on long-range benchmarks. Nonetheless, for a more reliable comparison **we have added an additional baseline of **Pythia with random initialization** in the revision**, denoted as Pythia 70M (Rand Init), and **expanded the discussion in section 3.5** following these addition.  The randomly initialized baseline from which clearly shows the benefits of text pretraining are more easily observed. Yet in some cases, the pretrained model fails to significantly outperform the randomly initialized model, unlike SPT, which provides benefits across the board.

Reviewer comment (Experiments):
Are S4/Transformers trained with the same number of epochs as S4+SPT/Transformers+SPT (including pretraining+finetuning)? If it's not, these results should be added. I'm asking to make sure the single-trained models are not undertrained.

Author response:
4. ```Are S4/Transformers trained with the same number of epochs as S4+SPT/Transformers+SPT```: This question was raised by all reviewers - please see our unified response above.

Reviewer comment (Experiments):
What is the point of the experiment with Pythia? For instance, what is the single Pythia row in Table 2 to be compared with?

Author response:
3. ```What is the point of the experiment with Pythia?...```: We thank the reviewer for pointing to the lack of clarity in the Pythia section. It is common practice in various areas of ML to take LLMs pretrained on text and directly adapt them to other modalities such as molecular data, tabular data, code and speech - researchers have reported significant benefits from this approach [1, 2, 3]. In the Pythia experiments, we wanted to examine if pretraining on a large text corpus would result in large gains on LRA or whether maintaining the modality between PT and FT tasks is important. We agree that comparing the Pythia results to the available Transformer results makes comparison difficult due to different architecture and model sizes. Therefore, in the revised version, **we have added an additional baseline of Pythia with random initialization**, denoted as Pythia 70M (Rand Init) and indeed observe benefits from pretraining on text, yet not on all tasks, which shows the importance of SPT leading to to better performance across the entire suite.

[1] Igor Melnyk, Vijil Chenthamarakshan, Pin-Yu Chen, Payel Das, Amit Dhurandhar, Inkit Padhi, Devleena Das, Reprogramming Pretrained Language Models for Antibody Sequence Infilling.  
[2] Herzig et al., TAPAS: Weakly Supervised Table Parsing via Pre-training.  
[3] Ao et al., SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for
Spoken Language Processing.

Reviewer comment (Evaluation):
Figure 3, in the smallest data setting it seems like SPT does not provide monotonically increasing gains as the data size reduces. Is there guidance on what scale of data SPT ends up resulting in poor initializations that the ones used in state space models.

Author response:
```... on what scale of data SPT ends up resulting in poor initializations that the ones used in state space models``` :  We clarify that the plots in Figure 3 show **relative gains** offered by SPT over trained from scratch baseline and for performing SPT standard S4 initialization was used. At extremely low data regimes (0.5% of the original data), the absolute model performance after finetuning on the downstream task is low and hence the relative gain offered by SPT is not significant - however it is not negative, i.e., **not inferior to the trained from-scratch baseline**. 0.5% of the original data amounts to 225 samples for the Image task and 125 samples for the Text task. As a general guidance, when performance on the pretraining task itself approaches random accuracy on the validation set, we would expect SPT not to be beneficial, but even for 125 samples in Text task, pre-training accuracy is ~ 50% which is significantly higher than chance performance for character-level language modeling. We have added the original dataset sizes to the caption of Figure 3.