Paper: Spaced Scheduling Enhances Instruction-Prompted Reasoning in Large Language Models (ICLR-2024)

Reviewer comment (Theory):
The proposed method does not appear to be motivated with a memory model; compare **Amiri et al.** or **https://arxiv.org/pdf/1602.07032.pdf**, both works seem to motivate their proposals based on a memory model.

Author response:
**Re: memory model motivation**: We have added the memory model discussion to both related work and the introduction.

Reviewer comment (Experiments):
In my opinion, the empirical part should have at least a comparison to another spaced scheduling method (compare **Amiri et al.**).

Author response:
**Re: Comparison to Amiri et al.**: We have added the suggested comparison. The results are in Table 1, and a detailed analysis is in Section 5.1 on page 8. Also, based on feedback from other reviewers, we've added 4 more baselines (Table 1 and section 4.4 for details), enhancing the demonstration of our work's benefits.

Reviewer comment (Novelty):
The novelty claim needs a better positioning with respect to previous work.

Author response:
**Re: Discussion of related work** Based on your comment, we have reworked the entire related work and introduction section as we can't claim novelty on using Spaced Repetition in DL.

Reviewer comment (Experiments):
The empirical investigation feels limited.

Author response:
**Re: Comparison to Amiri et al.**: We have added the suggested comparison. The results are in Table 1, and a detailed analysis is in Section 5.1 on page 8. Also, based on feedback from other reviewers, we've added 4 more baselines (Table 1 and section 4.4 for details), enhancing the demonstration of our work's benefits. **Re not comparing to other approaches**: The main reasons that we did not initially include a comparison to other CL or pruning work is: (1) most prior work either focuses on basic classification task or MNT. Thus comparing these methods means significant changes to 
the original algorithm since older work uses LSTM (such as in Amiri et al.)  MLP, or encode-only transformer models (Bert, Roberta) which is different from decoder-only model used recently.  For example, the implementation of an entropy-based active learning with an 
encoder + classification head is almost distinct from implementing it for a decoder-only. (2) to the best of our knowledge, none of the existing work addresses large scale multitask  (e.g, Flan v2 has 1k+tasks). (3)  to the best of our knowledge, none of the existing work focuses on IFT with large-scale models (7B+) which present other challenges such as the possibility of erasing 
per-trained knowledge as opposed to training an LSTM from scratch to perform sentiment analysis. **However, we now have 5 baselines for comparison**.

Reviewer comment (Experiments):
Could you elaborate on why there is not a comparison to another spaced scheduler or to other approaches that dynamically schedule the training examples? (e.g. **Kreutzer et al.**)

Author response:
**Re not comparing to other approaches**: The main reasons that we did not initially include a comparison to other CL or pruning work is: (1) most prior work either focuses on basic classification task or MNT. Thus comparing these methods means significant changes to 
the original algorithm since older work uses LSTM (such as in Amiri et al.)  MLP, or encode-only transformer models (Bert, Roberta) which is different from decoder-only model used recently.  For example, the implementation of an entropy-based active learning with an 
encoder + classification head is almost distinct from implementing it for a decoder-only. (2) to the best of our knowledge, none of the existing work addresses large scale multitask  (e.g, Flan v2 has 1k+tasks). (3)  to the best of our knowledge, none of the existing work focuses on IFT with large-scale models (7B+) which present other challenges such as the possibility of erasing 
per-trained knowledge as opposed to training an LSTM from scratch to perform sentiment analysis. **However, we now have 5 baselines for comparison**.

Reviewer comment (Experiments):
Table 1: The performance decrease on some tasks (e.g. World Knowledge) might be due to instruction tuning on the Mercury dataset. Since one is interested in comparing instruction-tuning strategies, it might be worth considering using the **LLAMA-2 Mercury**, which is instruction-tuned without spaced scheduling, as the baseline and reporting the gains/losses with respect to **LLAMA-2 Mercury**.

Author response:
**Re: Table 1** While that would make the result a bit easier to read, we believe that will hide the reduction of catastrophic forgetting of our approach. However, we have merged the table into one better clarity.

Reviewer comment (Evaluation):
The experimental results are not so good. Stating that there is an overall performance improvement seems risky because there was a performance gain in four out of seven evaluations.

Author response:
**Re: the experimental results are not so good:**  The reason we claim the overall performance is better is that our method is better on 4 out of 7 tasks. While that might seem ""risky"" we believe that the method is showing signification improvement, because 
   - When our method improves a particular type of task, the gains are substantial. (e.g., 15 acc points for math)
   - The values we show in Table 1 are averages across multiple datasets (e.g., reading comprehension contains 
8 datasets), thus an increase of 1.2 for example is very significant. 
   - On all the benchmarks where our method scored lower than the pre-trained model, our method is always better compared to the random training,  thus the claim that our method improves over random training remains valid. 
   -  On the main capabilities that our method improved (reasoning), we performed a paired t-test to prove the 
statistical significance of our results. Since other reviewers saw our results as strengths  (e.g, Review 1: "Spaced repetition consistently improves accuracy across all 5 benchmarks  and 2 models, demonstrating robust gains.", Reviewer 2: "quantitive results on 4 benchmark  suite show the effectiveness of the proposed methods versus baseline or random mixture methods;"),  please let us know what parts we should clarify, and we will be happy to make the needed updates when we submit the camera-ready version

Reviewer comment (Experiments):
While there are numerous hyperparameters within the proposed algorithm, the impact of their variations on performance has not been analyzed. Some ablation studies on hyperparameters, e.g., $s_t$ and $\rho_0$, seem necessary. This is necessary to determine whether this method is insensitive to hyperparameters.

Author response:
**Re: About the hyperparameters**: Thank you for the valuable suggestion we agree that this was indeed missing from the initial submission. We have added an ablation study of the hyperparameters in the appendix."

Reviewer comment (Theory):
There are often instances where explanations are missing. For instance, it would be advisable for the authors to explain why the formula for the minimum score threshold based on competency is $z_t \leftarrow z_{max} - \kappa -1$.

Author response:
**Re: explanations are missing**: We clarified the score threshold as you suggested in section 3.2, phase 2. Further, we reworked the algorithm section to make sure that all the aspects are clear."

Reviewer comment (Evaluation):
How do you define the minimum score to deem a response good enough and the threshold repetitions to deem an example learned?

Author response:
**Re: The minimum score value, and initial competency**: We clarified how we set that value  in section 4.1 (TLDR; it is based on SuperMemo default value)"

Reviewer comment (Experiments):
Could you explain in more detail how you define the data categories?

Author response:
**Re: the data categories**: We rely on any categorization that exists already in the dataset. A simple categorization is to use the dataset ID (since we only need the value for the stratification). Another possibility is to use categorization in the dataset. For example, the open orca dataset has GPT-4 and GPT-3.5 data splits. In such a case, we use the aforementioned categories
 rather than the dataset ID. We have added this clarification to section 3.2, phase 1.

Reviewer comment (Experiments):
Was there no paper using CL in instruction tuning for LLM previously? If there was, please explain the reason for not comparing with them.

Author response:
**Re CL prior work**: "To our knowledge, there is no CL work for IFT. However, we have reworked the related work section
to add additional studies that intersect with our work."

Reviewer comment (Experiments):
There is a missing comparison in Table 1 versus Tulu as mentioned in 4.2 for the effectiveness of MERCURY versus original Tulu paper. Besides, MT-Bench or other human-involved evaluations might also be good to show the comprehensive effectiveness of the proposed methods.

Author response:
**Re missing comparison in Table 1 versus Tulu**: The only reason why we don't include comparison is Tulu in Table 1 is the fact that Tulu is based on Llama, and our model is based on  LLama2, which gives an unfair advantage to our method since Llama2 is superior to Llama1.  However, with the dataset changes described in the main comment, we have included a Tulu V2 comparison that 
corresponds to training Llama2 on the tulu dataset."

Reviewer comment (Experiments):
Another concern of the proposed method is that it seems the benefits are enlarged for Math/Code with both MERCURY / Space Scheduling. However, when adding OpenOrca dataset only will contribute to that effect should be ablated.

Author response:
**About ablating OpenOrca dataset**: We agree that ideally, we should ablate the addition of each dataset. However, due the amount of work we spent adding 5 baselines and reworking the related work and introduction and  we did not have enough time and compute budget to run these experiments. However, we will be happy to include this in the camera-ready version. Further, we believe that you are bringing up a valuable point and we think that supplementing our method with signals from evaluation sets during training might catch such an issue, where the dropping mechanism can remove examples that might degrade performance. 
However, due to the rebuttal time schedule, we were not able to experiment more with this.

Reviewer comment (Experiments):
The scalability of the proposed method is questionable but it is great to show the purposed methods are based on LoRA.

Author response:
**Re: the scalability of the method**: We've added section 3.3, which shows the math equations that govern the overhead, and complete details of how we derived the equation in Appendix A.6. Further, we added a paragraph (in section 5.1, page 8) in the quantitative results that shows the numerical values, proving that the proposed method is on part with random sampling for 7B and more efficient for the 13B for the tested setup.

Reviewer comment (Evaluation):
Could the authors imply the decreased performance on MMLU / World-Knowledge, is that due to the introduce of OpenOrca or other datasets may pose negative impact on the overall performance?

Author response:
**Re: about the effect open orca** on MMLU, Looking at Table 1 in the Tulu paper, you can notice that Self-instruct is the dataset that affects negatively MMLU. However, we believe that the score drop on tasks that rely on latent knowledge decreases as a function of IFT dataset, that is, there is a point in time that will need to choose between teaching the model a new skill or keeping the world knowledge that might become stale.

Reviewer comment (Experiments):
Baselines: Though it is encouraging that this model improves over vanilla fine-tuning, this is not the only work in improving data quality/curriculum for fine-tuning. In terms of static pruning methods, one naive baseline is to filter sentences with too high or low perplexity (similar to high/low difficulty), as done [for pretraining](https://arxiv.org/abs/2309.04564). In terms of active learning methods, [Data diets](https://arxiv.org/abs/2306.03208) performs a very similar algorithm to the one in this work, dynamically pruning based on a notion of sample importance (the related work of this paper also provides other references). I believe the paper should reflect this in two ways.

Author response:
Re **1-Baselines**: Thanks for suggesting the naive baseline. We have a perplexity-based naive and the data diet baselines. The results are in Table 1, and a detailed analysis is in Section 5.1 on page 8. Also, based on feedback from other reviewers, we've added 4 more baselines (Table 1 and section 4.4 for details), enhancing the demonstration of our work's benefits."

Reviewer comment (Novelty):
The related work and contextualization of the current paper make it seem that this problem has not been studied before, which can be misleading. Better contextualizing the work in terms of prior research in this area will help highlight the novel contributions by this paper.

Author response:
Re **' related work and contextualization**:  We've revised the related work section and parts of the introduction (2nd paragraph). This clarifies our work's position as an extension of existing CL and pruning research and emphasizes our novel contributions."

Reviewer comment (Experiments):
Though the results provide an ablation study, it is unclear where prior work lies in this spectrum. Regardless of the motivation, this paper is providing a new method, and it is important to contextualize its gains with respect to prior work. Though there are too many baselines to evaluate all, I would appreciate seeing a reasonable baseline to confirm that along some axis, this work pushes along various fine-tuning tradeoffs.

Author response:
Re **1-Baselines**: Thanks for suggesting the naive baseline. We have a perplexity-based naive and the data diet baselines. The results are in Table 1, and a detailed analysis is in Section 5.1 on page 8. Also, based on feedback from other reviewers, we've added 4 more baselines (Table 1 and section 4.4 for details), enhancing the demonstration of our work's benefits."

Reviewer comment (Evaluation):
Overhead: This algorithm should induce extra time overhead since examples have to be scored, and "currently difficult" examples may have to go through the model multiple times. The authors should report the time taken by both algorithms to evaluate this slowdown so one can evaluate whether this accuracy improvement is worth the additional cost.

Author response:
Re **2-Overhead**: We've added section 3.3, which shows the math equations that govern the overhead, and complete details of how we derived the equation in Appendix A.6. Further, we added a paragraph (in section 5.1, page 8) in the quantitative results that shows the numerical values,  proving that the proposed method is on part with random sampling for 7B and more efficient for the 13B for the tested setup.

Re **currently difficult examples** note that our dropping mechanism removes these as they become intractable examples if the model is 
failing to predict them after many attempts properly.

Reviewer comment (Theory):
Connection to spaced repetition in psychology: From reading the paper, it is not clear to me how connected the algorithm is to spaced repetition learning for humans. According to the paper, spaced repetition says that "brains retain information more effectively when we learn in multiple, spread-out sessions". However, the actual algorithm proposed does not do this, and frames example selection under dynamic filtering based on example difficulty. Even if the analysis in Section 5.3 implies that the method is implicitly setting a curriculum, it is not clear to me how this connects to spaced repetition. I wonder what value the psychological motivation provides in the context of this work, and if there is a connection I'm missing in this regard.

Author response:
Re: **3 -Connection to spaced repetition in psychology**: We agree that this was not clear in our initial submission. We have updated the related work section (subset Spaced Repetition, 2nd page) to include the work that motivated using the human memory model in DL. We also reflected this in the introduction.

Reviewer comment (Experiments):
Is it possible to see the ablation study for all the benchmarks, or is it only possible to report for MMLU and BBH?

Author response:
Re: **Question 1** -We already provided the benchmarks' ablation. However, the results were split into two tables. For clarity, Table 1 now contains all the results.

Reviewer comment (Experiments):
Is there any ablation result on the importance of (a) doing this dynamically instead of statically with the pretrained model and (b) adding backdropped data into the dataset?

Author response:
Re: **Question 2- (a)**: Due to the amount of work induced by adding the 4 baselines, we did not have a chance to add it. But we will happily include it in the camera-ready version.

Re: **Question 2- (b)**: The backdropped data is added to a training set in the next epoch. However, we agree that this was not clear in the initial submission. We have updated the algorithm to make it more clear. You can see this in Algorithm 1, line 18; we the 
useful examples used in the previous epoch.