Paper: What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinement (ICLR-2024)

Reviewer comment (Evaluation):
The computational efficiency discussion is very good, but reporting actual runtimes for some example settings would also be helpful.

Author response:
We thank the reviewer for thoughtful and positive comments.

**Q: Reporting actual runtimes for some example settings would also be helpful. E.g. tuning the head only should be faster than the whole model, so I think the asymptotic complexity only tells half the story.**

We thank the reviewer for the suggestion. We agree that tuning the head is faster than tuning the whole model, and it is also more efficient to obtain ground truth forgotten examples when tuning the head only (assuming that pre-head outputs of upstream training examples can be cached). We discussed computation complexity of two cases separately in our initial version of Sec. 5.4, and concluded that our forecasting methods have better advantages when tuning the entire model.

We additionally obtained the number of FLOP (number of floating point operations) of forecasting methods as an empirical statistic that measures computational efficiency, which is agnostic to the hardware that we use. We experimented with FLAN-T5-large with 100 examples sampled from each of the 36 pre-training tasks while fine-tuning the entire model.

Method | FLOP 
-- | --
Representation-based | $1.35e^{10}$
Trainable Logit-based | $2.15e^{11}$
 
As a comparison, the FLOP of running inference on a single example is $2.51e^{11}$ flops. Without forecasting methods, obtaining ground truth forgotten examples requires about $9.02e^{14}$ flops. Our forecasting methods are much more efficient.

We have included the results in Appendix D and added pointers where we originally discussed computation complexity in Sec. 5.4.

Reviewer comment (Evaluation):
Reporting actual runtimes will help practitioners consider how feasible this method is for their own application.

Author response:
We thank the reviewer for thoughtful and positive comments.

**Q: Reporting actual runtimes for some example settings would also be helpful. E.g. tuning the head only should be faster than the whole model, so I think the asymptotic complexity only tells half the story.**

We thank the reviewer for the suggestion. We agree that tuning the head is faster than tuning the whole model, and it is also more efficient to obtain ground truth forgotten examples when tuning the head only (assuming that pre-head outputs of upstream training examples can be cached). We discussed computation complexity of two cases separately in our initial version of Sec. 5.4, and concluded that our forecasting methods have better advantages when tuning the entire model.

We additionally obtained the number of FLOP (number of floating point operations) of forecasting methods as an empirical statistic that measures computational efficiency, which is agnostic to the hardware that we use. We experimented with FLAN-T5-large with 100 examples sampled from each of the 36 pre-training tasks while fine-tuning the entire model.

Method | FLOP 
-- | --
Representation-based | $1.35e^{10}$
Trainable Logit-based | $2.15e^{11}$
 
As a comparison, the FLOP of running inference on a single example is $2.51e^{11}$ flops. Without forecasting methods, obtaining ground truth forgotten examples requires about $9.02e^{14}$ flops. Our forecasting methods are much more efficient.

We have included the results in Appendix D and added pointers where we originally discussed computation complexity in Sec. 5.4.

Reviewer comment (Writing):
Writing is quite poor for the methods section. Even after repeated reading, I could not understand how exactly $h$ is being learned for the trainable logit based forecasting. How does one train a LM which maps inputs to h(x), where h(x) is supposed to model the gradient of the model at x wrt $\theta$. I cannot find any details about this or some reference about this.

Author response:
**Q: Writing is quite poor for the methods section. Even after repeated reading, I could not understand how exactly h is being learned for the trainable logit based forecasting.**

We thank the reviewer for bringing up the clarity issue. In the updated version, we added algorithm boxes for logit-based and representation-based forecasting methods to enhance clarity of the methods in Appendix C.

Regarding the question about how $h(x)$ is computed: the logit-based forecasting method requires a kernel $\Theta$ that measures similarity between two examples. The first-order analysis in the beginning of Sec. 3.2 suggests that $\Theta$ should be dot products between gradients of the parameters, which is computationally infeasible, because it requires output length * vocab size backward passes to obtain all required gradients. We therefore propose to approximate with a lower-dimensional learnable kernel $\tilde{\Theta}=h(x)h(x)^T$. Here $h(\cdot)$ is a trainable LM encoder (either BART0 or FLAN-T5, depending on our experiment setup) and $h(x)$ has a much smaller dimension than actual gradients; $h(x)$ is explicitly trained to predict logit updates with a margin loss objective in Eq. 3.

Reviewer comment (Presentation):
Authors should clarify in Table 1 that they are predicting a minority class of forgotten examples in a binary classification setting. Hence F1 scores lower than 50% still make sense. In general, the writeup is quite poor.

Author response:
**Q: Authors should clarify in Table 1 that they are predicting a minority class of forgotten examples in a binary classification setting. Hence F1 scores lower than 50% still make sense.**

We thank the reviewer for the suggestion. We have updated the captions in Table 1 to highlight the skewed nature of the class distribution.

Reviewer comment (Evaluation):
The experiments can benefit from some quantitative results about the computational efficiency, for example in Table 4 what is the overhead of the approach compared to replay w/random.

Author response:
We thank the reviewer for the question. In the new version, we included the number of FLOP (floating point operations)  of forecasting methods in Appendix (assuming 100 examples per pretraining task and full fine-tuning of FLAN-T5-large) and added pointers in Sec. 5.4. We believe the FLOP are good empirical statistics that complement theoretical complexity analysis already presented in Sec. 5.4.

Method | FLOP 
-- | --
Representation-based | $1.35e^{10}$
Trainable Logit-based | $2.15e^{11}$
 
As a reference, without forecasting methods, obtaining ground truth forgotten examples requires   about $9.02e^{14}$ Flop.

Furthermore, the number of replayed examples is controlled across all replay-based methods in Table 4. Therefore, the results above are the only overhead of replaying forecasted examples compared to replay w/random.

Reviewer comment (Experiments):
The authors describe several prior works on forecasting (albeit not in the LM space); it would be interesting to experimentally compare these methods to the proposals.

Author response:
We discussed several related works that analyze features of forgotten examples and learning dynamics in Sec. 6. However, none of them studies the challenge of “forecasting”example forgetting. We believe we are the first to propose the task, set up baselines, and evaluate the performance of forecasting.

Reviewer comment (Evaluation):
The experiment results reveal that the improvement over the baseline is quite marginal.

Author response:
Thank you for pointing this out. We also acknowledged in abstract that logit-based forecasting performs well on BART0 but not on T5 models. 

Nevertheless, the other algorithm we presented, representation-based forecasting, improves over baseline consistently. In fact, the performance improvement is clearer when we replay forecasted examples (Table 4). As an example, we reduce EM drop ratio from 1.489 to 0.301 on FLAN-T5 LoRA.

As a pioneering work on forecasting forgotten examples, we also hope our findings can inspire further algorithmic improvement.

Reviewer comment (Writing):
The relation between the further-training and forgetting is not quite clearly explained.

Author response:
We thank the reviewers for the comments. Our work is inspired by one of many possible definitions of interpretability: the degree to which a human can consistently predict the model’s result [1]. We believe our trainable logit-based forecasting method has made a step towards this definition of interpretability. The approach states that the logit updates on $x_{PT}$ upstream example depend on (1) the logit updates of the online learning example $x_R$ and (2) certain similarity $\Theta$ between two examples. Therefore, our study implies factors such as (1) $x_{R}$ with significant logit change on the label space of the upstream example $x_{PT}$ (2) $\Theta$ that is far from 0 may contribute to forgetting. 

We believe understanding and interpreting forgetting is a challenging problem that requires long-term effort. We will explore more factors that contribute to forgetting in future work.


[1] Molnar, C. Interpretable Machine Learning, 2022

Reviewer comment (Experiments):
Are the predicted samples sensitive to the training order of the same set of samples?

Author response:
We perform additional experiments to address the question from the reviewer. We notice that examples that are forgotten at the end of the stream are not sensitive to the order of the examples. We see the standard deviation of F1 scores of the forecasting methods are small as we alternate the order of training examples (over 5 different orders).

Method | Average(std) forecasting F1
--- | ---
Threshold | 28.91 $\pm$ 0.4
Representation-based | 31.24 $\pm$ 0.9

Reviewer comment (Evaluation):
How the forgotten samples are related to the training data, learning rate, etc?

Author response:
We thank the reviewer for the question. Certain factors, such as learning rate, have a clear effect on edit success rate and EM drop ratio (forgetting) in continual model updates, as shown below for FLAN-T5-large (full fine-tuning).

Learning rate | Edit success | EM Drop Ratio.
-- | -- | --
1e-4 |  95.7 | 24.897
1e-5 (default) |  95.7 | 3.302
2e-6 |  93.5 | 1.820

When we evaluate the forecasting model trained in the default setup on other learning rate setups, we notice that the precision scores almost remain the same, indicating a common subset of examples are forgotten in three setups; while recall scores differ, because a great number of examples are forgotten only when using larger learning rates. We added the new results in the Appendix E.