Paper: Time-Varying Propensity Score to Bridge the Gap between the Past and Present (ICLR-2024)

Reviewer comment (Novelty):
The first weakness comes from how this work is situated in the literature, specifically the effort to distance this work from related work in continual learning. Specifically, on page 3 in Remark 2. I would argue that your setting is exactly encapsulated by continual learning frameworks. For a framework https://arxiv.org/pdf/2012.13490.pdf is a good resource. Specifically, definition 1 is a good starting point. While these definitions take a reinforcement learning perspective, they have counterparts in supervised settings as well, and can be readily adjusted to the supervised setting.

Author response:
We think there is a simple misunderstanding here. We don’t try to distance our work from continual learning literature but rather we study a different problem and there is a very clear distinction between what we do and continual learning. In particular, we focus on the problem of “how to select which past samples are similar to the current distribution of data”. We use this to predict only the immediate next task, namely $p_\{T+1\}$. Continual learning focuses on how to update a model to a new task, and it seeks to predict accurately on all past tasks. The two are totally different problems, but both are important and valuable problems. Further, we have 100s of tasks in our experiments, current continual learning methods are ill-suited to address such a large number of evolving tasks.

Reviewer comment (Writing):
I’m not sure I’m understanding the construction of your dataset D, and subsequently p(t). To me, each datum is received sequentially and therefore t_i is unique for each entry. This means p(t) is just a uniform distribution for the number of timesteps (of course this is not entirely accurate as t is assumed to be from the reals and not integers, but in effect this is what I understand). Given how you use this distribution, I’m assuming this is not the case. But it is not clear what t is here, how does t relate to how the data was generated? How is your dataset constructed? This confusion continues throughout the paper including all of section 2, and when defining objectives in section 3.

Author response:
Let us repeat a bit more elaborately the construction in Section 2.1. Suppose the dataset is $D =\\{x_i, t_i\\}^N_i$. The time $t$ is a modeling choice, it can be an integer or a real number. The time $t_i$ simply refers to the time at which a datum was recorded, say the Unix time. Let us suppose it is a real number, like we have done in the paper. If each datum is received at a unique time then all the $t_i$s are different. Like we described in the paper, the distribution $p(t) = \frac{1}{N} \sum_{i=1}^N \delta_{t_i}(t)$, i.e., the empirical distribution of the times recorded in the dataset. It is important to note that some times may not have any samples; this is not an issue when we take expectations/integrals over time $t$ because the distribution $p(t)$ will have zero probability over such times. If multiple samples were recorded at some instant, there would simply be two Dirac deltas in this summation. All the theory remains the same.

Reviewer comment (Writing):
The three lines before remark 4 are very confusing, and don’t clarify how \( g_\theta(x,t) \) is estimated. Do you mean t’ instead of s’? This comes up again after remark 4 under equation 10. It is not clear how the dataset is being generated to estimate \( g_\theta \), and the subtle difference from equation 8 & 9 is completely unclear (yes different distributions of the data are used, but how). It is very possible Algorithm 2 will clarify this, but I think a clarification on what the time in the original dataset (see 1) will deal with a lot of the confusion here.

Author response:
Thanks, yes, s’ should be t’. This has been fixed. Please see the response to the following question.

Let us explain algorithm 2 to clarify how data is generated.
 
To train our propensity score, we need to create a triplet $(x_j,t',t)$ for each of the samples in the dataset. In particular, if $t_j$ denotes the true time from which $x_j$ is observed, then we want our training data to have $t' = t^j$ ( and $t$ equals to a random time) in half of the training examples with label 1 and $t'$ equals random time in the other half (with $t =t_j$) with label -1.  Once this dataset is generated, the rest is to train a binary classifier using equation 19.

Please let us now if this is still unclear, happy to provide more explanation.

Reviewer comment (Evaluation):
You did not explain how relative accuracy is defined. I’m left to assume from the first line of figure 1’s caption that they are normalized by the performance of the everything benchmark, but how is not stated in the main paper. If this is normalized according to the accuracy of the everything baseline, I think these results might be misleading. The paper would benefit from a discussion on how well the baseline is actually performing to give more perspective on the other method’s performance. It is also unclear how the models are being tested in these experiments. Are you testing the accuracy of the model on the data generated at t+1? How is this data gathered? Is this same data then added to the dataset? There are many details missing, likely related to the confusion mentioned above.

Author response:
Yes, the caption defines the relative accuracy which is a standard metric in literature to present such results. We’ll repeat the exact same definition in the main text paper as well. Our result simply states that if we compare all methods with Everything how much performance decreases or increases with respect to the Everything baseline.

The reviewer's suggestion that the results might be misleading is perplexing to us. Could you please elaborate on this point? Does the reviewer suggest that because relative accuracy is not mentioned in the body of the paper but only in the caption?

Considering the extensive number of different experiments presented in the paper, we believe that providing detailed metrics and findings within the respective captions offers a more engaging and comprehensive reading experience.

We are not sure if we understand the reviewer's comment here. We believe the figures clearly show how well the different methods perform compared to each other, and provide a complete overview of their effectiveness.

Yes, we test and evaluate a trained model on unseen data (i.e. not seen during training time) at time $t+1$.  This has been explained in many places in the paper, namely remark 2, section 3.1, and the last paragraph of introduction. 

As detailed in section 3.1, each data point in the plots for the supervised learning experiments corresponds to a distinct experiment. We train a separate model from scratch for each experiment, leading to a varying number of experiments per dataset. For instance, the Gaussian dataset involves 160 distinct models and experiments, while the CLEAR dataset encompasses 500 separate experiments and models!
Please also see our response above about “Data generation”.

Reviewer comment (Reproducibility):
How were the hyperparameters chosen for your method and for the competitors? This question applies to all experiments as I don’t see it mentioned in the paper.

Author response:
Our method has the exact same hyper-parameters as those of Everything. Note that Fine-tune and recent use the exact same hyper-parameters as those of Everything except that they have different learning rates (this is because the lower learning rate for fine-tuning works better) and number of training epochs per time step given these are different methods than Everything. Table 1 and Table 2 in the appendix contain this information. We conducted a minimal random hyper-parameters search for the experiments using a validation set (based on Everything) in this paper and we mostly follow standard and readily available settings for the experiments whenever it is applicable. We will update section D of appendix to provide more details about the hyper-parameters and also release the code upon publication.

Reviewer comment (Experiments):
How does the amount of computation for your method relate to the amount of computation for the baselines (for both training and inference)? I notice that your method has a larger batch size than the baselines for both the reinforcement learning and supervised learning settings. Why was this chosen?

Author response:
No, our method and baselines use the exact same batch size. For RL experiments, we use the exact same batch size as SAC which is 256 and for supervised learning we use 128 as shown in Table 1 and Table 2. The batch sizes for fitting the propensity score is 512 in both supervised and reinforcement learning. We would like to note that **all** training details are exactly the same between our method and and the base method.

Reviewer comment (Evaluation):
In the reinforcement learning experiment, what are the shaded regions signifying? Also, these should be made darker. Currently it looks like there is significant overlap in both the experiments meaning there is not a statistically significant difference between your method and SAC.

Author response:
Following RL's literature for continuous control, we used 10 different random seeds to average each RL experiment, and the shaded areas represent the standard deviation across 10 random seeds. While some instances exhibit overlapping shaded areas, our method consistently demonstrates lower variation across runs, indicating its better stability. This is particularly crucial for methods susceptible to instability due to distributional shifts between the policy used for data collection and the policy undergoing optimization (a.k.a. off-policy methods in RL). Our results effectively demonstrate the ability of our method to lessen the detrimental effects of off-policyness.

Reviewer comment (Experiments):
The baselines considered for the supervised learning experiments are reasonable, but the RL experiments don't consider these baselines. As written, it's unclear if the time-vary propensity score is any better than simpler methods. Also, ROBEL DClaw is missing the SAC + Context baseline.

Author response:
>**The baselines considered for the supervised learning experiments are reasonable, but the RL experiments don't consider these baselines. As written, it's unclear if the time-vary propensity score is any better than simpler methods.** 

Thanks for the question. The distinct nature of RL and supervised learning methods necessitates modifications to the underlying RL algorithm to accommodate the same baselines as supervised learning. In particular, SAC, an off-policy algorithm, utilizes a replay buffer to store data during exploration and samples uniformly during training. Implementing a baseline like fine-tune in SAC necessitates an additional replay buffer that only retains recent data, introducing further alterations and new hyperparameters to the SAC algorithm. In our RL experiment, we design experiments such that it requires minimal changes to the original RL method and it doesn't change how the underlying RL algorithm works as our goal in this paper is to show rather applicability and efficacy of our method in RL, rather than proposing a new RL algorithm.  
 
>**Also, ROBEL DClaw is missing the SAC + Context baseline.** 

No, it is not missing. We already included this in the Figure 11 of appendix.  We also have another additional experiment for ROBEL DCLAW where tasks are sampled differently to show the effectiveness of our method in another setting. Figure 10 shows the result of this experiment. This experiment is another piece of evidence that shows the effectiveness of our method.

We also like to note that when we combine the SAC or SAC + Context with our time varying propensity score, we see that our method consistently demonstrates lower variation across runs, indicating its better stability ( see Figure 10 and 11 for examples). This is particularly crucial for RL methods susceptible to instability due to distributional shifts between the policy used for data collection and the policy undergoing optimization (a.k.a. off-policy methods in RL). Our results effectively demonstrate the ability of our method to lessen the detrimental effects of off-policyness of off-policy methods like SAC.

Reviewer comment (Presentation):
The shaded regions for the RL curves should be darker; as of now, they are very difficult to see.

Author response:
>**The shaded regions for the RL curves should be darker; as of now, they are very difficult to see.**

That is a very easy fix and we've now updated the figures.

Reviewer comment (Writing):
Section 2.1: "for some time large final time T"

Author response:
>**Typos.**

Thank you. We have fixed these issues in the paper

Reviewer comment (Writing):
Missing right parenthesis after Eq. 4: TV(p_{T+dt}, p_T)

Author response:
>**Typos.**

Thank you. We have fixed these issues in the paper

Reviewer comment (Writing):
Section 2.2 last paragraph: "is subtle when deals with data"

Author response:
>**Typos.**

Thank you. We have fixed these issues in the paper

Reviewer comment (Writing):
Just before Remark 5: "We call it “Finetune” make the objective"

Author response:
>**Typos.**

Thank you. We have fixed these issues in the paper

Reviewer comment (Writing):
Could the authors explain the CIFAR shifts described by Eq. 21? I'm hoping for a plain English description of the shift.

Author response:
>**Could the authors explain the CIFAR shifts described by Eq. 21? I'm hoping for a plain English description of the shift.**

This formula describes the label shift used in our experiments. At each time t, the task consists of images from two particular classes, say class 6 (10% samples) and class 7 (90% samples). In this case, at the step (time t+1) causes a new class to be added to the task, i.e., it consists of class 7 (90%) and class 8 ($10%$). At time t+2, the task consists of class 7 (80% samples) and class 8 (20% samples). At any time t, the test data consists of images sampled from the task at time t+1. It is important to note that there is a clear distribution [label] shift between the train and test distributions due to this construction and therefore it is an extremely direct test of our proposed approach. This shift is also gradual, in some cases the label distribution changes and in some changes a new class is introduced, but the new class only accounts for 10% samples at the instant of introduction. In simple words, the data distribution consists of two classes and the support of this distribution shifts over all the 10 CIFAR-10 classes, as it gradually undergoes label shifts.

Also, there is a typo in that formula where H should be T.  This has been fixed.

Reviewer comment (Theory):
The assumption that data evolves gradually which might not be true in many real world situations and limits the scope of application of the work. Consider for e.g. the sudden shifts in the data distribution. Would the proposed approach be extendable to such settings? What do we need to cover this setting as well?

Author response:
While we agree with the reviewer that there are other kinds of shifts that can change data, the gradual shift assumption is relevant for many real-world scenarios where data is collected from evolving distributions that change over an extended period. For instance, drift in environmental variables such as temperature caused by day/night or seasonal changes affects both the forecasting systems and the systems that collect the data [1]. There are also many examples in oceanography, e.g., currents and other relevant oceanographic variables such as salinity and temperature in a coastal area change slowly with time [2,3] and modifying the forecasts for these variables is challenging because of the drift in the data. Even for problems in healthcare, there are many situations where data drifts gradually, e.g., the different parts of the brain atrophy slowly as the brain ages for both healthy subjects and those with dementia; tracking these changes and distinguishing between them is very useful for staging the disease and deciding treatments [4,5]. Viruses can also mutate and drift over time, which can make them resistant to existing treatments [6,7,8]. We hope that this addresses your concern about the motivation of the problem: gradual drift data is indeed a problem with broad applications. There is enormous economical and intellectual value in addressing graduate data drift.

[1] Learning under Concept Drift: A Review. IEEE Transactions on Knowledge and Data Engineering, vol. 31, no. 12, pp. 2346-2363, 1 Dec. 2019, doi: 10.1109/TKDE.2018.2876857.

[2] Operational Forecast System, Center for Operational Oceanographic Products and Services. https://tidesandcurrents.noaa.gov/models.html

[3] Changing sea levels: effects of tides, weather and climate. David Pugh. Cambridge University Press. 

[4] Saito Y, Kamagata K, et al., Temporal Progression Patterns of Brain Atrophy in Corticobasal Syndrome and Progressive Supranuclear Palsy Revealed by Subtype and Stage Inference (SuStaIn). Front Neurol. 2022. 

[5] Rongguang Wang, et al.,  Adapting Machine Learning Diagnostic Models to New Populations Using a Small Amount of Data: Results from Clinical Neuroscience. 2023. https://arxiv.org/abs/2308.03175

[6] Ewen Callaway. The coronavirus is mutating — does it matter?, September 2020.

[7] William T Harvey, et al., COVID-19 Genomics UK (COG-UK) Consortium, et al. Sars-cov-2 variants, spike mutations and immune escape. Nature Reviews Microbiology, 19(7):409–424, 2021.

[8] C.J. Russell. Orthomyxoviruses: Structure of antigens. In Reference Module in Biomedical Sciences. Elsevier, 2016. ISBN 978-0-12-801238-3.

Reviewer comment (Evaluation):
Evaluating on test data seems very counterintuitive as typical settings would not have this privileged information except during training. Can you provide how this specific choice of evaluation can be overcome? I find it quite limiting to do so.

Author response:
There seems to be a simple misunderstanding here. We are not sure why the reviewer finds the test setting to be counterintuitive and why they think our method has access to privileged information during the test time which is otherwise only available during training time. Could you please provide more details on these matters so that we can address their concerns effectively?


In the meantime, let us clarify further the test and training time settings. For the purposes of calculating the propensity score, we create the datasets $\{(x,+1), (x’,-1)\}$ using the training samples. This estimator is run on all the training samples from $p_T$ and $p_t$ for $t < T$ to calculate the propensity of each of the $p_t$ with respect to $p_T$. **No test samples are ever used during training time (as is appropriate)**. A model is trained using the weighted version of the dataset (which now has samples from t = 0 to t = T) and is used to make inference on samples from $p_{T+1}$. No samples from $p_{T+1}$ are used to train the propensity estimator or the model. 

Moreover, we emphasize that there is no information about when, where, or what shifts happen in the past, present, or future during both training and test times and our method doesn’t know them as prior knowledge. More importantly, this is why it is remarkable that our method can identify continuously changing data distributions without knowing when or how the distribution changed.

Reviewer comment (Experiments):
How is the `Finetune` baseline different from typical online-meta learning approaches? Finetune typically would continue learning a hypothesis on all past tasks and then adapts it further using data from the new task at hand that is in round T $p_T$, for e.g. in [1] and [2]. Is it primarily that in your method the desiderata is essentially different i.e. to obtain a small risk on pT + dt — as opposed to obtaining a small population risk on all tasks. Is it possible to get the best of both worlds instead i.e. get small risk on the latest task/future task while maintaining low risk across ALL tasks that the agent has seen, as otherwise there is a recency bias to only caring about the immediate next data point.

Author response:
That is a great question. The main difference between Finetune and online meta learning is that online meta learning methods are formulated as meta-learning problems and they use test data for the adaptation. For the fine-tune baseline, we don’t have access to the test data to do adaptation and we don't formulate our problem like a meta-learning objective. 

Exactly, that is correct. Afterall, that is the most realistic problem in situations where the data distribution changes; the model must work well for the new data, not on average over all past data.

That is certainly possible and that is a good suggestion. In this work, we mainly focus on the setting where the problem is not formulated as a meta-learning objective and there is no access to the test-data to do adaptation which is closer to real-world setting. Our results confirm the effectiveness and applicability of our proposed method. However, combining our method with meta-learning based techniques can be a future research direction, but that is beyond the scope of this paper. In general, one should expect we cannot both have a small risk on the latest task and small risk on all past tasks, a number of papers have argued this, e.g., [10]. There can be situations when the tasks have solutions that are very similar to each other, and in such cases one will indeed have small risk on the latest task as well as past tasks (but there is also no effective distribution shift in such problems). 

Finally, we would like to bring the reviewer’s attention to the experiments in Figure 4 of the appendix where we adapted the "Meta-Weight-net" method of [9] to our setting and ran experiments comparing it to our method. As shown in Figure 4 of the appendix, our method outperforms Meta-Weight-net significantly, which provides additional evidence of the efficacy of our proposed method.

[9]  Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, Deyu Meng Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting,. NeurIPS 2019.


[10] Rahul Ramesh, Pratik Chaudhari, Model Zoo: A Growing "Brain" That Learns Continually,. ICLR 2022.