Paper: Modeling Annotation Delay In Continual Learning (ICLR-2024)

Reviewer comment (Novelty):
There exist overclaim on the contribution of this paper. The problem setting of continuous learning or online learning with label delay is observed and studied over a decades ago and there are many both theoretical and empirical studies on this problem, please see [1,2,3,4] and reference in these papers. There is lack of discussion and comparison with this line of research.

Author response:
The label delay and the literature the reviewer is referring to all fall in the online optimization/learning literature. While there is an overlap with our work that is focused on online continual learning, the setup here is far more challenging. Please read our response in the general comments above on the differences between the 2 domains both in the definition of the problems and the algorithms used. As we explain in detail in [F] existing work does not overlap or does not scale to this regime. More specifically, two of the four suggested works do not even consider time-evolving distributions. The largest dataset of the 4 works [9,10,11,12] consists of 2310 data points with 19 features. Our work is the first one to consider algorithms in the large scale (39M high resolution images) setting with over 10k sample delay and the first one to evaluate backward transfer in the label delay setting.

Reviewer comment (Novelty):
Results in section 4 provides a good validation on the recent images dataset in continuous learning that the multiple gradient descents in historical data may not benefit generliazation on newly arrived unlabled data due to distribution change. However, in my opinion, it is overclaiming to be a contribution as it is a known result. If there are not enough data at each time stamp (typically in continuous learning setting) and lack of well-designed regulazation term, the performance will even degrade with increasing number of gradient descent because the model overfits to such limited number of previous data with different distribution.

Author response:
There is no prior art that reports this phenomena at such scale. While the observed behaviour is expected, we disagree with the reviewer that this is a known result. We do experiments on 17,000x more samples of 2,900x more dimensions than the most complex experimental setup across all suggested works - this inevitably leads to significantly different findings and contributions. Rejecting new findings on the basis of not citing outdated work puts a significant burden on scientific progress.

Reviewer comment (Experiments):
Experimental comparisons are unfair. Self-supervised learning and test-time adaptation methods are designed for the learning scenario where the learner does not have access to historical labeled data, or in some test-time training algorithms, the learner can only modify the training process in the training time. Comparing these algorithms with importance sampling on historical data is unfair, since in the latter case we can obtain and use the ground truth label of the historical data. It is more reasonable to consider rehearsal-based continuous learning algorithms as a comparison.

Author response:
"Self-supervised learning and test-time adaptation methods are designed for the learning scenario where the learner does not have access to historical labeled data, or in some test-time training algorithms, the learner can only modify the training process in the training time." 

TTA and SSL methods do not assume access to prior data however they are particularly tailored to handle domain shift, which is our hypothesized root cause of the degradation in online accuracy. Even though they are targeting this domain shift (using the newest unlabelled data) they are incapable of bridging this gap without labels. Moreover, all TTA and SSL methods we compare against, as stated in Section 5.1 - Experimental setup, in fact are augmented with the memory replay strategy (illustrated in Figure [E]) in which these methods aim to preserve performance on prior data through memory rehearsal while adapting to new data.

Reviewer comment (Experiments):
What is the performance of the proposed importance sampling method compared to rehearsal based continuous learning algorithms in the setting of continuous learning with label delay?

Author response:
We would like to ask for clarification, as in which methods should we directly compare against? 
More specifically, we take the best performing method from [16] namely Experience Replay (ER)[19]. In the reported experiments of [16] (e.g. Figure 3) the rehearsal based continual learning methods like MIR[17] and GSS[18], requested by the reviewer, are the *two worst performing methods* when compared against ER[19] (which is chosen to be the Naïve baseline in our paper). Since our experimental setup is identical to [16], we can directly report these results in our work. Furthermore, in Figure 3 of [16], notice that the performance gap between ER-GSS and ER-MIR in the non-delayed scenario is larger than (in Figure 1 of our paper) the gap between the non-delayed ER and the d=100 ER method.

Reviewer comment (Evaluation):
It's surprising the proposed Importance Weighted Memory Sampling (IWMS) method performs even better than the naive algorithm with no delay. IWMS aims to match the distribution of the selected samples to the newest data distribution, how could a model trained on it outperform a model trained on the newest data with labels? More discussion is needed regarding its exceptional outperformance.

Author response:
It can be due to multiple factors: 
- Refer to our Ablation studies for overfitting on an outdated distribution (also mentioned by reviewer jtG8): when multiple iterations are taken, the Naive baseline might overfit on the supervised data which does not hurt the on-line accuracy when the delay is small but with increasing delay the distribution shift is increased too. This is why even RR can significantly outperform Naive (Figure 5 left in our paper).
- In the original experiments the memory size was limited (40K), please refer to our experiments with 2^19 memory size in the updated paper. We observe that IWMS only converges to Naive w/o delay and cannot outperform it neither on CLOC nor CGLM.

Reviewer comment (Evaluation):
Can you explain why the proposed IWMS performance is so different between CLOC and CGLM? Especially, why CGLM can outperform the Naive algorithm with no delay?

Author response:
Certainly! Our two key observations are:
- There is little to no correlation of the image content to the label on CLOC that may lead to such extensive differences in the results, however evaluations on this dataset is expected in the Online Continual Learning literature. See the supplementary material for examples of extremely low quality images in CLOC. 
- Furthermore, the difference in performance between the two datasets is possibly due to excessive concept drift in CLOC (reinforced by findings in [15]) while close to no concept drift in CGLM. While both dataset show covariate shift, CGLM does not exhibit concept drift (input to a previously different category appears under a new label) whereas CLOC does. For example a sunny beach could belong to Australia during December, while in June a visually very similar image would be a lot more likely to be taken in Spain or the US. As pointed out by reviewer nqA5 and prior work [4], detecting and handling concept drift before observing the actual labels is theoretically impossible. While our method successfully tackles the challenges of covariate shift, in case of concept drift our method recalls samples belonging to all potential candidate classes (regardless of their future ground truth label) which may be the ideal solution to the problem without any knowledge about the future, however is less robust against unpredictable changes in the stream. Such unpredictable changes are more prominent in CLOC, thus the difference in performance.

We will include this explanation in the main paper and provide details in the supplementary materials.
However, if the reviewer is referring to semi-supervised methods that use pseudo-labeling, we have provided further  experiments in Figure [B,C,D].

Reviewer comment (Reproducibility):
In page 8, memory buffer is introduced without detailed description. What samples are included in the buffer, and what's the frequency to update the buffer?

Author response:
The buffer follows the same First-In-First-Out implementation as it does in prior work [14,16,19] which we have already highlighted in the experimental setup, Section 4.1.

Reviewer comment (Novelty):
I think that the critical weakness of the paper is that it does not provide a convincing comparison of its contribution with respect to existing work. There is no discussion of existing work on label delay.

Author response:
4. **it does not provide a convincing comparison of its contribution with respect to existing work.**

We thank the reviewer for bringing up a list of literature. We find the shared work extremely insightful. However, we  argue that the majority of prior art referenced by the reviewer (which we commend) is in fact within the online learning (from optimization perspective) literature as opposed to *continual learning* online/offline literature that is a concern of the paper. The main issue seems to be the contextualization of our work.

Among them, [4] and [8] are the most related to our work. However, they cannot compete in our limited computational budget as highlighted by the paper [4]: “the main challenge in adopting such a strategy to a streaming [online] scenario is that it requires multiple passes over the input data”. 

Pseudo labeling is one important perspective of [B], and we had already done the experiments of using pseudo labeling. The experiments failed since
- The base classifier C has really low accuracy from the beginning, the quality of the pseudo labels were extremely poor, always leading to a feedback loop on training wrong labels.
- The correct predictions often have low confidence scores (top-1 softmax scores), and it makes the pseudo-label selection, proposed in [B], unstable. Furthermore, in the attached Figure [C,D], by increasing the delay it becomes less and less probable that a correct prediction will have a high enough confidence score

We have a more detailed description of each of the literatures in our provided link [G]. Nevertheless, we will cite the ones that address our problem in particular. If the reviewer considers this a relevant baseline to compare against, we are happy to include the full results in the updated version of the paper.

Reviewer comment (Novelty):
The problem of label delay has definitely already been studied in the literature. I think that the statement that no prior work addressed this problem is too strong, and a (big) section of related work should be dedicated to those existing works.

Author response:
4. **it does not provide a convincing comparison of its contribution with respect to existing work.**

We thank the reviewer for bringing up a list of literature. We find the shared work extremely insightful. However, we  argue that the majority of prior art referenced by the reviewer (which we commend) is in fact within the online learning (from optimization perspective) literature as opposed to *continual learning* online/offline literature that is a concern of the paper. The main issue seems to be the contextualization of our work.

Among them, [4] and [8] are the most related to our work. However, they cannot compete in our limited computational budget as highlighted by the paper [4]: “the main challenge in adopting such a strategy to a streaming [online] scenario is that it requires multiple passes over the input data”. 

Pseudo labeling is one important perspective of [B], and we had already done the experiments of using pseudo labeling. The experiments failed since
- The base classifier C has really low accuracy from the beginning, the quality of the pseudo labels were extremely poor, always leading to a feedback loop on training wrong labels.
- The correct predictions often have low confidence scores (top-1 softmax scores), and it makes the pseudo-label selection, proposed in [B], unstable. Furthermore, in the attached Figure [C,D], by increasing the delay it becomes less and less probable that a correct prediction will have a high enough confidence score

We have a more detailed description of each of the literatures in our provided link [G]. Nevertheless, we will cite the ones that address our problem in particular. If the reviewer considers this a relevant baseline to compare against, we are happy to include the full results in the updated version of the paper.

Reviewer comment (Novelty):
There are several relevant papers on the topic of label delay that should be discussed, including but not limited to:  
[A]: Mesterharm, Chris. "On-line learning with delayed label feedback." International Conference on Algorithmic Learning Theory. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005.  
[B]: Gomes, Heitor Murilo, et al. "A survey on semi-supervised learning for delayed partially labelled data streams." ACM Computing Surveys 55.4 (2022): 1-42.  
[C]: Gao, Haoran, and Zhijun Ding. "A Novel Machine Learning Method for Delayed Labels." 2022 IEEE International Conference on Networking, Sensing and Control (ICNSC). IEEE, 2022.  
[D]: Plasse, Joshua, and Niall Adams. "Handling delayed labels in temporally evolving data streams." 2016 IEEE International Conference on Big Data (Big Data). IEEE, 2016.  
[E]: Hu, Hanqing, and Mehmed Kantardzic. "Sliding Reservoir Approach for Delayed Labeling in Streaming Data Classification." (2017).  
[F]: Souza, Vinicius MA, et al. "Classification of evolving data streams with infinitely delayed labels." 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA). IEEE, 2015.

Author response:
4. **it does not provide a convincing comparison of its contribution with respect to existing work.**

We thank the reviewer for bringing up a list of literature. We find the shared work extremely insightful. However, we  argue that the majority of prior art referenced by the reviewer (which we commend) is in fact within the online learning (from optimization perspective) literature as opposed to *continual learning* online/offline literature that is a concern of the paper. The main issue seems to be the contextualization of our work.

Among them, [4] and [8] are the most related to our work. However, they cannot compete in our limited computational budget as highlighted by the paper [4]: “the main challenge in adopting such a strategy to a streaming [online] scenario is that it requires multiple passes over the input data”. 

Pseudo labeling is one important perspective of [B], and we had already done the experiments of using pseudo labeling. The experiments failed since
- The base classifier C has really low accuracy from the beginning, the quality of the pseudo labels were extremely poor, always leading to a feedback loop on training wrong labels.
- The correct predictions often have low confidence scores (top-1 softmax scores), and it makes the pseudo-label selection, proposed in [B], unstable. Furthermore, in the attached Figure [C,D], by increasing the delay it becomes less and less probable that a correct prediction will have a high enough confidence score

We have a more detailed description of each of the literatures in our provided link [G]. Nevertheless, we will cite the ones that address our problem in particular. If the reviewer considers this a relevant baseline to compare against, we are happy to include the full results in the updated version of the paper.

Reviewer comment (Novelty):
How does your paper stand with respect to existing work on label delay? (See Weakness).

Author response:
4. **it does not provide a convincing comparison of its contribution with respect to existing work.**

We thank the reviewer for bringing up a list of literature. We find the shared work extremely insightful. However, we  argue that the majority of prior art referenced by the reviewer (which we commend) is in fact within the online learning (from optimization perspective) literature as opposed to *continual learning* online/offline literature that is a concern of the paper. The main issue seems to be the contextualization of our work.

Among them, [4] and [8] are the most related to our work. However, they cannot compete in our limited computational budget as highlighted by the paper [4]: “the main challenge in adopting such a strategy to a streaming [online] scenario is that it requires multiple passes over the input data”. 

Pseudo labeling is one important perspective of [B], and we had already done the experiments of using pseudo labeling. The experiments failed since
- The base classifier C has really low accuracy from the beginning, the quality of the pseudo labels were extremely poor, always leading to a feedback loop on training wrong labels.
- The correct predictions often have low confidence scores (top-1 softmax scores), and it makes the pseudo-label selection, proposed in [B], unstable. Furthermore, in the attached Figure [C,D], by increasing the delay it becomes less and less probable that a correct prediction will have a high enough confidence score

We have a more detailed description of each of the literatures in our provided link [G]. Nevertheless, we will cite the ones that address our problem in particular. If the reviewer considers this a relevant baseline to compare against, we are happy to include the full results in the updated version of the paper.