Paper: TiC-CLIP: Continual Training of CLIP Models (ICLR-2024)

Reviewer comment (Experiments):
The paper is commendable in its focus as a benchmark study, emphasizing data and existing baselines without introducing new methodologies. The utilization of well-known and straightforward baselines is executed competently. However, the paper could have significantly bolstered its strength by incorporating benchmarks from related works such as Continual-CLIP [1] and the methods outlined in "Robust fine-tuning of zero-shot models" [2]. Including these comparisons would have provided a more comprehensive evaluation, highlighting the benchmark's effectiveness in contrast to existing state-of-the-art approaches.

Author response:
We thank the reviewer for their appreciation of our benchmark study. We would like to clarify that we indeed perform experiments with methods close to both the references the reviewer pointed out. In particular, we perform experiments with Patching (Ilharco et al., 2022) and LwF (Li & Hoiem, 2017; Ding et al., 2022). To clarify: (i) We employ Patching which is a follow up of Wise-FT that adapts Wise-FT to continual learning settings where we observe more than 1 task in sequence; (ii)  We adapt LwF for multimodal models which closely resembles the method in Zheng et al. 2023 (equation 4 performs LwF on both text and vision backbone).

Reviewer comment (Experiments):
An additional area of improvement lies in the paper's scope, particularly concerning the fine-tuning stage. While the study adeptly focuses on the continual pre-training phase, it would have been more insightful and impactful to extend the analysis to the fine-tuning stage. Specifically, investigating how well the obtained checkpoint at a given time transfers to standard vision benchmarks, beyond simple retrieval tasks, would have provided a more nuanced understanding of the model's performance. This consideration is essential for gauging the real-world applicability and adaptability of the continual learning approach outlined in the paper.

Author response:
We perform zero-shot evaluations with 28 existing standard vision benchmarks to evaluate our CLIP models (results are in Table 2 and the standard 28 datasets are listed in Table 4). Our observations highlight that continual pretraining improves on standard tasks (see Figure 1-right and Table 2). Moreover, existing work highlights that zero-shot evaluation is strongly correlated with linear-probe performance (Gadre et al. 2023; Figures 16,17 in Section O). This finding, together with our zero shot evaluation results on 28 datasets, highlight the real-world applicability of continual pretraining for linear probing evaluations. 

If we misunderstood your comment, we request the reviewer to provide clarification. We would be happy to run additional experiments if the reviewer has suggestions. We are also happy to run linear probing experiments if the reviewer suggests that these comparisons would strengthen the paper.

Reviewer comment (Experiments):
This benchmark lacks various types of continual learning methods [1]: elastic weights consolidation methods, progressive neural network methods, dynamic architecture methods, etc. Therefore, the experiments of this benchmark is relative weak and insufficient.

Author response:
While we included references to these papers, we do not compare with other methods such as  elastic weights consolidation methods and  progressive neural network methods because these methods significantly increase the computation cost (refer to Appendix A.1). Several existing works argue that in constrained-compute scenarios, these methods perform poorly than other CL methods, e.g., LwF and Patching, that are computationally more efficient (Ilharco et al. 2022, Wallingford et al. 2023). We did compare with LwF which often performs empirically better than or similar to EWC (Zheng et al. 2023, Ni et al. 2023). Our findings highlight that this method increases the compute and doesn't improve over our baselines. 

Moreover, in our work, we show that simply by increasing the compute budget by 1.2$\times$ for cumulative baseline, we can bridge the gap completely and achieve performance close to or better than the Oracle method. In contrast, methods like EWC significantly increase the compute budget, i.e., at least by a factor of 2 (since we need to compute gradients for two models) excluding the time to estimate the eigenvalues which can also incur significant costs (Ding et al. 2022). 
We also thank the reviewer for sharing the reference on the toolbox. We have now included this in our work.

Reviewer comment (Experiments):
This paper lacks some in-depth analysis of vision-language models solving continual learning. Vision-language models enable various novel model tuning paradigms, such as prompt tuning, vision prompt tuning, parameter-efficient tuning, etc. If these aspects are ignored when discussing the solving of continual learning with the CLIP model, then this benchmark is a bit over-claimed.

Author response:
We would like to clarify and further highlight the distinct nature of our work in comparison to existing research. Unlike the majority of prior studies which primarily focus on task/domain/class incremental setups, our work diverges significantly. In these previous works, such as those by Ni et al. 2023, Ding et al. 2022, and Zheng et al. 2023, the emphasis has been on handling small-scale downstream task data, typically of a very small order than pretraining dataset size. These methods tend to produce models specialized in individual domains. However, in our work, we focus on pre-training (general knowledge acquisition) of the base model with continually evolving internet data. We create the first benchmark with 12.7 billion timestamped img-text pairs that enables systematic study of continual pretraining of CLIP models. This benchmark paves the way for a systematic exploration of the continual pretraining methods  in CLIP models. 

While fine-tuning a limited set of parameters may be an effective strategy for tailoring a model to a confined dataset from a specific domain, this approach is less viable for maintaining and updating "general-purpose" foundational models (e.g., CLIP) with dynamically evolving internet data. In our setup, the proportional increase in new data relative to the pre-existing dataset necessitates comprehensive model updates.

*(New result)* As per your suggestion, we performed an additional experiment to assess the efficacy of updating selected model parameters versus full-scale fine-tuning with sequential training on the TiC-YFCC dataset. In particular, we compare LiT [1] where we perform full-finetuning on the first time step and then perform LiT for subsequent time steps on Tic-Datacomp (M). We observed that while updating a few parameters does avoid catastrophic forgetting on old timestamps, this happens at the cost of reduced performance on all tasks. For instance, the final Imagenet performance drops substantially from 24.0 (w full finetuning) to 7.3 (w LiT).  This highlights limitations of finetuning only a small number of parameters while trying to keep general purpose foundation models up to date. 

Moreover, we would like to emphasize that the main goal of the paper is NOT to introduce a new continual learning technique, rather introduce the first benchmark for continual pretraining where we pretrain foundation models as internet data evolves. For this setup, we present extensive results using standard continual learning approaches that are suitable for pre-training and highlight a simple method that enables continual pretrain of CLIP models without retraining models from scratch when new data arrives.

We would be happy to answer any further questions you have. If you do not have any further questions, we hope that you might consider raising your score.

Reviewer comment (Experiments):
**Sequential and cumulative models behave quite differently between TiC-YFCC15M and TiC-Datacomp** [Critical]  
- The paper nicely illustrates that YFCC15M has strong distribution shifts in Figure 15.  
- However, does TiC-DataComp have significant distribution shifts?  
    - The case for continually training CLIP primarily relies on Datacomp-like data having strong distribution shifts.  
    - I suspect the case there is far weaker than YFCC15M (I am worried it's too small to make this setting exciting).  
- Can the authors create a plot similar to Figure 15 for TiC-DataComp or some other mechanism to analyze distribution shift in TiC-Datacomp?

Author response:
> **W1) Sequential and cumulative models behave quite differently between TiC-YFCC15M and TiC-Datacomp**

We give a brief answer here and provide a details as we answer the sequence of questions below. The primary reasons are two (see Sec B.7 for detailed discussion):

(i) the nature of the distribution shift observed in YFCC and its constructed dynamic retrieval task; we observe that models trained on Tic-YFCC suffer from relatively larger drops due to catastrophic forgetting than Tic-Datacomp (see updated plots for Sequential in Fig 6 in App B.1).

(ii) compute used at each time step per data available at each time step. Overall YFCC is 2x smaller than Tic-Datacomp (M) but the compute we used in both TiC-YFCC and TiC-Datacomp setup is of similar order (in fact, it is slightly higher in TiC-YFCC). We re-ran the experiments for Tic-YFCC by reducing the compute. In the updated runs, we observe that the gap between ID performances of Sequential and Cumulative-All vanishes .  

We also emphasize that the goal of our study is to highlight the problem with the distribution shift for data collected from different data sources (e.g., Flickr, Common Crawl, Reddit). While we agree that the distribution shift severity is different in different data sources, we show that the problem exists for all different data sources. Similar to Figure 4 (right), we have now added heatmaps with other methods and datasets on dynamic retrieval evaluation tasks (Figure 6). For a fixed training time (a specific row), we clearly see performance drop on evaluation time steps after the training time step, highlight distribution shift in these benchmarks.

- > **… does TiC-DataComp have significant distribution shift** 

Yes, we observed that TiC-Datacomp has distribution shift problems as shown by the performance difference between forward transfer and backward transfer (in Table 2, Figure 4 (right) and Figure 6 (c,d)). We clearly observe that performance on dynamic tasks from future time steps is worse than current or previous time steps. Figure 3 presents some examples highlighting how distribution evolves over time in TiC-Datacomp. For example, with time images from novel concepts appear (e.g., COVID-19). 

- > **Can the authors create a plot similar to Figure 15 for TiC-DataComp or some other mechanism to analyze distribution shift in TiC-Datacomp?**

We have included a plot highlighting the nature of the shift in TiC-Datacomp in updated Figure 16. We observe similar trends for FID score on feature distribution for TIc-Datacomp. For TV distance, the distance increases initially and then saturates for later timesteps.

Reviewer comment (Evaluation):
**Further unexplained variations between Sequential and -All**  
- In TiC-YFCC15M and TiC-RedCaps, the performance of forward transfer is significantly affected, but that is not the case in TiC-Datacomp.  
- Similarly, the gap between backward transfer and ID retrieval is far smaller between them in TiC-Datacomp. Why is this the case?

Author response:
- > **In TiC-YFCC15M and TiC-RedCaps, …  forward transfer is significantly affected, but … not … in TiC-Datacomp. Similarly, gap between backward transfer and ID retrieval is far smaller between them in TiC-Datacomp. Why is this the case?** 

We primarily attribute this to the data source differences between these constructed benchmarks. While we agree that the distribution shift severity is higher in TiC-YFCC and TiC-Redcaps, we show that this problem exists even for TiC-Datacomp. We have now included Figure 6 where we plot the performance evaluation matrix $\mathcal{E}$ for all of these datasets.

Reviewer comment (Evaluation):
**Fig1 (mid) and Figure 10 present a different picture than Table 2**  
- Comparing backward transfer retrieval to ID performance for Cumulative exp (oracle) in Table 2  
    - Intuitively, ID performance should be higher (equal).  
- To what degree is the drop in forward transfer attributable to tasks getting harder vis-a-vis encountering new distributions.  
- Hypothesis: Past tasks are easier! Can the authors help shed some light on why the difference?

Author response:
- > **Fig1 (mid) and Figure 10 present a different picture than Table 2. … In Table 2, intuitively, ID performance should be higher (equal)** 

Your observation about the discrepancy in the forward and backward transfer metrics is insightful. Thank you for highlighting this. Originally, our methodology involved averaging the entries in the upper and lower diagonal of our performance matrix $\mathcal{E}. This approach inadvertently results in the backward transfer metric being influenced by later evaluation time steps resulting in backward transfer performance numbers slightly larger than ID performance.

To address this issue, we've revised our metric calculation method by deviating from prior work Lin et al 2021. Now, we normalize the data in each row by subtracting the ID performance as in Díaz-Rodríguez et al., 2018) This adjustment ensures a more balanced and accurate representation across all training time steps. We have also adjusted the forward transfer metric similarly (Table in Appendix ). We have updated the draft with these change (App E). In the final version, we will include these results in addition to the existing results.  

We have also included additional plots that capture the performance matrix \mathcal{E} as in Figure 4 (right) on all datasets in the appendix (Figure 6 in App B.1).

Reviewer comment (Experiments):
**Major results simply not present** [Critical]  
I cannot find any results for LAIONNet-like dynamic classification task. Did I miss something?

Author response:
- > **I cannot find any results for LAIONNet-like dynamic classification task.**

We apologize for missing the table for this result. We have included this in the updated draft and have added a reference to that in the main paper.  The trends described in Sec 3.4 hold true. We have added Table 8 in App B.6 and Figure 15 in C.6.

Reviewer comment (Experiments):
**Missing ablation a)**  
- “Same maximum LR works best across all runs when using cosine schedule” – Didn't find supporting evidence.

Author response:
- >**Same maximum LR works best across all runs when using cosine schedule**

Table 7 in the updated draft provides evidence to this finding (we have added the missing reference in the main paper). Earlier draft also included this table with a typo in one column – we have fixed that.

Reviewer comment (Experiments):
**Missing ablation b)**  
- “Warm up helps training on data from the first time step, but hurts on subsequent time steps.” – After correcting the minor shift issue, the table seems to robustly support the opposite conclusion. This seems surprising to me! I would have not expected warmup to help as the init is a very good one. Warmup mostly mitigates over-updating to high gradients when starting from a poor (random) init in my view.

Author response:
- >**Warm up helps training on data from the first time step, but hurts on subsequent time steps – After correcting the minor shift issue, the table  … support the opposite conclusion.**

Sorry for the confusion here. We copied the column names incorrectly (you can check the mismatch between column names and the corresponding results between Table 6 and Table 2). We have now fixed this table in the updated draft which matches the stated finding in our paper.

Reviewer comment (Experiments):
**Missing ablation c)**  
- “Given a fixed buffer size for each past step, we observe minimal to no difference between random subsampling and other strategies.” – Didn't find supporting evidence.

Author response:
- > **Given a fixed buffer size for each past step, we observe minimal to no difference between random subsampling and other strategies. – Didn't find supporting evidence**

We only performed preliminary analysis for this. In our preliminary experiments, we explored the efficacy of subsampling old data based on the alignment between text and image content from previous time steps. Specifically, when training a model at time step t+1 , we used the model from the end of time step t to assess this alignment. We employed two distinct subsampling methods: 
1. Retaining half of the data with the lowest alignment scores, based on the premise that these data points might be more challenging to learn and require additional gradient steps.
2. Retaining half of the data with the highest alignment scores, under the assumption that these represent higher quality data, as indicated by the stronger alignment between text and image pairs.

We applied these methods to the TiC-YFCC dataset and evaluated their performance against a baseline of random sampling. The outcomes revealed minimal differences: less than 0.2% variation in Imagenet performance and under 0.5% in dynamic retrieval performance across different time steps. Given that these minor improvements came with a significant computational cost—requiring a full forward pass to compute alignment post each training epoch—they exceeded our compute budget constraints. As a result, we opted for random sampling in our research. These findings have been included in the appendix for further reference (App B.4). We leave investigation on improved subsampling techniques for future work.

Reviewer comment (Experiments):
**Serious Design Flaws in Continual Learning Setup** [Critical] — *4-7 timesteps are too few*  
- I am worried the major findings would significantly change when tested on 20/50 timesteps but with the same computational cost per timestep, as the streams become far more non-i.i.d in that case.  
- Why? Because memory buffers become far smaller, making it very hard in my experience to bridge the gap between CL methods and -all.  
- Similarly, I am concerned there will be a far larger gap between -all and Oracle.  
- This critically affects some of the core findings presented in this work.

Author response:
> **W3) Serious Design Flaws in Continual Learning Setup**
- > **4-7 timesteps are too few … major findings would significantly change when tested on 20/50 timesteps**

We acknowledge the reviewer's perspective that extending our analysis to 20/50 timesteps could yield different findings. However, we contend that our chosen 4-7 timestep range (one model a year for up to 7 years) is reflective of certain real-world scenarios, particularly in fields where data is subject to rigorous quality filterings and models are subject to rigorous evaluations between deployments. Such models are expected to adapt to slow changes in the data distribution (year to year) and tolerate faster but small changes (month to month). Examples of such settings include image classification for autonomous driving, where updates are thoroughly tested in controlled environments to ensure safety and compliance with regulations. Even models, like ChatGPT undergo updates at a relatively smaller frequencies (After its initial release about an year ago, GPT-4 was only recently updated with new knowledge).   

Moreover, **our work is  the first benchmark for continual training at the scale of 12.7 B image-text pairs** (e.g., CLOC [Cai et al. 2021] has only 39M samples, see others in Table 4). This initial exploration into the realm of 4-7 timesteps  multi-year web-scale data can pave the way for future work to explore the more challenging settings with 50-60 time steps. We have prioritized the large-scale aspect of our benchmark and tapped the largest sources of publicly available image-text data. Our benchmarks effectively span 14 years from 2008-2022. As such, creating a benchmark with more than 14 years with data prior to 2008 might be infeasible as of now as the scale and quality of earlier data is much less.

Alternatively, our benchmark itself provides a testbed to experiment with finer granularity of months (Sec 2.3). Our dataset release will include time stamped data at the granularity of months. We believe that it would be an interesting direction for future work to explore continual learning at finer granularities in our TiC-DataComp benchmark where we have 70-80 time steps. In such regimes, we imagine that one can explore techniques that alternate between full model updates (which can be done less frequently) and finetuning a few parameters to adapt to more rapid distribution shifts (which can be done more frequently).