Paper: PATHS: Parameter-wise Adaptive Two-Stage Training Harnessing Scene Transition Mask Adapters for Video Retrieval (ICLR-2024)

Reviewer comment (Experiments):
5.4 ablates the param selection strategies, which is great; also it shows stage-1’s importance in Tab 4. However, it’s still unclear what the performance would be like if we apply stage-2 ONLY to the common param selection strategies (i.e. skip stage-1). This might make it clearer how important stage 1 and 2 is respectively?

Author response:
Thank you for your remarks regarding the clarity of our two-stage method's performance and the potential concerns of overfitting.

Regarding the ablation of parameter selection strategies in Section 5.4, we have conducted experiments using X-CLIP and CLIP4Clip as backbones to assess the isolated impact of the second stage without the first (parameter-wise method). Unfortunately, due to space constraints and the structure of Table 4, these results were not included in the manuscript. When the second stage of our method was applied without the parameter-wise initial stage (with initialization done epoch-wise), we observed a performance of Table below. We recognize the importance of this information and will include these details in the appendix of our revised manuscript for a comprehensive understanding.

| Model | R@1 | R@5 | MeanR |
|-------|--------|-------|--------|
| X-CLIP | 47.7 | 73.8 | 13.4 |
|CLIP4Clip| 43.9 | 72.2 | 15.1 |

Reviewer comment (Reproducibility):
Appendix seems not uploaded?

Author response:
Thank you for informing us regarding the appendix. The appendix was initially uploaded only through the supplementary material. We would like to append the appendix at the end of main draft for better readability. Once again, we appreciate your attention to this matter and assistance in improving our submission.

Reviewer comment (Presentation):
IIUC, the red curve of Fig 3 is comparable with the green curve of Fig 2 (both are X-CLIP eval), but it seems there’s some difference (e.g. in the end of epoch 5)? If my understanding is correct, maybe it would be clearer if we plot the green curve of Fig 2 in Fig 3 as well?

Author response:
We appreciate your observation regarding the comparison between the red curve in Figure 3 and the green curve in Figure 2, both of which represent evaluations of the X-CLIP model. The red curve in Figure 3 illustrates the performance at points selected by our parameter-wise method. As such, it does exhibit differences from the green curve in Figure 2, which represents evaluations conducted at fixed intervals (per 50 epochs).

The x-axis in Figure 2 corresponds to these fixed intervals, whereas the x-axis in Figure 3 corresponds to evaluations conducted five times per epoch, with K set to 5. This difference in evaluation frequency makes a direct overlay of the two curves challenging. However, we recognize the need for clearer comparative visualization to facilitate reader understanding. We will explore ways to integrate these two plots more effectively and update the figures accordingly in our revised manuscript.

Reviewer comment (Writing):
In 4.1 “Two-stage Process”: in the 2nd stage, if params are frozen, why do we still need to “load these parameters back into the model at the end of each epoch to perform the pivoting”? And could you please explain a bit what “perform the pivoting” exactly means?

Author response:
Thank you for inquiring about the process described in Section 4.1 of our manuscript. During the second stage, while the original parameters of the model are indeed frozen, we continue to train the adapter. In this stage, we monitor and pivot the ranking of relative changes within the adapter's internal group of parameters.

Pivoting, in this context, refers to restoring the model's parameters to the state with the highest recorded performance at the end of each epoch. This is achieved by reloading the saved parameters, allowing the model to revert to the optimal point before continuing the training process. This technique helps to fine-tune the model by leveraging the most influential parameter settings discovered during the training epochs.

We will ensure that this explanation is clarified within our manuscript to assist readers in understanding the nuanced training dynamics of our proposed two-stage process.

Reviewer comment (Writing):
Typo in 2.2: “adapters have been *unsed* for progressive learning…”

Author response:
We are deeply thankful for your astute attention to detail concerning the typographical error found in our manuscript. This has been immediately corrected in the text.

Reviewer comment (Reproducibility):
Some details are not clear. For example, how does the ranking work? How does a two-stage scheme w/o STMA work (no adapter and freeze the backbone)? How to split scenes in STMA?

Author response:
Thank you for highlighting areas in our manuscript where further clarity is needed. Regarding the parameter group ranking, it operates as outlined in Section 4.1: Quantifying Dynamics subsection. We monitor the magnitude of change within each parameter group and compare it to the changes in other groups. In other words, Ranking works by comparing the amount of change in the "parameter group". Based on these observations, we use three methods—$BP$, $SP$, and $USP$—to determine the final evaluation points. This process allows us to assess and rank the parameter groups effectively. 

In the case of operating without the Scene Transition Mask Adapter (STMA), our two-stage scheme would not freeze the original parameters but would instead continue additional training. We acknowledge that this was not sufficiently detailed in our initial description and will make the necessary amendments to ensure this is clearly articulated in our revised manuscript. 

As for scene splitting within STMA, this is elaborated in Appendix A.3. We calculate the similarity between video frames and identify where the largest difference in similarity occur, using these points to segment the video. This method allows the STMA to process and learn from the inherent structure of the video content effectively. We will incorporate these clarifications into the revised version of our manuscript to ensure these processes are transparent and comprehensible for all readers.

Reviewer comment (Evaluation):
The method seems to be tricky, which overfits a specific test set.

Author response:
Thank you for your comment on the potential overfitting of our method to a specific test set. We want to clarify that at no point during the training phase do we utilize the test set. Our methodology strictly follows the experimental protocol. That is to say, we strictly adheres to using validation datasets across all datasets, and the parameters selected for final evaluation are those that yield the highest performance on the validation set. The results reported are derived from assessments conducted on the test dataset using these optimally tuned parameters. This approach ensures that our method is evaluated fairly and without bias towards the test set.

Reviewer comment (Theory):
In Page 5, `Motivation` part, `When the model starts to diverge after passing the optimum point, the model parameter values exhibit strong fluctuations. This often involves rearranging parameters in terms of importance (or the value of parameters)`, how can the author get the conclusion?

Author response:
We apologize for the confusion, we originally meant to use "fluctuation" to describe "rank fluctuation".

As elaborated in the Motivation section on page 5 of our manuscript, the statement *"When the model starts to diverge after passing the optimum point, the model parameter values exhibit strong fluctuations. This often involves rearranging parameters in terms of importance"* is predicated on the previous sentences on manuscript: *"The solution we proposed is based on our key observation. When the model converges, parameters in the neural network exhibit stable behavior. Here, we focus on the other way, i.e., when the model diverges."*

Our hypothesis is grounded in the observation that parameters in a neural network stabilize as the model converges. Conversely, we posit that significant parameter’s rank fluctuations suggest divergence, which indicates instability. This premise has shaped our development of the PATHS and guided our empirical efforts to substantiate its effectiveness.

Reviewer comment (Reproducibility):
The elaboration in Section 4.2 on the proposed **Co-Attention** module in STMA is not clear enough. The sentence '*the co-attention layer takes different queries, keys, and values to enable the learning and updating of two pieces of information regarding each other*' is confusing. What are the settings of QKV in your Co-Attention? Considering this module is part of the core designs, more formulation or illustration is needed for better understanding.

Author response:
We appreciate the feedback regarding the description of the Co-Attention module. To clarify, our Co-Attention module is designed to update the values of one modality by taking it as a query and using the keys from another modality. Specifically, when we have two scenes, say $A$ and $B$, scene $A$ serves as the query, and the keys from scene $B$ are utilized to update the values corresponding to scene $B$. In light of your comment, we recognize the need for a more detailed explanation and will update the manuscript to include a clearer description of the QKV settings in our Co-Attention module.

Reviewer comment (Experiments):
According to the paper, the authors set the hyperparameter $K$ as 5. How do the authors choose this value? More ablation studies on $K$ should be conducted.

Author response:
Thank you for your question regarding selecting our study's hyperparameter value set to 5. As mentioned in comment 1 of reviewer 1, We acknowledge the trade-off between the upper bound of performance and the computational overhead that comes with increasing the value of this $K$. We chose $5$ to maintain computational efficiency while capturing a significant portion of the performance gains. We posit that a higher evaluation frequency could further enhance performance in cases with larger training datasets and more learning steps. However, for the scope of our experiments, we found that setting this $K$ to $5$ provided a suitable balance between performance and computational resource management. We appreciate the suggestion for additional ablation studies on this $K$ and will consider this for future research to further elucidate its impact. 

The suggested ablation experiments (different K) are currently underway.  We wanted to respond to other questions first in a timely manner, and would like to come back once we get the results.

---
Below, we continue our response with the results.

We are very thankful for your constructive suggestion. As indicated in the table, there is a trend that performance improves as the size of K increases. However, our objective was not to find the ideal value of K but to identify a value that minimizes computational overhead and can increase the potential of the model. This was the rationale for fixing K at 5 across all experiments. We genuinely appreciate your precise observation and will ensure this point is addressed in our manuscript to aid our readers’ understanding. Thank you.

| Model |Num K| R@1 | R@5 | MeanR |
|-------|-------|--------|-------|-------|
| X-CLIP | Baseline | 47.2 | 73.5 | 13.8 | 
| |K = 3| 48.1 | 73.2 | 13.5 |
| |K = 5 | 48.4 | 73.7 | 13.2 |
||K = 10 | 48.4 | 73.3 | 13.3 |
|CLIP4Clip| Baseline| 43.0 | 71.9 | 15.6|
||K = 3| 44.3 | 72.2 | 14.6 |
||K = 5|44.5 | 72.2 | 14.6 |
||K = 10| 45.0 | 72.3 | 15.0 |

Reviewer comment (Theory):
What if in some certain samples, there does not exist a scene transition or contains more than one transition? Can the proposed STMA be applied to all possible conditions?

Author response:
As detailed in Appendix A.6, our STMA has demonstrated the capability to adapt seamlessly, even in instances where scene transitions are absent or when multiple transitions are present. In cases when there isn’t any scene transition, the query and key within the Co-Attention mechanism would contain similar frames, leading to a learning pattern akin to that of a vanilla attention layer. Moreover, even with numerous scene transitions, the Co-Attention Layer allows the segmented $A$ scenes to attend to the corresponding $B$ scenes effectively. As illustrated in Figure 4, $X$ involves the attention of the "entire video" through a temporal encoder, which prioritizes important frames before concatenation. Subsequently, the alignment attention, which operates similarly to general attention mechanisms, refines the focus as depicted in Figure 9, ensuring the model's attention is directed towards the most relevant scenes. We have endeavored to ensure our model's robustness across various scenarios, and the empirical results support its effectiveness.

Reviewer comment (Novelty):
The contribution of the STMA adaptor seems marginal. For example in Table 1, the gain from STMA is only 0.2 point.

Author response:
We appreciate the reviewer’s observation regarding the perceived marginal contribution of the STMA adapter, as reflected in Table 1. However, it is basically designed as an adapter for transfer learning. If only STMA is applied in stage 1, the STMA module may not seem relatively important. This is because the complexity of the model increases by adding an adapter while the model is not optimized. However, it is different in stage 2. For $w/o STMA$ in Table 1 at stage 2, $R@1$ performs well, but you can see that $R@5$, $R@10$, and $MeanR$ are overfitted compared to the baseline. Our parameter-wise two-stage method pivots based on the $R@1$ performance on the validation set during the second stage, which could indeed lead to overfitting on $R@5$, $R@10$, and $MeanR$. To counteract this, we integrate the “scene” information into the model learning process, which not only mitigates the risk of overfitting but also contributes to an incremental increase in $R@1$.

Reviewer comment (Experiments):
In the ablation section, all BP, SP, USP methods exhibit very similar results. It is hard to tell if one quantifying strategy is better than others, raising a question whether the quantifying strategy is a main component of the proposed method.

Author response:
Thank you for your insightful comments regarding the ablation study and the performance of $BP$, $SP$, and $USP$ methods. It's worth noting that these methods are employed to determine the parameter ranking within our framework. We do not explicitly confirm one strategy over others, as we intend to provide a flexible approach that can accommodate various quantifying methods. We believe many viable methods could be utilized effectively in this context. Our presentation of these methods aims to highlight the empirical observation that monitoring the extent of parameter rank fluctuations can indicate performance improvements at most rank fluctuation points. By showcasing these strategies, we suggest a direction for future research to quantify parameter changes to understand and optimize model performance. This part of our work offers a groundwork for further strategies to develop and refine.

Reviewer comment (Writing):
Typo "raking" --> "ranking" in section 5.4

Author response:
Thank you for pointing out the typographical error. Your attention to detail is greatly appreciated, and we will ensure the correction is made in the revised manuscript.

Reviewer comment (Experiments):
While the proposed method focuses on the retrieval task, would PATHS method also applicable to other video-text tasks such as video captioning or video QA?

Author response:
We thank the reviewer for brining this up; PATHS can be applied to other tasks, where we have tested for video QA in Section A.1 in the appendix. We regret that the appendix wasn't appended at the end of main draft, but uploaded separately through supplementary material.  We apologize for any inconvenience.