Paper: Unsupervised open-vocabulary action recognition with an autoregressive model (ICLR-2024)

Reviewer comment (Writing):
Presentation of Methodology: My primary concern lies in the method section's exposition. While the core idea of the proposed method is intuitive and straightforward, the section seems overburdened with intricate notations. This makes it somewhat challenging to follow. A more streamlined and intuitive presentation could significantly enhance the paper's readability and impact.

Author response:
Thank you for your feedback. We will improve the clarity of the exposition further by the camera ready.

Reviewer comment (Reproducibility):
While the author mentions that video-video similarity scores, leveraging the frozen CLIP encoder, can be pre-computed and re-utilized throughout training, Algorithm 1, line 12, indicates these scores are calculated during training. Could you clarify this discrepancy?

Author response:
Line 12 is executed every R-epochs, when the pseudo-labels are reassigned. You are correct, as only the text changes, the remaining features and scores that don't change (i.e. video-to-video) are cached in practice during the 1st run. We will clarify this in the text.

Reviewer comment (Reproducibility):
Regarding the CLIP image encoder's usage for retrieval, is there any temporal modeling involved? Is the process simply averaging the frame-wise representation?

Author response:
Yes, that's correct. For the type of videos used (which are 3-15 seconds) this isn't generally a problem. We explored multiple avenues on this (e.g.: aggregated scoring on a frame-by-frame basis, using X-CLIP video encoder etc), but found the performance to be similar.

Reviewer comment (Experiments):
Is it feasible to concurrently train the video retrieval model?

Author response:
Thank you for your question and suggestion. We believe that it’s possible to train a retrieval component too, following BLIP modeling for example, where the LM is used both as an encoder and decoder. In this work, we only focused however on the generative component, adapting it for action recognition.

Reviewer comment (Presentation):
While the paper offers valuable contributions, oversights such as typos, inconsistency, and table misalignments detract from its overall quality and can be distracting for readers. Maintaining consistent formatting throughout the paper can greatly enhance its quality.

Author response:
We will improve this further, incorporating your suggestions. We note however that Fig. 1 describes the overall framework at a high level, while Alg. 1 already offers an exact description of how our method works. We will also address all the remaining cosmetic issues, carefully proofreading the paper.

Reviewer comment (Presentation):
The method is primarily described in prose, making it somewhat challenging to grasp. Incorporating relevant figures could make the content more accessible and easier to follow.

Author response:
We will improve this further, incorporating your suggestions. We note however that Fig. 1 describes the overall framework at a high level, while Alg. 1 already offers an exact description of how our method works. We will also address all the remaining cosmetic issues, carefully proofreading the paper.

Reviewer comment (Writing):
In Section 5's second paragraph, avoid using abbreviations for "resolution."

Author response:
We will improve this further, incorporating your suggestions. We note however that Fig. 1 describes the overall framework at a high level, while Alg. 1 already offers an exact description of how our method works. We will also address all the remaining cosmetic issues, carefully proofreading the paper.

Reviewer comment (Presentation):
For Table 2, the emphasis (bold) should be on 50.1. Row entries should maintain consistency, e.g., "RISE(ours)."

Author response:
We will improve this further, incorporating your suggestions. We note however that Fig. 1 describes the overall framework at a high level, while Alg. 1 already offers an exact description of how our method works. We will also address all the remaining cosmetic issues, carefully proofreading the paper.

Reviewer comment (Presentation):
In Table 4 (b), the value 49.8 of the column for "$N_I=4$" is likely a typo since the best result is reported as 49.7, and the bold formatting for 49.8 should be removed. This may cause misunderstanding of the experiment result.

Author response:
We will improve this further, incorporating your suggestions. We note however that Fig. 1 describes the overall framework at a high level, while Alg. 1 already offers an exact description of how our method works. We will also address all the remaining cosmetic issues, carefully proofreading the paper.

Reviewer comment (Presentation):
In Table 8, the partition line should begin at the number 2.

Author response:
We will improve this further, incorporating your suggestions. We note however that Fig. 1 describes the overall framework at a high level, while Alg. 1 already offers an exact description of how our method works. We will also address all the remaining cosmetic issues, carefully proofreading the paper.

Reviewer comment (Reproducibility):
Would the authors be open to releasing the supporting code for their paper?

Author response:
Thank you, we aim to release it upon acceptance.

Reviewer comment (Evaluation):
The performance gap between the proposed method and discriminative methods with similar pretraining models (like X-CLIP) is huge (on the standard zero-shot benchmark, 65.2 vs. 53.0, Table 6).

Author response:
We respectfully disagree with this assessment. This is cherry-picking from our results and is not representative of the overall performance achieved by our approach. This is the only case where our approach is outperformed by X-CLIP. __We outperform all prior works for zero-shot on HMDB-51, UCF-101 and K220 (1-vs-620 setting) and for few-shot recognition on UCF-101 and HDMB-51 - which are all the settings reported on X-CLIP too__. Note that we outperform it despite the fact that our approach uses pseudo-labels while X-CLIP for example, uses human-annotated labels.

Reviewer comment (Writing):
The paper writing/clarity needs to be greatly improved.

Author response:
Fig. 1 was intended as a high-level overview. The exact details can be followed in Alg. 1. We will add where possible such annotations to improve its readability.

Regarding Table 2 - As the setting from Table 2 is for zero-shot, the training and pretraining are the same step. We will rename it for further clarity.

Regarding whether, it should state LAION-115M & K-400, we included only the last dataset used for brevity. For example, BLIP vision encoder is pretrained on Imagenet, while their LM is a pretrained BERT, but they are usually not mentioned explicitly. We will add a note mentioning that the initial checkpoint is a BLIP model, hence was pretrained on the data from BLIP.

Yes, FM stands for foundation model. We will introduce this.

Regarding H \cdot W + 1, this was a typo. We will fix it.

Reviewer comment (Presentation):
The overview figure (Figure 1) is not clear. There is no H, g_v, g_t, language modeling loss, etc. in the figure.

Author response:
Fig. 1 was intended as a high-level overview. The exact details can be followed in Alg. 1. We will add where possible such annotations to improve its readability.

Regarding Table 2 - As the setting from Table 2 is for zero-shot, the training and pretraining are the same step. We will rename it for further clarity.

Regarding whether, it should state LAION-115M & K-400, we included only the last dataset used for brevity. For example, BLIP vision encoder is pretrained on Imagenet, while their LM is a pretrained BERT, but they are usually not mentioned explicitly. We will add a note mentioning that the initial checkpoint is a BLIP model, hence was pretrained on the data from BLIP.

Yes, FM stands for foundation model. We will introduce this.

Regarding H \cdot W + 1, this was a typo. We will fix it.

Reviewer comment (Presentation):
In Table 2, “Dataset” should be “Pretrained Dataset” and for the RISE row shouldn’t it be “LAION-115M & Kinetics-400” since it is initialized from BLIP?

Author response:
Fig. 1 was intended as a high-level overview. The exact details can be followed in Alg. 1. We will add where possible such annotations to improve its readability.

Regarding Table 2 - As the setting from Table 2 is for zero-shot, the training and pretraining are the same step. We will rename it for further clarity.

Regarding whether, it should state LAION-115M & K-400, we included only the last dataset used for brevity. For example, BLIP vision encoder is pretrained on Imagenet, while their LM is a pretrained BERT, but they are usually not mentioned explicitly. We will add a note mentioning that the initial checkpoint is a BLIP model, hence was pretrained on the data from BLIP.

Yes, FM stands for foundation model. We will introduce this.

Regarding H \cdot W + 1, this was a typo. We will fix it.

Reviewer comment (Presentation):
The presentation of this paper is poor.

Author response:
Fig. 1 was intended as a high-level overview. The exact details can be followed in Alg. 1. We will add where possible such annotations to improve its readability.

Regarding Table 2 - As the setting from Table 2 is for zero-shot, the training and pretraining are the same step. We will rename it for further clarity.

Regarding whether, it should state LAION-115M & K-400, we included only the last dataset used for brevity. For example, BLIP vision encoder is pretrained on Imagenet, while their LM is a pretrained BERT, but they are usually not mentioned explicitly. We will add a note mentioning that the initial checkpoint is a BLIP model, hence was pretrained on the data from BLIP.

Yes, FM stands for foundation model. We will introduce this.

Regarding H \cdot W + 1, this was a typo. We will fix it.

Reviewer comment (Evaluation):
The authors use zero-shot benchmarks for evaluation but claim "open-vocabulary".

Author response:
We agree in principle, however, there are no datasets that could appropriately measure this, and collecting one is a major endeavor and out of the scope of this work. We believe however generative models, as proposed in this work, are a natural fit for this, as they are not limited to a selection from a predefined set of labels. In a sense, our user study offers a glimpse of this, as the users are asked to assess whether a text produced describes the action or not, but as we used the K220 data, this will still not increase the variability of the classes themselves. We will add this discussion into the updated manuscript, addressing also the use of the term.

Reviewer comment (Novelty):
The novelty of the proposed approach is limited. Basically, it is a mixture of BLIP and CLIP iterating over multiple times throughout the training. CLIP will generate new text features because of new text generated by the updated BLIP model after each iteration. It looks like the model is finetuned on the Action Recognition dataset.

Author response:
We politely disagree, this is a misrepresentation of our work.

Firstly, our work is the first to explore the idea of generative V&L models for open world action recognition - a novel concept and idea that was not previously proposed nor investigated, and we show it can match and even outperform discriminative methods without making use of human annotations/labels.
 
Secondly, the description is both inaccurate and an oversimplification. Our process consists of a bi-level retrieval: video-to-video and  frame-to-text, which retrieves the appropriate pseudo-captions at a dataset level. We empirically show the retrieval module, which is non-trivial, has a massive impact in the final performance (+9%, see Table 3).  There are also no new text features computed after each iteration, they are only recomputed after a given number of epochs.  

Finally, no human annotated labels were used for training.

Reviewer comment (Experiments):
(Section 1) Table-1 - Are the approaches trained from scratch or fine-tuned? How is the testing done? What is zero-shot accuracy without any fine-tuning at all? Because in Section 3.1 it is mentioned that the model is initialized with pre-trained weights.

Author response:
In Table 1 all approaches are fine-tuned on Kinetics-400 using various forms of supervision (each row specifies the type of supervision and training strategy).  We already report the performance of the initial BLIP/GIT pretrained model (i.e. without any finetuning) in Table 2 on the HMDB-51 dataset. For K220, the performance of such a model without fine-tuning is ~10%. Our approach is massively better than this. The initial models are generally not sufficiently discriminative to form class-separable text. In fact, this is one aspect that our bi-level retrieval seeks to address.

Reviewer comment (Reproducibility):
From the algorithm it looks like the video-video and video-text similarity scores are calculated after every training iteration. Given the dataset size, how is it not a bottleneck, because it will be a huge operation to generate captions for all training videos and then calculate similarity between all data-points? (Lines 10 and 12).

Author response:
We don't do this, this is a misunderstanding. Firstly, as shown in Alg. 1 this is repeated after R epochs, not every iteration as stated. Secondly, the video-video similarity scores and the video features are cached, as they don’t change during training. Thirdly, as shown in Alg. 1 the text-to-video similarities are computed only between the captions of the videos that are close in the video-video space. All in all, given that most of the data is cached, the process is repeated after a few epochs, and it considers only similarities within a neighborhood, the process is computationally feasible, taking a similar amount of time with that of running one training epoch.

Reviewer comment (Experiments):
Why is it even needed to calculate video similarity after each iteration? It won’t change as CLIP encoders are frozen.

Author response:
We don't do this, this is a misunderstanding. Firstly, as shown in Alg. 1 this is repeated after R epochs, not every iteration as stated. Secondly, the video-video similarity scores and the video features are cached, as they don’t change during training. Thirdly, as shown in Alg. 1 the text-to-video similarities are computed only between the captions of the videos that are close in the video-video space. All in all, given that most of the data is cached, the process is repeated after a few epochs, and it considers only similarities within a neighborhood, the process is computationally feasible, taking a similar amount of time with that of running one training epoch.

Reviewer comment (Experiments):
(Section-3.4) Similarity Calculation - Why not similarity be calculated using CLIP models adapted for video such as [1]? They are better encoders in the case of videos.

Author response:
Our method is amenable to any vision-language model. We used a CLIP model simply because at the time it was by far the most widely used foundation model V&L, with well studied generalization properties. Thanks for pointing [1], it is relatively recent (CVPR’23) and we weren't aware of it at the time of developing our approach. Given that we already achieve state-of-the-art results, having scope for further improvements is actually not a negative but a good thing.

Reviewer comment (Experiments):
(Section 3.5) Retrieval - How does the proposed approach ensure that the captions selected each time introduce diversity via CLIP? Because the video-video similarity is calculated from a frozen encoder. The video will be the same, only the text will be different for that same video because BLIP is finetuned on K400 now.

Author response:
We are always maintaining a large pool of options, searching during the retrieval within the captions of other similar videos too. This ensures a high initial diversity. Such collapse can occur in practice only if the initial diversity is very low (not the case here) or if the model used for retrieval is inaccurate (i.e. unsuitable to match a sentence with a frame) - again not the case here. Our results and statistics about the data in the appendix support this assessment. We also show excellent experimental performance, which clearly would not be the case if the retrieval mechanism failed to maintain or improve diversity.

Reviewer comment (Experiments):
General Concern - BLIP/BLIP-2 is supposed to be made for tasks like image-retrieval which means it has a better capability to find the images given the phrases. Does this setting is more like a video-text retrieval task rather than solving action recognition? Why is it mentioned again and again there are no labels? If I generate a text for UCF101 class from BLIP for applying eye makeup - BLIP generates multiple texts for a woman putting on makeup. Now if we compare the similarity of this phrase with classes of UCF-101 it will be sure that the ApplyEyeMake up class will have high scores at the inference time. I personally tested it with Huggingface APIs.

Author response:
No, this is a classification problem, not a retrieval problem. Our solution is generative, i.e. it produces an action-descriptive output in an autoregressive manner. This is notably different from all prior action recognition methods."

Reviewer comment (Experiments):
(Section 6.1) Ablation study - (Table-2) Comparing an image model against a video model is not the right thing to do. For fair comparison, it should have been a BLIP model with adapter but no other additions. Only trained with class labels/original captions.

Author response:
Please see below the results of the requested experiments:

Training a generative model using class names results in near-complete overfitting to the base class names. This is one of the findings of our paper, shown in Table 1. Adding temporal adapters doesn't change the performance and such models score close to 0% top-1 accuracy (e.g. K220 1-vs-620 setting). For fine-tuning with the initial captions, without our approach, on HMDB the results are as follows: w\o temp. Adapter: 40.1%; with temporal adapter: 42.6%; Ours: 49.7%; Our approach is significantly better.

Reviewer comment (Novelty):
Oversimplification - I don’t think using complex terms to present a simple approach could be mentioned as novelty and that’s why I kept the presentation score low. There’s very limited novelty to this work. It’s just usage of the CLIP and BLIP image-based approach and there’s no adaptation for Videos. An action recognition task involves temporal understanding. I don’t see any novel component in the proposed approach, quantitatively or qualitatively which shows the temporal understanding. The reason why I’m saying HMDB and UCF couldn’t quantify temporal understanding is because if an action class is considered, the caption will always contain the original class description such as trampoline, cricket, etc. pick any frame from the video. Something-something will actually show that it focuses on the verb aspect which is considered as action. With the current approach, there’s no novelty in terms of temporal understanding for the task which is the basis of video action understanding.

Author response:
We politely disagree, this is a misrepresentation of our work.

Firstly, our work is the first to explore the idea of generative V&L models for open world action recognition - a novel concept and idea that was not previously proposed nor investigated, and we show it can match and even outperform discriminative methods without making use of human annotations/labels.
 
Secondly, the description is both inaccurate and an oversimplification. Our process consists of a bi-level retrieval: video-to-video and  frame-to-text, which retrieves the appropriate pseudo-captions at a dataset level. We empirically show the retrieval module, which is non-trivial, has a massive impact in the final performance (+9%, see Table 3).  There are also no new text features computed after each iteration, they are only recomputed after a given number of epochs.  

Finally, no human annotated labels were used for training.

Reviewer comment (Experiments):
Looking at the second contribution, caption diversity selection. I don’t think it’s the CLIP which is selecting a better caption (CLIP based retrieval), it’s just the better caption generated by BLIP as training progresses. There’s no criteria to improve the selection other than similarity scores which depend on text generated by BLIP. Both of the contributions are pretty weak at this point of time, that’s why I believe the paper needs more work.

Author response:
We are always maintaining a large pool of options, searching during the retrieval within the captions of other similar videos too. This ensures a high initial diversity. Such collapse can occur in practice only if the initial diversity is very low (not the case here) or if the model used for retrieval is inaccurate (i.e. unsuitable to match a sentence with a frame) - again not the case here. Our results and statistics about the data in the appendix support this assessment. We also show excellent experimental performance, which clearly would not be the case if the retrieval mechanism failed to maintain or improve diversity.

Reviewer comment (Experiments):
(Table 4-b) Why is it shown only for 4 iterations? The authors did 6 iterations. 60 epochs and each iteration are done after 10 epochs.

Author response:
We use 3, not 4 for the final model. The 4-th is shown to showcase that performance saturates after, so they are not run. The 1st training round is longer (30 epochs), hence the total of 60.

Reviewer comment (Experiments):
Does the model have any temporal understanding? Result on Something-Something v2 would have highlighted these capabilities.

Author response:
We follow the evaluation protocol and datasets from prior work (i.e. X-CLIP). SS-v2 is generally not a good fit for zero-shot based on text due to the unusually worded classes (e.g. : "Pretending to put [something] onto [something]"). We validate this already on HMDB-51 which has already a temporal nature (compared with UCF-101).

Reviewer comment (Experiments):
Ablations on UCF even in the appendix would have been more convincing. HMDB is a small dataset.

Author response:
We are already reporting results on UCF-101, HMDB-51, K-620 (2 settings). We don't see why the conclusions would change for the ablations when measured on UCF-101. For prior methods, generally, these performances correlate. Nevertheless, we followed your suggestion:
 
For different values of K: K=1 (62.6); K=3 (63.93); K=5 (63.98);

For different number of retrieval steps: N=1 (59.6); N=2 (62.75); N=3 (63.93); N=4 (63.95).

These results are in line with the ones for HMDB-51.