Paper: Look, Remember and Reason: Grounded Reasoning in Videos with Language Models (ICLR-2024)

Reviewer comment (Experiments):
In the experiments, the authors primarily focus on conducting investigations using synthetic datasets, particularly the ACRE dataset. However, it raises concerns about the generalizability of the conclusions/findings obtained from synthetic datasets to real-world datasets.

Author response:
**Synthetic datasets:**  We would like to emphasize that recent work has shown that even on synthetic datasets such as ACRE, state of the art LLMs such as GPT-4 do not perform well (Gendron et. al. 2023). Moreover, we include results on the challenging real-world Something-Else dataset in Section 4.2. We additionally include results on the challenging real-world STAR (Wu et. al. 2021) dataset below and in Appendix B in the paper. 

Specifically, note that the results in Fig. 4 and Fig. 7 (Appendix) on Something-Else clearly show that our LRR model can deal with visually complex real-world scenarios. Our LRR model can deal with: severe occlusions, e.g., the silver ball in Fig. 4 bottom row is occluded by the hand; severe deformations, e.g., the piece of paper in Fig. 7 second row which is torn into two pieces; severe changes in appearance, e.g., the pen in Fig. 7 third row. It outperforms the state of the art STIN + OIE + NL model and the recently proposed Video-ChatGPT multi-modal LLM by a significant margin.

We additionally include results on the visually complex STAR dataset in Appendix B. The STAR benchmark (https://bobbywu.com/STAR/) consists of questions built upon real-world videos associated with human actions and interactions. Our LRR model with an OPT-350M LM backbone **achieves 66.7% accuracy** and outperforms prior state of the art models such as SeViLA (Yu et. al., 2023), Internvideo (Wang et.al. 2022) and BLIP-2 (Li et. al. 2023). Further details can be found in Appendix B, where we also include ablations of our LRR model (w/o Surrogate tasks) which highlights the advantage of using surrogate tasks during training.

| Method | Int. | Seq. | Pre. | Fea. | Overall|
| -------- | :-------: | :-------: | :-------: | :-------: | :-------: | 
| Internvideo | 62.7 | 65.6 | 54.9 | 51.9 | 58.7 |
| BLIP-2 | 65.4 | 69.0 | 59.7 | 54.2 | 62.0 |
| SeViLA | 63.7 | 70.4 | 63.1 | **62.4** | 64.9 | 
| LRR (ours) | **79.6** | **71.3** | **63.2** | 52.8 | **66.7** |

Note that methods such as SeViLA are trained on a much larger set of image/video - text pairs and, use the 4.1B parameter BLIP as a video encoder and the 3B parameter Flan-T5 XL as an language model (c.f. Section 4.4 in Yu et al. (2023)). In contrast, our LRR model is trained on a much smaller dataset but one that includes low-level surrogate tasks to endow the model with the requisite low-level visual capabilities.

Finally, in terms of visual complexity, CATER (especially the moving camera split) is complex and challenging as shown in Fig. 5. The results in Fig. 5 and Table 3 indicate that our LRR model can not only detect occlusions for tracking the snitch, but also understand camera motion based just on visual observations, e.g., motion of the background.
To conclude, the results on Something-Else, STAR, CATER and ACRE clearly show that our LRR model can deal with highly varied complex real-world visual inputs and reasoning tasks.

Reviewer comment (Experiments):
The experimental results primarily focus on classical models. However, the generalizability of the conclusions/findings derived from these classical models to more powerful transformer-based models, such as the models mentioned in *Related Work* part, remains a concern.

Author response:
**Classical models:** We would like to emphasize that we include powerful transformer-based models in our evaluation. In Table 1, we include the recently proposed state of the art OpenFlamingo (Awadalla et al., 2023) multimodal LLM as a baseline. Furthermore, IV-CL (Sun et al., 2023) and ALOE (Ding et. al., 2021) in Table 1 are also large transformer based models. In Table 2, we include the recently proposed state of the art Video-ChatGPT (Maaz et al., 2023) multimodal LLM as a baseline. In Table 3, we compare to the large transformer based IV-CL (Sun et al., 2023) and ALOE (Ding et. al., 2021) models. In Table 5 (Appendix), we compare to SeViLA (Yu et. al., 2023), Internvideo (Wang et.al. 2022) and BLIP-2 (Li et. al. 2023), all of which are large multi-modal transformer based models. Note that, we are first to propose the use of language models for solving complex reasoning tasks such as causal and spatiotemporal compositional reasoning, while ensuring that we always compare to state of art models across all tasks. We will clarify further in the final version.

Reviewer comment (Writing):
Throughout the paper, there's a recurring mention of "low-level surrogate tasks". Could the authors elucidate the definition of these low-level tasks? Moreover, how do they differ from high-level tasks?

Author response:
**Low-level surrogate tasks:** We consider tasks like object recognition, tracking and re-identification as low-level tasks. These are computer vision tasks which play a crucial role in solving high-level reasoning tasks, like those found in ACRE, CATER, Something-Else and STAR. For example, in the case of Something-Else, the ability to recognize the performed action in Fig. 4 (top) rests upon grounding our LRR model to the motion and interaction of the hand and the stapler. This makes object recognition and tracking constituent low-level capabilities for recognizing the (high-level) compositional action. We have updated the text in Section 3.4 to make this clearer. We would be glad to receive further feedback to help improve the text further for the final version.

Reviewer comment (Writing):
The Look, Remember, Reason (LRR) model framework is innovative. However, there seems to be a gap in explicitly correlating this framework with the actual operations carried out in the method. The unique contributions of the "Remember" and "Reason" steps, in particular, are not clearly highlighted. It would be beneficial for the readers if the authors can provide a clearer mapping of these steps to their corresponding operations.

Author response:
**"Remember" and "Reason" steps:** We apologize for any issues with clarity. As mentioned in Section 3.3, in the “remember” step our LRR model remembers low-level visual information by storing it within the context window of the LLM backbone. In the “reason” step, our LRR model aggregates this low-level visual information to arrive at the final answer (see also Fig. 1). This aggregation happens implicitly through the attention mechanism of the backbone LLM model. We have improved the clarity of the text in Section 1, Section 3.3 and 3.4 in the updated pdf. The updated text is in blue. We would be glad to receive additional feedback to help improve the text further for the final version.

Reviewer comment (Reproducibility):
Will the codebase for the presented method be made publicly available?

Author response:
**Codebase:** Code will be released, pending legal review.

Reviewer comment (Reproducibility):
Regarding the results of Video-ChatGPT on the Something-Else dataset: Were these results replicated by the authors? I couldn't find a direct reference to such results in the original Video-ChatGPT paper.

Author response:
**Video-ChatGPT:** The Video-ChatGPT model was tested by ourselves. We have updated Table 2 to indicate this. We use the publicly available codebase (https://github.com/mbzuai-oryx/Video-ChatGPT). We obtained the best results by finetuning the Video-ChatGPT model based on the information in the publicly available codebase and the paper (see Maaz et. al. 2023). We have updated the text in Section 4.2 (in blue).

Reviewer comment (Experiments):
The datasets used are rather simple with low visual complexity, such as CATER.

Author response:
We thank the reviewer for the helpful comments and for recognizing that our LRR model demonstrates strong performance on multiple challenging visual reasoning datasets by grounding the language model in low-level visual details.

**Low visual complexity:** We would like to emphasize that recent work has shown that even on synthetic datasets such as ACRE, state of the art LLMs such as GPT-4 do not perform well (Gendron et. al. 2023). Moreover, note that we include results on the challenging real-world Something-Else dataset in Section 4.2. We additionally include results on the challenging STAR (Wu et. al. 2021) dataset below and in Appendix B in the paper.

Specifically, note that the results in Fig. 4 and Fig. 7 (Appendix) on Something-Else clearly show that our LRR model can deal with visually complex real-world scenarios. Our LRR model can deal with: severe occlusions, e.g., the silver ball in Fig. 4 bottom row is occluded by the hand; severe deformations, e.g., the piece of paper in Fig. 7 second row which is torn into two pieces; severe changes in appearance, e.g., the pen in Fig. 7 third row. It outperforms the state of the art STIN + OIE + NL model and the recently proposed Video-ChatGPT multi-modal LLM by a significant margin.

We additionally include results on the visually complex STAR dataset in Appendix B. The STAR benchmark (https://bobbywu.com/STAR/) consists of questions built upon real-world videos associated with human actions and interactions. Our LRR model with an OPT-350M LM backbone **achieves 66.7% accuracy** and outperforms prior state of the art models such as SeViLA (Yu et. al., 2023), Internvideo (Wang et.al. 2022) and BLIP-2 (Li et. al. 2023). Further details can be found in Appendix B, where we also include ablations of our LRR model (w/o Surrogate tasks) which highlights the advantage of using surrogate tasks during training.

| Method | Int. | Seq. | Pre. | Fea. | Overall|
| -------- | :-------: | :-------: | :-------: | :-------: | :-------: | 
| Internvideo | 62.7 | 65.6 | 54.9 | 51.9 | 58.7 |
| BLIP-2 | 65.4 | 69.0 | 59.7 | 54.2 | 62.0 |
| SeViLA | 63.7 | 70.4 | 63.1 | **62.4** | 64.9 | 
| LRR (ours) | **79.6** | **71.3** | **63.2** | 52.8 | **66.7** |

Note that methods such as SeViLA are trained on a much larger set of image/video - text pairs and, use the 4.1B parameter BLIP as a video encoder and the 3B parameter Flan-T5 XL as an language model (c.f. Section 4.4 in Yu et al. (2023)). In contrast, our LRR model is trained on a much smaller dataset but one that includes low-level surrogate tasks to endow the model with the requisite low-level visual capabilities.

Finally, we would like to emphasize that CATER, especially the moving camera split, is visually complex and challenging as shown in Fig. 5. The results in Fig. 5 and Table 3 indicate that our LRR model can not only detect occlusions for tracking the snitch, but also understand camera motion based just on visual observations, e.g., motion of the background.
To conclude, the results on Something-Else, STAR and CATER clearly show that our LRR model can deal with highly varied complex real-world visual inputs and reasoning tasks.

Reviewer comment (Writing):
Writing, section 3 and Figure 2 is a little unclear and hard to follow.

Author response:
**Writing, section 3 and Figure 2 is a little unclear and hard to follow:** We apologize for any issues with the clarity of the text in Section 3 and Figure 2. We have made extensive changes to Section 1 and 3 to improve clarity of the text, which we highlight in blue. In Section 1 we have improved the explanation of low-level surrogate tasks and, the “remember” and “reason” steps of our LRR framework. In Section 3, we have improved the overview paragraph of LRR model, focusing on the role of surrogate tasks in our LRR framework. In Section 3.1, we have provided more details of the inputs to and outputs from our LRR model. In Section 3.2, we have clarified more details of our spatial and temporal attention components. In Section 3.3, we have clarified our top-down attention scheme. Finally, in Section 3.4 we have endeavored to define and explain our low-level surrogate tasks more clearly. We will continue to improve the manuscript and we would be glad to receive further and more specific feedback about parts of the text and figures to help improve it further for the final version.

Reviewer comment (Experiments):
For different surrogate tasks, where do the ground-truth answers such as localization or box come from?

Author response:
**For different surrogate tasks, where do the ground-truth answers such as localization or box come from?** We employ the ground-truth annotations, such as localization or bounding boxes, provided by the corresponding datasets, to train our LRR model end-to-end. As shown in Fig. 1 and Fig. 4, the Something-Else dataset provides bounding box annotations and tracks of hands and objects involved in the action (c.f. Section 4 in Materzynska et. al. 2020). These annotations are also used by the state of the art STIN + OIE + NL model (see Table 2). Similarly, for the ACRE dataset, the object class annotations and the state of the blicket machine are provided, and for the CATER dataset the tracks of the objects in the scene are provided. These annotations are also used by the state of art ALOE model (Ding et al. 2021; see Tables 1 and 3). Thus, we only use annotations provided with the target datasets, and these annotations are also used by the prior state of the art models, ensuring a fair comparison. In case such annotations are not available, due to the recent advances in perception models reaching human-level performance it is possible to use off-the-shelf vision models (Ren et al., 2017; Tang et al., 2017; Ye et al., 2022) as discussed in Section 3.4. It is worth noting that the annotations need to be used only at training time and not at test time, so the computational efficiency of models generating the annotations is not critical for any real-world use. Finally, in the case of STAR (see Appendix B in the updated paper PDF and response to all reviewers), we observe that training jointly on multiple tasks, some of which do and some of which don’t provide annotations, the model is able to acquire the required low-level visual skills from those tasks that do provide them and transfer them to those that don’t. This allows the jointly trained model to reach very high accuracy on the scarcely annotated STAR dataset.