Paper: LLCP: Learning Latent Causal Processes for Reasoning-based Video Question Answer (ICLR-2024)

Reviewer comment (Novelty):
The novelty of the paper is limited. While the paper draws motivation from causal modeling, both the theoretical analysis and the implementation of TMGM using conditional VAEs do not adequately realize the causal processes in Eq. 1.

Author response:
**A1:** We appreciate this comment which helped us clarify both theoretical and technical details and highlight our contributions.  We would like to highlight that under the assumptions we make, our TMGM model is able to learn the causal processes. We have included detailed proof in the revised Appendix A2. 
We have conducted an experiment to verify whether we can learn the causal processes in Appendix A6.3. 
We also analyzed the relations between motivation and implementation in the following. The discussion on how the implementation details can constrain/leverage these assumptions 

1. **Key Motivation**
   Our key motivation is to learn the model by answering reasoning questions without the usage of textual annotations. To achieve that, our LLCP consists of two stages, one is learning the causal processes with the temporal multivariate generative model, and the other is applying the discovered causal processes to conduct inference for answering reasoning-based questions. The first stage relies on the assumptions of Granger Causality, and the latter one leverages the shared feature spaces and causal inference. 

2. **Granger Causality in Learning the Causal Processes**
   - **Assumption:** We assume that the past values of one variable can predict the future values of another, indicating a directional relationship in a time-series context and implying no instantaneous effects in the temporal dynamics. Variables $\mathbf{z}_t$ are independent given the historical state $Pa(\mathbf{z}_t)$, as aligned with our data generation process.
   - **Implementation:** To substantiate this, we employ auto-regression based on the concept of Granger Causality. This approach predicates future events on historical data, validating our assumption about predictability. The **identification of our proposed model** is further supported by a proposition and proof in Appendix A2. 

3. **Causal Inference**
   - **Root cause analysis** We conduct the root cause analysis by identifying the changed local causal mechanisms.  The insight is if one variable's change cannot be explained by its parents, then it is the root cause of the change. 
   - **Counterfactual** Counterfactual inference is an inference method that leverages the causal structures to predict the possible outcome in the counterfactual conditions.

Reviewer comment (Theory):
It is not clear how the authors enforce the independence constraints, both spatially and temporally, on the noise variables by the three subnetworks f_N, f_H and f_E. The fusion of these three networks encodes spatio-temporal dynamics across objects (agents) but it is unclear if this has any bearing on causal relations.

Author response:
**A2:** Thanks very much for this question. It definitely helped improve the readability of our paper. We respectively answer the questions in terms of independence and model-fusion below. We have revised the manuscript accordingly to make the details clearer as shown in Section 2.1.

**Independence.** We enforce the independence by the KL divergence loss in Equation (3) in the revised version (original Equation (2) ). 
Specifically, we constrain the learned posterior distribution with a prior distribution where $  p(\bf{\hat z}_t) | Pa(\bf{\hat z}_t) )= \prod _{i=1}^{M} p({\bf{\hat z}_t}_i) | Pa({\bf{\hat z}_t}_i) ) $ which implies the following conditional independent properties. Given the historical information, the variables in the current time step are independent. By this, with the log function, we have
 $  \log(p(\bf{\hat z}_t) | Pa(\bf{\hat z}_t) )) = \sum _{i=1}^{M} \log(p({\bf{\hat z}_t}_i) | Pa({\bf{\hat z}_t}_i) )) $.
With the KL loss, we encourage the posterior also with these conditional independent properties. 

**Subnetworks.** By the design of three subnetworks, we decompose the causal processes into local mechanisms. First, this indicates prior knowledge that the transition functions are different for historical states, neighborhood interactions, and environmental clues. By this decomposition, we respectively learn each submodule leveraging the modularity property of the causal mechanisms, which makes the learning process easier.

Reviewer comment (Experiments):
Experiments only compare LLCP with weaker variants of VAEs which offer little to no temporal modeling. In addition, the proposed method largely underperforms supervised methods as indicated by Table 5 and Table 6.

Author response:
**A3** We are grateful for your careful review and constructive suggestions to improve the completeness of our experiments. In light of your suggestion, we have further considered several latest VAE variants like SKD [R1] and SlowVAE [R2], which are shown in the following table. According to the experiment results, we can find that our method can still enjoy significant improvement. Please refer to Table A3 for details.

| Models   | function change |          | Structure change |          |
|----------|-----------------|----------|------------------|----------|
|          | recall          | f1 score | recall           | f1 score |
| LSTM VAE | 52.11           | 27.81    | 37.75            | 20.78    |
| VAE      | 51.92           | 27.33    | 31.08            | 19.35    |
| SlowVAE  | 54.41           | 29.03    | 38.12            | 21.39    |
| SKD      | 58.93           | 34.76    | 35.72            | 19.79    |
| LLCP     | 63.34           | 63.01    | 45.04            | 44.19    |

For the comparison with supervised learning methods, we would like to respectfully highlight that our method doesn't use any textual annotations, which causes the results to appear lower. For a more fair comparison, we also compared LLCP with other unsupervised learning such as CLIP [R3] and OpenFlamingo [R4] and found a significant improvement. Please refer to Appendix A7.2 for more details. 

| Model                       | Accuracy |
|-----|-------------|
| OpenFlamingo-3B (2 examples)   | 28.7        |
| OpenFlamingo-3B (4 examples)   | 29.1        |
| VidIL                       | in processing  |
| CLIP                        | 27.7        |
| LLCP                        | 33.7        |

In addition, when further utilizing the textual annotations, we outperformed other supervised learning methods, such as the baseline HCRN, by a large margin (5\%). Please kindly refer to Appendix A5 and Table A2 for the implementation details and experimental results.

|                    | B     | A     | I     | C     | F     | R     | Avg   |
|--------------------|-------|-------|-------|-------|-------|-------|-------|
| HCRN               | 34.17 | 50.29 | 33.40  | 40.73 | 44.58 | 50.09 | 36.26 |
| LLCRP              | 38.95 | 44.98 | 32.43 | 48.64 | 41.70  | 47.16 | 41.22 |



[R1] Multifactor Sequential Disentanglement via Structured Koopman Autoencoders. Nimrod Berman and Ilan Naiman and Omri Azencot. The Eleventh International Conference on Learning Representations,2023.

[R2] Klindt D, Schott L, Sharma Y, et al. Towards nonlinear disentanglement in natural data with temporal sparse coding[J]. arXiv preprint arXiv:2007.10930, 2020.

Reviewer comment (Theory):
Regarding the first task of root cause analysis, it is unclear if there is any reference for the definition of root causes in Sec. 3.2. The provided definition seems to be applicable only to anomaly/outlier detection.

Author response:
**A4** Thanks for your question which helped improve the readability. 
The "root cause" in root cause analysis is defined as the fundamental, underlying, system-related reason to explain the failure. It denotes that changing this factor can solve the problem from the root, which implies the changing of this factor cannot be explained by its cause variables[R3, R4].
 In this paper, we consider the "root cause" from the perspective of causality and apply this to answer cause-based reasoning questions.

[R3]Tariq S Abdelhamid and John G Everett. Identifying root causes of construction accidents. Journal of construction engineering and management, 126(1):52–60, 2000

[R4] OSHA Fact Sheet. The importance of root cause analysis during incident investigation. URL https://www.osha.gov/Publications/OSHA3895.pdf, 2016.

Reviewer comment (Writing):
In Sec. 3.4, it is unclear what happens if counterfactual questions are formulated in an open-ended format, i.e., “what if”.

Author response:
**A5** Thanks a lot. Similar to conventional VideoQA methods, the differences between the multi-choice and open-ended formats lie in the predictor module. We use classifier head and generative head respectively for multi-choice and open-ended formats. It indicates that we can train or pre-train a generative head to allow open-ended questions.

Reviewer comment (Experiments):
The outlier analysis at inference times relies on extracted features from pretrained models (i.e., CLIP) which can be brittle if tested data is outside of the trained distribution. In addition, using a language parser to parse user queries is not ideal. It is unclear if the authors have considered an alternative solution towards this.

Author response:
**A6** We appreciate this question which improved the potential of our method for real-world applications. We have two potential strategies to solve the possible OOD data. First, the CLIP model shows the excellent transferability [R5]. We can efficiently conduct the prompt learning or adapter tuning to align the pretrained CLIP into the test data distribution. Second, we may follow the conventional supervised methods to learn cross-modality matching and take the LLCP as the plugging module to learn causal presentations.  As shown in Appendix A5, we find that the LLCP can benefit from the annotations to achieve better performance, which shows the lights to solve the difficulty caused by no annotations.  

[R5] Zhou K, Yang J, Loy C C, et al. Learning to prompt for vision-language models[J]. International Journal of Computer Vision, 2022, 130(9): 2337-2348.

Reviewer comment (Presentation):
The presentation is not satisfying enough. It is hard for me to figure out the connection between motivation and implementation details, especially how the method can guarantee that the learned relation is causal rather than temporal, and the terminologies of variables. Please see "questions" for detailed comments.

Author response:
Thanks for your valuable comments which have helped clarify our motivation, assumptions, and technical details. 
In light of your suggestions, we list the assumptions and discuss how the implementation details can constrain/leverage these assumptions in Appendix A3.1. 
Besides, we further provide the theoretical guarantee of why the learned model follows the Granger Causality in Section 2.1 and Appendix A2. 

Our method's connection between motivation and implementation is deeply rooted in the core assumptions of the **LLCP** model, which are crucial for understanding causality in the temporal dynamic system.
1. **Key Motivation**
   Our key motivation is to learn the model for answering reasoning questions without the usage of textual annotations. To achieve that, our LLCP consists of two stages, one is learning the causal processes with the temporal multivariate generative model, and the other is applying the discovered causal processes to conduct inference for answering reasoning-based questions. The first stage relies on the assumptions of Granger Causality, and the latter one leverages the shared feature spaces and causal inference. 

2. **Granger Causality Principle**
   - **Assumption:** We assume that the past values of one variable can predict the future values of another, indicating a directional relationship in a time-series context and implying no instantaneous effects in the temporal dynamics. Variables $\mathbf{z}_t$ are independent given the historical state $Pa(\mathbf{z}_t)$, as aligned with our data generation process.
   - **Implementation:** To substantiate this, we employ auto-regression based on the concept of Granger Causality. This approach predicates future events on historical data, validating our assumption about predictability. The identification of our proposed model is further supported by a proposition and proof in Section A2. 

3. **Shared Feature Spaces in Visual and Textual Domains:**
   - **Assumption:** We posit that the feature spaces for visual and textual domains are shared, facilitating the matching of the same variables across these domains.
   - **Implementation:** To corroborate this, we integrate the pre-trained CLIP model as a feature extractor for both video objects and textual elements (questions and answers). Its application ensures that both visual and textual data are represented in congruent feature spaces, thus affirming the feasibility of matching variables across these domains.

4. **Causal Inference**
   - **Root cause analysis** We conduct the root cause analysis by identifying the changed local causal mechanisms.  The insight is if one variable's change cannot be explained by its parents, then it is the root cause of the change. 
   - **Counterfactual** Counterfactual inference is an inference method that leverages the causal structures to predict the possible outcome in the counterfactual conditions.

**Terminologies of variables** 
We guess that the reviewer is curious about the terminologies of variables. This term denotes an object with randomness (probability distribution). Here, our observed variables are the features of objects, such as vehicles, whose randomness comes from different characteristics and states. While the latent variables are hidden, which contribute to the data generation process.

Reviewer comment (Theory):
Another main concern is whether the model is causality-based or just capture temporal relations. In addition to the concept of causal relation, I didn't see how the implementation reflect the tools of causal inference or causal reasoning. Therefore, I am wondering whether the causal understanding ability is over-claimed. Experimental results can verify the ability empirically, but theoretical explanations or guarantees are missing.

Author response:
We are grateful for the suggestions which help clarify our theoretical soundness. 
We would like to respectfully highlight that the proposed LLCP framework is a causal framework, which consists of two stages: causal discovery (learn the causal structures) and causal inferences (use the causal structures).
In light of your suggestion, we have included the identification theorem to show that our proposed model can learn the causal relations by proposition and proof in Appendix A2. 

**Learn the causal structures**
We assume that data follows the Granger Causality, which implies that the past values of one variable can be used to predict the future values of another, suggesting a directional relationship between these variables in a time series context. With this assumption, we provide proof to show that the learned causal relations are identifiable. Please refer to Appendix A2 and the revised Section 2.1 for details. 

**Leverage the causal structures**
In the stage of causal inference, we propose to conduct the root cause analysis and counterfactual predictions respectively for cause-based and effect-based reasoning tasks. The former leverages the modularity of the causal mechanisms to locate the root cause by matching the observed agents with the local generative process. The counterfactual prediction is also a classical causal inference tool to provide personalized prediction under counterfactual conditions. Please kindly refer to Chapter 7, Section 1 of Judea Pearl's book “Causality” [R1], for more details. 

[R1] Pearl J. Causality[M]. Cambridge university press, 2009.

Reviewer comment (Experiments):
The ablation studies in Table 4 mainly demonstrate the contributions of sub-networks rather than the role of historical states and environment. What if we replace the historical states and environment with wrong ones? What is the performance then? That would show that the performance drop is due to the lack of visual information rather than fewer parameters.

Author response:
We greatly appreciate this suggestion which improved the completeness of our ablation studies. In light of your suggestions, we further conducted the experiments by replacing the historical states and environment with random vectors. 
As shown in Table 3 in the revised version, we summarized the performance of all versions of LLCP, including using random features of historical states and environments.
First, using random historical feature $f_{N}$ and environmental feature $f_{E}$ hurts the performance, which demonstrates both these two modules help learn the causal generation process. 
Second, we found that the parameters also influence the performance since the performance of using the random feature is a bit better than the total removal. 
Thanks again for your insightful advice!

Reviewer comment (Presentation):
What does the red node in Figure 1 (b) mean? What it is red during test pipeline but not highlighted during training pipeline?

Author response:
Thank you very much for this question. It helps us to further clarify the illustration and improve the readability. The red node in Figure 1 (b) means the root cause (the agent causing the accident). We have added this in the captions of the revised version. For the training stage, we learn the causal mechanisms from regular videos in which **no** accident is there. In the inference stage, we apply the causal inference to identify the root cause and answer the corresponding questions.

Reviewer comment (Theory):
The abstract mentioned that the proposed LLCP employs a temporal multivariate generative model to understand the causal patterns. I am not aware of how the *temporal* model can discover *causal* relations. I am wondering how to guarantee that the learned pattern are causal relations rather than temporal correlations using the so-called temporal model? In the method part, I am not aware of how the learned pattern are causality rather than correlation.

Author response:
Thanks for your questions.  Please refer to the Answer 2 above for the details. We assume the variables in temporal dynamics follow the Granger Causality, which implies that the past values of one variable can be used to predict the future values of another, suggesting a directional relationship between these variables in a time series context. Thus, we can directly obtain the causality (directional) rather than correlation (unidirectional). We provided a theoretical guarantee in Appendix A2 and Section 2.1 in the revised version.

Reviewer comment (Writing):
I didn't find strict definitions of historical state, neighborhoods, and environment variables. Could the authors provide a precise and accurate definition, or give examples of these three variables? For example, are they features of a single frame, an object, or a set of tracked frames? How does the object tracking model obtain these three variables in temporal and spatial dimensions? It seems that Figure 4 provide an example, but it appears a bit late when I was reading Sec. 3.1 but didn't find the examples.

Author response:
Thanks a lot for the valuable suggestions which improved the readability of our paper. Here, all of the historical state, neighborhoods, and environment variables are the variables in the temporal dynamic system.  The variables here can be understood as the agents in traffic events, such as vehicles, pedestrians, or traffic signals.
Below are the detailed definitions:
For a given variable $x_{t, i}$, its historical state is the variable with the same identity but in the past frame, i.e., $x_{t', i} | t'<t$. Similarly, its neighborhoods denote other variables (not experimental variables) in the previous frame, such as $x_{t',j}| t'<t, j \neq i$. The environment variables denote the environmental context, such as traffic lights, zebra crossing, and traffic signals. We have included these detailed definitions in the revised Section 2.1 (Section 3.1 in the revised paper).

Reviewer comment (Writing):
In Ex. (2), is x_{t,i} an image (or region of interest) or feature vector?

Author response:
Thanks for this question.  $x_{t,i}$ is a feature vector, which is obtained by the ROI pooling on the CLIP feature tensor for the detected bounding box of a certain object, such as a vehicle.

Reviewer comment (Experiments):
According to Figure 4, the question seems to be the input of text side rather than video side. In this case, how can we determine the object of interest when extract the visual information and make the prediction of visual states? Is it reasonable to extract question-independent visual feature to answer the question?

Author response:
We sincerely appreciate this question, which led to a new light to further improve our methods.  In light of your suggestion, we included a justification for the rationality and motivation for training without textual annotations in Appendix A3.3. 
Please note that the usages of textual information in training and inference are different. We respectfully believe the inference needs the questions to identify the focus of interest, while the learning process can benefit from annotations but not necessary. We also extended our method to leverage the QA annotations. The detailed method, implementations, and experiments can be found in Appendix A5.


**Training Without Textual Pairs**
It would be really good if we could always have enough annotated question-answering pairs as the supervision. It would help the learning of the reasoning model. However. the annotated question-answering pairs are not easy to obtain, especially for tasks requiring domain knowledge such as traffic accident attribution. 
For humans, learning to reason can be effectively achieved without cross-modality supervision, relying solely on observation induction within shared feature spaces. This concept aligns with causality theories[R2 R3], suggesting the feasibility of uncovering latent variables and causal relationships unsupervised. 
Additionally, models trained with textual annotations are susceptible to language biases, where they might learn shortcuts based on linguistic patterns rather than understanding the content. For instance, if all questions in the training set beginning with "What is the reason" have similar answers, the model might generalize inaccurately, associating any question with this format to a specific answer, such as "White car speeding", regardless of context. This phenomenon, known as language bias, has been extensively discussed in question-answering research[R4, R5].

**Use of Text Queries in the Inference Phase**
Despite the avoidance of textual pairs in training, text queries play a crucial role during the inference phase. Our model, trained exclusively on visual data, can identify the visual causal relations of variables. During inference, textual queries are employed to first identify the local causal mechanisms we are interested in. Then, we leverage the causal relations to answer the detailed question. For example, if two accidents occur in the same video, our model will find causal mechanisms with two root causes. Then the query questions can help to identify the root cause we are more interested in. 

**Extend the Method to Leverage QA Pairs**
To learn the latent causal representation, we add two extra modules to the HCRN baseline, including an MLP Decoder and a Prior Aligner. The MLP Decoder uses HCRN's feature output to generate current observations, while the Prior Aligner assists in calculating KL divergence, essential for modeling transition functions and ensuring conditional independence with the prior. LLCRP, following HCRN's approach, first constructs latent causal representations using an encoder network identical to HCRN. Then, similar to HCRN, the output latent variables are learned with the supervision from language side. To plug the LLCP, we also apply a decoder to generate the observations from the latent space. To ensure the independence of latent variables, LLCRP imposes a KL divergence constraint between the posterior distribution of these variables and a predefined prior distribution. Finally, we also employ a reconstruction loss between the ground truth and generated observations.

As shown in the Table below, we report the performance of LLCRP on SUTD-TrafficQA and compare it with the original HCRN method. We can observe that learning the causal representation and process helps improve the accuracy of reasoning-based VideoQA tasks. On average, LLCRP achieves an improvement of almost 5% over the baseline HCRN.

| |B|A|I|C|F|R|Avg|
|-|-|-|-|-|-|-|-|
|HCRN|34.17|50.29|33.40|40.73|44.58|50.09|36.26|
|LLCRP|38.95|44.98|32.43|48.64|41.70|47.16|41.22|

[R2]Yao W, Sun Y, Ho A, et al. Learning temporally causal latent processes from general temporal data[C]. ICLR, 2021.

[R3]Yao W, Chen G, Zhang K. Temporally disentangled representation learning[J]. NeurIPS, 2022, 35: 26492-26503.

[R4] Desen Yuan. Language bias in visual question answering: A survey and taxonomy. arXiv preprint arXiv:2111.08531, 2021

[R5] Zhiquan Wen, Guanghui Xu, Mingkui Tan, Qingyao Wu, and Qi Wu. Debiased visual question answering from feature and sample perspectives. NeurIPS, 34:3784–3796, 2021

Reviewer comment (Presentation):
What does the arrows mean in Figure 5? Are they drew manually or automatically estimated by the model?

Author response:
The arrows denote the learned causal relations. $A \rightarrow B$ denotes that A causes B.  We manually draw the lines on the picture according to the learned causal structure.

Reviewer comment (Experiments):
The main concern is that the current setting is too far from realistic settings. The current evaluation setting is really more like hacking parts of existing datasets. The reviewer encourage the authors to make this work more complete.

Author response:
**A1:** We appreciate the suggestion which helped make this paper more complete and closer to realistic applications. 

**Make more complete experiments**
In light of this suggestion and the following detailed points, we have included more comprehensive evaluations, and provided more discussions as well as justifications for the experimental designs. Specifically, 
1) we extended our method by using QA annotations (as a plug-in module) and compared it with the baseline in Appendix A5;
2) we added the comparisons and discussions with the zero-shot multi-modal LLM models, such as OpenFlamingo, in Appendix A7.2;
3) we included more justifications on rationality and motivation for training without textual annotations in Appendix A3.3;
4) we clarified the experimental settings of Causal-VideoQA;
5) we analyzed a principle way to use LLCP for broader settings.

**Justify the reality of the current setting**
We appreciate your variable questions about the experimental settings and enjoy further discussions on its reality. 
We respectfully believe that our experimental settings focus on a more difficult and realistic scenario instead of a hacking of the existing datasets. 
Though textual annotations can be meticulously prepared for the training of the reasoning models to enhance the model's ability, in many applications, such as traffic accident reasoning, it is unrealistic to prepare large-scale well-organized textual annotations for the reasoning questions, since it is usually time-consuming, labor-intensive and requires expert knowledge.
Therefore, it is realistic and important to address the reasoning QA problem in an unsupervised manner. However, there are no published datasets for this problem. To solve this problem, we employ the existing reasoning QA datasets with textual annotation and compare them with the existing supervised methods, which also shows the advantages of our methods.

Reviewer comment (Experiments):
It is too constraint to only evaluate the proposed method when it has no access to QA labels. If the obtain model really captures the casual relationship in videos, plugging it into existing methods in the supervised training setting can show a much more broad application of the proposed method.

Author response:
**A2:** We are grateful for this suggestion which makes our evaluations more complete. In light of your suggestions, we proposed to plug LLCP into existing methods in the supervised training setting, such as the HCRN [1] method, and provided the experimental results on the SUTD-TrafficQA datasets. The proposed method plugging the LLCP learns the both latent causal representation and process, which is called LLCRP. Please kindly refer to Section A5 in the revised appendix for the detailed model design and experimental evaluations. Here, we provide a brief summary of LLCRP and the experimental comparison.

**Implementation Details** 
To learn the latent causal representation, we add two extra modules to the HCRN baseline, including an MLP Decoder and a Prior Aligner. The MLP Decoder uses HCRN's feature output to generate current observations, while the Prior Aligner assists in calculating KL divergence, essential for modeling transition functions and ensuring conditional independence with the prior. LLCRP, following HCRN's approach, first constructs latent causal representations using an encoder network identical to HCRN. 
Then, similar to HCRN, the output latent variables are learned with the supervision from language side. 
To plug the LLCP, we also apply a decoder to generate the observations from the latent space. 
To ensure the independence of latent variables, LLCRP imposes a KL divergence constraint between the posterior distribution of these variables and a predefined prior distribution. Finally, we also employ a reconstruction loss between the ground truth and generated observations.

**Experimental Results** 
As shown in the Table below, we report the performance of LLCRP on the SUTD-TrafficQA dataset and compare it with the original HCRN method. We can observe that learning the causal representation and process helps improve the accuracy of reasoning-based VideoQA tasks. On average, LLCRP achieves an improvement of almost 5\% over the baseline HCRN. 

|  | B| A| I | C | F | R | Avg |
|--|---|--|---|--|--|--|--|
| HCRN| 34.17 | 50.29 | 33.4  | 40.73 | 44.58 | 50.09 | 36.26 |
| LLCRP| 38.95 | 44.98 | 32.43 | 48.64 | 41.70  | 47.16 | 41.22 |

Reviewer comment (Experiments):
It is also not realistic to exclude the text query from the training process since the fusion between visual and textual input is also the crucial design to really solve the VQA problem. For example, if there are two accidents going on in the video, the current framework will have systematical flaw as the model is not conditioned on the question text which specifies which accident it is about.

Author response:
**A3:** Thanks for your insightful comments. The discussions on the leverage of text query definitely help elaborate the underlying insights of our method. We have refined the paper to add a separate subsection A3.3 in the appendix for this discussion. 

**The textual information in training and inference**
We discuss the usage of text query from two points of view, including the training process and the inference process. We respectfully think that text queries are extremely useful during the testing phase for pinpointing questions and finding precise answers (such as the example in the question). However, in the training phase, the use of text queries, or text pair supervision, is also useful but not indispensable, if we have the aligned feature spaces of visual and textual domain. In our implementation, LLCP also involves the textual query to identify the question in the inference phase, even if the textual pairs are not used as supervision. In the provided example, our model is trained on visual data only, which will identify two potential accident candidates. During the inference phase, we utilize the query question to first pinpoint the candidates of interest. Subsequently, we use the selected candidates to accurately respond to the query.


**Rationality and motivation for training without textual pairs**
First, for human beings, we can effectively learn reasoning ability without cross-modality supervision by only leveraging the induction of observations. Similarly, the theorems in the field of causality show that we can identify the latent variables and causal relations in an unsupervised way. They can serve as the support of the rationality to learn the causal relations without the textual supervision. Then we show this ability of learning to reason without annotations is well-motivated in real-world applications. First, on the difficulty of data collection, the annotated textual pairs are more than the unannotated videos. It would raise extra costs to annotate the data with neural language. Second, the model is easy to be misled by the short-cut in the textual annotations. For example, for all questions in the training set beginning with "What is the reason", the answer is similar, such as "White car speeding".  Then the model is easy to learn this shortcut and responds to the "White car speeding" for any questions beginning with "What is the reason". This issue caused by the training data is called language bias, which is widely discussed in the field of question answering [R1, R2].


[R1] Desen Yuan. Language bias in visual question answering: A survey and taxonomy. arXiv preprint arXiv:2111.08531, 2021

[R2] Zhiquan Wen, Guanghui Xu, Mingkui Tan, Qingyao Wu, and Qi Wu. Debiased visual question answering from feature and sample perspectives. Advances in Neural Information Processing Systems, 34:3784–3796, 2021

Reviewer comment (Experiments):
Despite the effort of re-training many existing methods, it is not well-justified why it is necessary to discard important features like motion or object as used in Causal-Vid-QA, which brings all the models to a low-performance scheme.

Author response:
**A4:** Thanks for the suggestion to further clarify and justify the experimental settings. We double-checked the implementations of the re-training methods and confirmed that all motion and object features are used in the re-training process. For the methods using the motion features, such as HCRN, we apply the clip average of CLIP features as the motion features.  For the methods using the textual object features to build graphs, such as B2A, we also follow the original strategy to accomplish this. The main difference between the results of our re-training methods and that reported in Causal-Vid-QA is the basic features, i.e., we use the CLIP features for both visual and textual domains in the re-training while the original ones used ResNet101 and 3D-ResNeXt-101 visual features and GloVe/BERT textual features. As shown in Table A7 in the revised version (Table A6 in the original paper), we have included a comparison between the results of re-training methods and reported results. Compared with the methods using GloVe features, our re-training ones even improve the performances. Actually, the performance is sensitive to the textual features as the order of performance: BERT > CLIP > GloVe. It makes sense since the backbone cost is different. 


Comparisons under different features (backbones) are unfair, which motivated us to re-train all the methods with CLIP features. We apply the CLIP features since they satisfy the assumption well that visual and textual feature spaces are shared. We show that under the same backbone features, our unsupervised LLCP can achieve comparable results with other supervised ones, and significantly improve the baseline CLIP method.

Reviewer comment (Experiments):
There is no proper comparison with methods that do not require QA data like but not including to [a,b,c]. The authors should also acknowledge and at least provide comparison with some of these relevant methods to really provide the audience a correct and comprehensive understanding of the relevant solution to this setting.

Author response:
**A5:** We appreciate the valuable suggestion that includes the discussion with LLM-based methods. In light of your suggestion, we have added the comparison between our method with Flamingo[a] on SUTD-TrafficQA and are trying our best to evaluate VidIL[b].  We didn't include the comparison with [c] since its code is not publicly released. We applied the open-source alternative OpenFlamingo[R3] instead of Flamingo[a] in our implementation.  We provided the experimental comparisons and a brief discussion below. Please refer to Appendix A7.2 for a detailed comparison and discussion. 

We evaluate all the models on SUTD-TrafficQA with a zero-shot setting. 
For OpenFlamingo, we used the 2/4 textual demo examples (no visual data due to zero-shot) to guide the outputs. Here is an illustration of the prompt.
```
for i in range(len(questions)):
    token_inputs.append(
        " Question: What might have happened moments ago? 
         Candidate 1: The blue truck hit the white sedan from the back. 
        Candidate 2: The white sedan crashed into the blue truck. 
        Candidate 3: The blue truck did an emergency brake. 
        Candidate 4: The white sedan lost its control. 
        Answer: The blue truck did an emergency brake.
       

        <image>
        Question:{} Candidate1:{} Candidate2:{} Candidate3:{} candidate4: {} Answer:".format(
            questions[i], 
            ans_candidates[0][i], 
            ans_candidates[1][i], 
            ans_candidates[2][i], 
            ans_candidates[3][i]
        )
    )
```
As shown in the Table below, we summarize the average accuracy of OpenFlamingo, CLIP, and LLCP. We found that these LLM models don't work as well as we expected since no particular knowledge is used. It is interesting and may inspire the following research. 
Currently, the evaluation of VidIL is in the process. We currently face some issues with leveraging GPT3 to get the response. We are trying our best to reproduce the strategy used in the method, but it still needs some time. The results will be updated later once we finish the evaluation. 
Please kindly note that VidIL is a few-shot method, not zero-shot. But we are glad to add it to the comparison since it can be converted to the zero-shot setting. 

| Model | Accuracy |
|--|--|
| OpenFlamingo-3B (2 examples)   | 28.7  |
| OpenFlamingo-3B (4 examples)   | 29.1 |
| VidIL | in processing  |
| CLIP  | 27.7  |
| LLCP   | 33.7   |

[R3] Awadalla A, Gao I, Gardner J, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models[J]. arXiv preprint arXiv:2308.01390, 2023.

Reviewer comment (Experiments):
Once a and b are done, it is also necessary to provide additional comparison on broader VideoQA datasets to understand the importance of causal learning process in videos for broader videoQA tasks, which is really beneficial for the community.

Author response:
**A6:** Thanks for your suggestions about the comparisons on broader VideoQA datasets. We appreciate your kind expectations for the improvement of our method on the broader VideoQA tasks. 

**Why we didn't consider broader VideoQA datasets**
We would like to respectfully highlight that LLCP is a framework designed for the reasoning-based VideoQA task.
It is based on a pre-trained cross-modality matching model, which demonstrates that given the shared visual and textual representation space, we can **further** learn the reasoning ability without any textual annotations.  It indicates that we assume that the pre-trained cross-modality matching models, such as CLIP or OpenFlamingo can solve the broader VideoQA tasks (described questions). 
There are two justifications for this assumption. 

1) Most current VideoQA models for described tasks are trained as cross-modality matching. 

2) Existing zero-shot methods (the models are pre-trained with cross-modality matching and are used for inference in a zero-shot manner) perform well on these reasoning-free tasks. 

However, as shown in Table A6, these pre-trained models (CLIP and OpenFlamingo) don't perform well on the reason-based benchmarks since they require a reasoning process which is not easy to get from the large-scale pretraining dataset. 


**The general procedure using LLCP for reasoning-free tasks**
In light of your suggestion, we propose a general procedure to apply the LLCP to solve the broader reasoning-free VideoQA tasks. For a given query, we first classify whether this question is reasoning-based (it is easy since the reasoning-based questions have clear patterns). Then, we use the pre-trained cross-modality aligner for the reasoning-free tasks and use LLCP for the reasoning-based ones. Actually, the results in Table 4 in the revised version (original Table 5) are obtained by this procedure. We compared both the reasoning-free ("Basic Understanding", "Event Forecasting", "Reverse Forecasting") and reasoning-based  ("Counterfactual Inference", "Introspection", and "Attribution") questions. We take the re-weighting average on all types of questions in Table 4 in the revised version.

Reviewer comment (Writing):
Minor: Title, related work: Video Question Answer -> Video Question Answering.

Author response:
**A7:** Thanks a lot for pointing it out!! We have corrected this typo in the revised version. Please kindly see our revised Appendix A1.