Paper: Actions-to-Action: Inductive Attention for Egocentric Video Action Anticipation (ICLR-2024)

Reviewer comment (Writing):
The quality of paper writing is not always up to standard. Some sentences are a bit overselling, unclear, or not adequate for scientific writing. For example: “by integrating gaze information within the observed frames”: is the paper referring to a specific work here? As far as I know, gaze analysis for intention prediction has not been systematically investigated in egocentric vision.

Author response:
We have revised the following sentences: “Furthermore, video action anticipation frequently employs egocentric videos, which not only harmonize the perspectives from diverse subjects but also implicitly unveil their intentions by integrating gaze information within the observed frames” to “Furthermore, video action anticipation frequently employs egocentric videos, which harmonize perspectives from diverse subjects and implicitly unveil their intentions. This is achieved by integrating elements such as coarse-grained visual attention, indicated by the camera’s heading direction, within the observed frames.“

Reviewer comment (Writing):
The statement “Unlike action recognition, which primarily relies on patter recognition” seems to imply that action anticipation does not rely on pattern recognition, which is not accurate, as neural networks are anyway pattern recognition machines.

Author response:
We agree reviewer’s insightful comment. We revised the original sentences “Unlike action recognition, which primarily relies on pattern recognition, video action anticipation explores the complex nature of potential future actions, each with multiple possibilities.” to “While action recognition focuses on classifying current actions through pattern recognition, video action anticipation uses these patterns to predict the complex nature of potential future actions, each with multiple possibilities.”

Reviewer comment (Evaluation):
The claim “our model can infer causation from abstract concepts” is a bit overselling and does not seem to be shown or proved in any of the experiments.

Author response:
We have down-tone our sentences and rewritten “Mimicking human reasoning, our model can infer causation from abstract concepts without relying on finer details like pixels.” to “Our model detects video patterns to predict future actions, using past conditions to enhance understanding of complex interactions, focusing more on overall patterns than on specific details like pixels.”

Reviewer comment (Evaluation):
The statement “our innovative model sets new performance benchmarks” does not seem accurate, as the proposed model appears to outperform competitors only by small margins. It would be better to highlight that it may point to a promising direction for future models.

Author response:
We revised the original sentences from “Experimental results and in-depth analyses confirm that our innovative model sets new performance benchmarks, while also competing effectively in both model size and computational efficiency.” to “Experimental results and in-depth analyses indicate that our model achieves competitive performance gains and suggests a promising direction for future models in the field. Additionally, it maintains an effective balance in model size and computational efficiency.“

Reviewer comment (Writing):
In section 3, the set X is later referred to as $X_{T-t_s:t}$, which is a bit confusing.

Author response:
To improve the readability, we revised the sentence to “at every timestep $t$, the model uses the observed input up to time $t$, denoted as $X_{*:t}$, to predict the subsequent action”

Reviewer comment (Theory):
It is not clear how equation (1) is an accurate description of existing methods, whether recurrent or transformer-based. The equation seems to suggest that models following this formulation explicitly plug in the last observed action for predicting the future one. However, methods that directly predict future actions, such as vanilla LSTMs, do not do that.

Author response:
We revise the equation (1) to be more general and should be much clearer to highlight our design is to add the explicit feedback from the classifier of past timestep and utilize it in the attention mechanism.

Reviewer comment (Writing):
In equation (2), it is unclear whether $\hat y_i$ is a probability or a predicted label.

Author response:
We apologize for the confusion. Both $y_\*$ and $\hat{y_{\*}}$ represent probabilities. However, $y_*$ is the ground-truth label, where only the top-1 action is annotated, thus the probability distribution can be considered a Dirac distribution. We have revised all the sentences which mentioned $y_*$ or $\hat{y}_*$ to a probability rather than described as a label.

Reviewer comment (Writing):
In Section 5.3.1, the sentence “IAM demonstrated notable e enhancement over several formidable baselines” could be revised, particularly the term “formidable,” which may be unsuitable in a scientific context.

Author response:
We thank the reviewer's suggestion. We have revised by using “well-established” rather than “formidable”.

Reviewer comment (Writing):
Throughout the paper, there is no clear motivation for the use of the term “inductive” in “inductive attention.” This could be clarified.

Author response:
We appreciate your observation regarding the term "inductive" used in the context of our "inductive attention" mechanism. The choice of this term was intended to reflect the specific way in which our model processes and learns from video data.

In our model, "inductive" refers to the process of making generalized predictions about future actions based on specific observed patterns and sequences within the video data. This is akin to the inductive reasoning process in human cognition, where conclusions or generalizations are drawn from specific instances or observations.

Our model learns to anticipate future actions by identifying and generalizing patterns observed in the training data. It extrapolates from these specific instances to make broader predictions, a process that mirrors the inductive reasoning used in human learning and decision-making. In addition, the term "inductive" in the context of attention mechanism signifies that the model selectively focuses on certain aspects of the input data (observed actions and interactions) to inductively predict future actions. This involves not just recognizing patterns but also using these patterns to infer what actions are likely to occur next.

By using the term "inductive," we aim to emphasize the model's ability to generalize from observed data to predict future actions, a key aspect that differentiates it from other attention-based models which may focus more on direct pattern matching or deductive reasoning processes.

Reviewer comment (Theory):
There is little insight into why the model works. While the architecture is described in Equations (11)-(14), there is no discussion as to why the introduced modifications are adequate or what kind of processing they intuitively bring to the model. There is also a lack of explanation for why the attention mechanism is tweaked in its particular way.

Author response:
We thank the reviewers for their interest in our inductive attention model. We have revised Section 3 of our manuscript to provide a clearer overview of our proposal. The inductive attention is designed to utilize previous predictions as prior input for the next timestep. By doing so, we first (1) extend our recurrent model to a higher order, explicitly forming a context window during each recurrent update; (2) utilize attention to aggregate the higher order, assigning the query (Q) with the previous prediction. Correspondingly, the keys (K) are also in the action space (obtained from the prediction of that time) to represent each past recurrent state. All these steps are mentioned in Tables 6 and 7. In the revised version, we separated parts of the original Table 6 into Table 7 for a more comprehensive view.

Reviewer comment (Experiments):
While ablation studies detail the weight of each macro-component to performance (Table 6), it would have been interesting to see a more detailed ablation into the various modifications introduced by each of the equations (11)-(14) with respect to a baseline attention architecture.

Author response:
We thank the reviewers for their interest in our inductive attention model. We have revised Section 3 of our manuscript to provide a clearer overview of our proposal. The inductive attention is designed to utilize previous predictions as prior input for the next timestep. By doing so, we first (1) extend our recurrent model to a higher order, explicitly forming a context window during each recurrent update; (2) utilize attention to aggregate the higher order, assigning the query (Q) with the previous prediction. Correspondingly, the keys (K) are also in the action space (obtained from the prediction of that time) to represent each past recurrent state. All these steps are mentioned in Tables 6 and 7. In the revised version, we separated parts of the original Table 6 into Table 7 for a more comprehensive view.

Reviewer comment (Experiments):
The proposed algorithm outperforms competitors when a single modality is considered, but some approaches outperform it when multiple modalities are used. It would be interesting to see how the proposed approach performs when multiple modalities are considered, even with simple late fusion. This would help assess the generalizability of the approach to signals other than RGB images.

Author response:
We thank the reviewer for their interest in the multi-modality aspect of our research. Multi-modality is indeed a promising direction for our future work and warrants further exploration. Our proposed method is flexible enough to consider multi-modality in each recurrent update. For instance, the inductive attention can utilize different action structures (such as verbs or nouns) to select various parts of the recurrent state and modality.

In this work, we primarily focus on RGB and demonstrate that our model, relying solely on RGB video, can outperform most current methods with more efficient modeling. Additionally, Tables 1 and 2 include single-modal variants of the multi-modal baseline. For example, AVT+/AVT++ and AFFT/AFFT+ are compared, and our method surpasses their single-modal configurations.

Reviewer comment (Evaluation):
There seems to be less significant improvement in the performance with the inductive attention mechanism module. For example, in Table 1, when comparing Swin-IAM with MeMViT 32x3 there is only a performance improvement of 0.4% on actions, almost none on verbs, and 0.2% on nouns.

Author response:
While the quantitative improvements shown in Table 1 might seem modest (0.4% on actions, almost none on verbs, and 0.2% on nouns), these gains are significant in this task's context. In a highly competitive field where models already perform at high levels, these incremental improvements underscore our approach's effectiveness in refining the model's predictive capabilities.

Reviewer comment (Writing):
There can be a grammar check run on the paper text. For example, the first line of section 3, problem statement can be edited. Additionally, multi-modal and multi-model terms have been used interchangeably in abstract and results table. The last line of the abstract can also be checked - 'multi-modality models using only RGB visual inputs' whereas multi-modal approaches have more modalities than visual input.

Author response:
Thank you for pointing out the grammatical issues and the inconsistency in the use of terms in our manuscript. We have carefully revised the first sentence of Section 3 to ensure it clearly and accurately conveys our intended message. The revised sentence now reads “Video action anticipation is defined as a task where an arbitrary length, $t_s$, of video inputs $X = \{x_i \in \R^{C \times H \times W} | i = T-t_s, \dots, T\}$, starting from timestep $T-t_s$ to $T$, is used to predict the target action that happens in the future, $\tau_a$ seconds later.”

Regarding the last line of the abstract, we have revised it to accurately reflect the nature of our models. The revised line now reads: “...our proposed model surpasses most multi-modal models by only using RGB visual inputs…”. This change clarifies that our models, while primarily utilizing RGB visual inputs, result in competitive performance to the models using more modalities.

Reviewer comment (Presentation):
The association with object tracking in the introduction and Figure 1 also seems a bit misplaced as it seems for the reader that object tracking would be used for the task of action anticipation, which is not actually used.

Author response:
We apologize for any confusion caused. Figure 1 is intended to illustrate the underlying similarity between action anticipation and object tracking. In object tracking, the previous bounding box location is used as a prior to predict the next bounding box. Similarly, action anticipation can benefit from feeding back past action predictions and conditioning them to the target action. This concept forms the main motivation for our inductive attention mechanism.

Reviewer comment (Novelty):
Originality is somewhat limited as using prior predictions to condition the target prediction has been applied to other problems. See reference

Author response:
Thank you for pointing out the existing applications where prior predictions are used to condition target predictions. We acknowledge that this concept is not entirely new in the broader field of machine learning and has indeed been explored in various contexts. However, our work distinguishes itself in several key aspects, particularly in the domain of egocentric video anticipation.

First Application in Egocentric Video Anticipation: To the best of our knowledge, ours is the first study to apply the concept of using prior predictions to condition target predictions specifically to the egocentric video anticipation problem. This application is novel in this context and addresses unique challenges inherent to egocentric video analysis.

Innovative Model Design: Our paper introduces a novel model that synergistically combines higher-order recurrent and attention mechanisms. This integration is specifically tailored to leverage the dynamics of egocentric videos, a domain where temporal context and rapid shifts in the visual field present unique challenges.

Improved Generalization and Efficiency: The results from our experiments demonstrate that utilizing prior predictions in our specific model architecture not only enhances generalization across different scenarios but also leads to a more efficient model design. This efficiency is quantifiable in terms of the reduced number of model parameters required, which is a significant advancement in resource-constrained environments.

Empirical Evidence (see Figure 3 and new Figure 7 and 8 in the supplementary material): We provide empirical evidence to support our claims, for example the topmost positive and negative prior learned by the model, showcasing how our approach outperforms existing methods in terms of accuracy and efficiency, particularly in the context of egocentric video anticipation.

In conclusion, while the foundational concept of using prior predictions may have precedents in other domains, our application of this idea to egocentric video anticipation, coupled with our novel model design, represents a significant original contribution to the field. We believe our approach sets a new precedent in this specific area and opens up avenues for further innovative research.

Reviewer comment (Novelty):
The choice of egocentric action anticipation problem for the proposed method is not well motivated. There is no inherent advantage of the proposed method for egocentric action anticipation, compared to other video prediction problems, e.g. physical interaction/dynamics, 3PV action anticipation, gaze anticipation/prediction etc.

Author response:
Thank you for your query regarding the specific focus on the egocentric action anticipation problem in our study. The choice of this domain was driven by both the unique challenges it presents and the specific advantages our proposed method offers in addressing these challenges.
Egocentric action anticipation involves predicting future actions from a first-person perspective, which is fundamentally different from third-person views (3PV) or other video prediction tasks. This perspective is characterized by rapid, unpredictable motion, a high degree of camera wearer's interaction with the environment, and a need for understanding subtle cues from the wearer's viewpoint.

1. Advantages of Our Method in Egocentric Context:

- Handling Perspective Shifts: Our model's integration of higher-order recurrent and attention mechanisms is particularly effective in managing the perspective shifts typical in egocentric videos. This perspective shift can be unintentional (i.e., distraction) or intentional (relevant to the target action), and learned form the data.

- Leveraging Temporal Context: The egocentric video stream provides a rich temporal context that our model exploits, using prior predictions to inform future anticipations. This is particularly beneficial in first-person videos where sequential actions are closely linked.
Efficient Processing of Dense Interactions: Egocentric videos often contain dense interactions with objects and environments. Our model's efficient design allows it to process these interactions without the need for extensive computational resources.

2. Comparative Advantage over Other Domains:

- Physical Interaction/Dynamics: While our model could potentially be applied to physical interaction dynamics, the first-person perspective offers a more direct and immediate view of interactions, making it a more challenging and thus insightful testbed for our method.
3PV Action Anticipation: Third-person views present different challenges and often lack the immediacy and intimacy of interaction present in egocentric videos, which our method is specifically designed to handle.
- Gaze Anticipation/Prediction: Although gaze anticipation is related, it often focuses on predicting the focal point of attention rather than the broader spectrum of actions, which is the focus of our model.
In summary, our decision to focus on the egocentric action anticipation problem was motivated by the unique challenges it presents and the suitability of our proposed method in addressing these challenges. We believe that the insights gained from this domain can be instrumental in advancing the broader field of video action anticipation.

Reviewer comment (Novelty):
Significance is average. While the problem of action anticipation is interesting, it is not clear how the proposed method can be applied to other related problems.

Author response:
The field of video action anticipation involves predicting target actions with limited observed information. Our method demonstrates efficiency in addressing this challenge. The core idea is to utilize historical predictions of previous actions as a prior, conditioning the input. While our proposed model can be adapted to other related problems sharing similar properties, some tailored modifications may be necessary. For instance, gaze prediction might lean more towards regression than classification. Additionally, third-person view (3PV) action anticipation could overlook critical features such as the subject's intention.

Reviewer comment (Experiments):
I will be interested to see the results comparison for S=1.

Author response:
We have updated Table 7 (originally is part of table 6) in our manuscript to include these results, thereby providing a more comprehensive view of the performance impact when the inductive attention component is not utilized.

In the absence of inductive attention, which is a combination of higher-order processing and the use of predictions as priors, we observed the overall action score drop of 1.3%/1.0%/1.1% decreased in the different backbone configuration in terms of $S=1$.

These results highlight the significant contribution of the inductive attention mechanism to our model's performance. The reduction in action scores when inductive attention is removed underscores its effectiveness, particularly in enhancing the model's ability to accurately anticipate actions in the S=1 setting.

This detailed analysis strengthens our assertion that the inductive attention mechanism, with its integration of higher-order processing and predictive priors, is a crucial component of our model, contributing substantially to its overall efficacy in action anticipation tasks.

Reviewer comment (Evaluation):
In Table 6, the ablation study shows that one of the major designs of this paper (predictions as prior) doesn't play a major role in the final performance improvement. The performance improvement is highly attributed to some design choices like a better backbone and class weighting.

Author response:
Thank you for your observation regarding the ablation study in Table 6 (in the revised manuscript table 6 and 7). Our paper introduces an innovative inductive attention model, fundamentally integrating higher-order recurrent and attention mechanisms with a focus on using predictions as priors (serve as attention query and keys). It's important to recognize these elements as integral to our algorithm's design and performance.

Class weighting is indeed widely adopted for the long-tailed distributed dataset such as EPIC-Kitchens, as carried out in MeMViT (Wu et al. 2022). While it falls outside the primary scope of our research, it undeniably enhances our method by ensuring a more equitable exposure of different categories within the training set.

To provide a clearer picture of our method's effectiveness, we have revised Table 6 with the following enhancements: (1) Inclusion of Unseen and Tail Scores: Alongside overall scores, this allows for a more comprehensive evaluation (2) Distinct Presentation of Different Backbones: Ensuring that the impact of backbone choice is transparent and understandable. (3) Aggregation of Components: We've combined “Prior predictions serve as attention query (Q) and keys (K) ” and “Extend to higher-order (S=1 to S=30)” of the table 7 into a single “Inductive Attention” category in the table 6. This change reflects the interdependence of these features, as inductive attention cannot function in isolation.

It's important to note that all improvements in our model build upon the results achieved through class weighting. Thus, directly comparing the incremental improvements of our method with those attributed to class weighting is not straightforward, as the enhancements are not linearly additive.

Finally, we emphasize that our model not only shows significant improvements over previous works in terms of overall, unseen, and tail action scores but also achieves this with considerably fewer model parameters. This aspect underscores the efficiency and innovation of our approach.

Reviewer comment (Evaluation):
Lack of analysis and visualization of the proposed mechanism. For example, how do previous predictions affect future action anticipation?

Author response:
Thank you for highlighting the need for a deeper analysis and visualization of our proposed inductive attention mechanism, especially regarding the influence of previous predictions on future action anticipation. We have included new evidence in Figure 3, and Figure 7 and 8 in the supplementary material, showing the topmost positive and negative previous prediction entry inductive attention referenced related to the final prediction. It is important to mention that without the previous predictions as prior, the model can only base on the recurrent hidden states $h_t$ and as shown in the Table 6, resulting in -1.3%, -1.0%, -1.1% absolute overall action scores drop.

We believe these additions will greatly enhance the understanding of our proposed mechanism and its efficacy in improving action anticipation in egocentric video analysis.

Reviewer comment (Experiments):
Is the proposed model able to also handle the long-term action anticipation tasks (i.e. predicting multiple future actions) defined in Ego4D (grauman2022ego4d) and EgoTOPO (nagarajan2020ego).

Author response:
Thank you for your question regarding the applicability of our proposed model to long-term action anticipation tasks, as exemplified in datasets like Ego4D (Grauman et al., 2022) and EgoTOPO (Nagarajan et al., 2020).

In our current work, the focus has been predominantly on short-term action anticipation with a temporal anticipation window $\tau_a$ set to 1s/0.5s. This decision was based on a well-established evaluation protocol of the dataset design of EPIC-Kitchens/EGTEA Gaze+ evaluation protocols from previous works.

Long-term action anticipation, especially in datasets like Ego4D, introduces additional challenges. These include dealing with more extended temporal dependencies and the need for a broader contextual understanding to accurately predict a sequence of future actions.
We recognize the importance and potential of extending our model to these long-term tasks. As such, we have earmarked this as a key area for future research. Our planned extensions will involve adapting and enhancing the current model to handle longer temporal sequences and more complex action interdependencies characteristic of datasets like Ego4D and EgoTOPO.