Original: U NIFIED R ECURRENCE M ODELING FOR V IDEO A CTION A NTICIPATION
Revised: U NIFIED R ECURRENCE M ODELING FOR V IDEO A CTION A NTICIPATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Forecasting future events based on evidence of current conditions is an innate skill of human beings, and key for predicting the outcome of any decision making.
Revised: Forecasting future events based on evidence of current conditions is an innate skill of human beings, and key for predicting the outcome of any decision making.

Original: In artiﬁcial vision for example, we would like to predict the next human action before it actually happens, without observing the future video frames associated to it.
Revised: In artiﬁcial vision for example, we would like to predict the next human action before it actually happens, without observing the future video frames associated to it.

Original: Computer vision models for action anticipation are expected to collect the subtle evidence in the preamble of the target actions.
Revised: Computer vision models for action anticipation are expected to collect the subtle evidence in the preamble of the target actions.

Original: In prior studies recurrence modeling often leads to better performance, and the strong temporal inference is assumed to be a key element for reasonable prediction.
Revised: In prior studies recurrence modeling often leads to better performance, and the strong temporal inference is assumed to be a key element for reasonable prediction.

Original: To this end, we propose a uniﬁed recurrence modeling for video action anticipation by generalizing the recurrence mechanism from sequence into graph representation via message passing.
Revised: To this end, we propose a uniﬁed recurrence modeling for video action anticipation by generalizing the recurrence mechanism from sequence into graph representation via message passing.

Original: The information ﬂow in space-time can be described by the interaction between vertices and edges, and the changes of vertices for each incoming frame reﬂects the underlying dynamics.
Revised: The information ﬂow in space-time can be described by the interaction between vertices and edges, and the changes of vertices for each incoming frame reﬂects the underlying dynamics.

Original: Our model leverages self-attention for all building blocks in the graph modeling, and we introduce different edge learning strategies that can be end-to-end optimized while updating the vertices.
Revised: Our model leverages self-attention for all building blocks in the graph modeling, and we introduce different edge learning strategies that can be end-to-end optimized while updating the vertices.

Original: Our experimental results demonstrate that our modeling method is light-weight, efﬁcient, and outperforms all previous works using the large-scale EPIC-Kitchen dataset.
Revised: Our experimental results demonstrate that our modeling method is light-weight, efﬁcient, and outperforms all previous works using the large-scale EPIC-Kitchen dataset.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Video action recognition is a long-standing problem in computer vision.
Revised: Video action recognition is a long-standing problem in computer vision.

Original: It predicts the action from experienced frames by understanding the contexts in the observations.
Revised: It predicts the action from experienced frames by understanding the contexts in the observations.

Original: However, having the prediction after the whole observation of action is insufﬁcient to evaluate the outcome of any decision making.
Revised: However, having the prediction after the whole observation of action is insufﬁcient to evaluate the outcome of any decision making.

Original: Predicting future human activities and interactions of objects are receiving research interests in these years, in assisting navigation systems (OhnBar et al., 2018)
Revised: Predicting future human activities and interactions of objects are receiving research interests in these years, in assisting navigation systems (OhnBar et al., 2018)

Original: , robotics (Park et al., 2016), entertainment (Liang et al., 2015; Taylor et al., 2020) and autonomous vehicles (Hirakawa et al., 2018).
Revised: , robotics (Park et al., 2016), entertainment (Liang et al., 2015; Taylor et al., 2020) and autonomous vehicles (Hirakawa et al., 2018).

Original: Different from action recognition, the video action anticipation is required to give the predictions strictly before the action frames being observed.
Revised: Different from action recognition, the video action anticipation is required to give the predictions strictly before the action frames being observed.

Original: Models, in this case, are expected to collect the subtle evidence in the preamble of target actions (Furnari et al., 2018; Furnari & Farinella, 2020;
Revised: Models, in this case, are expected to collect the subtle evidence in the preamble of target actions (Furnari et al., 2018; Furnari & Farinella, 2020;

Original: Rodin et al., 2021).
Revised: Rodin et al., 2021).

Original: In the high-level viewpoint, the objective of anticipating is not only to understand the actual content in the observations but seeking to associate the minor details to the next possible action.
Revised: In the high-level viewpoint, the objective of anticipating is not only to understand the actual content in the observations but seeking to associate the minor details to the next possible action.

Original: Figure 1 illustrates the deﬁnition of the video anticipation problem.
Revised: Figure 1 illustrates the deﬁnition of the video anticipation problem.

Original: In the absence of target action frames, pretrained models for the robust feature extraction from
Revised: In the absence of target action frames, pretrained models for the robust feature extraction from

Original: RGB inputs are often deployed on video anticipation, either pretrained from image classiﬁcation or video action recognition.
Revised: RGB inputs are often deployed on video anticipation, either pretrained from image classiﬁcation or video action recognition.

Original: On top of the pretrained models, recurrent neural networks are the widely adopted options to model the temporal relationship in anticipation problems and lead to better performance than clip based modeling methods.
Revised: On top of the pretrained models, recurrent neural networks are the widely adopted options to model the temporal relationship in anticipation problems and lead to better performance than clip based modeling methods.

Original: Unlike the action recognition settings, where the relevant frames associated with the target action are not necessary in the end of observation and may locate in any action duration.
Revised: Unlike the action recognition settings, where the relevant frames associated with the target action are not necessary in the end of observation and may locate in any action duration.

Original: video anticipation predicts the action that always performs in the future of observations, making the accumulated information gains aligned with the temporal orders.
Revised: video anticipation predicts the action that always performs in the future of observations, making the accumulated information gains aligned with the temporal orders.

Original: To this end, we propose a uniﬁed recurrence modeling for video action anticipation by generalizing the recurrence mechanism from a sequence into graph representation via message passing.
Revised: To this end, we propose a uniﬁed recurrence modeling for video action anticipation by generalizing the recurrence mechanism from a sequence into graph representation via message passing.

Original: We utilize self-attention as the universal building block in the graph modeling.
Revised: We utilize self-attention as the universal building block in the graph modeling.

Original: In the graph point of view, selfattention can be treated as the information routing between vertices.
Revised: In the graph point of view, selfattention can be treated as the information routing between vertices.

Original: The attention weight derived by the scaled dot-product, which computes the correlation of vertices, can be explained as an adjacency estimation implicitly.
Revised: The attention weight derived by the scaled dot-product, which computes the correlation of vertices, can be explained as an adjacency estimation implicitly.

Original: However, in this way, the representation of edges is limited and purely based on the similarity in input tensors.
Revised: However, in this way, the representation of edges is limited and purely based on the similarity in input tensors.

Original: Therefore, we propose three explicitly edge learning strategies trying to escape to the trivial estimation by bringing the ﬂexibility of edge representations.
Revised: Therefore, we propose three explicitly edge learning strategies trying to escape to the trivial estimation by bringing the ﬂexibility of edge representations.

Original: They are end-to-end optimized and can be jointly trained with the main task.
Revised: They are end-to-end optimized and can be jointly trained with the main task.

Original: In the experimental results, we show our proposed uniﬁed recurrent modeling outperforms several state-of-the-art methods on largescale dataset EPIC
Revised: In the experimental results, we show our proposed uniﬁed recurrent modeling outperforms several state-of-the-art methods on largescale dataset EPIC

Original: Kitchen on video anticipation.
Revised: Kitchen on video anticipation.

Original: Combining with edge learning strategies, the performance can even boost to the next level.
Revised: Combining with edge learning strategies, the performance can even boost to the next level.

Original:  R ELATED WORKS 2. V IDEO A CTION A NTICIPATION
Revised:  R ELATED WORKS 2. V IDEO A CTION A NTICIPATION

Original: Early works in video anticipation model the problem with recurrent neural networks (Gao et al., 2017; Farha et al., 2018; Miech et al., 2019).
Revised: Early works in video anticipation model the problem with recurrent neural networks (Gao et al., 2017; Farha et al., 2018; Miech et al., 2019).

Original: Some prior works also leverage the future frames for learning the representations (Vondrick et al., 2016; Fernando & Herath, 2021).
Revised: Some prior works also leverage the future frames for learning the representations (Vondrick et al., 2016; Fernando & Herath, 2021).

Original: A self-regulated learning framework for action anticipation in the egocentric video is presented in (Qi et al., 2021).
Revised: A self-regulated learning framework for action anticipation in the egocentric video is presented in (Qi et al., 2021).

Original: RU-LSTM (Furnari & Farinella, 2019) deploys two LSTMs and behaves like an encoder-decoder, where the ﬁrst progressively summarizes the observed together with the second that unrolls over future predictions without observing.
Revised: RU-LSTM (Furnari & Farinella, 2019) deploys two LSTMs and behaves like an encoder-decoder, where the ﬁrst progressively summarizes the observed together with the second that unrolls over future predictions without observing.

Original: (Osman et al., 2021;
Revised: (Osman et al., 2021;

Original: Tai et al., 2021) both integrate unrolling mechanism as RU-LSTM does, but replace the rolling part in RU-LSTM with SlowFast (Feichtenhofer et al., 2019) and higher-order recurrent networks, respectively.
Revised: Tai et al., 2021) both integrate unrolling mechanism as RU-LSTM does, but replace the rolling part in RU-LSTM with SlowFast (Feichtenhofer et al., 2019) and higher-order recurrent networks, respectively.

Original: (Sener et al., 2020) aggregates the predictions by pooling over different granularity of temporal segments.
Revised: (Sener et al., 2020) aggregates the predictions by pooling over different granularity of temporal segments.

Original: (Girdhar & Grauman, 2021) combine causal self-attention with the look ahead prediction on successive frames.
Revised: (Girdhar & Grauman, 2021) combine causal self-attention with the look ahead prediction on successive frames.

Original: 2. M ESSAGE P ASSING N EURAL N ETWORK
Revised: 2. M ESSAGE P ASSING N EURAL N ETWORK

Original: The concept of message passing is ﬁrst published in (Gilmer et al., 2017), where it was originally designed for molecular property prediction.
Revised: The concept of message passing is ﬁrst published in (Gilmer et al., 2017), where it was originally designed for molecular property prediction.

Original: It assumes an undirected graph structure with dataindependent, equal edge contributions.
Revised: It assumes an undirected graph structure with dataindependent, equal edge contributions.

Original: To address this limitation, (Ma et al., 2020) deploys two encoders separately for vertex and edge estimation and aggregates them by a self-attention readout.
Revised: To address this limitation, (Ma et al., 2020) deploys two encoders separately for vertex and edge estimation and aggregates them by a self-attention readout.

Original: Some prior works use attention or dedicated network design to learn the directed edge representations to improve the model capability (Jørgensen et al., 2018;
Revised: Some prior works use attention or dedicated network design to learn the directed edge representations to improve the model capability (Jørgensen et al., 2018;

Original: Withnall et al., 2020; Gong & Cheng, 2019).
Revised: Withnall et al., 2020; Gong & Cheng, 2019).

Original: Recently, (Arnab et al., 2021b) express the non-local attention (Wang et al., 2018) and GAT (Veliˇckovi´c et al., 2017) as message passing functions and apply them to video understanding task.
Revised: Recently, (Arnab et al., 2021b) express the non-local attention (Wang et al., 2018) and GAT (Veliˇckovi´c et al., 2017) as message passing functions and apply them to video understanding task.

Original: Differently, we view the message passing framework as the generalized recurrent models and specialize it for edge representations learning in this work.
Revised: Differently, we view the message passing framework as the generalized recurrent models and specialize it for edge representations learning in this work.

Original: 2. S ELF -A TTENTION
Revised: 2. S ELF -A TTENTION

Original: (Vaswani et al., 2017)
Revised: (Vaswani et al., 2017)

Original: ﬁrst propose a recurrence free sequence learning architecture by stacking several self-attention layers, which can achieve remarkable performance in the NLP domain.
Revised: ﬁrst propose a recurrence free sequence learning architecture by stacking several self-attention layers, which can achieve remarkable performance in the NLP domain.

Original: (Dehghani et al., 2018) demonstrates that the self-attention can be treated as the recurrent unit which unfolds to input sequences to processes with shared weights.
Revised: (Dehghani et al., 2018) demonstrates that the self-attention can be treated as the recurrent unit which unfolds to input sequences to processes with shared weights.

Original: On the other hand, (Dosovitskiy et al., 2020) propose Vision Transformer (ViT), an architecture with only self-attention for image classiﬁcation.
Revised: On the other hand, (Dosovitskiy et al., 2020) propose Vision Transformer (ViT), an architecture with only self-attention for image classiﬁcation.

Original: (Zhou et al., 2021) improves ViT by re-attending the multi-heads information in the post-softmax step to enable the deeper conﬁguration.
Revised: (Zhou et al., 2021) improves ViT by re-attending the multi-heads information in the post-softmax step to enable the deeper conﬁguration.

Original: Some recent studies explore ViT based models on video action recognition (Arnab et al., 2021a; Bertasius et al., 2021), and also video anticipation (Girdhar & Grauman, 2021).
Revised: Some recent studies explore ViT based models on video action recognition (Arnab et al., 2021a; Bertasius et al., 2021), and also video anticipation (Girdhar & Grauman, 2021).

Original: Unlike these prior works, our proposed model processes the video in a ﬂexible graph representation and ﬁts into the message passing framework.
Revised: Unlike these prior works, our proposed model processes the video in a ﬂexible graph representation and ﬁts into the message passing framework.

Original: It is lightweight and only contains few self-attention layers which sequentially process each timestep.
Revised: It is lightweight and only contains few self-attention layers which sequentially process each timestep.

Original:  M ETHODOLOGY 3. B ACKGROUNDS 3.1. M ESSAGE P ASSING
Revised:  M ETHODOLOGY 3. B ACKGROUNDS 3.1. M ESSAGE P ASSING

Original: Given an undirected graph G , the Message Passing algorithm involves two-phase forwarding processes, message passing phase with message function M and update function U , and readout phase with readout function R .
Revised: Given an undirected graph G , the Message Passing algorithm involves two-phase forwarding processes, message passing phase with message function M and update function U , and readout phase with readout function R .

Original: The message passing phase can run arbitrary T step with the following deﬁnition:
Revised: The message passing phase can run arbitrary T step with the following deﬁnition:

Original: [Equation]
Revised: [Equation]

Original: where v is the vertex in G , and N ( v ) deﬁnes the neighbors of v .
Revised: where v is the vertex in G , and N ( v ) deﬁnes the neighbors of v .

Original: a vw is the connection strength bonding between vertex v and w .
Revised: a vw is the connection strength bonding between vertex v and w .

Original: The readout phase then extract the features of the whole graph, abstracted from the message passing phase, at time T ,
Revised: The readout phase then extract the features of the whole graph, abstracted from the message passing phase, at time T ,

Original: [Equation]
Revised: [Equation]

Original: Our proposed method inherits these three core functions (e.g., message, update, and readout).
Revised: Our proposed method inherits these three core functions (e.g., message, update, and readout).

Original: However, the most notable difference is that we treat the whole anticipation predicting as a chain of messages conveyed in space-time, with the constant strengthening of the signal by the RGB inputs in each time-step.
Revised: However, the most notable difference is that we treat the whole anticipation predicting as a chain of messages conveyed in space-time, with the constant strengthening of the signal by the RGB inputs in each time-step.

Original: The readout function is called when the prediction is required at any time t .
Revised: The readout function is called when the prediction is required at any time t .

Original: 3.1. S ELF -A TTENTION
Revised: 3.1. S ELF -A TTENTION

Original: Self-Attention (SA) forms the Q, K, V tokens (for query, key, and value) from the same source input x 1 .
Revised: Self-Attention (SA) forms the Q, K, V tokens (for query, key, and value) from the same source input x 1 .

Original: The output of the attention is the linear combination of the value, based on the attention weight computed by the scaled dot-product between the query and key followed by a softmax.
Revised: The output of the attention is the linear combination of the value, based on the attention weight computed by the scaled dot-product between the query and key followed by a softmax.

Original: [Equation]
Revised: [Equation]

Original: where scaled factor D default to the input feature dimension, W iQ , W iK , W iV are the trainable embeddings.
Revised: where scaled factor D default to the input feature dimension, W iQ , W iK , W iV are the trainable embeddings.

Original: We use superscript i to note the embeddings are associated to each self-attention layer.
Revised: We use superscript i to note the embeddings are associated to each self-attention layer.

Original: Multi-Head Self Attention (MHSA) performs n -way self-attention in parallel, where n is the total number of heads.
Revised: Multi-Head Self Attention (MHSA) performs n -way self-attention in parallel, where n is the total number of heads.

Original: An additional aggregation function, with parameters W agg , is adopted to fuse the information computed from each head.
Revised: An additional aggregation function, with parameters W agg , is adopted to fuse the information computed from each head.

Original: [Equation]
Revised: [Equation]

Original: where H i =
Revised: where H i =

Original: SA i ( x ) where the [ ., . ] presents the concatenation.
Revised: SA i ( x ) where the [ ., . ] presents the concatenation.

Original: Following the transformer-style architecture, a Feed-Forward Network (FFN) performs after each attention layer to project the attention output and bring the non-linearity.
Revised: Following the transformer-style architecture, a Feed-Forward Network (FFN) performs after each attention layer to project the attention output and bring the non-linearity.

Original: The FFN computes:
Revised: The FFN computes:

Original: [Equation]
Revised: [Equation]

Original:  where σ can be any arbitrary nonlinear function.
Revised: ) where σ can be any arbitrary nonlinear function.

Original: Our proposed model utilizes multi-head self-attention and deeply integrates into the message passing design.
Revised: Our proposed model utilizes multi-head self-attention and deeply integrates into the message passing design.

Original: We use multi-head self-attention as the information routing between vertices of a graph.
Revised: We use multi-head self-attention as the information routing between vertices of a graph.

Original: Note in this case, the resulting graph is bi-directed, because the dot-product between Q and K is not symmetric.
Revised: Note in this case, the resulting graph is bi-directed, because the dot-product between Q and K is not symmetric.

Original: 3. U NIFIED R ECURRENT M ODELING
Revised: 3. U NIFIED R ECURRENT M ODELING

Original: Self-Attention Block : We implement the self-attention building block in prenorm style (Xiong et al., 2020), as implemented in many transformer-style architecture, where the normalization layer is placed before MHSA and FFN and not counts in shortcuts.
Revised: Self-Attention Block : We implement the self-attention building block in prenorm style (Xiong et al., 2020), as implemented in many transformer-style architecture, where the normalization layer is placed before MHSA and FFN and not counts in shortcuts.

Original: Figure 2 left shows the self-attention block
Revised: Figure 2 left shows the self-attention block

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We can optionally expose the edge information, in matrix A , into SABlock ( . ; A ) .
Revised: We can optionally expose the edge information, in matrix A , into SABlock ( . ; A ) .

Original: In this case, the extension deﬁnes MHSA ( x ; A ) where the matrix A is fused into the step after the softmax of scaled dot-product computation.
Revised: In this case, the extension deﬁnes MHSA ( x ; A ) where the matrix A is fused into the step after the softmax of scaled dot-product computation.

Original: We rewrite the equation 5 and equation 6 to
Revised: We rewrite the equation 5 and equation 6 to

Original: [Equation]
Revised: [Equation]

Original:  where H i
Revised: (11) where H i

Original: [Equation]
Revised: [Equation]

Original: Note A is unique and shared in multi-heads self-attention.
Revised: Note A is unique and shared in multi-heads self-attention.

Original: Although the probability comes with postsoftmax becomes twice larger, such difference may not cause an issue because trainable weights,
Revised: Although the probability comes with postsoftmax becomes twice larger, such difference may not cause an issue because trainable weights,

Original: W agg , W V , are able to absorb the impact.
Revised: W agg , W V , are able to absorb the impact.

Original: Recurrent Cell : Figure 2 shows the overall architecture.
Revised: Recurrent Cell : Figure 2 shows the overall architecture.

Original: Given the frame features x t at time t , with shape ( H , W , C ), which is transformed by a backbone model. We deﬁne the vertices e tv by taking a nonlinear transformation of x t and ﬂatten into 1D, resulting in the shape ( HW , C ).
Revised: Given the frame features x t at time t , with shape ( H , W , C ), which is transformed by a backbone model, we deﬁne the vertices e tv by taking a nonlinear transformation of x t and ﬂatten into 1D, resulting in the shape ( HW , C ).

Original: The position encoding W pe is added on top of e tv .
Revised: The position encoding W pe is added on top of e tv .

Original: We then leverage SABlock ( . ; A ) to message function M with optionally matrix A , and SABlock ( . ) to update function U for computing the hidden states h 
Revised: We then leverage SABlock ( . ; A ) to message function M with optionally matrix A , and SABlock ( . ) to update function U for computing the hidden states h tv ,

Original: [Equation]
Revised: [Equation]

Original: where f se is a Squeeze-and-Excitation (SE) module (Hu et al., 2018) performs on vertices dimension, and f h is a linear transformer to reduce the feature size from concatenated 2 C into C .
Revised: where f se is a Squeeze-and-Excitation (SE) module (Hu et al., 2018) performs on vertices dimension, and f h is a linear transformer to reduce the feature size from concatenated 2 C into C .

Original: Note we bound the hidden states by the tanh to ensure the value stability in temporal propagation.
Revised: Note we bound the hidden states by the tanh to ensure the value stability in temporal propagation.

Original: For the readout function, we leverage SABlock ( . ) to abstract the information from the hidden state representation h tv ,
Revised: For the readout function, we leverage SABlock ( . ) to abstract the information from the hidden state representation h tv ,

Original: [Equation]
Revised: [Equation]

Original: where f scale is a linear function to scale the values of h tv .
Revised: where f scale is a fully-connected transformation to scale the values of h tv .

Original: We assume every vertex in the graph G is accessible by any other vertex to maintain the maximal possibility without prior.
Revised: We assume every vertex in the graph G is accessible by any other vertex to maintain the maximal possibility without prior.

Original: In this case where N ( v ) ≡ v in equation 1.
Revised: In this case, N ( v ) ≡ v in equation 1.

Original: The edges, presents by adjacency matrix A = { a vw ; ∀ w ∈ N ( v ) , v ∈ G } , can be optionally provided, explicitly , else implicitly derived in self-attention.
Revised: The edges, presents by adjacency matrix A = { a vw ; ∀ w ∈ N ( v ) , v ∈ G } , can be optionally provided, explicitly , else implicitly derived in self-attention.

Original: Implicit Edge Estimation : The scaled dot-product original in the self-attention operator can be treated as implicit estimation of edges, which computes the pairwise similarity by correlation measurement of vertices on-the-ﬂy with the parametric trainable embeddings of inputs.
Revised: Implicit Edge Estimation : The scaled dot-product original in the self-attention operator can be treated as implicit estimation of edges, which computes the pairwise similarity by correlation measurement of vertices on-the-ﬂy with the parametric trainable embeddings of inputs.

Original: Explicit Edge Estimation : On the other hand, edge information can also be provided by feeding a adjacency matrix A during the attention computation as discussed in equation 11 and equation 12.
Revised: Explicit Edge Estimation : On the other hand, edge information can also be provided by feeding a adjacency matrix A during the attention computation as discussed in equation 11 and equation 12.

Original: All edge learning strategies introduced  followingare all belong to explicit edge estimation category.
Revised: All edge learning strategies introduced in the following all belong to explicit edge estimation category.

Original: 3. E DGE L EARNING
Revised: 3. E DGE L EARNING

Original: We propose three different learning strategies to construct the edge information (see Figure 3).
Revised: We propose three different learning strategies to construct the edge information (see Figure 3).

Original: Edge attention decouples the attention operator in message function into vertex and edge estimation separately.
Revised: Edge attention decouples the attention operator in message function into vertex and edge estimation separately.

Original: Class token projection performs the outer-product of a trainable vector with supervision signal from class labels, and template bank obtains the edge matrix by the linear combination of trainable templates, based on a selecting module conditions on inputs.
Revised: Class token projection performs the outer-product of a trainable vector with supervision signal from class labels, and template bank obtains the edge matrix by the linear combination of trainable templates, based on a selecting module conditions on inputs.

Original: 3.3. E DGE ATTENTION
Revised: 3.3. E DGE ATTENTION

Original: We can model the adjacency matrix by deploying a dedicated SABlock for edge matrix.
Revised: We can model the adjacency matrix by deploying a dedicated SABlock for edge matrix.

Original: In this case, we put two separate self-attention layers in parallel, one for vertex estimation (with subscript v ) and another one for edge estimation (with subscript e ).
Revised: In this case, we put two separate self-attention layers in parallel, one for vertex estimation (with subscript v ) and another one for edge estimation (with subscript e ).

Original: Applying to equation 15, we have
Revised: Applying to equation 15, we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) where the (cid:98) A t is the estimated edge information used in the message function.
Revised: where the (cid:98) A t is the estimated edge information used in the message function.

Original: 3.3. C LASS TOKEN PROJECTION
Revised: 3.3. C LASS TOKEN PROJECTION

Original: With class token accompanied with supervision signals received, we can span an edge matrix by applying outer-product of class token.
Revised: With class token accompanied with supervision signals received, we can span an edge matrix by applying outer-product of class token.

Original: Consider a class token, x cls , with dimension ( 1 , C ), the projection can be done by a linear transformation, f proj , from C dimension to N .
Revised: Consider a class token, x cls , with dimension ( 1 , C ), the projection can be done by a linear transformation, f proj , from C dimension to N .

Original: The outer-product of the token then spans it into a matrix with shape ( N , N ), which serves as the edge matrix.
Revised: The outer-product of the token then spans it into a matrix with shape ( N , N ), which serves as the edge matrix.

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where the ⊗ is the outer-product operator.
Revised: (21) where the ⊗ is the outer-product operator.

Original: 3.3. T EMPLATE BANK
Revised: 3.3. T EMPLATE BANK

Original: We deﬁne a template bank B with bank size S indicates the number of templates contained.
Revised: We deﬁne a template bank B with bank size S indicates the number of templates contained.

Original: Each template in the bank is the map with the shape ( N , N ) where N is the number of vertices.
Revised: Each template in the bank is the map with the shape ( N , N ) where N is the number of vertices.

Original: During the forward process, the edge information is formed by weighted sum of S templates by a selecting module, f select ( . ) , a Multi-Layer Perception (MLP) followed by a sigmoid to produce the selecting ratio conditioned on inputs and apply to the bank:
Revised: During the forward process, the edge information is formed by weighted sum of S templates by a selecting module, f select ( . ) , a Multi-Layer Perception (MLP) followed by a sigmoid to produce the selecting ratio conditioned on inputs and apply to the bank:

Original: [Equation]
Revised: [Equation]

Original: where e tv is the average of e tv over the vertex dimension.
Revised: where e tv is the average of e tv over the vertex dimension.

Original: The edge estimation (cid:98)
Revised: The edge estimation (cid:98)

Original: A t , in the combination of the template bank B , is then fed into the message function the equation 15.
Revised: A t , in the combination of the template bank B , is then fed into the message function the equation 15.

Original:  E XPERIMENTS 4. I MPLEMENTATION D ETAILS
Revised:  E XPERIMENTS 4. I MPLEMENTATION D ETAILS

Original: We based the pretrained BN-Inception model of (Furnari & Farinella, 2019) as backbone.
Revised: We based the pretrained BN-Inception model of (Furnari & Farinella, 2019) as backbone and kept it frozen during training.

Original: All input frames are resized to 256x454 and fed through a proposed model followed by an unrolling classiﬁer forming the verb, noun, and action prediction.
Revised: All input frames are resized to 256x454 and fed through a proposed model followed by an unrolling classiﬁer forming the verb, noun, and action prediction.

Original: The unrolling classiﬁer is a widely adopted design in action anticipation (Furnari & Farinella, 2019; Osman et al., 2021;
Revised: The unrolling classiﬁer is a widely adopted design in action anticipation (Furnari & Farinella, 2019; Osman et al., 2021;

Original: Tai et al., 2021), which unfolds the ﬁnal output over the unobserved interval with a LSTM, till the moment where the action is expected to start, and then computes the predictions.
Revised: Tai et al., 2021), which unfolds the ﬁnal output over the unobserved interval with a LSTM, till the moment where the action is expected to start, and then computes the predictions.

Original: The outputs are supervised by comparing predictions with ground-truth labels with cross-entropy in the last 8 anticipate interval for each individual verb, noun, and action.
Revised: The outputs are supervised by comparing predictions with ground-truth labels with cross-entropy in the last 8 anticipate interval for each individual verb, noun, and action.

Original: Note in the class token projection scheme, the class token for action is used, otherwise mean-pool over the output from readout function is performed.
Revised: Note in the class token projection scheme, the class token for action is used, otherwise mean-pool over the output from readout function is performed.

Original: For training details, RandAugment (Cubuk et al., 2020) augmentation scheme is applied in all experiments.
Revised: For training details, RandAugment (Cubuk et al., 2020) augmentation scheme is applied in all experiments.

Original: AdaBelief (Zhuang et al., 2020) in combination with look-ahead optimizer (Zhang et al., 2019) is adopted.
Revised: AdaBelief (Zhuang et al., 2020) in combination with look-ahead optimizer (Zhang et al., 2019) is adopted.

Original: Weight decay is set to 0.001.
Revised: Weight decay is set to 0.001.

Original: Learning rate is initially set to 1e-4 and cosine annealed to 1e-7 on the last 25% of epochs.
Revised: Learning rate is initially set to 1e-4 and cosine annealed to 1e-7 on the last 25% of epochs.

Original: We use 4 × NVIDIA V100 32GB GPUs for training, with batch size set to 32 and run in total 50 epochs.
Revised: We use 4 × NVIDIA V100 32GB GPUs for training, with batch size set to 32 and run in total 50 epochs.

Original: [Table]
Revised: [Table]

Original: 4. EPIC-K ITCHEN D ATASET
Revised: 4. EPIC-K ITCHEN D ATASET

Original: EPIC-Kitchen 55 (EK55) (Damen et al., 2018) is a large scale egocentric video dataset, captured by 32 subjects in 32 kitchens.
Revised: EPIC-Kitchen 55 (EK55) (Damen et al., 2018) is a large scale egocentric video dataset, captured by 32 subjects in 32 kitchens.

Original: The data split scheme is inherited from (Furnari & Farinella, 2019) for action anticipation tasks, resulting in 23492 action segments for training and 4979 for validation.
Revised: The data split scheme is inherited from (Furnari & Farinella, 2019) for action anticipation tasks, resulting in 23492 action segments for training and 4979 for validation.

Original: All unique verb-noun pairs, of 125 verbs and 352 nouns.
Revised: All unique verb-noun pairs, of 125 verbs and 352 nouns.

Original: deﬁne 2513 action categories.
Revised: deﬁne 2513 action categories.

Original: The top-1 and top-5 accuracy at 1 s are the major evaluation metrics used on EK55.
Revised: The top-1 and top-5 accuracy at 1 s are the major evaluation metrics used on EK55.

Original: We also include the speciﬁc performance metrics at 1 s , with additional mean top-5 recall reported.
Revised: We also include the speciﬁc performance metrics at 1 s , with additional mean top-5 recall reported.

Original: We sample 14 frames from each clip with a ﬁxed stride of α s = 0 .
Revised: We sample 14 frames from each clip with a ﬁxed stride of α s = 0 .

Original: 25 (4 fps), resulting in 3.5 s context in each sample.
Revised: 25 (4 fps), resulting in 3.5 s context in each sample.

Original: We use τ a to indicate anticipation time.
Revised: We use τ a to indicate anticipation time.

Original: 4. E XPERIMENTAL R ESULTS
Revised: 4. E XPERIMENTAL R ESULTS

Original: Baselines .
Revised: Baselines .

Original: We consider the baselines methods, all in RGB inputs, including Deep Multimodal Regressor (DMR), (Vondrick et al., 2016), TSN-based models MCE (Furnari et al., 2018) and ATSN (Damen et al., 2018), deep network trained with top-k classiﬁer (SVM-Top3/5) (Berrada et al., 2018), Verb-Noun Marginal Cross Entropy (VNMCE) (Furnari et al., 2018), and several LSTM variants, Encoder-Decoder LSTM (ED) (Gao et al., 2017), Feedback Network LSTM (FN) (Geest & Tuytelaars, 2018), LSTM with Ranking Loss (RL) (Ma et al., 2016), and LSTM with Exponential Anticipation Loss (EL)
Revised: We consider the baselines methods, all in RGB inputs, including Deep Multimodal Regressor (DMR), (Vondrick et al., 2016), TSN-based models MCE (Furnari et al., 2018) and ATSN (Damen et al., 2018), deep network trained with top-k classiﬁer (SVM-Top3/5) (Berrada et al., 2018), Verb-Noun Marginal Cross Entropy (VNMCE) (Furnari et al., 2018), and several LSTM variants, Encoder-Decoder LSTM (ED) (Gao et al., 2017), Feedback Network LSTM (FN) (Geest & Tuytelaars, 2018), LSTM with Ranking Loss (RL) (Ma et al., 2016), and LSTM with Exponential Anticipation Loss (EL)

Original: (Jain et al., 2016).
Revised: (Jain et al., 2016).

Original: We also compare the state-of-the-art RollingUnrolling LSTM (RU)
Revised: We also compare the state-of-the-art RollingUnrolling LSTM (RU)

Original: (Furnari & Farinella, 2020), ImagineRNN by predicting future feature (Wu et al., 2020), Higher-Order Recurrent Space-Time Transformer (HORST)
Revised: (Furnari & Farinella, 2020), ImagineRNN by predicting future feature (Wu et al., 2020), Higher-Order Recurrent Space-Time Transformer (HORST)

Original: (Tai et al., 2021), SelfRegulation Learning (SRL) (Qi et al., 2021), SlowFast with unrolling classiﬁer (SF-RU)
Revised: (Tai et al., 2021), SelfRegulation Learning (SRL) (Qi et al., 2021), SlowFast with unrolling classiﬁer (SF-RU)

Original: (Osman et al., 2021), and previous winners in EPIC-Kitchens anticipation challenge (in Table 3), ActionBanks in 2020 challenge (Sener et al., 2020) and Anticipative Video Transformer (AVT) in(Girdhar & Grauman, 2021).
Revised: (Osman et al., 2021), and previous winners in EPIC-Kitchens anticipation challenge (in Table 3), ActionBanks in 2020 challenge (Sener et al., 2020) and Anticipative Video Transformer (AVT) in(Girdhar & Grauman, 2021).

Original: Some methods cannot query the accuracy at any time (i.e., DMR,
Revised: Some methods cannot query the accuracy at any time (i.e., DMR,

Original: ATSN, MCE, VN-CE, SVM-Top3/5, VNMCE) and some are constraint by the speciﬁc sampling rates (i.e., SF-RU, AVT).
Revised: ATSN, MCE, VN-CE, SVM-Top3/5, VNMCE) and some are constraint by the speciﬁc sampling rates (i.e., SF-RU, AVT).

Original: The α s in SF-RU presents the data sampling rate used in the algorithm.
Revised: The α s in SF-RU presents the data sampling rate used in the algorithm.

Original: Results .
Revised: Results .

Original: From Table 1, we can observe that our proposed method with implicit edge estimation already performs strongly compared to previous methods on EK55.
Revised: From Table 1, we can observe that our proposed method with implicit edge estimation already performs strongly compared to previous methods on EK55.

Original: Using edge learning strategies discussed in section 3.3, we found template bank and class token projection methods with signiﬁcant improvements.
Revised: Using edge learning strategies discussed in section 3.3, we found template bank and class token projection methods with signiﬁcant improvements.

Original: Table 2 shows the speciﬁc performance at 1 s , which is the widely used evaluation criteria in the EPIC-Kitchen, we provide the top-5 accuracy, and also top-5 recall, for each individual verb, noun, and action.
Revised: Table 2 shows the speciﬁc performance at 1 s , which is the widely used evaluation criteria in the EPIC-Kitchen, we provide the top-5 accuracy, and also top-5 recall, for each individual verb, noun, and action.

Original: We can clearly see our proposed uniﬁed recurrent modeling outperforms previous methods, not only on the accuracy but also on the recall.
Revised: We can clearly see our proposed uniﬁed recurrent modeling outperforms previous methods, not only on the accuracy but also on the recall.

Original: Equipped with the explicit edge
Revised: Equipped with the explicit edge

Original: [Table] [Table]
Revised: [Table] [Table]

Original: estimation, template bank and class token projection even shows signiﬁcant improvements.
Revised: estimation, template bank and class token projection even shows signiﬁcant improvements.

Original: Tablereveals top-1 accuracy at 1 s and compared with additional candidates who won the EPIC-Kitchen ofﬁcial challenge in recent two years (ActionBanks and AVT).
Revised: Tablereveals top-1 accuracy at 1 s and compared with additional candidates who won the EPIC-Kitchen ofﬁcial challenge in recent two years (ActionBanks and AVT).

Original: Despite the AVT trained with advanced backbone deployed, and with different sampling rate
Revised: Despite the AVT trained with advanced backbone deployed, and with different sampling rate

Original: α s = 1 (1 fps) for 10 s for each sample
Revised: α s = 1 (1 fps) for 10 s for each sample

Original: , our proposed methods are with superior accuracy in both top-1 and top-5 accuracy over all the methods using the same BN-Inception backbone.
Revised: , our proposed methods are with superior accuracy in both top-1 and top-5 accuracy over all the methods using the same BN-Inception backbone.

Original: It is worth noting although the edge attention is not as strong as template bank and class token projection proposals in top-5 comparison, it gets the best amongst all top-1 accuracy.
Revised: It is worth noting although the edge attention is not as strong as template bank and class token projection proposals in top-5 comparison, it gets the best amongst all top-1 accuracy.

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: From the previous section, we found edge learning with template bank achieved overall best according to accuracy and recall among all experiments.
Revised: From the previous section, we found edge learning with template bank achieved overall best according to accuracy and recall among all experiments.

Original: In this section we compare and discuss the two major hyper-parameters set in this model, the bank size and the feature dimension.
Revised: In this section we compare and discuss the two major hyper-parameters set in this model, the bank size and the feature dimension.

Original: 4.4. B ANK S IZE
Revised: 4.4. B ANK S IZE

Original: Table 4 shows the experimental results on setting different bank sizes for the template bank edge learning.
Revised: Table 4 shows the experimental results on setting different bank sizes for the template bank edge learning.

Original: We manipulate the bank size ranges from 1 to 2048.
Revised: We manipulate the bank size ranges from 1 to 2048.

Original: In the case of bank size being equal to 1, a unique edge matrix is shared globally across all samples and each timestep.
Revised: In the case of bank size being equal to 1, a unique edge matrix is shared globally across all samples and each timestep.

Original: It can be explained as placing a strong regularization to the intermediate representations before feeding into the message function, forcing the vertex constrained to the speciﬁc representation that is able to co-work with the
Revised: It can be explained as placing a strong regularization to the intermediate representations before feeding into the message function, forcing the vertex constrained to the speciﬁc representation that is able to co-work with the

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: global edge matrix.
Revised: global edge matrix.

Original: When the bank size is larger than 1, the constraint becomes weaker, bringing additional ﬂexibility to the representations.
Revised: When the bank size is larger than 1, the constraint becomes weaker, bringing additional ﬂexibility to the representations.

Original: We can see when the bank size is greater than 64, top1/5 accuracy and top-5 recall are showing better results than the global template used.
Revised: We can see when the bank size is greater than 64, top1/5 accuracy and top-5 recall are showing better results than the global template used.

Original: The peak performance is at bank size 512, and showing no improvement with more templates.
Revised: The peak performance is at bank size 512, and showing no improvement with more templates.

Original: 4.4. F EATURE D IMENSIONS
Revised: 4.4. F EATURE D IMENSIONS

Original: Table 5 shows the performance difference while varying the width of the proposed model.
Revised: Table 5 shows the performance difference while varying the width of the proposed model.

Original: By default, the feature dimension is set to 1024, which is aligned with the output dimension of BNInception backbone.
Revised: By default, the feature dimension is set to 1024, which is aligned with the output dimension of BNInception backbone.

Original: We adjust the feature dimension from 0 .
Revised: We adjust the feature dimension from 0 .

Original: 25 x to 4 x. This adjustment also affects the feature dimension used in multi-head self-attentions.
Revised: 25 x to 4 x. This adjustment also affects the feature dimension used in multi-head self-attentions.

Original: From the table results, we can observe the best conﬁguration falls in the 1024 feature dimension, which is equivalent to setting each attention head to 128 feature dimensions.
Revised: From the table results, we can observe the best conﬁguration falls in the 1024 feature dimension, which is equivalent to setting each attention head to 128 feature dimensions.

Original: This ablation suggests setting the appropriate feature dimension is needed for a speciﬁc task.
Revised: This ablation suggests setting the appropriate feature dimension is needed for a speciﬁc task.

Original: 4. P ARAMETERS AND C OMPUTATION E FFICIENCY
Revised: 4. P ARAMETERS AND C OMPUTATION E FFICIENCY

Original: Our model contains only three multi-head self-attention layers with linear transformations for dimension reduction or selection.
Revised: Our model contains only three multi-head self-attention layers with linear transformations for dimension reduction or selection.

Original: Compared to other transformer-based models, which often stack multiple self-attention layers, our modeling is lightweight and efﬁcient.
Revised: Compared to other transformer-based models, which often stack multiple self-attention layers, our modeling is lightweight and efﬁcient.

Original: The total parameters in our model, excluding the backbone, classiﬁer, and normalization layers, are only about 21 C 2 (21M when C =1024).
Revised: The total parameters in our model, excluding the backbone, classiﬁer, and normalization layers, are only about 21 C 2 (21M when C =1024).

Original: The template bank brings about additional 6.5M parameters, including a selector with parameters 14 C ( S + C ) , and templates that occupy SN 2 .
Revised: The template bank brings about additional 6.5M parameters, including a selector with parameters 14 C ( S + C ) , and templates that occupy SN 2 .

Original: In the EPIC-Kitchen dataset with ( 256 , 454 ) input dimension based on BN-Inception backbone, N is 112 .
Revised: In the EPIC-Kitchen dataset with ( 256 , 454 ) input dimension based on BN-Inception backbone, N is 112 .

Original: The total computation is about 40 GFLOPs per timestep.
Revised: The total computation is about 40 GFLOPs per timestep.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we present a uniﬁed recurrent modeling which only uses self-attention as building block to represent features through graph, and which leverages edge learning to estimate vertex semantics.
Revised: In this work we present a uniﬁed recurrent modeling which only uses self-attention as building block to represent features through graph, and which leverages edge learning to estimate vertex semantics.

Original: On the large-scale egocentric EPIC-Kitchen dataset for the video action anticipation, we surpass the current state-of-the-art in action anticipation.
Revised: On the large-scale egocentric EPIC-Kitchen dataset for the video action anticipation, we surpass the current state-of-the-art in action anticipation.

Original: Proposed model is simple, lightweight, and thus embodies the ﬂexibility for further extensions.
Revised: Proposed model is simple, lightweight, and thus embodies the ﬂexibility for further extensions.