Original: Bringing Image Structure to Video via Frame-Clip Consistency of Object Tokens
Revised: Bringing Image Structure to Video via Frame-Clip Consistency of Object Tokens

Original: Abstract
Revised: Abstract

Original: Recent action recognition models have achieved impressive results by integrating objects, their locations and interactions.
Revised: Recent action recognition models have achieved impressive results by integrating objects, their locations and interactions.

Original: However, obtaining dense structured annotations for each frame is tedious and time-consuming, making these methods expensive to train and less scalable.
Revised: However, obtaining dense structured annotations for each frame is tedious and time-consuming, making these methods expensive to train and less scalable.

Original: At the same time, if a small set of annotated images is available, either within or outside the domain of interest, how could we leverage these for a video downstream task?
Revised: At the same time, if a small set of annotated images is available, either within or outside the domain of interest, how could we leverage these for a video downstream task?

Original: We propose a learning framework
Revised: We propose a learning framework

Original: StructureViT (SViT for short), which demonstrates how utilizing the structure of a small number of images only available during training can improve a video model.
Revised: StructureViT (SViT for short), which demonstrates how utilizing the structure of a small number of images only available during training can improve a video model.

Original: SViT relies on two key insights.
Revised: SViT relies on two key insights.

Original: First, as both images and videos contain structured information, we enrich a transformer model with a set of object tokens that can be used across images and videos.
Revised: First, as both images and videos contain structured information, we enrich a transformer model with a set of object tokens that can be used across images and videos.

Original: Second, the scene representations of individual frames in video should “align” with those of still images.
Revised: Second, the scene representations of individual frames in video should “align” with those of still images.

Original: This is achieved via a Frame-Clip Consistency loss, which ensures the ﬂow of structured information between images and videos.
Revised: This is achieved via a Frame-Clip Consistency loss, which ensures the ﬂow of structured information between images and videos.

Original: We explore a particular instantiation of scene structure, namely a Hand-Object Graph , consisting of hands and objects with their locations as nodes, and physical relations of contact/no-contact as edges.
Revised: We explore a particular instantiation of scene structure, namely a Hand-Object Graph , consisting of hands and objects with their locations as nodes, and physical relations of contact/no-contact as edges.

Original: SViT shows strong performance improvements on multiple video understanding tasks and datasets.
Revised: SViT shows strong performance improvements on multiple video understanding tasks and datasets, including the ﬁrst place in the Ego4D CVPR’22 Point of No Return Temporal

Revised: Localization Challenge.

Original: For code and pretrained models, visit the project page at https://eladb3.github.io/SViT/.
Revised: For code and pretrained models, visit the project page at https://eladb3.github.io/SViT/

Original:  Introduction
Revised:  Introduction

Original: Semantic understanding of videos is a key challenge for machine vision and artiﬁcial intelligence.
Revised: Semantic understanding of videos is a key challenge for machine vision and artiﬁcial intelligence.

Original: It is intuitive that video models should beneﬁt from incorporating scene structure, e.g., the objects that appear in a video, their attributes, and the way they interact.
Revised: It is intuitive that video models should beneﬁt from incorporating scene structure, e.g., the objects that appear in a video, their attributes, and the way they interact.

Original: Indeed, several works [5, 25, 26, 34, 35, 42, 81, 83, 89] have demonstrated that incorporating structured representations into models improves both performance and sample efﬁciency.
Revised: Indeed, several works [5, 25, 26, 34, 35, 42, 81, 83, 89] have demonstrated that incorporating structured representations into models improves both performance and sample efﬁciency.

Original: In particular, [34, 81, 89] showed how to incorporate structured representations by utilizing static image object annotations in videos.
Revised: In particular, [34, 81, 89] showed how to incorporate structured representations by utilizing static image object annotations in videos.

Original: Recently, vision transformers (ViT) [20] have emerged as the leading model for many vision applications [4, 21, 12].
Revised: Recently, vision transformers (ViT) [20] have emerged as the leading model for many vision applications [4, 21, 12].

Original: This begs the question, how can structured representations be leveraged for video tasks in a video transformer?
Revised: This begs the question, how can structured representations be leveraged for video tasks in a video transformer?

Original: Past works [4, 21] have proposed video transformer models but these approaches were not focused on learning structured representations.
Revised: Past works [4, 21] have proposed video transformer models but these approaches were not focused on learning structured representations.

Original: A recent line of works [34, 81, 89] utilized objects for structured approaches. [83] leveraged the decoder with CNNs region crop features
Revised: A recent line of works [34, 81, 89] 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  from SlowFast [22], [34] proposed a new block that incorporates object representations, and [89] proposed a structured model for action detection.
Revised: utilized objects for structured approaches. [83] leveraged the decoder with CNNs region crop features from SlowFast [22], [34] proposed a new block that incorporates object representations, and [89] proposed a structured model for action detection.

Original: However, these approaches are limited because they use external pretrained detectors for the frames, which require additional processing time (also during inference).
Revised: However, these approaches are limited because they use external pretrained detectors for the frames, which require additional processing time (also during inference).

Original: Moreover, they ignore the potential beneﬁts of the connection between the detection and the other tasks, and it is unclear how to use them with more detailed structured annotations.
Revised: Moreover, they ignore the potential beneﬁts of the connection between the detection and the other tasks, and it is unclear how to use them with more detailed structured annotations.

Original: Our approach, described next, builds on these prior works and proposes an effective way to leverage scene structure in video transformers , using a relatively small static image supervision.
Revised: Our approach, described next, builds on these prior works and proposes an effective way to leverage scene structure in video transformers , using a relatively small static image supervision.

Original: Most structured transformer-based models for videos require structured annotations during training.
Revised: Most structured transformer-based models for videos require structured annotations during training.

Original: For example, frame-level object bounding boxes are considered available in [5, 7, 34, 35, 59, 81].
Revised: For example, frame-level object bounding boxes are considered available in [5, 7, 34, 35, 59, 81].

Original: When it comes to structured annotations of static images, there are numerous datasets with annotations such as boxes, visual relationships, attributes, and segmentations [10, 49, 59].
Revised: When it comes to structured annotations of static images, there are numerous datasets with annotations such as boxes, visual relationships, attributes, and segmentations [10, 49, 59].

Original: Since we cannot always expect to have annotated images that perfectly align with our downstream video task, how can these resources be used to build better transformer-based video models?
Revised: Since we cannot always expect to have annotated images that perfectly align with our downstream video task, how can these resources be used to build better transformer-based video models?

Original: In this work, we propose such an approach that is speciﬁcally designed for transformer architectures and offers an effective mechanism for improving video transformers by leveraging images with structured annotations that are not necessarily associated with the video domain and can be within or outside the domain of interest.
Revised: In this work, we propose such an approach that is speciﬁcally designed for transformer architectures and offers an effective mechanism for improving video transformers by leveraging images with structured annotations that are not necessarily associated with the video domain and can be within or outside the domain of interest.

Original: A natural ﬁrst step towards image-video knowledge sharing is to use the same transformer model to process both: namely, to view images as “single frame videos”, and feed them to a video transformer model.
Revised: A natural ﬁrst step towards image-video knowledge sharing is to use the same transformer model to process both: namely, to view images as “single frame videos”, and feed them to a video transformer model.

Original: However, this still leaves two key questions: how to model structured information, and how to account for domain mismatch between images and videos.
Revised: However, this still leaves two key questions: how to model structured information, and how to account for domain mismatch between images and videos.

Original: Towards this end, we introduce two key concepts: (i) A set of transformer “object-tokens” – additional tokens initialized from learned embeddings (we refer to these embeddings as “object prompts”) that are meant to capture objectcentric information in both still image and video.
Revised: Towards this end, we introduce two key concepts: (i) A set of transformer “object-tokens” – additional tokens initialized from learned embeddings (we refer to these embeddings as “object prompts”) that are meant to capture objectcentric information in both still image and video.

Original: These tokens can be supervised based on the scene structure of images, and that information will also be propagated to videos.
Revised: These tokens can be supervised based on the scene structure of images, and that information will also be propagated to videos.

Original: To formalize the image structure, we proposed the HAnd-Object Graph (HAOG), a simple representation of the interactions between hands and objects in the scene.
Revised: To formalize the image structure, we proposed the HAnd-Object Graph (HAOG), a simple representation of the interactions between hands and objects in the scene.

Original: (ii)
Revised: (ii)

Original: A novel “Frame-Clip Consistency loss” that ensures consistency between the “object-tokens” when they are part of a video vs. a still image.
Revised: A novel “Frame-Clip Consistency loss” that ensures consistency between the “object-tokens” when they are part of a video vs. a still image.

Original: Moreover, as we show in our experiments, our approach does not require any alignment between object categories appearing in still images and the ones appearing in the videos of the target downstream task.
Revised: Moreover, as we show in our experiments, our approach does not require any alignment between object categories appearing in still images and the ones appearing in the videos of the target downstream task.

Original: We name our proposed approach StructureViT ( Bringing Scene Structure from Images to Vi deo via Frame-Clip
Revised: We name our proposed approach StructureViT ( Bringing Scene Structure from Images to Vi deo via Frame-Clip

Original: Consistency of Object T okens , or SViT for short).
Revised: Consistency of Object T okens , or SViT for short).

Original: See Figure 1 for an overview.
Revised: See Figure 1 for an overview.

Original: Prior work has explored several other video & image models.
Revised: Prior work has explored several other video & image models.

Original: The I3D model [13] was one of the ﬁrst successful attempts to leverage ImageNet architectures in video.
Revised: The I3D model [13] was one of the ﬁrst successful attempts to leverage ImageNet architectures in video.

Original: They proposed to simply convert image 2D classiﬁcation models into 3D ConvNets by inﬂating all the ﬁlters and pooling kernels.
Revised: They proposed to simply convert image 2D classiﬁcation models into 3D ConvNets by inﬂating all the ﬁlters and pooling kernels.

Original: Here, our focus is on learning the shared structured representations between the two domains in order to transfer this knowledge into videos.
Revised: Here, our focus is on learning the shared structured representations between the two domains in order to transfer this knowledge into videos.

Original: Several other methods have explored simultaneous image and video training in the context of multi-task [6] and multi-modal [27] learning.
Revised: Several other methods have explored simultaneous image and video training in the context of multi-task [6] and multi-modal [27] learning.

Original: However, these works did not utilize structural information, nor did they promote consistency between image and video predictions – which, as we show, leads to signiﬁcant gains in downstream video task performance.
Revised: However, these works did not utilize structural information, nor did they promote consistency between image and video predictions – which, as we show, leads to signiﬁcant gains in downstream video task performance.

Original: To summarize, our main contributions are as follows: (i) we propose a new method for exploiting structured information present in images in order to improve performance of video understanding tasks; (ii) we propose a novel concept of “object tokens” - a form of learned prompts to a video & image transformer, which during training are associated to the available structure representation of the still images (HAOG).
Revised: To summarize, our main contributions are as follows: (i) we propose a new method for exploiting structured information present in images in order to improve performance of video understanding tasks; (ii) we propose a novel concept of “object tokens” - a form of learned prompts to a video & image transformer, which during training are associated to the available structure representation of the still images (HAOG).

Original: (iii) we introduce a new Frame-Clip Consistency loss, which promotes consistency between image-level and video-level predictions made by a shared transformer backbone.
Revised: (iii) we introduce a new Frame-Clip Consistency loss, which promotes consistency between image-level and video-level predictions made by a shared transformer backbone.

Original: We show how this loss helps to drive performance improvements in downstream video tasks even in cases when image labels are unrelated to the video task; (iv) we demonstrate improved performance on several video understanding benchmarks, highlighting the effectiveness of the proposed approach.
Revised: We show how this loss helps to drive performance improvements in downstream video tasks even in cases when image labels are unrelated to the video task; (iv) we demonstrate improved performance on several video understanding benchmarks, highlighting the effectiveness of the proposed approach.

Original:  The SViT Approach
Revised:  The SViT Approach

Original: Our approach SViT learns structured representation that is exhibited in both static images and video.
Revised: Our approach SViT learns structured representation that is exhibited in both static images and video.

Original: We consider the setting where the main goal is to learn video understanding downstream tasks while leveraging structured image data.
Revised: We consider the setting where the main goal is to learn video understanding downstream tasks while leveraging structured image data.

Original: In this paper, we focus on the following video tasks: action recognition, spatio-temporal action detection and object state classiﬁcation & localization.
Revised: In this paper, we focus on the following video tasks: action recognition, spatio-temporal action detection and object state classiﬁcation & localization.

Original: In training, we assume that we have access to task-speciﬁc video labels and structured scene annotations for images, Hand-Object Graph s.
Revised: In training, we assume that we have access to task-speciﬁc video labels and structured scene annotations for images, Hand-Object Graph s.

Original: Based on the structured representations obtained by using the object tokens and regularized by the Frame-Clip Consistency loss , the inference is performed only for the downstream video task without explicitly using the structured image annotations.
Revised: Based on the structured representations obtained by using the object tokens and regularized by the Frame-Clip Consistency loss , the inference is performed only for the downstream video task without explicitly using the structured image annotations.

Original: Further, we begin by describing the Hand-Object Graph (HAOG) (Section 2.1), then introduce our
Revised: Further, we begin by describing the Hand-Object Graph (HAOG) (Section 2.1), then introduce our

Original: SViT model (Section 2.2), and the Frame-Clip Consistency loss function (Section 2.3).
Revised: SViT model (Section 2.2), and the Frame-Clip Consistency loss function (Section 2.3).

Original: Our method is schematically illustrated in Figure 2.
Revised: Our method is schematically illustrated in Figure 2.

Original: 2.1 The Hand-Object Graph
Revised: 2.1 The Hand-Object Graph

Original: As mentioned above, our motivation is to bring scene structure from still images into video.
Revised: As mentioned above, our motivation is to bring scene structure from still images into video.

Original: In order to achieve this, one component of this work is the construction of a semantic representation of the interactions between the hands and objects in the scene.
Revised: In order to achieve this, one component of this work is the construction of a semantic representation of the interactions between the hands and objects in the scene.

Original: Speciﬁcally, we propose to use a graph-structure we call Hand-Object Graph (HAOG), see Figure 1.
Revised: Speciﬁcally, we propose to use a graph-structure we call Hand-Object Graph (HAOG), see Figure 1.

Original: The nodes of the HAOG represent two hands and two objects with their locations, whereas the edges correspond to physical properties such as contact/no contact.
Revised: The nodes of the HAOG represent two hands and two objects with their locations, whereas the edges correspond to physical properties such as contact/no contact.

Original: Formally, an HAOG is a tuple ( O, E ) deﬁned as follows:
Revised: Formally, an HAOG is a tuple ( O, E ) deﬁned as follows:

Original: Nodes O : A set O of n objects.
Revised: Nodes O : A set O of n objects.

Original: Human-object interaction scenes, which we focus on in this work, usually consist of up to two hands and objects.
Revised: Human-object interaction scenes, which we focus on in this work, usually consist of up to two hands and objects.

Original: Thus our node categories are “hand” and “object”; they may also contain attributes (e.g., spatial coordinates).
Revised: Thus our node categories are “hand” and “object”; they may also contain attributes (e.g., spatial coordinates).

Original: We assume that each image comes with the bounding boxes describing the locations of the two hands, and the objects they interact with.
Revised: We assume that each image comes with the bounding boxes describing the locations of the two hands, and the objects they interact with.

Original: The bounding boxes for the left and right hand are denoted as b 1 , b 2 2 R 4 , respectively, while for the two corresponding objects interacting with the hands are denoted as b 3 , b 4 .
Revised: The bounding boxes for the left and right hand are denoted as b 1 , b 2 2 R 4 , respectively, while for the two corresponding objects interacting with the hands are denoted as b 3 , b 4 .

Original: We also assume access to four binary variables e 1 , . .
Revised: We also assume access to four binary variables e 1 , . .

Original: , e 4 2 { 0 , 1 } that indicate whether the corresponding hands or objects in fact exist in the image (e.g., if e 1 = 

Revised: , e 4 2 { 0 , 1 } that indicate whether the corresponding hands or objects in fact exist in the image (e.g., if e 1 =

Original: the left hand is not in the image, and b 1 should be ignored).
Revised: the left hand is not in the image, and b 1 should be ignored).

Original: Edges E : The edges are represented as labeled directed edges between hand and object nodes.
Revised: Edges E : The edges are represented as labeled directed edges between hand and object nodes.

Original: Edge vocabulary describes the physical properties of the scene.
Revised: Edge vocabulary describes the physical properties of the scene.

Original: In our current modeling, each edge is characterized by a physical property of “contact” or “no-contact”.
Revised: In our current modeling, each edge is characterized by a physical property of “contact” or “no-contact”.

Original: We denote two binary variables c 1 , c 2 2 { 0 , 1 } that specify if each of the hands is in direct contact with the corresponding object (the left hand b 1 can only interact with the object b 3 , similar for the right hand).
Revised: We denote two binary variables c 1 , c 2 2 { 0 , 1 } that specify if each of the hands is in direct contact with the corresponding object (the left hand b 1 can only interact with the object b 3 , similar for the right hand).

Original: In principle, we can also include properties such as distances between objects, 3D hand poses, etc., where available.
Revised: In principle, we can also include properties such as distances between objects, 3D hand poses, etc., where available.

Original: We note that images annotated consistent with our deﬁned HAOG format can be found in several datasets [31, 59, 72], and we leverage these in our experiments.
Revised: We note that images annotated consistent with our deﬁned HAOG format can be found in several datasets [31, 59, 72], and we leverage these in our experiments.

Original: 2.2 The SViT Transformer Model
Revised: 2.2 The SViT Transformer Model

Original: We next present the SViT model: a video transformer that employs object tokens as a means to capture structure in both images and videos.
Revised: We next present the SViT model: a video transformer that employs object tokens as a means to capture structure in both images and videos.

Original: We begin by reviewing the video transformer architecture, which
Revised: We begin by reviewing the video transformer architecture, which

Original: [Figure]
Revised: [Figure]

Original: HAOGloss
Revised: HAOGloss

Original: [Figure]
Revised: [Figure]

Original: ObjectToken
Revised: ObjectToken

Original: PatchToken our model extends.
Revised: PatchToken our model extends.

Original: Next, we explain how we process both images and videos.
Revised: Next, we explain how we process both images and videos.

Original: Finally, we describe object tokens and how they are used in videos and still images.
Revised: Finally, we describe object tokens and how they are used in videos and still images.

Original: Shared Video & Image Transformer .
Revised: Shared Video & Image Transformer .

Original: Video transformers [4, 11, 21, 34] extend the Vision Transformer model to the temporal domain.
Revised: Video transformers [4, 11, 21, 34] extend the Vision Transformer model to the temporal domain.

Original: Similar to vision transformers, the input is ﬁrst “patchiﬁed” but with temporally extended 3-D convolution (instead of 2-D) producing a down-sampled tensor X of size T ⇥ H ⇥ W ⇥ d .
Revised: Similar to vision transformers, the input is ﬁrst “patchiﬁed” but with temporally extended 3-D convolution (instead of 2-D) producing a down-sampled tensor X of size T ⇥ H ⇥ W ⇥ d .

Original: We refer to this as “patch tokens”.
Revised: We refer to this as “patch tokens”.

Original: Then, spatio-temporal position embeddings are added for providing location and time information.
Revised: Then, spatio-temporal position embeddings are added for providing location and time information.

Original: Next, multiple stacked self-attention blocks are applied repeatedly on the down-sampled tensor X to produce the ﬁnal vector representation using mean pooling over the patch tokens.
Revised: Next, multiple stacked self-attention blocks are applied repeatedly on the down-sampled tensor X to produce the ﬁnal vector representation using mean pooling over the patch tokens.

Original: In our approach, we want to be able to process batches of images or videos.
Revised: In our approach, we want to be able to process batches of images or videos.

Original: A key desideratum in this context is to be able to input both videos and still-images into the same model.
Revised: A key desideratum in this context is to be able to input both videos and still-images into the same model.

Original: Towards this end, our ﬁrst change is to rely only on 2-D convolutions in the ﬁrst transformer step instead of
Revised: Towards this end, our ﬁrst change is to rely only on 2-D convolutions in the ﬁrst transformer step instead of

Original: 3-D. This allows single images to be used as inputs without padding, and does not hurt performance in practice.
Revised: 3-D. This allows single images to be used as inputs without padding, and does not hurt performance in practice.

Original: Object tokens .
Revised: Object tokens .

Original: Our goal is to learn a shared structured representation between the video and the image domains.
Revised: Our goal is to learn a shared structured representation between the video and the image domains.

Original: We do this by adding transformer tokens to represent objects.
Revised: We do this by adding transformer tokens to represent objects.

Original: These tokens are functionally similar to the patch tokens, with two exceptions.
Revised: These tokens are functionally similar to the patch tokens, with two exceptions.

Original: First, they are initialized with additional trained embeddings, which we referred to as “object prompts”.
Revised: First, they are initialized with additional trained embeddings, which we referred to as “object prompts”.

Original: We consider the “object prompts” as a bridge between the images and the videos since they are being used in prediction tasks across both domains.
Revised: We consider the “object prompts” as a bridge between the images and the videos since they are being used in prediction tasks across both domains.

Original: Second, they are used to predict the structured representations when those are available (here, we use the Hand-Object Graphs structure for still images).
Revised: Second, they are used to predict the structured representations when those are available (here, we use the Hand-Object Graphs structure for still images).

Original: An object token is used to predict properties of the corresponding node in the HAOG, and the concatenation of two object tokens is used to predict edge properties.
Revised: An object token is used to predict properties of the corresponding node in the HAOG, and the concatenation of two object tokens is used to predict edge properties.

Original: Next, we describe the model more formally.
Revised: Next, we describe the model more formally.

Original: Let n be the maximum number of modeled objects per frame (or still image).
Revised: Let n be the maximum number of modeled objects per frame (or still image).

Original: We have n tokens for each frame, and thus a total of T ⇥ n tokens.
Revised: We have n tokens for each frame, and thus a total of T ⇥ n tokens.

Original: The token for object i at frame t is initialized with the vector o i + r t where o i 2 R d is a learned object prompt and r t 2 R d is a learned temporal positional embedding (the same one used for initializing the patch tokens).
Revised: The token for object i at frame t is initialized with the vector o i + r t where o i 2 R d is a learned object prompt and r t 2 R d is a learned temporal positional embedding (the same one used for initializing the patch tokens).

Original: With these new tokens, we have T ⇥ ( H ⇥ W + n ) tokens (i.e, vectors in R d ), and these together will go through the standard self-attention layers.
Revised: With these new tokens, we have T ⇥ ( H ⇥ W + n ) tokens (i.e, vectors in R d ), and these together will go through the standard self-attention layers.

Original: We denote the operator that outputs the ﬁnal representation of the object tokens by F O , where for a video V we have F O ( V ) 2 R T ⇥ n ⇥ d and for an image I we have F O ( I ) 2 R n ⇥ d .
Revised: We denote the operator that outputs the ﬁnal representation of the object tokens by F O , where for a video V we have F O ( V ) 2 R T ⇥ n ⇥ d and for an image I we have F O ( I ) 2 R n ⇥ d .

Original: The F O ( I ) representation is used to predict structured representations for single images.
Revised: The F O ( I ) representation is used to predict structured representations for single images.

Original: We also include a loss that makes single frame representations align with those of clip representations (see Section 2.3).
Revised: We also include a loss that makes single frame representations align with those of clip representations (see Section 2.3).

Original: The operator that outputs the ﬁnal representation vector used for the video downstream task is denoted by
Revised: The operator that outputs the ﬁnal representation vector used for the video downstream task is denoted by

Original: F CLS , where for video V we have F CLS ( V ) 2 R d .
Revised: F CLS , where for video V we have F CLS ( V ) 2 R d .

Original: Finally, our method can be used on top of the most common video transformers (MViT [21]
Revised: Finally, our method can be used on top of the most common video transformers (MViT [21]

Original: TimeSformer model it performs well empirically
Revised: TimeSformer model it performs well empirically

Original: 2.3 Losses and Training
Revised: 2.3 Losses and Training

Original: During training we have a set of images annotated with HAOGs and videos with downstream task labels.
Revised: During training we have a set of images annotated with HAOGs and videos with downstream task labels.

Original: We use these as inputs to our model (Section 2.2) and optimize the losses described below.
Revised: We use these as inputs to our model (Section 2.2) and optimize the losses described below.

Original: Video loss .
Revised: Video loss .

Original: Each training video V has a corresponding ground-truth category Y 2 { 1 , . .
Revised: Each training video V has a corresponding ground-truth category Y 2 { 1 , . .

Original: . , K } .
Revised: . , K } .

Original: As noted above, we have a vector representation for the entire video denoted by F CLS ( V ) .
Revised: As noted above, we have a vector representation for the entire video denoted by F CLS ( V ) .

Original: Thus, we simply use a neural network to predict logits for Y .
Revised: Thus, we simply use a neural network to predict logits for Y .

Original: Speciﬁcally, our predicted logits are: ˆ
Revised: Speciﬁcally, our predicted logits are: ˆ

Original: Y =
Revised: Y =

Original: FC ( F CLS ( V ))
Revised: FC ( F CLS ( V ))

Original: 2 R K , where FC is a trainable fully connected network.
Revised: 2 R K , where FC is a trainable fully connected network.

Original: We then consider a standard cross-entropy between the predicted logits ˆ Y and true labels Y .
Revised: We then consider a standard cross-entropy between the predicted logits ˆ Y and true labels Y .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: HAOG loss .
Revised: HAOG loss .

Original: The transformer outputs a set of n object tokens for each image.
Revised: The transformer outputs a set of n object tokens for each image.

Original: Since we have supervision for these objects, we predict structured representations from the tokens, and introduce a loss to optimize these predictions.
Revised: Since we have supervision for these objects, we predict structured representations from the tokens, and introduce a loss to optimize these predictions.

Original: Recall our notation from Section 2.1.
Revised: Recall our notation from Section 2.1.

Original: We let the transformer use n = 4 object tokens which correspond to the two hands (bounding boxes b 1 , b 2 ) and two manipulated objects (bounding boxes b 3 , b 4 ).
Revised: We let the transformer use n = 4 object tokens which correspond to the two hands (bounding boxes b 1 , b 2 ) and two manipulated objects (bounding boxes b 3 , b 4 ).

Original: The j th token is used to predict a corresponding bounding box b j , as well as the existence variable e j .
Revised: The j th token is used to predict a corresponding bounding box b j , as well as the existence variable e j .

Original: Formally, we let:
Revised: Formally, we let:

Original: [Equation]
Revised: [Equation]

Original: where FC bb , FC e are fully connected networks with four and one output respectively, F jO ( I ) is the j th object token, and I is the input image.
Revised: where FC bb , FC e are fully connected networks with four and one output respectively, F jO ( I ) is the j th object token, and I is the input image.

Original: Formally, this deﬁne the loss of the nodes as following:
Revised: Formally, this deﬁne the loss of the nodes as following:

Original: [Equation]
Revised: [Equation]

Original: where L1 is the standard L1 distance, and GIoU is used as in [69].
Revised: where L1 is the standard L1 distance, and GIoU is used as in [69].

Original: Furthermore, we predict the contact variable from the concatenation of the corresponding hand and object tokens.
Revised: Furthermore, we predict the contact variable from the concatenation of the corresponding hand and object tokens.

Original: Namely for j 2 { 1 , 2 } :
Revised: Namely for j 2 { 1 , 2 } :

Original: [Equation]
Revised: [Equation]

Original: where FC c is a fully connected with one output.
Revised: where FC c is a fully connected with one output.

Original: Thus, we deﬁne the loss of the edges as follows:
Revised: Thus, we deﬁne the loss of the edges as follows:

Original: [Equation]
Revised: [Equation]

Original: Finally, to obtain the loss, we compare the predictions and ground-truth labels as follows:
Revised: Finally, to obtain the loss, we compare the predictions and ground-truth labels as follows:

Original: [Equation]
Revised: [Equation]

Original: Frame-Clip Consistency Loss .
Revised: Frame-Clip Consistency Loss .

Original: Since we have different losses for images and video, the model could ﬁnd a way to minimize the image loss in a way that only applies to the images and does not transfer to video.
Revised: Since we have different losses for images and video, the model could ﬁnd a way to minimize the image loss in a way that only applies to the images and does not transfer to video.

Original: 1 To avoid this, we need to make sure that the video representation contains the same type of information as the still images, and in particular the structured information carried by the object tokens.
Revised: 1 To avoid this, we need to make sure that the video representation contains the same type of information as the still images, and in particular the structured information carried by the object tokens.

Original: To achieve this, we explicitly enforce the object tokens to be consistent across still images (frames) and videos (clips) using a frame-clip Consistency loss.
Revised: To achieve this, we explicitly enforce the object tokens to be consistent across still images (frames) and videos (clips) using a frame-clip Consistency loss.

Original: To calculate the loss, we process each video twice: once as a clip, and once as a batch of T frames.
Revised: To calculate the loss, we process each video twice: once as a clip, and once as a batch of T frames.

Original: Namely, a clip V consisting of T frames I 1 , ..., I T , will be processed once as F O ( V ) and once as a list of images ( F O ( I 1 ) , ..., F O ( I T )) .
Revised: Namely, a clip V consisting of T frames I 1 , ..., I T , will be processed once as F O ( V ) and once as a list of images ( F O ( I 1 ) , ..., F O ( I T )) .

Original: This results in two groups of T ⇥ n object tokens, the ﬁrst are the clip object tokens , denoted as, O clip :=
Revised: This results in two groups of T ⇥ n object tokens, the ﬁrst are the clip object tokens , denoted as, O clip :=

Original: F O ( V ) 2 R T ⇥ n ⇥ d , the second are the frame object tokens , denoted as O frames :=
Revised: F O ( V ) 2 R T ⇥ n ⇥ d , the second are the frame object tokens , denoted as O frames :=

Original: [ F O ( v 1 ) , ..., F O ( v T )] 2 R T ⇥ n ⇥ d .
Revised: [ F O ( v 1 ) , ..., F O ( v T )] 2 R T ⇥ n ⇥ d .

Original: Since both groups originate from the same input, each clip object token has a single corresponding frame object token.
Revised: Since both groups originate from the same input, each clip object token has a single corresponding frame object token.

Original: Namely, there is an alignment between the clip object tokens and the frame object tokens.
Revised: Namely, there is an alignment between the clip object tokens and the frame object tokens.

Original: Following the intuition that each clip object token should contain the information from its corresponding frame object token and potentially more (the temporal context), we minimize the L 1 distance between each clip object token and frame object token.
Revised: Following the intuition that each clip object token should contain the information from its corresponding frame object token and potentially more (the temporal context), we minimize the L 1 distance between each clip object token and frame object token.

Original: Namely:
Revised: Namely:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Overall loss .
Revised: Overall loss .

Original: The total loss consists of L Con , L HAOG and L V id , where L V id is the loss of the main task.
Revised: The total loss consists of L Con , L HAOG and L V id , where L V id is the loss of the main task.

Original: Each of the three terms in the loss is multiplied by a hyper-parameter (  Con ,  HAOG ,  V id ), and the total loss is the weighted combination of the three terms:
Revised: Each of the three terms in the loss is multiplied by a hyper-parameter (  Con ,  HAOG ,  V id ), and the total loss is the weighted combination of the three terms:

Original: [Equation]
Revised: [Equation]

Original:  Experiments and Results
Revised:  Experiments and Results

Original: We begin by describing the datasets, implementation details, and baselines and variants.
Revised: We begin by describing the datasets, implementation details, and baselines and variants.

Original: We then evaluate our approach on several benchmarks.
Revised: We then evaluate our approach on several benchmarks.

Original: Speciﬁcally, we consider the following tasks: Compositional Action Recognition (Section 3.4), Object State Change Classiﬁcation & Localization (Section 3.5), Action Recognition (Section 3.6), and Spatio-Temporal Action Detection (Section 3.7).
Revised: Speciﬁcally, we consider the following tasks: Compositional Action Recognition (Section 3.4), Object State Change Classiﬁcation & Localization (Section 3.5), Action Recognition (Section 3.6), and Spatio-Temporal Action Detection (Section 3.7).

Original: 3.1 Datasets
Revised: 3.1 Datasets

Original: We ﬁrst describe the datasets used for the downstream video tasks, and those used as “auxiliary” datasets with annotated images.
Revised: We ﬁrst describe the datasets used for the downstream video tasks, and those used as “auxiliary” datasets with annotated images.

Original: We use the following video datasets: (1) Something-Something v2 (SSv2) [30] is a dataset containing 174 action categories of common human-object interactions.
Revised: We use the following video datasets: (1) Something-Something v2 (SSv2) [30] is a dataset containing 174 action categories of common human-object interactions.

Original: (2) SomethingElse [59] which exploits the compositional structure of SSv2, where a combination of a verb and a noun deﬁnes an action.
Revised: (2) SomethingElse [59] which exploits the compositional structure of SSv2, where a combination of a verb and a noun deﬁnes an action.

Original: We follow the ofﬁcial compositional split from [59], which assumes the set of noun-verb pairs available for training is disjoint from the set given at test time.
Revised: We follow the ofﬁcial compositional split from [59], which assumes the set of noun-verb pairs available for training is disjoint from the set given at test time.

Original: (3) Ego4D [31] is a new large-scale dataset of more than 3,670 hours of video data, capturing the daily-life scenarios of more than 900 unique individuals from nine different countries around the world.
Revised: (3) Ego4D [31] is a new large-scale dataset of more than 3,670 hours of video data, capturing the daily-life scenarios of more than 900 unique individuals from nine different countries around the world.

Original: (4) Diving48 [54] contains 48 ﬁne-grained categories of diving activities.
Revised: (4) Diving48 [54] contains 48 ﬁne-grained categories of diving activities.

Original: (5) Atomic Visual
Revised: (5) Atomic Visual

Original: Actions (AVA) [32] is a benchmark for human action detection.
Revised: Actions (AVA) [32] is a benchmark for human action detection.

Original: We report Mean Average Precision (mAP) on AVA-V2.2.
Revised: We report Mean Average Precision (mAP) on AVA-V2.2.

Original: For “auxiliary” image datasets we use the Ego4D, and the 100 Days of
Revised: For “auxiliary” image datasets we use the Ego4D, and the 100 Days of

Original: Hands (100DOH) [72] datasets.
Revised: Hands (100DOH) [72] datasets.

Original: We collected 79,921 annotated images from 100DOH, and 57, annotated images from Ego4D. 2 The AVA or SSv2 datasets are also used as “auxiliary” images in some experiments.
Revised: We collected 79,921 annotated images from 100DOH, and 57, annotated images from Ego4D. 2 The AVA or SSv2 datasets are also used as “auxiliary” images in some experiments.

Original: For more details, see Section D.1 in the supplementary.
Revised: For more details, see Section D.1 in the supplementary.

Original: For the image datasets, the image annotations are based on frames selected from videos, but these are often sparsely selected from the videos, so it is natural to treat them as annotated still images.
Revised: For the image datasets, the image annotations are based on frames selected from videos, but these are often sparsely selected from the videos, so it is natural to treat them as annotated still images.

Original: 3.2 Implementation Details
Revised: 3.2 Implementation Details

Original: SViT is implemented in PyTorch, and the code will be released in our project page.
Revised: SViT is implemented in PyTorch, and the code will be released in our project page.

Original: Our training recipes and code are based on the MViTv2 model, and were taken from https://github.com/ facebookresearch/mvit .
Revised: Our training recipes and code are based on the MViTv2 model, and were taken from https://github.com/ facebookresearch/mvit .

Original: We pretrain the SViT model on the K400 [45] video dataset with our auxiliary image datasets.
Revised: We pretrain the SViT model on the K400 [45] video dataset with our auxiliary image datasets.

Original: Then, we ﬁnetune on the target video understanding task (detailed in
Revised: Then, we ﬁnetune on the target video understanding task (detailed in

Original: Section 3.1) together with the auxiliary image datasets and the SViT loss.
Revised: Section 3.1) together with the auxiliary image datasets and the SViT loss.

Original: Each training batch contains 64 images and 64 videos in order to minimize the overall loss in Equation 8.
Revised: Each training batch contains 64 images and 64 videos in order to minimize the overall loss in Equation 8.

Original: For more details about datasets and evaluation see Section C of Supplementary.
Revised: For more details about datasets and evaluation see Section C of Supplementary.

Original: 3.3 Baselines and SViT variants
Revised: 3.3 Baselines and SViT variants

Original: Baselines .
Revised: Baselines .

Original: In the experiments, we compare SViT to several models from prior work reported for the corresponding datasets.
Revised: In the experiments, we compare SViT to several models from prior work reported for the corresponding datasets.

Original: These include the following approaches: I3D [13], SlowFast [22], and the state-of-the-art transformers – Mformer [66] and MViTv2 [53].
Revised: These include the following approaches: I3D [13], SlowFast [22], and the state-of-the-art transformers – Mformer [66] and MViTv2 [53].

Original: In addition, we provide the MViTv multi-task (MViTv2 MT) baseline for comparing a naive multi-tasking approach to our model.
Revised: In addition, we provide the MViTv multi-task (MViTv2 MT) baseline for comparing a naive multi-tasking approach to our model.

Original: This is a version of MViTv2 which performs video classiﬁcation and HAOG prediction from images.
Revised: This is a version of MViTv2 which performs video classiﬁcation and HAOG prediction from images.

Original: The
Revised: The

Original: MViTv2 model is enhanced with two heads: the ﬁrst head is composed of an MLP that predicts the video labels given a ﬁnal representation vector, the second head consists of MLPs that predict the
Revised: MViTv2 model is enhanced with two heads: the ﬁrst head is composed of an MLP that predicts the video labels given a ﬁnal representation vector, the second head consists of MLPs that predict the

Original: HAOG information (i.e., hands and object boxes, and contact information).
Revised: HAOG information (i.e., hands and object boxes, and contact information).

Original: SViT Variants .
Revised: SViT Variants .

Original: As mentioned above, we use several sources of auxiliary image datasets.
Revised: As mentioned above, we use several sources of auxiliary image datasets.

Original: In some cases, we have auxiliary images that are frames from video understanding datasets.
Revised: In some cases, we have auxiliary images that are frames from video understanding datasets.

Original: For example, in Ego4D, we have both videos and frames annotated from some of these videos.
Revised: For example, in Ego4D, we have both videos and frames annotated from some of these videos.

Original: In this sense, the
Revised: In this sense, the

Original: [Table]
Revised: [Table]

Revised: Compositional

Revised: [Table]

Original: auxiliary data is “in-domain” for some of the video tasks.
Revised: auxiliary data is “in-domain” for some of the video tasks.

Original: In order to evaluate the importance of this effect, we explore “Different Domain” ( SViT-DD ) training, which avoids using images from a given video task as auxiliary during ﬁnetuning and pretraining.
Revised: In order to evaluate the importance of this effect, we explore “Different Domain” ( SViT-DD ) training, which avoids using images from a given video task as auxiliary during ﬁnetuning and pretraining.

Original: Similarly, “In Domain” ( SViT-ID ) refers to the case where we use only images from the video task during ﬁnetuning, but not necessarily during pretraining .
Revised: Similarly, “In Domain” ( SViT-ID ) refers to the case where we use only images from the video task during ﬁnetuning, but not necessarily during pretraining .

Original: We consider another variant that does not use the auxiliary images during ﬁnetuning at all (but does use them during pre-training), and refer to it as SViT-SFT .
Revised: We consider another variant that does not use the auxiliary images during ﬁnetuning at all (but does use them during pre-training), and refer to it as SViT-SFT .

Original: Finally, to explore the effect of data-size in the In Domain setting, we evaluate a version SViT ⇥ X% that only uses X% of the in-domain ﬁnetuning data.
Revised: Finally, to explore the effect of data-size in the In Domain setting, we evaluate a version SViT ⇥ X% that only uses X% of the in-domain ﬁnetuning data.

Original: For MViTv2 MT, the auxiliary images correspond to the “In Domain” setting (i.e., same setting as for SViT-ID).
Revised: For MViTv2 MT, the auxiliary images correspond to the “In Domain” setting (i.e., same setting as for SViT-ID).

Original: Auxiliary Images datasets .
Revised: Auxiliary Images datasets .

Original: For SViT-DD, we generally use 100DOH and Ego4D for pretraining and ﬁnetuning, while for the Ego4D tasks, we only use 100DOH.
Revised: For SViT-DD, we generally use 100DOH and Ego4D for pretraining and ﬁnetuning, while for the Ego4D tasks, we only use 100DOH.

Original: SViT-ID uses 100DOH and Ego4D for pretraining and in-domain downstream frames for ﬁnetuning.
Revised: SViT-ID uses 100DOH and Ego4D for pretraining and in-domain downstream frames for ﬁnetuning.

Original: The SViT-SFT pretraining setting is similar to that of SViT-DD.
Revised: The SViT-SFT pretraining setting is similar to that of SViT-DD.

Original: SViT ⇥ X% setting is similar to that of SViT-ID.
Revised: SViT ⇥ X% setting is similar to that of SViT-ID.

Original: For more details, see Section D.1 in the supplementary.
Revised: For more details, see Section D.1 in the supplementary.

Original: 3.4 Compositional & Few-Shot Action Recognition
Revised: 3.4 Compositional & Few-Shot Action Recognition

Original: Several video datasets deﬁne an action as a combination of a verb and a noun.
Revised: Several video datasets deﬁne an action as a combination of a verb and a noun.

Original: One challenge then is to recognize combinations that were not seen during training.
Revised: One challenge then is to recognize combinations that were not seen during training.

Original: This “compositional” setting was explored in the “SomethingElse” dataset [59], where verb-noun combinations in the test data do not occur in the training data.
Revised: This “compositional” setting was explored in the “SomethingElse” dataset [59], where verb-noun combinations in the test data do not occur in the training data.

Original: The split contains 174 classes with 54,919/54,876 videos for training/validation.
Revised: The split contains 174 classes with 54,919/54,876 videos for training/validation.

Original: We also evaluate on the few-shot compositional action recognition task in [59] (See supplementary).
Revised: We also evaluate on the few-shot compositional action recognition task in [59] (See supplementary).

Original: Table 1 reports the results for these two tasks.
Revised: Table 1 reports the results for these two tasks.

Original: All of our variants outperform all prior works and the
Revised: All of our variants outperform all prior works and the

Original: MViTv2 MT on the Compositional and Few-shot task.
Revised: MViTv2 MT on the Compositional and Few-shot task.

Original: From the table, we can see that using 2% of the in-domain data as auxiliary images (SViT ⇥ 2%), is almost comparable to using all the available in-domain images (SViT-ID).
Revised: From the table, we can see that using 2% of the in-domain data as auxiliary images (SViT ⇥ 2%), is almost comparable to using all the available in-domain images (SViT-ID).

Original: Furthermore, the results are comparable to SViT-DD (0.7 difference) when trained on auxiliary images from different domains.
Revised: Furthermore, the results are comparable to SViT-DD (0.7 difference) when trained on auxiliary images from different domains.

Original: In this way, our approach optimizes the use of structural information across domains.
Revised: In this way, our approach optimizes the use of structural information across domains.

Original: Last, SViT-DD outperforms SViT-SFT when using the same data, demonstrating the value of ﬁnetuning video and image together.
Revised: Last, SViT-DD outperforms SViT-SFT when using the same data, demonstrating the value of ﬁnetuning video and image together.

Original: 3.5 Object State Change Classiﬁcation and Localization
Revised: 3.5 Object State Change Classiﬁcation and Localization

Original: Hands and objects are key elements in human activity.
Revised: Hands and objects are key elements in human activity.

Original: Two tasks related to hand-object interaction have recently been introduced in the Ego4D [31] dataset.
Revised: Two tasks related to hand-object interaction have recently been introduced in the Ego4D [31] dataset.

Original: The ﬁrst is temporal localization, which is deﬁned as ﬁnding key frames in a video clip that indicate a change in object state.
Revised: The ﬁrst is temporal localization, which is deﬁned as ﬁnding key frames in a video clip that indicate a change in object state.

Original: The second is object state change classiﬁcation, which indicates whether an object state has changed or not.
Revised: The second is object state change classiﬁcation, which indicates whether an object state has changed or not.

Revised: SViT won the ﬁrst place at the Ego4D CVPR’22 Point of No Return Temporal Localization Challenge.

Original: Table 2a reports results on the temporal localization and object state change classiﬁcation tasks on the
Revised: Table 2a reports results on the temporal localization and object state change classiﬁcation tasks on the

Original: Ego4D dataset.
Revised: Ego4D dataset.

Original: We observe that SViT-ID, SViT ⇥ 2% and SViT-DD perform better than SViT-SFT and are very close.
Revised: We observe that SViT-ID, SViT ⇥ 2% and SViT-DD perform better than SViT-SFT and are very close.

Original: These results are consistent with those presented in Section 3.4, which indicates that our method successfully leverages structural information across domains, and that a joint image and video ﬁnetuning is beneﬁcial.
Revised: These results are consistent with those presented in Section 3.4, which indicates that our method successfully leverages structural information across domains, and that a joint image and video ﬁnetuning is beneﬁcial.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: not use any additional structured annotation with † .
Revised: not use any additional structured annotation with † .

Original: 3.6 Action Recognition
Revised: 3.6 Action Recognition

Original: Table 2b and Table 2c report the results for standard action recognition task on the SSv2 and
Revised: Table 2b and Table 2c report the results for standard action recognition task on the SSv2 and

Original: Diving48 datasets respectively.
Revised: Diving48 datasets respectively.

Original: It can be seen that in Diving48 dataset, our method improves over the MViT-v2 baseline by 6.7%, outperforming all the methods.
Revised: It can be seen that in Diving48 dataset, our method improves over the MViT-v2 baseline by 6.7%, outperforming all the methods.

Original: This demonstrates that our method has some ability to generalize the learned structural information of hand-object interactions even to ﬁne-grained “human-motion” tasks such as Diving48.
Revised: This demonstrates that our method has some ability to generalize the learned structural information of hand-object interactions even to ﬁne-grained “human-motion” tasks such as Diving48.

Original: We also note that Diving48 has no structured annotations available, thus SViT-ID (and SViT ⇥ 2% ) can not be provided.
Revised: We also note that Diving48 has no structured annotations available, thus SViT-ID (and SViT ⇥ 2% ) can not be provided.

Original: On SSv2, SViT-ID improves the baseline by 1.5%, SViT ⇥ 2% improves by 1.3%, and SViT-DD improves by 1.1%.
Revised: On SSv2, SViT-ID improves the baseline by 1.5%, SViT ⇥ 2% improves by 1.3%, and SViT-DD improves by 1.1%.

Original: As we observed signiﬁcant improvements on the more challenging SomethingElse setup, we hypothesize that SomethingElse is more likely to beneﬁt from the ability to generalize to unseen combinations of objects and hands exhibited by SViT.
Revised: As we observed signiﬁcant improvements on the more challenging SomethingElse setup, we hypothesize that SomethingElse is more likely to beneﬁt from the ability to generalize to unseen combinations of objects and hands exhibited by SViT.

Original: We provide visualizations of the HAOGs predicted by SViT in Section B of the supplementary.
Revised: We provide visualizations of the HAOGs predicted by SViT in Section B of the supplementary.

Original: 3.7 Spatio-temporal Action Detection
Revised: 3.7 Spatio-temporal Action Detection

Original: [Table]
Revised: [Table]

Original: Table 3 report the results for spatio-temporal action detection on the AVA dataset.
Revised: Table 3 report the results for spatio-temporal action detection on the AVA dataset.

Original: In the literature, the action detection task on AVA is formulated as a two stage prediction procedure.
Revised: In the literature, the action detection task on AVA is formulated as a two stage prediction procedure.

Original: The ﬁrst step is the detection of bounding boxes, which are obtained through an off-the-shelf pretrained person detector.
Revised: The ﬁrst step is the detection of bounding boxes, which are obtained through an off-the-shelf pretrained person detector.

Original: The second step involves predicting the action being performed at each detected bounding box.
Revised: The second step involves predicting the action being performed at each detected bounding box.

Original: The performance is benchmarked on the end result of these steps and is measured with the Mean Average Precision (MAP) metric.
Revised: The performance is benchmarked on the end result of these steps and is measured with the Mean Average Precision (MAP) metric.

Original: Typically, for fair comparison, the detected person boxes are kept identical across approaches and hence, the ﬁnal performance depends directly on the ability of the approach to utilize the video and box information.
Revised: Typically, for fair comparison, the detected person boxes are kept identical across approaches and hence, the ﬁnal performance depends directly on the ability of the approach to utilize the video and box information.

Original: We observe that all SViT variants improve over the MViT-v2 baseline.
Revised: We observe that all SViT variants improve over the MViT-v2 baseline.

Original: Speciﬁcally, SViT-ID improves the
Revised: Speciﬁcally, SViT-ID improves the

Original: MViT-v2 baseline by 1.7, SViT-DD improves by 1.5, and SViT-SFT improves by 1.0.
Revised: MViT-v2 baseline by 1.7, SViT-DD improves by 1.5, and SViT-SFT improves by 1.0.

Original: 3.8 Ablations
Revised: 3.8 Ablations

Original: We perform a comprehensive ablation study on the compositional action recognition task [59] on the
Revised: We perform a comprehensive ablation study on the compositional action recognition task [59] on the

Original: SomethingElse dataset to measure the contribution of the different SViT components (Table 4).
Revised: SomethingElse dataset to measure the contribution of the different SViT components (Table 4).

Original: Components of the SViT model .
Revised: Components of the SViT model .

Original: In Table 4a, we ablate our model to show the signiﬁcance of the individual components.
Revised: In Table 4a, we ablate our model to show the signiﬁcance of the individual components.

Original: First, we train a standard MViTv2 in a multi-task setting (MViTv2 MT), where we use videos for video action recognition, and images for HAOG prediction.
Revised: First, we train a standard MViTv2 in a multi-task setting (MViTv2 MT), where we use videos for video action recognition, and images for HAOG prediction.

Original: There is a small improvement (+0.5) over the baseline.
Revised: There is a small improvement (+0.5) over the baseline.

Original: We then take the MViTv2 and add the object tokens and corresponding SViT loss; this leads to a more signiﬁcant improvement (+1.7).
Revised: We then take the MViTv2 and add the object tokens and corresponding SViT loss; this leads to a more signiﬁcant improvement (+1.7).

Original: Last, when adding both the consistency loss and the object tokens, we observe the largest improvement (+2.5).
Revised: Last, when adding both the consistency loss and the object tokens, we observe the largest improvement (+2.5).

Original: [Table]
Revised: [Table]

Original: The amount of in-domain data .
Revised: The amount of in-domain data .

Original: In Table 4b
Revised: In Table 4b

Original: [Table]
Revised: [Table]

Original: we demonstrate the effect of the amount of in-domain image data used during training.
Revised: we demonstrate the effect of the amount of in-domain image data used during training.

Original: Speciﬁcally, since “SomethingElse” includes HAOG annotations for each frame
Revised: Speciﬁcally, since “SomethingElse” includes HAOG annotations for each frame

Original: [Table]
Revised: [Table]

Original: we train our method with different amount of images ( 2% , 25% , 50% , and 100% ) from the “SomthingElse” dataset.
Revised: we train our method with different amount of images ( 2% , 25% , 50% , and 100% ) from the “SomthingElse” dataset.

Original: It can be seen that our approach takes advantage of the structured information within a small amount of data and, as a result, has enhanced sample-efﬁciency.
Revised: It can be seen that our approach takes advantage of the structured information within a small amount of data and, as a result, has enhanced sample-efﬁciency.

Original: Robustness to multiple domains .
Revised: Robustness to multiple domains .

Original: In Table 4c We performed experiments on SomethingElse (groundtruth annotations available) where hands or objects were not visible in order to verify the usefulness of our approach: (i) After ﬁltering the videos containing (annotated) hands, we tested our model and the MViTv2 model on the ﬁltered hand split.
Revised: In Table 4c We performed experiments on SomethingElse (groundtruth annotations available) where hands or objects were not visible in order to verify the usefulness of our approach: (i) After ﬁltering the videos containing (annotated) hands, we tested our model and the MViTv2 model on the ﬁltered hand split.

Original: MViTv2 achieved an accuracy of 64.6 while our model achieved 66.5.
Revised: MViTv2 achieved an accuracy of 64.6 while our model achieved 66.5.

Original: This implies an improvement of +1.9.
Revised: This implies an improvement of +1.9.

Original: (ii) After ﬁltering the videos containing (annotated) objects in more than 40% of the frames, we tested our model and the MViTv2 model on the ﬁltered object split.
Revised: (ii) After ﬁltering the videos containing (annotated) objects in more than 40% of the frames, we tested our model and the MViTv2 model on the ﬁltered object split.

Original: MViTv2 achieved an accuracy of 64.9 while our model achieved 67.0.
Revised: MViTv2 achieved an accuracy of 64.9 while our model achieved 67.0.

Original: This implies an improvement of +2.1.
Revised: This implies an improvement of +2.1.

Original: We can observe that our model outperforms the baseline even when there are no objects or hands in the videos, demonstrating the robustness of our approach.
Revised: We can observe that our model outperforms the baseline even when there are no objects or hands in the videos, demonstrating the robustness of our approach.

Original: The total improvement (+1.9 and +2.1) is also a little bit lower than before the ﬁltering (+2.5), which indicates that there has been a slight degradation.
Revised: The total improvement (+1.9 and +2.1) is also a little bit lower than before the ﬁltering (+2.5), which indicates that there has been a slight degradation.

Original: Even so, it is still valuable to use the hand-object graphs.
Revised: Even so, it is still valuable to use the hand-object graphs.

Original: How important are the HAOGs .
Revised: How important are the HAOGs .

Original: To examine how important is the information provided by the
Revised: To examine how important is the information provided by the

Original: HAOG, we suggest learning HAOGs without any useful information.
Revised: HAOG, we suggest learning HAOGs without any useful information.

Original: Thus, we run an experiment in which the HAOGs are completely random.
Revised: Thus, we run an experiment in which the HAOGs are completely random.

Original: This means that, for each image, a random HOAG is generated by sampling the boxes and their relationships uniformly.
Revised: This means that, for each image, a random HOAG is generated by sampling the boxes and their relationships uniformly.

Original: We refer to this experiment as
Revised: We refer to this experiment as

Original: SViT-Random-ID.
Revised: SViT-Random-ID.

Original: The SViT-Random-ID result on SomethingElse is 50.6 compared to 63.3 in the
Revised: The SViT-Random-ID result on SomethingElse is 50.6 compared to 63.3 in the

Original: SViT-ID baseline.
Revised: SViT-ID baseline.

Original: This demonstrates that predicting the actual HAOG attributes is important but that not many annotations are necessary for the auxiliary task.
Revised: This demonstrates that predicting the actual HAOG attributes is important but that not many annotations are necessary for the auxiliary task.

Original: One explanation may be that learning the shared structured representation between the domains is sample efﬁcient and does not require many annotated images, which we view as a big advantage of our approach.
Revised: One explanation may be that learning the shared structured representation between the domains is sample efﬁcient and does not require many annotated images, which we view as a big advantage of our approach.

Original:  Related Work
Revised:  Related Work

Original: Joint Training with Images and Videos .
Revised: Joint Training with Images and Videos .

Original: Vision Transformers [20] are versatile architectures designed to be ﬂexible in terms of input sequence length.
Revised: Vision Transformers [20] are versatile architectures designed to be ﬂexible in terms of input sequence length.

Original: Aside from the data “‘patchiﬁcation”’ layer, the rest of the transformer is agnostic to the domain of the input.
Revised: Aside from the data “‘patchiﬁcation”’ layer, the rest of the transformer is agnostic to the domain of the input.

Original: Consequently, an exciting direction of joint training image-video has emerged in computer vision, a direction previously less natural with convolutional architectures.
Revised: Consequently, an exciting direction of joint training image-video has emerged in computer vision, a direction previously less natural with convolutional architectures.

Original: Speciﬁcally, this allows for training network simultaneously on image and video, with most of the network parameters shared across both domains.
Revised: Speciﬁcally, this allows for training network simultaneously on image and video, with most of the network parameters shared across both domains.

Original: Recently, multi-modal models became widely popular, from shared backbones [15, 57, 61] to separate encoders for each input modality [74, 77], and even non-vision modalities [39, 40].
Revised: Recently, multi-modal models became widely popular, from shared backbones [15, 57, 61] to separate encoders for each input modality [74, 77], and even non-vision modalities [39, 40].

Original: In our case, we focus on video and image input, similar to [6, 27, 87].
Revised: In our case, we focus on video and image input, similar to [6, 27, 87].

Original: In contrast to these works, our training approach utilizes two domains, while one of them (image) is used only during training and is supervised by a different task.
Revised: In contrast to these works, our training approach utilizes two domains, while one of them (image) is used only during training and is supervised by a different task.

Original: Additionally, as we demonstrate here, we are not greatly impacted by domain mismatches between images and videos, as in previous works.
Revised: Additionally, as we demonstrate here, we are not greatly impacted by domain mismatches between images and videos, as in previous works.

Original: In a related area of research, multi-task learning [14] involves developing models that predict output for multiple tasks using a common input.
Revised: In a related area of research, multi-task learning [14] involves developing models that predict output for multiple tasks using a common input.

Original: Our work is different since it uses two inputs from different domains, images and videos.
Revised: Our work is different since it uses two inputs from different domains, images and videos.

Original: Similarly, our work can be viewed as a form of auxiliary task, as already shown in many works [29, 38, 73].
Revised: Similarly, our work can be viewed as a form of auxiliary task, as already shown in many works [29, 38, 73].

Original: However, it differs in that the shared object prompts are utilized to learn the shared structured representation between video and image domains, which enhances the self-attention layers with the structured representation for the main video task.
Revised: However, it differs in that the shared object prompts are utilized to learn the shared structured representation between video and image domains, which enhances the self-attention layers with the structured representation for the main video task.

Original: Learning from video using other modalities .
Revised: Learning from video using other modalities .

Original: Researchers have proposed different approaches to learn from videos using audio and natural language [1, 3, 19, 62, 64, 65, 68, 82].
Revised: Researchers have proposed different approaches to learn from videos using audio and natural language [1, 3, 19, 62, 64, 65, 68, 82].

Original: Vision tasks like Video Captioning [19, 24, 65, 82] and Visual Quesion Answering [2, 79] require understanding of both visual and natural language.
Revised: Vision tasks like Video Captioning [19, 24, 65, 82] and Visual Quesion Answering [2, 79] require understanding of both visual and natural language.

Original: Another line of vision and language works explored object labels, supervision of textual descriptions, or the use of region crops [50, 52], in contrast to SViT, which does not require any of these components.
Revised: Another line of vision and language works explored object labels, supervision of textual descriptions, or the use of region crops [50, 52], in contrast to SViT, which does not require any of these components.

Original: Additionally, Since high quality captions are expensive, researchers have proposed self-supervised ways to learn from both modalities without labeled data [62, 61, 68].
Revised: Additionally, Since high quality captions are expensive, researchers have proposed self-supervised ways to learn from both modalities without labeled data [62, 61, 68].

Original: In these previous works, the two modalities must be consistent and aligned (e.g., text, audio, and RGB should be consistent for same sample).
Revised: In these previous works, the two modalities must be consistent and aligned (e.g., text, audio, and RGB should be consistent for same sample).

Original: However, in our work, we do not require that the annotated images be precisely aligned with our downstream video task.
Revised: However, in our work, we do not require that the annotated images be precisely aligned with our downstream video task.

Original: Last, other new works [16, 70] explored different structured representations for videos, different from what we do, which is to leverage structure from images to videos.
Revised: Last, other new works [16, 70] explored different structured representations for videos, different from what we do, which is to leverage structure from images to videos.

Original: Structured Models .
Revised: Structured Models .

Original: Recently structured models have been successfully applied in many computer vision applications: video relation detection [55, 71, 76], vision and language [17, 51, 52, 78], relational reasoning [8, 9, 36, 48, 41, 67, 85, 86], human-object interactions [23, 44, 84], and even image & video generation [7, 33, 43].
Revised: Recently structured models have been successfully applied in many computer vision applications: video relation detection [55, 71, 76], vision and language [17, 51, 52, 78], relational reasoning [8, 9, 36, 48, 41, 67, 85, 86], human-object interactions [23, 44, 84], and even image & video generation [7, 33, 43].

Original: A recent line of works [5, 25, 26, 34, 35, 42, 58, 60, 63, 75, 80, 81, 83, 89] focuses on capturing spatio-temporal interactions for video understanding.
Revised: A recent line of works [5, 25, 26, 34, 35, 42, 58, 60, 63, 75, 80, 81, 83, 89] focuses on capturing spatio-temporal interactions for video understanding.

Original: Some of these works [34, 81, 83, 89] used static image object annotations to build the representations.
Revised: Some of these works [34, 81, 83, 89] used static image object annotations to build the representations.

Original: However, these works differ from ours.
Revised: However, these works differ from ours.

Original: [83] proposed the Object Transformers (OT) for long form video understanding, which model a video as a set of objects by extracting region crops from SlowFast [22] based on bounding boxes from an external detector.
Revised: [83] proposed the Object Transformers (OT) for long form video understanding, which model a video as a set of objects by extracting region crops from SlowFast [22] based on bounding boxes from an external detector.

Original: Contrary to this work, our transformer contains both patch tokens and object tokens, whereas the OT contains only object tokens.
Revised: Contrary to this work, our transformer contains both patch tokens and object tokens, whereas the OT contains only object tokens.

Original: Additionally, our model learns the object-centric representation without the need for an external detector.
Revised: Additionally, our model learns the object-centric representation without the need for an external detector.

Original: Another work [89], proposed a structured model for action detection that explicitly utilizes image annotations.
Revised: Another work [89], proposed a structured model for action detection that explicitly utilizes image annotations.

Original: They use a GCN [47] to reason about actors and objects interaction, on top of an I3D network.
Revised: They use a GCN [47] to reason about actors and objects interaction, on top of an I3D network.

Original: Unlike this work, we examine how image annotations can be exploited as part of a video transformer model.
Revised: Unlike this work, we examine how image annotations can be exploited as part of a video transformer model.

Original: Video Understanding Models .
Revised: Video Understanding Models .

Original: We work on several well-benchmarked down-stream video tasks such as compositional action recognition on Something-Else dataset [59] and action recognition with focus on temporal cues de-emphasizing appearance such as in Something-Something V2 dataset [30] and
Revised: We work on several well-benchmarked down-stream video tasks such as compositional action recognition on Something-Else dataset [59] and action recognition with focus on temporal cues de-emphasizing appearance such as in Something-Something V2 dataset [30] and

Original: Diving48 [54].
Revised: Diving48 [54].

Original: Several recent video transfromer works performed well on SSv2, such as ViViT [4],
Revised: Several recent video transfromer works performed well on SSv2, such as ViViT [4],

Original: MViT [21], MFormer [66], TimeSformer [11], MViTv2 [53],and Video Swin [56].
Revised: MViT [21], MFormer [66], TimeSformer [11], MViTv2 [53],and Video Swin [56].

Original: We choose to work with MViTv2, although our method can be used on top of any of these.
Revised: We choose to work with MViTv2, although our method can be used on top of any of these.

Original: Additionally, we also adapt our proposed method to interesting new tasks on the Ego4D dataset [31].
Revised: Additionally, we also adapt our proposed method to interesting new tasks on the Ego4D dataset [31].

Original: We believe such ﬁne-grained video understanding tasks require richer scene and object semantic reasoning, a requirement well afforded by our proposed Hand-Object Graph.
Revised: We believe such ﬁne-grained video understanding tasks require richer scene and object semantic reasoning, a requirement well afforded by our proposed Hand-Object Graph.

Original:  Conclusion
Revised:  Conclusion

Original: Video understanding is a key element of human visual perception, but modeling remains a challenge for machine vision.
Revised: Video understanding is a key element of human visual perception, but modeling remains a challenge for machine vision.

Original: In this work, we demonstrated the importance of learning from a scene structure from a small set of images to facilitate video learning within or outside the domain of interest.
Revised: In this work, we demonstrated the importance of learning from a scene structure from a small set of images to facilitate video learning within or outside the domain of interest.

Original: According to our empirical study, our SViT approach improves performance on four video understanding tasks.
Revised: According to our empirical study, our SViT approach improves performance on four video understanding tasks.

Original: Note, that we did not prioritize making the HAOG annotations richer, i.e., it may be possible to add other physical properties to improve structure modeling.
Revised: Note, that we did not prioritize making the HAOG annotations richer, i.e., it may be possible to add other physical properties to improve structure modeling.

Original: In addition, we noticed that leveraging the structure is effective even with 2% of the in-domain frames, as well as when applying it from outside the domain.
Revised: In addition, we noticed that leveraging the structure is effective even with 2% of the in-domain frames, as well as when applying it from outside the domain.

Original: We leave for future work how different physical properties can be more effectively utilized in more complex structures.
Revised: We leave for future work how different physical properties can be more effectively utilized in more complex structures.

Original: We do not anticipate a speciﬁc negative impact, but, as with any Machine Learning method, we recommend to exercise caution.
Revised: We do not anticipate a speciﬁc negative impact, but, as with any Machine Learning method, we recommend to exercise caution.

Original: Acknowledgements
Revised: Acknowledgements

Original: This project has received funding from the European Research Council (ERC) under the European
Revised: This project has received funding from the European Research Council (ERC) under the European

Original: Unions Horizon 2020 research and innovation programme (grant ERC HOLI 819080).
Revised: Unions Horizon 2020 research and innovation programme (grant ERC HOLI 819080).

Original: Prof. Darrell’s group was supported in part by DoD including DARPA’s PTG and LwLL programs, as well as BAIR’s industrial alliance programs.
Revised: Prof. Darrell’s group was supported in part by DoD including DARPA’s PTG and LwLL programs, as well as BAIR’s industrial alliance programs.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data are proprietary.

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See line 341-342.
Revised: [Yes] See line 341-342.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] Sec Section 3.1 (the Dataset Section).
Revised: [Yes] Sec Section 3.1 (the Dataset Section).

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See the Dataset Section in supplementary.
Revised: [Yes] See the Dataset Section in supplementary.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether
Revised: [No] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] See the Dataset Section in supplementary.
Revised: [Yes] See the Dataset Section in supplementary.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [Yes] See the Dataset Section in supplementary.
Revised: [Yes] See the Dataset Section in supplementary.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]