Original: Bringing Image Structure to VideoTransformers via Frame-Clip Consistency of Object Tokens
Revised: Bringing Image Structure to Video via Frame-Clip Consistency of Object Tokens

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Recent action recognition models have achieved impressive results by integratingobjects, their locations and interactions.
Revised: Recent action recognition models have achieved impressive results by integrating objects, their locations and interactions.

Original: However, obtaining dense structuredannotations for each frame is tedious and time-consuming, making these methodsexpensive to train and less scalable.
Revised: However, obtaining dense structured annotations for each frame is tedious and time-consuming, making these methods expensive to train and less scalable.

Original: At the same time, if a small set of annotatedimages is available, either within or outside the domain of interest, how could weleverage these for a video downstream task?
Revised: At the same time, if a small set of annotated images is available, either within or outside the domain of interest, how could we leverage these for a video downstream task?

Original: We propose a learning framework
Revised: We propose a learning framework

Original: StructureViT (SViT for short), which demonstrates how utilizing the structure of asmall number of images only available during training can improve a video model.
Revised: StructureViT (SViT for short), which demonstrates how utilizing the structure of a small number of images only available during training can improve a video model.

Original: SViT relies on two key insights.
Revised: SViT relies on two key insights.

Original: First, as both images and videos contain structuredinformation, we enrich a transformer model with a set of object tokens that can beused across images and videos.
Revised: First, as both images and videos contain structured information, we enrich a transformer model with a set of object tokens that can be used across images and videos.

Original: Second, the scene representations of individualframes in video should “align” with those of still images.
Revised: Second, the scene representations of individual frames in video should “align” with those of still images.

Original: This is achieved via a Frame-Clip Consistency loss, which ensures the ﬂow of structured informationbetween images and videos.
Revised: This is achieved via a Frame-Clip Consistency loss, which ensures the ﬂow of structured information between images and videos.

Original: We explore a particular instantiation of scene structure,namely a Hand-Object Graph , consisting of hands and objects with their locationsas nodes, and physical relations of contact/no-contact as edges.
Revised: We explore a particular instantiation of scene structure, namely a Hand-Object Graph , consisting of hands and objects with their locations as nodes, and physical relations of contact/no-contact as edges.

Original: SViT shows strongperformance improvements on multiple video understanding tasks and datasets.
Revised: SViT shows strong performance improvements on multiple video understanding tasks and datasets.

Revised: For code and pretrained models, visit the project page at https://eladb3.github.io/SViT/ .

Original:  Introduction
Revised:  Introduction

Original: Semantic understanding of videos is a key challenge for machine vision and artiﬁcial intelligence.
Revised: Semantic understanding of videos is a key challenge for machine vision and artiﬁcial intelligence.

Original: Itis intuitive that video models should beneﬁt from incorporating scene structure, e g., the objects thatappear in a video, their attributes and the way they interact.
Revised: It is intuitive that video models should beneﬁt from incorporating scene structure, e.g., the objects that appear in a video, their attributes, and the way they interact.

Original: Indeed, several works [5, 25, 26, 33, 34,41, 57, 59, 62, 74, 79, 80, 82, 88] have demonstrated that incorporating structured representationsinto models improves both performance and sample efﬁciency.
Revised: Indeed, several works [5, 25, 26, 34, 35, 42, 81, 83, 89] have demonstrated that incorporating structured representations into models improves both performance and sample efﬁciency.

Original: In particular, [33, 80, 88] showed howto incorporate structured representations by utilizing static image object annotations in videos.
Revised: In particular, [34, 81, 89] showed how to incorporate structured representations by utilizing static image object annotations in videos.

Original: Recently, vision transformers (ViT) [20] have emerged as the leading model for many vision applications [4, 21, 12].
Revised: Recently, vision transformers (ViT) [20] have emerged as the leading model for many vision applications [4, 21, 12].

Original: This begs the question, how can structured representations be used to leveragevideo transformers for video tasks?
Revised: This begs the question, how can structured representations be leveraged for video tasks in a video transformer?

Original: Past works [4, 21] have proposed video transformer modelsbut these approaches were not focused on learning structured representations.
Revised: Past works [4, 21] have proposed video transformer models but these approaches were not focused on learning structured representations.

Original: A recent line ofworks [33, 80, 88] utilized objects for structured approaches.
Revised: A recent line of works [34, 81, 89] utilized objects for structured approaches.

Original: [82] leveraged the decoder with CNNsregion crop features  from SlowFast [22], and [33] proposed a new block that incorporates objectrepresentations.
Revised: [83] leveraged the decoder with CNNs region crop features [Figure] [Figure] from SlowFast [22], [34] proposed a new block that incorporates object representations, and [89] proposed a structured model for action detection.

Original: However, these approaches are limited because they use external pretrained detectorsfor the frames, which require additional processing time (also during inference), they ignore thepotential beneﬁts of the connection between the detection and the other tasks, and it is not clear howto use them with more detailed structured annotations.
Revised: However, these approaches are limited because they use external pretrained detectors for the frames, which require additional processing time (also during inference). Moreover, they ignore the potential beneﬁts of the connection between the detection and the other tasks, and it is unclear how to use them with more detailed structured annotations.

Original: Finally, [88] proposed a structured modelfor action detection that explicitly utilizes image annotations, but not in a transformer architecture.

Original: Our approach, described next, builds on these prior works and proposes an effective way to leveragescene structure in video transformers, using  static image supervision.
Revised: Our approach, described next, builds on these prior works and proposes an effective way to leverage scene structure in video transformers , using a relatively small static image supervision.

Original: Most structured transformer-based models for videos require structured annotations during training.
Revised: Most structured transformer-based models for videos require structured annotations during training.

Original: For example, frame-level object bounding boxes are considered available in [5, 7,33, 34, 58, 80]. [Figure]
Revised: For example, frame-level object bounding boxes are considered available in [5, 7, 34, 35, 59, 81].

Original: [Figure] When it comes to structured annotations of static images, there are numerous datasets with annotationssuch as boxes, visual relationships, attributes, and segmentations [10, 48, 58].
Revised: When it comes to structured annotations of static images, there are numerous datasets with annotations such as boxes, visual relationships, attributes, and segmentations [10, 49, 59].

Original: Since we cannot alwaysexpect to have annotated images that perfectly align with our downstream video task, how can theseresources be used to build better transformer-based video models?
Revised: Since we cannot always expect to have annotated images that perfectly align with our downstream video task, how can these resources be used to build better transformer-based video models?

Original: In this work, we propose suchan approach, which is particularly designed for transformer architectures and offers an effectivemechanism to leverage image structure to improve video transformers.
Revised: In this work, we propose such an approach that is speciﬁcally designed for transformer architectures and offers an effective mechanism for improving video transformers by leveraging images with structured annotations that are not necessarily associated with the video domain and can be within or outside the domain of interest.

Original: A natural ﬁrst step towards image-video knowledge sharing is to use the same transformer model toprocess both: namely, to view images as “single frame videos”, and feed them to a video transformermodel.
Revised: A natural ﬁrst step towards image-video knowledge sharing is to use the same transformer model to process both: namely, to view images as “single frame videos”, and feed them to a video transformer model.

Original: However, this still leaves two key questions: how to model structured information, and howto account for domain mismatch between images and videos.
Revised: However, this still leaves two key questions: how to model structured information, and how to account for domain mismatch between images and videos.

Original: Towards this end, we introduce twokey concepts: (i) A set of transformer “object-tokens” – additional tokens initialized from learnedembeddings (we refer to these embeddings as “object prompts”) that are meant to capture objectcentric information in both still image and video.
Revised: Towards this end, we introduce two key concepts: (i) A set of transformer “object-tokens” – additional tokens initialized from learned embeddings (we refer to these embeddings as “object prompts”) that are meant to capture objectcentric information in both still image and video.

Original: These tokens can be supervised based on the scenestructure of images, and that information will also be propagated to videos.
Revised: These tokens can be supervised based on the scene structure of images, and that information will also be propagated to videos.

Original: To formalize the imagestructure, we proposed the HAnd-Object Graph (HAOG), a simple representation of the interactionsbetween hands and objects in the scene.
Revised: To formalize the image structure, we proposed the HAnd-Object Graph (HAOG), a simple representation of the interactions between hands and objects in the scene.

Original: (ii)
Revised: (ii)

Original: A novel “Frame-Clip Consistency loss” that ensuresconsistency between the “object-tokens” when they are part of a video vs. a still image.
Revised: A novel “Frame-Clip Consistency loss” that ensures consistency between the “object-tokens” when they are part of a video vs. a still image.

Original: Moreover, aswe show in our experiments, our approach does not require any alignment between object categoriesappearing in still images and the ones appearing in the videos of the target downstream task.
Revised: Moreover, as we show in our experiments, our approach does not require any alignment between object categories appearing in still images and the ones appearing in the videos of the target downstream task.

Original: We nameour proposed approach StructureViT ( Bringing Scene Structure from Images to Vi deo via Frame-Clip
Revised: We name our proposed approach StructureViT ( Bringing Scene Structure from Images to Vi deo via Frame-Clip

Original: Consistency of Object T okens , or SViT for short).
Revised: Consistency of Object T okens , or SViT for short).

Original: See Figure 1 for an overview.
Revised: See Figure 1 for an overview.

Original: Prior work has explored several other video & image models.
Revised: Prior work has explored several other video & image models.

Original: The I3D model [13] was one of the ﬁrstsuccessful attempts to leverage ImageNet architectures in video.
Revised: The I3D model [13] was one of the ﬁrst successful attempts to leverage ImageNet architectures in video.

Original: They proposed to simply convertimage 2D classiﬁcation models into 3D ConvNets by inﬂating all the ﬁlters and pooling kernels.
Revised: They proposed to simply convert image 2D classiﬁcation models into 3D ConvNets by inﬂating all the ﬁlters and pooling kernels.

Original: Here, our focus is on learning the shared structured representations between the two domains in orderto transfer this knowledge into videos.
Revised: Here, our focus is on learning the shared structured representations between the two domains in order to transfer this knowledge into videos.

Original: Several other methods have explored simultaneous image andvideo training in the context of multi-task [6] and multi-modal [27] learning.
Revised: Several other methods have explored simultaneous image and video training in the context of multi-task [6] and multi-modal [27] learning.

Original: However, these worksdid not utilize structural information, nor did they promote consistency between image and videopredictions – which, as we show, leads to signiﬁcant gains in downstream video task performance.
Revised: However, these works did not utilize structural information, nor did they promote consistency between image and video predictions – which, as we show, leads to signiﬁcant gains in downstream video task performance.

Original: To summarize, our main contributions are as follows: (i) we propose a new method for exploitingstructured information present in images in order to improve performance of video understandingtasks; (ii) we propose a novel concept of “object tokens” - a form of learned prompts to a video &image transformer, which during training are associated to the available structure representation ofthe still images (HAOG).
Revised: To summarize, our main contributions are as follows: (i) we propose a new method for exploiting structured information present in images in order to improve performance of video understanding tasks; (ii) we propose a novel concept of “object tokens” - a form of learned prompts to a video & image transformer, which during training are associated to the available structure representation of the still images (HAOG).

Original: (iii) we introduce a new Frame-Clip Consistency loss, which promotesconsistency between image-level and video-level predictions made by a shared transformer backbone.
Revised: (iii) we introduce a new Frame-Clip Consistency loss, which promotes consistency between image-level and video-level predictions made by a shared transformer backbone.

Original: We show how this loss helps to drive performance improvements in downstream video tasks even incases when image labels are unrelated to the video task; (iv) we demonstrate improved performanceon several video understanding benchmarks, highlighting the effectiveness of the proposed approach.
Revised: We show how this loss helps to drive performance improvements in downstream video tasks even in cases when image labels are unrelated to the video task; (iv) we demonstrate improved performance on several video understanding benchmarks, highlighting the effectiveness of the proposed approach.

Original:  The SViT Approach
Revised:  The SViT Approach

Original: Our approach SViT learns structured representation that is exhibited in both static images and video.
Revised: Our approach SViT learns structured representation that is exhibited in both static images and video.

Original: We consider the setting where the main goal is to learn video understanding downstream taskswhile leveraging structured image data.
Revised: We consider the setting where the main goal is to learn video understanding downstream tasks while leveraging structured image data.

Original: In this paper, we focus on the following video tasks: actionrecognition and object state classiﬁcation & localization.
Revised: In this paper, we focus on the following video tasks: action recognition, spatio-temporal action detection and object state classiﬁcation & localization.

Original: In training, we assume that we have accessto task-speciﬁc video labels and structured scene annotations for images, Hand-Object Graph s.
Revised: In training, we assume that we have access to task-speciﬁc video labels and structured scene annotations for images, Hand-Object Graph s.

Original: Based on the structured representations obtained by using the object tokens and regularized by the Frame-Clip Consistency loss , the inference is performed only for the downstream video task withoutexplicitly using the structured image annotations.
Revised: Based on the structured representations obtained by using the object tokens and regularized by the Frame-Clip Consistency loss , the inference is performed only for the downstream video task without explicitly using the structured image annotations.

Original: Further, we begin by describing the Hand-Object Graph (HAOG) (Section 2.1), then introduce our
Revised: Further, we begin by describing the Hand-Object Graph (HAOG) (Section 2.1), then introduce our

Original: SViT model (Section 2.2), and the Frame-Clip Consistency loss function (Section 2.3).
Revised: SViT model (Section 2.2), and the Frame-Clip Consistency loss function (Section 2.3).

Original: Our methodis schematically illustrated in Figure 2.
Revised: Our method is schematically illustrated in Figure 2.

Original: 2.1 The Hand-Object Graph
Revised: 2.1 The Hand-Object Graph

Original: As mentioned above, our motivation is to bring scene structure from still images into video.
Revised: As mentioned above, our motivation is to bring scene structure from still images into video.

Original: Inorder to achieve this, one component of this work is the construction of a semantic representationof the interactions between the hands and objects in the scene.
Revised: In order to achieve this, one component of this work is the construction of a semantic representation of the interactions between the hands and objects in the scene.

Original: Speciﬁcally, we propose to usea graph-structure we call Hand-Object Graph (HAOG), see Figure 1.
Revised: Speciﬁcally, we propose to use a graph-structure we call Hand-Object Graph (HAOG), see Figure 1.

Original: The nodes of the HAOGrepresent two hands and two objects with their locations, whereas the edges correspond to physicalproperties such as contact/no contact.
Revised: The nodes of the HAOG represent two hands and two objects with their locations, whereas the edges correspond to physical properties such as contact/no contact.

Original: Formally, an HAOG is a tuple ( O, E ) deﬁned as follows:
Revised: Formally, an HAOG is a tuple ( O, E ) deﬁned as follows:

Original: Nodes O : A set O of n objects.
Revised: Nodes O : A set O of n objects.

Original: Human-object interaction scenes, which we focus on in this work,usually consist of up to two hands and objects.
Revised: Human-object interaction scenes, which we focus on in this work, usually consist of up to two hands and objects.

Original: Thus our node categories are “hand” and “object”;they may also contain attributes (e.g., spatial coordinates).
Revised: Thus our node categories are “hand” and “object”; they may also contain attributes (e.g., spatial coordinates).

Original: We assume that each image comes withthe bounding boxes describing the locations of the two hands, and the objects they interact with.
Revised: We assume that each image comes with the bounding boxes describing the locations of the two hands, and the objects they interact with.

Original: Thebounding boxes for the left and right hand are denoted as b 1 , b 2 2 R 4 , respectively, while for the twocorresponding objects interacting with the hands are denoted as b 3 , b 4 .
Revised: The bounding boxes for the left and right hand are denoted as b 1 , b 2 2 R 4 , respectively, while for the two corresponding objects interacting with the hands are denoted as b 3 , b 4 .

Original: We also assume access tofour binary variables e 1 , . .
Revised: We also assume access to four binary variables e 1 , . .

Original: , e 4 2 { 0 , 1 } that indicate whether the corresponding hands or objects infact exist in the image (e.g., if e 1 = 

Revised: , e 4 2 { 0 , 1 } that indicate whether the corresponding hands or objects in fact exist in the image (e.g., if e 1 =

Original: the left hand is not in the image, and b 1 should be ignored).
Revised: the left hand is not in the image, and b 1 should be ignored).

Original: Edges E : The edges are represented as labeled directed edges between hand and object nodes.
Revised: Edges E : The edges are represented as labeled directed edges between hand and object nodes.

Original: Edgevocabulary describes the physical properties of the scene.
Revised: Edge vocabulary describes the physical properties of the scene.

Original: In our current modeling, each edge ischaracterized by a physical property of “contact” or “no-contact”.
Revised: In our current modeling, each edge is characterized by a physical property of “contact” or “no-contact”.

Original: We denote two binary variablesc 1 , c 2 2 { 0 , 1 } that specify if each of the hands is in direct contact with the corresponding object (theleft hand b 1 can only interact with the object b 3 , similar for the right hand).
Revised: We denote two binary variables c 1 , c 2 2 { 0 , 1 } that specify if each of the hands is in direct contact with the corresponding object (the left hand b 1 can only interact with the object b 3 , similar for the right hand).

Original: In principle, we can alsoinclude properties such as distances between objects, 3D hand poses, etc., where available.
Revised: In principle, we can also include properties such as distances between objects, 3D hand poses, etc., where available.

Original: We note that images annotated consistent with our deﬁned HAOG format can be found in severaldatasets [31, 58, 71], and we leverage these in our experiments.
Revised: We note that images annotated consistent with our deﬁned HAOG format can be found in several datasets [31, 59, 72], and we leverage these in our experiments.

Original: 2.2 The SViT Transformer Model
Revised: 2.2 The SViT Transformer Model

Original: We next present the SViT model: a video transformer that employs object tokens as a means to capturestructure in both images and videos.
Revised: We next present the SViT model: a video transformer that employs object tokens as a means to capture structure in both images and videos.

Original: We begin by reviewing the video transformer architecture, whichour model extends.
Revised: We begin by reviewing the video transformer architecture, which [Figure] HAOGloss [Figure]

Revised: ObjectToken

Revised: PatchToken our model extends.

Original: Next, we explain how we process both images and videos.
Revised: Next, we explain how we process both images and videos.

Original: Finally, we describeobject tokens and how they are used in videos and still images.
Revised: Finally, we describe object tokens and how they are used in videos and still images.

Original: Shared Video & Image Transformer .
Revised: Shared Video & Image Transformer .

Original: Video transformers [4, 11, 21, 33] extend the Vision Transformer model to the temporal domain.
Revised: Video transformers [4, 11, 21, 34] extend the Vision Transformer model to the temporal domain.

Original: Similar to vision transformers, the input is ﬁrst “patchiﬁed” butwith temporally extended 3-D convolution (instead of 2-D) producing a down-sampled tensor X ofsize T ⇥ H ⇥ W ⇥ d .
Revised: Similar to vision transformers, the input is ﬁrst “patchiﬁed” but with temporally extended 3-D convolution (instead of 2-D) producing a down-sampled tensor X of size T ⇥ H ⇥ W ⇥ d .

Original: We refer to this as “patch tokens”.
Revised: We refer to this as “patch tokens”.

Original: Then, spatio-temporal position embeddingsare added for providing location and time information.
Revised: Then, spatio-temporal position embeddings are added for providing location and time information.

Original: Next, multiple stacked self-attention blocksare applied repeatedly on the down-sampled tensor X to produce the ﬁnal vector representation usingmean pooling over the patch tokens.
Revised: Next, multiple stacked self-attention blocks are applied repeatedly on the down-sampled tensor X to produce the ﬁnal vector representation using mean pooling over the patch tokens.

Original: In our approach, we want to be able to process batches of images or videos.
Revised: In our approach, we want to be able to process batches of images or videos.

Original: A key desideratum inthis context is to be able to input both videos and still-images into the same model.
Revised: A key desideratum in this context is to be able to input both videos and still-images into the same model.

Original: Towards this end,our ﬁrst change is to rely only on 2-D convolutions in the ﬁrst transformer step instead of 3-D. Thisallows single images to be used as inputs without padding, and does not hurt performance in practice.[Figure]
Revised: Towards this end, our ﬁrst change is to rely only on 2-D convolutions in the ﬁrst transformer step instead of 3-D. This allows single images to be used as inputs without padding, and does not hurt performance in practice.

Original: Object tokens .
Revised: Object tokens .

Original: Our goal is to learn a shared structured representation between the video and theimage domains.
Revised: Our goal is to learn a shared structured representation between the video and the image domains.

Original: We do this by adding transformer tokens to represent objects.
Revised: We do this by adding transformer tokens to represent objects.

Original: These tokens arefunctionally similar to the patch tokens, with two exceptions.
Revised: These tokens are functionally similar to the patch tokens, with two exceptions.

Original: First, they are initialized with additionaltrained embeddings, which we referred to as “object prompts”.
Revised: First, they are initialized with additional trained embeddings, which we referred to as “object prompts”.

Original: We consider the “object prompts” asa bridge between the images and the videos since they are being used in prediction tasks across bothdomains.
Revised: We consider the “object prompts” as a bridge between the images and the videos since they are being used in prediction tasks across both domains.

Original: Second, they are used to predict the structured representations when those are available(here, we use the Hand-Object Graphs structure for still images).
Revised: Second, they are used to predict the structured representations when those are available (here, we use the Hand-Object Graphs structure for still images).

Original: An object token is used to predictproperties of the corresponding node in the HAOG, and the concatenation of two object tokens isused to predict edge properties.
Revised: An object token is used to predict properties of the corresponding node in the HAOG, and the concatenation of two object tokens is used to predict edge properties.

Original: Next, we describe the model more formally.
Revised: Next, we describe the model more formally.

Original: Let n be the maximum number of modeled objects perframe (or still image).
Revised: Let n be the maximum number of modeled objects per frame (or still image).

Original: We have n tokens for each frame, and thus a total of T ⇥ n tokens.
Revised: We have n tokens for each frame, and thus a total of T ⇥ n tokens.

Original: The tokenfor object i at frame t is initialized with the vector o i + r t where o i 2 R d is a learned object promptand r t 2 R d is a learned temporal positional embedding (the same one used for initializing the patchtokens).
Revised: The token for object i at frame t is initialized with the vector o i + r t where o i 2 R d is a learned object prompt and r t 2 R d is a learned temporal positional embedding (the same one used for initializing the patch tokens).

Original: With these new tokens, we have T ⇥ ( H ⇥ W + n ) tokens (i.e, vectors in R d ), and thesetogether will go through the standard self-attention layers.
Revised: With these new tokens, we have T ⇥ ( H ⇥ W + n ) tokens (i.e, vectors in R d ), and these together will go through the standard self-attention layers.

Original: We denote the operator that outputs the ﬁnal representation of the object tokens by F O , where fora video V we have F O ( V ) 2 R T ⇥ n ⇥ d and for an image I we have F O ( I ) 2 R n ⇥ d .
Revised: We denote the operator that outputs the ﬁnal representation of the object tokens by F O , where for a video V we have F O ( V ) 2 R T ⇥ n ⇥ d and for an image I we have F O ( I ) 2 R n ⇥ d .

Original: The F O ( I )representation is used to predict structured representations for single images.
Revised: The F O ( I ) representation is used to predict structured representations for single images.

Original: We also include a lossthat makes single frame representations align with those of clip representations (see Section 2.3).
Revised: We also include a loss that makes single frame representations align with those of clip representations (see Section 2.3).

Original: Theoperator that outputs the ﬁnal representation vector used for the video downstream task is denoted by
Revised: The operator that outputs the ﬁnal representation vector used for the video downstream task is denoted by

Original: F CLS , where for video V we have F CLS ( V ) 2 R d .
Revised: F CLS , where for video V we have F CLS ( V ) 2 R d .

Original: Finally, our method can be used on top of the most common video transformers (MViT [21],
Revised: Finally, our method can be used on top of the most common video transformers (MViT [21]

Original: TimeSformer [11], Mformer [65], Video Swin [55]). For our experiments, we use the MViTv2 [52]model because it performs well empirically.
Revised: TimeSformer model it performs well empirically

Original: 2.3 Losses and Training
Revised: 2.3 Losses and Training

Original: During training we have a set of images annotated with HAOGs and videos with downstream tasklabels.
Revised: During training we have a set of images annotated with HAOGs and videos with downstream task labels.

Original: We use these as inputs to our model (Section 2.2) and optimize the losses described below.
Revised: We use these as inputs to our model (Section 2.2) and optimize the losses described below.

Original: Video loss .
Revised: Video loss .

Original: Each training video V has a corresponding ground-truth category Y 2 { 1 , . .
Revised: Each training video V has a corresponding ground-truth category Y 2 { 1 , . .

Original: . , K } .
Revised: . , K } .

Original: As noted above, we have a vector representation for the entire video denoted by F CLS ( V ) .
Revised: As noted above, we have a vector representation for the entire video denoted by F CLS ( V ) .

Original: Thus,we simply use a neural network to predict logits for Y .
Revised: Thus, we simply use a neural network to predict logits for Y .

Original: Speciﬁcally, our predicted logits are:ˆ Y =
Revised: Speciﬁcally, our predicted logits are: ˆ Y =

Original: FC ( F CLS ( V ))
Revised: FC ( F CLS ( V ))

Original: 2 R K , where FC is a trainable fully connected network.
Revised: 2 R K , where FC is a trainable fully connected network.

Original: We then consider astandard cross-entropy between the predicted logits ˆ Y and true labels Y . 
Revised: We then consider a standard cross-entropy between the predicted logits ˆ Y and true labels Y . [Equation]

Original: [Equation]
Revised: [Equation]

Original: HAOG loss .
Revised: HAOG loss .

Original: The transformer outputs a set of n object tokens for each image.
Revised: The transformer outputs a set of n object tokens for each image.

Original: Since we havesupervision for these objects, we predict structured representations from the tokens, and introduce aloss to optimize these predictions.
Revised: Since we have supervision for these objects, we predict structured representations from the tokens, and introduce a loss to optimize these predictions.

Original: Recall our notation from Section 2.1.
Revised: Recall our notation from Section 2.1.

Original: We let the transformer usen = 4 object tokens which correspond to the two hands (bounding boxes b 1 , b 2 ) and two manipulated objects (bounding boxes b 3 , b 4 ).
Revised: We let the transformer use n = 4 object tokens which correspond to the two hands (bounding boxes b 1 , b 2 ) and two manipulated objects (bounding boxes b 3 , b 4 ).

Original: The j th token is used to predict a corresponding bounding box b j ,as well as the existence variable e j .
Revised: The j th token is used to predict a corresponding bounding box b j , as well as the existence variable e j .

Original: Formally, we let:
Revised: Formally, we let:

Original: [Equation]
Revised: [Equation]

Original: where FC bb , FC e are fully connected networks with four and one output respectively, F jO ( I ) is thej th object token, and I is the input image.
Revised: where FC bb , FC e are fully connected networks with four and one output respectively, F jO ( I ) is the j th object token, and I is the input image.

Original: Formally, this deﬁne the loss of the nodes as following:
Revised: Formally, this deﬁne the loss of the nodes as following:

Original: [Equation]
Revised: [Equation]

Original: where L1 is the standard L1 distance, and GIoU is used as in [68].
Revised: where L1 is the standard L1 distance, and GIoU is used as in [69].

Original: Furthermore, we predict thecontact variable from the concatenation of the corresponding hand and object tokens.
Revised: Furthermore, we predict the contact variable from the concatenation of the corresponding hand and object tokens.

Original: Namely forj 2 { 1 , 2 } :
Revised: Namely for j 2 { 1 , 2 } :

Original: [Equation]
Revised: [Equation]

Original: where FC c is a fully connected with one output.
Revised: where FC c is a fully connected with one output.

Original: Thus, we deﬁne the loss of the edges as follows:
Revised: Thus, we deﬁne the loss of the edges as follows:

Original: [Equation]
Revised: [Equation]

Original: Finally, to obtain the loss, we compare the predictions and ground-truth labels as follows:
Revised: Finally, to obtain the loss, we compare the predictions and ground-truth labels as follows:

Original: [Equation]
Revised: [Equation]

Original: Frame-Clip Consistency Loss .
Revised: Frame-Clip Consistency Loss .

Original: Since we have different losses for images and video, the modelcould ﬁnd a way to minimize the image loss in a way that only applies to the images and does nottransfer to video.
Revised: Since we have different losses for images and video, the model could ﬁnd a way to minimize the image loss in a way that only applies to the images and does not transfer to video.

Original: 1 To avoid this, we need to make sure that the video representation contains thesame type of information as the still images, and in particular the structured information carried bythe object tokens.
Revised: 1 To avoid this, we need to make sure that the video representation contains the same type of information as the still images, and in particular the structured information carried by the object tokens.

Original: To achieve this, we explicitly enforce the object tokens to be consistent acrossstill images (frames) and videos (clips) using a frame-clip Consistency loss.
Revised: To achieve this, we explicitly enforce the object tokens to be consistent across still images (frames) and videos (clips) using a frame-clip Consistency loss.

Original: To calculate the loss,we process each video twice: once as a clip, and once as a batch of T frames.
Revised: To calculate the loss, we process each video twice: once as a clip, and once as a batch of T frames.

Original: Namely, a clip Vconsisting of T frames I 1 , ..., I T , will be processed once as F O ( V ) and once as a list of images( F O ( I 1 ) , ..., F O ( I T )) .
Revised: Namely, a clip V consisting of T frames I 1 , ..., I T , will be processed once as F O ( V ) and once as a list of images ( F O ( I 1 ) , ..., F O ( I T )) .

Original: This results in two groups of T ⇥ n object tokens, the ﬁrst are the clip objecttokens , denoted as, O clip :=
Revised: This results in two groups of T ⇥ n object tokens, the ﬁrst are the clip object tokens , denoted as, O clip :=

Original: F O ( V ) 2 R T ⇥ n ⇥ d , the second are the frame object tokens , denotedas O frames :=
Revised: F O ( V ) 2 R T ⇥ n ⇥ d , the second are the frame object tokens , denoted as O frames :=

Original: [ F O ( v 1 ) , ..., F O ( v T )] 2 R T ⇥ n ⇥ d .
Revised: [ F O ( v 1 ) , ..., F O ( v T )] 2 R T ⇥ n ⇥ d .

Original: Since both groups originate from the same input,each clip object token has a single corresponding frame object token.
Revised: Since both groups originate from the same input, each clip object token has a single corresponding frame object token.

Original: Namely, there is an alignmentbetween the clip object tokens and the frame object tokens.
Revised: Namely, there is an alignment between the clip object tokens and the frame object tokens.

Original: Following the intuition that each clipobject token should contain the information from its corresponding frame object token and potentiallymore (the temporal context), we minimize the L 1 distance between each clip object token and frameobject token, after applying a fully-connected layer to each clip object token.
Revised: Following the intuition that each clip object token should contain the information from its corresponding frame object token and potentially more (the temporal context), we minimize the L 1 distance between each clip object token and frame object token.

Original: Namely:
Revised: Namely:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Overall loss .
Revised: Overall loss .

Original: The total loss consists of L Con , L HAOG and L V id , where L V id is the loss of the main190 task.
Revised: The total loss consists of L Con , L HAOG and L V id , where L V id is the loss of the main task.

Original: Each of the three terms in the loss is multiplied by a hyper-parameter (  Con ,  HAOG ,  V id ),and the total loss is the weighted combination of the three terms:
Revised: Each of the three terms in the loss is multiplied by a hyper-parameter (  Con ,  HAOG ,  V id ), and the total loss is the weighted combination of the three terms:

Original: [Equation]
Revised: [Equation]

Original:  Experiments and Results
Revised:  Experiments and Results

Original: We begin by describing the datasets, implementation details, and baselines and variants.
Revised: We begin by describing the datasets, implementation details, and baselines and variants.

Original: We thenevaluate our approach on several benchmarks for video understanding.
Revised: We then evaluate our approach on several benchmarks.

Original: Speciﬁcally, we consider thefollowing tasks: Compositional Action Recognition (Section 3.4), Object State Change Classiﬁcation& Localization (Section 3.5),and Action Recognition (Section 3.6).
Revised: Speciﬁcally, we consider the following tasks: Compositional Action Recognition (Section 3.4), Object State Change Classiﬁcation & Localization (Section 3.5), Action Recognition (Section 3.6), and Spatio-Temporal Action Detection (Section 3.7).

Original: 3.1 Datasets
Revised: 3.1 Datasets

Original: We ﬁrst describe the datasets used for the downstream video tasks, and those used as “auxiliary”datasets with annotated images.
Revised: We ﬁrst describe the datasets used for the downstream video tasks, and those used as “auxiliary” datasets with annotated images.

Original: We use the following video datasets: (1) Something-Somethingv2 (SSv2) [30] is a dataset containing 174 action categories of common human-object interactions.(2) SomethingElse [58] which exploits the compositional structure of SSv2, where a combinationof a verb and a noun deﬁnes an action.
Revised: We use the following video datasets: (1) Something-Something v2 (SSv2) [30] is a dataset containing 174 action categories of common human-object interactions. (2) SomethingElse [59] which exploits the compositional structure of SSv2, where a combination of a verb and a noun deﬁnes an action.

Original: We follow the ofﬁcial compositional split from [58], whichassumes the set of noun-verb pairs available for training is disjoint from the set given at test time.(3) Ego4D [31] is a new large-scale dataset of more than 3,670 hours of video data, capturing thedaily-life scenarios of more than 900 unique individuals from nine different countries around theworld.
Revised: We follow the ofﬁcial compositional split from [59], which assumes the set of noun-verb pairs available for training is disjoint from the set given at test time. (3) Ego4D [31] is a new large-scale dataset of more than 3,670 hours of video data, capturing the daily-life scenarios of more than 900 unique individuals from nine different countries around the world.

Original: (4) Diving48 [53] contains 48 ﬁne-grained categories of diving activities.
Revised: (4) Diving48 [54] contains 48 ﬁne-grained categories of diving activities.

Revised: (5) Atomic Visual

Revised: Actions (AVA) [32] is a benchmark for human action detection.

Revised: We report Mean Average Precision (mAP) on AVA-V2.2.

Original: For “auxiliary”image datasets we use the Ego4D, and the 100 Days of Hands (100DOH) [71] datasets.
Revised: For “auxiliary” image datasets we use the Ego4D, and the 100 Days of Hands (100DOH) [72] datasets.

Original: We collected79,921 annotated images from 100DOH, and 57,213 annotated images from Ego4D. 2 The collectedobject boxes from SSv2, 100DOH, and Ego4D are purely human annotated.
Revised: We collected 79,921 annotated images from 100DOH, and 57, annotated images from Ego4D. 2 The AVA or SSv2 datasets are also used as “auxiliary” images in some experiments.

Original: In SSv2 and Ego4D,contact relations between the object and hand are not annotated, so we assign the closest object to thehand for each hand.

Original: The contact relations for 100DOH are available in the dataset.
Revised: For more details, see Section D.1 in the supplementary.

Original: For the image datasets, the image annotations are based on frames selected from videos, but these areoften sparsely selected from the videos, so it is natural to treat them as annotated still images.
Revised: For the image datasets, the image annotations are based on frames selected from videos, but these are often sparsely selected from the videos, so it is natural to treat them as annotated still images.

Original: 3.2 Implementation Details
Revised: 3.2 Implementation Details

Original: SViT is implemented in PyTorch, we include the codein the supplementary and will release it uponacceptance.
Revised: SViT is implemented in PyTorch, and the code will be released in our project page.

Original: Our training recipes and code are based on the MViTv2 model, and were taken fromhttps://github.com/facebookresearch/mvit .
Revised: Our training recipes and code are based on the MViTv2 model, and were taken from https://github.com/ facebookresearch/mvit .

Original: We pretrain the SViT model on the K400 [44]video dataset with our auxiliary image datasets.
Revised: We pretrain the SViT model on the K400 [45] video dataset with our auxiliary image datasets.

Original: Then, we ﬁnetune on the target video understandingtask (detailed in Section 3.1) together with the auxiliary image datasets and the SViT loss.
Revised: Then, we ﬁnetune on the target video understanding task (detailed in Section 3.1) together with the auxiliary image datasets and the SViT loss.

Original: Eachtraining batch contains 64 images and 64 videos in order to minimize the overall loss in Equation 8.
Revised: Each training batch contains 64 images and 64 videos in order to minimize the overall loss in Equation 8.

Revised: For more details about datasets and evaluation see Section C of Supplementary.

Original: 3.3 Baselines and SViT variants
Revised: 3.3 Baselines and SViT variants

Original: Baselines .
Revised: Baselines .

Original: In the experiments, we compare SViT to several models from prior work reported for thecorresponding datasets.
Revised: In the experiments, we compare SViT to several models from prior work reported for the corresponding datasets.

Original: These include the following approaches: I3D [13], SlowFast [22], and thestate-of-the-art transformers – Mformer [65] and MViTv2 [52].
Revised: These include the following approaches: I3D [13], SlowFast [22], and the state-of-the-art transformers – Mformer [66] and MViTv2 [53].

Original: In addition, we provide the MViTv2multi-task (MViTv2 MT) baseline for comparing a naive multi-tasking approach to our model.
Revised: In addition, we provide the MViTv multi-task (MViTv2 MT) baseline for comparing a naive multi-tasking approach to our model.

Original: This isa version of MViTv2 which performs video classiﬁcation and HAOG prediction from images.
Revised: This is a version of MViTv2 which performs video classiﬁcation and HAOG prediction from images.

Original: The
Revised: The

Original: MViTv2 model is enhanced with two heads: the ﬁrst head is composed of an MLP that predicts thevideo labels given a ﬁnal representation vector, the second head consists of MLPs that predict the
Revised: MViTv2 model is enhanced with two heads: the ﬁrst head is composed of an MLP that predicts the video labels given a ﬁnal representation vector, the second head consists of MLPs that predict the

Original: HAOG information (i.e., hands and object boxes, and contact information).
Revised: HAOG information (i.e., hands and object boxes, and contact information).

Original: SViT Variants .
Revised: SViT Variants .

Original: As mentioned above, we use several sources of auxiliary image datasets.
Revised: As mentioned above, we use several sources of auxiliary image datasets.

Original: In somecases, we have auxiliary images that are frames from video understanding datasets.
Revised: In some cases, we have auxiliary images that are frames from video understanding datasets.

Original: For example,in Ego4D, we have both videos and frames annotated from some of these videos.
Revised: For example, in Ego4D, we have both videos and frames annotated from some of these videos.

Original: In this sense, theauxiliary data is “in-domain” for some of the video tasks.
Revised: In this sense, the [Table] auxiliary data is “in-domain” for some of the video tasks.

Original: In order to evaluate the importance of thiseffect, we explore “Different Domain” ( SViT-DD ) training, which avoids using images from a givenvideo task as auxiliary during ﬁnetuning and pretraining.
Revised: In order to evaluate the importance of this effect, we explore “Different Domain” ( SViT-DD ) training, which avoids using images from a given video task as auxiliary during ﬁnetuning and pretraining.

Original: Similarly, “In Domain” ( SViT-ID ) refers tothe case where we use only images from the video task during ﬁnetuning, but not necessarily duringpretraining .
Revised: Similarly, “In Domain” ( SViT-ID ) refers to the case where we use only images from the video task during ﬁnetuning, but not necessarily during pretraining .

Original: We consider another variant that does not use the auxiliary images during ﬁnetuning atall (but does use them during pre-training), and refer to it as SViT-SFT .
Revised: We consider another variant that does not use the auxiliary images during ﬁnetuning at all (but does use them during pre-training), and refer to it as SViT-SFT .

Original: Finally, to explore the effectof data-size in the In Domain setting, we evaluate a version SViT ⇥ X% that only uses X% of thein-domain ﬁnetuning data.
Revised: Finally, to explore the effect of data-size in the In Domain setting, we evaluate a version SViT ⇥ X% that only uses X% of the in-domain ﬁnetuning data.

Original: For MViTv2 MT, the auxiliary images correspond to the “In Domain”setting (i.e., same setting as for SViT-ID).
Revised: For MViTv2 MT, the auxiliary images correspond to the “In Domain” setting (i.e., same setting as for SViT-ID).

Original: Auxiliary Images datasets .
Revised: Auxiliary Images datasets .

Original: For SViT-DD, we generally use 100DOH and Ego4D for pretraining andﬁnetuning, while for the Ego4D tasks, we only use 100DOH.
Revised: For SViT-DD, we generally use 100DOH and Ego4D for pretraining and ﬁnetuning, while for the Ego4D tasks, we only use 100DOH.

Original: SViT-ID uses 100DOH and Ego4D forpretraining and in-domain downstream frames for ﬁnetuning.
Revised: SViT-ID uses 100DOH and Ego4D for pretraining and in-domain downstream frames for ﬁnetuning.

Original: The SViT-SFT pretraining setting issimilar to that of SViT-DD.
Revised: The SViT-SFT pretraining setting is similar to that of SViT-DD.

Original: SViT ⇥ X% setting is similar to that of SViT-ID.
Revised: SViT ⇥ X% setting is similar to that of SViT-ID.

Original: [Table] [Table]
Revised: For more details, see Section D.1 in the supplementary.

Original: 3.4 Compositional & Few-Shot Action Recognition
Revised: 3.4 Compositional & Few-Shot Action Recognition

Original: Several video datasets deﬁne an action as a combination of a verb and a noun.
Revised: Several video datasets deﬁne an action as a combination of a verb and a noun.

Original: One challenge then is torecognize combinations that were not seen during training.
Revised: One challenge then is to recognize combinations that were not seen during training.

Original: This “compositional” setting was exploredin the “SomethingElse” dataset [58], where verb-noun combinations in the test data do not occur inthe training data.
Revised: This “compositional” setting was explored in the “SomethingElse” dataset [59], where verb-noun combinations in the test data do not occur in the training data.

Original: The split contains 174 classes with 54,919/54,876 videos for training/validation.
Revised: The split contains 174 classes with 54,919/54,876 videos for training/validation.

Original: We also evaluate on the few-shot compositional action recognition task in [58] (See supplementary).
Revised: We also evaluate on the few-shot compositional action recognition task in [59] (See supplementary).

Original: Table 1 reports the results for these two tasks.
Revised: Table 1 reports the results for these two tasks.

Original: All of our variants outperform all prior works and the
Revised: All of our variants outperform all prior works and the

Original: MViTv2 MT on the Compositional and Few-shot task.
Revised: MViTv2 MT on the Compositional and Few-shot task.

Original: From the table, we can see that using 2% ofthe in-domain data as auxiliary images (SViT ⇥ 2%), is almost comparable to using all the availablein-domain images (SViT-ID).
Revised: From the table, we can see that using 2% of the in-domain data as auxiliary images (SViT ⇥ 2%), is almost comparable to using all the available in-domain images (SViT-ID).

Original: Furthermore, the results are comparable to SViT-DD (0.7 difference)when trained on auxiliary images from different domains.
Revised: Furthermore, the results are comparable to SViT-DD (0.7 difference) when trained on auxiliary images from different domains.

Original: In this way, our approach optimizes theuse of structural information across domains.
Revised: In this way, our approach optimizes the use of structural information across domains.

Original: Last, SViT-DD outperforms SViT-SFT when using thesame data, demonstrating the value of ﬁnetuning video and image together.
Revised: Last, SViT-DD outperforms SViT-SFT when using the same data, demonstrating the value of ﬁnetuning video and image together.

Original: 3.5 Object State Change Classiﬁcation and Localization
Revised: 3.5 Object State Change Classiﬁcation and Localization

Original: Hands and objects are key elements in human activity.
Revised: Hands and objects are key elements in human activity.

Original: Two tasks related to hand-object interactionhave recently been introduced in the Ego4D [31] dataset.
Revised: Two tasks related to hand-object interaction have recently been introduced in the Ego4D [31] dataset.

Original: The ﬁrst is temporal localization, whichis deﬁned as ﬁnding key frames in a video clip that indicate a change in object state.
Revised: The ﬁrst is temporal localization, which is deﬁned as ﬁnding key frames in a video clip that indicate a change in object state.

Original: The second isobject state change classiﬁcation, which indicates whether an object state has changed or not.
Revised: The second is object state change classiﬁcation, which indicates whether an object state has changed or not.

Original: Table 2a reports results on the temporal localization and object state change classiﬁcation tasks onthe Ego4D dataset.
Revised: Table 2a reports results on the temporal localization and object state change classiﬁcation tasks on the Ego4D dataset.

Original: We observe that SViT-ID and SViT-DD are very close, and that both SViT-ID and SViT-DD are better than SViT-SFT.
Revised: We observe that SViT-ID, SViT ⇥ 2% and SViT-DD perform better than SViT-SFT and are very close.

Original: These results are consistent with those presented in Section 3.4,which indicates that our method successfully leverages structural information across domains, andthat a joint image and video ﬁnetuning is beneﬁcial. 
Revised: These results are consistent with those presented in Section 3.4, which indicates that our method successfully leverages structural information across domains, and that a joint image and video ﬁnetuning is beneﬁcial. [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Revised: not use any additional structured annotation with † .

Original: 3.6 Action Recognition
Revised: 3.6 Action Recognition

Original: Table 2b and Table 2c report the results for standard action recognition task on the SSv2 and Diving48datasets respectively.
Revised: Table 2b and Table 2c report the results for standard action recognition task on the SSv2 and Diving48 datasets respectively.

Original: It can be seen that in Diving48 dataset, our method improves over the MViTv2 baseline by +6.7%, outperforming all the methods.
Revised: It can be seen that in Diving48 dataset, our method improves over the MViT-v2 baseline by 6.7%, outperforming all the methods.

Original: This demonstrates that our method hassome ability to generalize the learned structural information of hand-object interactions even toﬁne-grained “human-motion” tasks such as Diving48.
Revised: This demonstrates that our method has some ability to generalize the learned structural information of hand-object interactions even to ﬁne-grained “human-motion” tasks such as Diving48.

Original: We also note that Diving48 has no structuredannotations available, thus SViT-ID  can not be provided.
Revised: We also note that Diving48 has no structured annotations available, thus SViT-ID (and SViT ⇥ 2% ) can not be provided.

Original: On SSv2,we overall observe a smallerimprovement. SViT-ID improves the baseline by 0.8% while SViT-DD improves by 0.1%.
Revised: On SSv2, SViT-ID improves the baseline by 1.5%, SViT ⇥ 2% improves by 1.3%, and SViT-DD improves by 1.1%.

Original: As weobserved signiﬁcant improvements on the more challenging SomethingElse setup, we hypothesizethat the ability to generalize to unseen combinations of objects and hands exhibited by  SViT is lessneeded on SSv2, hence leading to relatively smaller improvements.
Revised: As we observed signiﬁcant improvements on the more challenging SomethingElse setup, we hypothesize that SomethingElse is more likely to beneﬁt from the ability to generalize to unseen combinations of objects and hands exhibited by SViT. We provide visualizations of the HAOGs predicted by SViT in Section B of the supplementary. 3.7 Spatio-temporal Action Detection [Table]

Original: 3.7 Ablations
Revised: Table 3 report the results for spatio-temporal action detection on the AVA dataset. In the literature, the action detection task on AVA is formulated as a two stage prediction procedure. The ﬁrst step is the detection of bounding boxes, which are obtained through an off-the-shelf pretrained person detector. The second step involves predicting the action being performed at each detected bounding box.

Revised: The performance is benchmarked on the end result of these steps and is measured with the Mean Average Precision (MAP) metric.

Revised: Typically, for fair comparison, the detected person boxes are kept identical across approaches and hence, the ﬁnal performance depends directly on the ability of the approach to utilize the video and box information.

Revised: We observe that all SViT variants improve over the MViT-v2 baseline.

Revised: Speciﬁcally, SViT-ID improves the

Revised: MViT-v2 baseline by 1.7, SViT-DD improves by 1.5, and SViT-SFT improves by 1.0.

Revised: 3.8 Ablations

Original: We perform a comprehensive ablation study on the compositional action recognition task [58] on the
Revised: We perform a comprehensive ablation study on the compositional action recognition task [59] on the

Original: SomethingElse dataset to measure the contribution of the different SViT components (Table 3).
Revised: SomethingElse dataset to measure the contribution of the different SViT components (Table 4).

Original: Components of the SViT model .
Revised: Components of the SViT model .

Original: In Table 3a, we ablate our model to show the signiﬁcance of theindividual components.
Revised: In Table 4a, we ablate our model to show the signiﬁcance of the individual components.

Original: First, we train a standard MViTv2 in a multi-task setting (MViTv2 MT),where we use videos for video action recognition, and images for HAOG prediction.
Revised: First, we train a standard MViTv2 in a multi-task setting (MViTv2 MT), where we use videos for video action recognition, and images for HAOG prediction.

Original: There is a smallimprovement (+0.5) over the baseline.
Revised: There is a small improvement (+0.5) over the baseline.

Original: We then take the MViTv2 and add the object tokens andcorresponding SViT loss; this leads to a more signiﬁcant improvement (+1.7).
Revised: We then take the MViTv2 and add the object tokens and corresponding SViT loss; this leads to a more signiﬁcant improvement (+1.7).

Original: Last, when addingboth the consistency loss and the object tokens, we observe the largest improvement (+2.5). 
Revised: Last, when adding both the consistency loss and the object tokens, we observe the largest improvement (+2.5). [Table]

Original: The amount of in-domain data .
Revised: The amount of in-domain data .

Original: In Table 3b, we demonstrate the effect of the amount of in-domainimage data used during training.
Revised: In Table 4b [Table] we demonstrate the effect of the amount of in-domain image data used during training.

Original: Speciﬁcally, since “SomethingElse” includes HAOG annotationsfor each frame, we train our method with different amount of images ( 2% , 25% , 50% , and 100% )from the “SomthingElse” dataset.
Revised: Speciﬁcally, since “SomethingElse” includes HAOG annotations for each frame [Table] we train our method with different amount of images ( 2% , 25% , 50% , and 100% ) from the “SomthingElse” dataset.

Original: It can be seen that our approach takes advantage of the structuredinformation within a small amount of data and, as a result, has enhanced sample-efﬁciency.
Revised: It can be seen that our approach takes advantage of the structured information within a small amount of data and, as a result, has enhanced sample-efﬁciency.

Original: HAOG attributes . 
Revised: Robustness to multiple domains . In Table 4c We performed experiments on SomethingElse (groundtruth annotations available) where hands or objects were not visible in order to verify the usefulness of our approach: (i) After ﬁltering the videos containing (annotated) hands, we tested our model and the MViTv2 model on the ﬁltered hand split. MViTv2 achieved an accuracy of 64.6 while our model achieved 66.5. This implies an improvement of +1.9.

Original: We use the object tokens to predict several aspects of the HAOG. In Table 3c weconsider several variations on this prediction. The model “SViT-Contact” does not predict the contactinformation, resulting in  0 .
Revised: (ii) After ﬁltering the videos containing (annotated) objects in more than 40% of the frames, we tested our model and the MViTv2 model on the ﬁltered object split. MViTv2 achieved an accuracy of 64.9 while our model achieved 67.0.

Original: 3% drop compared to SViT.
Revised: This implies an improvement of +2.1. We can observe that our model outperforms the baseline even when there are no objects or hands in the videos, demonstrating the robustness of our approach.

Original: The model “SViT+Geometry” adds to SViTadditional heads that predict distances between all HAOG objects (to explicitly add geometric bias). This does not affect performance.
Revised: The total improvement (+1.9 and +2.1) is also a little bit lower than before the ﬁltering (+2.5), which indicates that there has been a slight degradation. Even so, it is still valuable to use the hand-object graphs. How important are the HAOGs .

Original: Finally, we explore what happens when we do not provide theinformation about the identity of bounding boxes in the training data.
Revised: To examine how important is the information provided by the HAOG, we suggest learning HAOGs without any useful information.

Original: Namely, we treat them as fourboxes, and ask the model to match those to the object tokens, via a matching losss as in [12].
Revised: Thus, we run an experiment in which the HAOGs are completely random. This means that, for each image, a random HOAG is generated by sampling the boxes and their relationships uniformly.

Original: This ismodel “SViT-Contact -Corresp.”
Revised: We refer to this experiment as SViT-Random-ID.

Original: which is quite close to “SViT-Contact”, indicating that SViT canperhaps be trained with weaker labels.
Revised: The SViT-Random-ID result on SomethingElse is 50.6 compared to 63.3 in the SViT-ID baseline. This demonstrates that predicting the actual HAOG attributes is important but that not many annotations are necessary for the auxiliary task. One explanation may be that learning the shared structured representation between the domains is sample efﬁcient and does not require many annotated images, which we view as a big advantage of our approach.

Original:  Related Work
Revised:  Related Work

Original: Joint Training with Images and Videos .
Revised: Joint Training with Images and Videos .

Original: Vision Transformers [20] are versatile architecturesdesigned to be ﬂexible in terms of input sequence length.
Revised: Vision Transformers [20] are versatile architectures designed to be ﬂexible in terms of input sequence length.

Original: Aside from the data “‘patchiﬁcation”’ layer,the rest of the transformer is agnostic to the domain of the input.
Revised: Aside from the data “‘patchiﬁcation”’ layer, the rest of the transformer is agnostic to the domain of the input.

Original: Consequently, an exciting directionof joint training image-video has emerged in computer vision, a direction previously less natural withconvolutional architectures.
Revised: Consequently, an exciting direction of joint training image-video has emerged in computer vision, a direction previously less natural with convolutional architectures.

Original: Speciﬁcally, this allows for training network simultaneously on imageand video, with most of the network parameters shared across both domains.
Revised: Speciﬁcally, this allows for training network simultaneously on image and video, with most of the network parameters shared across both domains.

Original: Recently, multi-modalmodels became widely popular, from shared backbones [15, 56, 60] to separate encoders for eachinput modality [73, 76], and even non-vision modalities [38, 39].
Revised: Recently, multi-modal models became widely popular, from shared backbones [15, 57, 61] to separate encoders for each input modality [74, 77], and even non-vision modalities [39, 40].

Original: In our case, we focus on video and image input, similar to [6, 27, 86].
Revised: In our case, we focus on video and image input, similar to [6, 27, 87].

Original: In contrast to these works, our training approach utilizes twodomains, while one of them (image) is used only during training and is supervised by a different task.
Revised: In contrast to these works, our training approach utilizes two domains, while one of them (image) is used only during training and is supervised by a different task.

Original: Additionally, as we demonstrate here, we are not greatly impacted by domain mismatches betweenimages and videos, as in previous works.
Revised: Additionally, as we demonstrate here, we are not greatly impacted by domain mismatches between images and videos, as in previous works.

Original: In a related area of research, multi-task learning [14]involves developing models that predict output for multiple tasks using a common input.
Revised: In a related area of research, multi-task learning [14] involves developing models that predict output for multiple tasks using a common input.

Original: Our work isdifferent since it uses two inputs from different domains, images and videos.
Revised: Our work is different since it uses two inputs from different domains, images and videos.

Original: Similarly, our workcan be viewed as a form of auxiliary task, as already shown in many works [29, 37, 72].
Revised: Similarly, our work can be viewed as a form of auxiliary task, as already shown in many works [29, 38, 73].

Original: However,it differs in that the shared object prompts are utilized to learn the shared structured representationbetween video and image domains, which enhances the self-attention layers with the structuredrepresentation for the main video task.
Revised: However, it differs in that the shared object prompts are utilized to learn the shared structured representation between video and image domains, which enhances the self-attention layers with the structured representation for the main video task.

Original: Learning from video using other modalities .
Revised: Learning from video using other modalities .

Original: Researchers have proposed different approachesto learn from videos using audio and natural language [1, 3, 19, 61, 63, 64, 67, 81].
Revised: Researchers have proposed different approaches to learn from videos using audio and natural language [1, 3, 19, 62, 64, 65, 68, 82].

Original: Vision taskslike Video Captioning [19, 24, 64, 81] and Visual Quesion Answering [2, 78] require understandingof both visual and natural language.
Revised: Vision tasks like Video Captioning [19, 24, 65, 82] and Visual Quesion Answering [2, 79] require understanding of both visual and natural language.

Original: Another line of vision and language works explored objectlabels, supervision of textual descriptions, or the use of region crops [49, 51], in contrast to SViT,which does not require any of these components.
Revised: Another line of vision and language works explored object labels, supervision of textual descriptions, or the use of region crops [50, 52], in contrast to SViT, which does not require any of these components.

Original: Additionally, Since high quality captions areexpensive, researchers have proposed self-supervised ways to learn from both modalities withoutlabeled data [61, 60, 67].
Revised: Additionally, Since high quality captions are expensive, researchers have proposed self-supervised ways to learn from both modalities without labeled data [62, 61, 68].

Original: In these previous works, the two modalities must be consistent and aligned(e.g. , text, audio, and RGB should be consistent for same sample).
Revised: In these previous works, the two modalities must be consistent and aligned (e.g., text, audio, and RGB should be consistent for same sample).

Original: However, in our work, we do notrequire that the annotated images be precisely aligned with our downstream video task.
Revised: However, in our work, we do not require that the annotated images be precisely aligned with our downstream video task.

Original: Last, othernew works [16, 69] explored different structured representations for videos, different from what wedo, which is to leverage structure from images to videos.
Revised: Last, other new works [16, 70] explored different structured representations for videos, different from what we do, which is to leverage structure from images to videos.

Original: Structured Models .
Revised: Structured Models .

Original: Recently structured models have been successfully applied in many computervision applications: video relation detection [54, 70, 75], vision and language [17,50, 51, 77],relational reasoning [8, 9, 35, 47, 40, 66, 84, 85], human-object interactions [23, 43, 83], and evenimage & video generation [7, 32, 42].A recent line of works [5, 25, 26,33, 34, 41, 57, 59, 62, 74, 79,80, 82, 88] focuses on capturing spatio-temporal interactions for video understanding.
Revised: Recently structured models have been successfully applied in many computer vision applications: video relation detection [55, 71, 76], vision and language [17, 51, 52, 78], relational reasoning [8, 9, 36, 48, 41, 67, 85, 86], human-object interactions [23, 44, 84], and even image & video generation [7, 33, 43]. A recent line of works [5, 25, 26, 34, 35, 42, 58, 60, 63, 75, 80, 81, 83, 89] focuses on capturing spatio-temporal interactions for video understanding.

Original: Some of theseworks [33, 80, 82, 88] used static image object annotations to build the representations.
Revised: Some of these works [34, 81, 83, 89] used static image object annotations to build the representations.

Original: However,these works differ from ours.
Revised: However, these works differ from ours.

Original: [82] proposed the Object Transformers (OT) for long form videounderstanding, which model a video as a set of objects by extracting region crops from SlowFast [22]based on bounding boxes from an external detector.
Revised: [83] proposed the Object Transformers (OT) for long form video understanding, which model a video as a set of objects by extracting region crops from SlowFast [22] based on bounding boxes from an external detector.

Original: Contrary to this work, our transformer containsboth patch tokens and object tokens, whereas the OT contains only object tokens.
Revised: Contrary to this work, our transformer contains both patch tokens and object tokens, whereas the OT contains only object tokens.

Original: Additionally, ourmodel learns the object-centric representation without the need for an external detector.
Revised: Additionally, our model learns the object-centric representation without the need for an external detector.

Original: Anotherwork [88], proposed a structured model for action detection that explicitly utilizes image annotations.
Revised: Another work [89], proposed a structured model for action detection that explicitly utilizes image annotations.

Original: They use a GCN [46] to reason about actors and objects interaction, on top of an I3D network.
Revised: They use a GCN [47] to reason about actors and objects interaction, on top of an I3D network.

Original: Unlikethis work, we examine how image annotations can be exploited as part of a video transformer model.
Revised: Unlike this work, we examine how image annotations can be exploited as part of a video transformer model.

Original: Video Understanding Models .
Revised: Video Understanding Models .

Original: We work on several well-benchmarked down-stream video tasks suchas compositional action recognition on Something-Else dataset [58] and action recognition with focuson temporal cues de-emphasizing appearance such as in Something-Something V2 dataset [30] and
Revised: We work on several well-benchmarked down-stream video tasks such as compositional action recognition on Something-Else dataset [59] and action recognition with focus on temporal cues de-emphasizing appearance such as in Something-Something V2 dataset [30] and

Original: Diving48 [53].
Revised: Diving48 [54].

Original: Several recent video transfromer works performed well on SSv2, such as ViViT [4],
Revised: Several recent video transfromer works performed well on SSv2, such as ViViT [4],

Original: MViT [21], MFormer [65], TimeSformer [11], MViTv2 [52],and Video Swin [55].
Revised: MViT [21], MFormer [66], TimeSformer [11], MViTv2 [53],and Video Swin [56].

Original: We choose towork with MViTv2, although our method can be used on top of any of these.
Revised: We choose to work with MViTv2, although our method can be used on top of any of these.

Original: Additionally, wealso adapt our proposed method to interesting new tasks on the Ego4D dataset [31].
Revised: Additionally, we also adapt our proposed method to interesting new tasks on the Ego4D dataset [31].

Original: We believesuch ﬁne-grained video understanding tasks require richer scene and object semantic reasoning, arequirement well afforded by our proposed Hand-Object Graph.
Revised: We believe such ﬁne-grained video understanding tasks require richer scene and object semantic reasoning, a requirement well afforded by our proposed Hand-Object Graph.

Original:  Conclusion
Revised:  Conclusion

Original: Video understanding is a key element of human visual perception, but modeling remains a challenge for machine vision.
Revised: Video understanding is a key element of human visual perception, but modeling remains a challenge for machine vision.

Original: In this work, we demonstrated the importance of learning from a scenestructure from a small set of images to facilitate video learning within or outside the domain ofinterest.
Revised: In this work, we demonstrated the importance of learning from a scene structure from a small set of images to facilitate video learning within or outside the domain of interest.

Original: According to our empirical study, our SViT approach improves performance on three videounderstanding tasks.
Revised: According to our empirical study, our SViT approach improves performance on four video understanding tasks.

Original: Note, that we did not prioritize making the HAOG annotations richer, i.e., itmay be possible to add other physical properties to improve structure modeling.
Revised: Note, that we did not prioritize making the HAOG annotations richer, i.e., it may be possible to add other physical properties to improve structure modeling.

Original: In addition, wenoticed that leveraging the structure is effective even with 2% of the in-domain frames, as well aswhen applying it from outside the domain.
Revised: In addition, we noticed that leveraging the structure is effective even with 2% of the in-domain frames, as well as when applying it from outside the domain.

Original: We leave for future work how different physical propertiescan be more effectively utilized in more complex structures.
Revised: We leave for future work how different physical properties can be more effectively utilized in more complex structures.

Original: We do not anticipate a speciﬁc negativeimpact, but, as with any Machine Learning method, we recommend to exercise caution. 
Revised: We do not anticipate a speciﬁc negative impact, but, as with any Machine Learning method, we recommend to exercise caution. Acknowledgements

Revised: This project has received funding from the European Research Council (ERC) under the European

Revised: Unions Horizon 2020 research and innovation programme (grant ERC HOLI 819080).

Revised: Prof. Darrell’s group was supported in part by DoD including DARPA’s PTG and LwLL programs, as well as BAIR’s industrial alliance programs.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.
Revised: [No] The code and the data are proprietary.

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes](c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See line341-342.(d)
Revised: [Yes] See line 341-342.

Original:  Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A]3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes](c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: Yes (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] Sec Section 3.1 (theDataset Section).
Revised: [Yes] Sec Section 3.1 (the Dataset Section).

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See the Dataset Section in supplementary.
Revised: [Yes] See the Dataset Section in supplementary.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] See the Dataset Section in supplementary.
Revised: [Yes] See the Dataset Section in supplementary.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [Yes] See the Dataset Section in supplementary.
Revised: [Yes] See the Dataset Section in supplementary.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]