Original: Bringing Image  Structure to VideoTransformers via Frame-Clip Consistency of Object Tokens
Revised: Bringing Image Scene Structure to Video via Frame-Clip Consistency of Object Tokens

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Recent action recognition models have achieved impressive results by integratingobjects, their locations and interactions.
Revised: Recent action recognition models have achieved impressive results by integratingobjects, their locations and interactions.

Original: However, obtaining dense structuredannotations for each frame is tedious and time-consuming, making these methodsexpensive to train and less scalable.
Revised: However, obtaining dense structuredannotations for each frame is tedious and time-consuming, making these methodsexpensive to train and less scalable.

Original: At the same time, if a small set of annotatedimages is available, either within or outside the domain of interest, how could weleverage these for a video downstream task?
Revised: At the same time, if a small set of annotatedimages is available, either within or outside the domain of interest, how could weleverage these for a video downstream task?

Original: We propose a learning framework
Revised: We propose a learning framework

Original: StructureViT (SViT for short), which demonstrates how utilizing the structure of asmall number of images only available during training can improve a video model.
Revised: StructureViT (SViT for short), which demonstrates how utilizing the structure of asmall number of images only available during training can improve a video model.

Original: SViT relies on two key insights.
Revised: SViT relies on two key insights.

Original: First, as both images and videos contain structuredinformation, we enrich a transformer model with a set of object tokens that can beused across images and videos.
Revised: First, as both images and videos contain structuredinformation, we enrich a transformer model with a set of object tokens that can beused across images and videos.

Original: Second, the scene representations of individualframes in video should “align” with those of still images.
Revised: Second, the scene representations of individualframes in video should “align” with those of still images.

Original: This is achieved via a
Revised: This is achieved via a

Original: Frame-Clip Consistency loss, which ensures the ﬂow of structured informationbetween images and videos.
Revised: Frame-Clip Consistency loss, which ensures the flow of structured informationbetween images and videos.

Original: We explore a particular instantiation of scene structure,namely a Hand-Object Graph , consisting of hands and objects with their locationsas nodes, and physical relations of contact/no-contact as edges.
Revised: We explore a particular instantiation of scene structure,namely a Hand-Object Graph , consisting of hands and objects with their locationsas nodes, and physical relations of contact/no-contact as edges.

Original: SViT shows strongperformance improvements on multiple video understanding tasks and datasets.
Revised: SViT shows strongperformance improvements on multiple video understanding tasks and datasets.

Original:  Introduction
Revised:  Introduction

Original: Semantic understanding of videos is a key challenge for machine vision and artiﬁcial intelligence.
Revised: Semantic understanding of videos is a key challenge for machine vision and artificial intelligence.

Original: Itis intuitive that video models should beneﬁt from incorporating scene structure, e g., the objects thatappear in a video, their attributes and the way they interact.
Revised: Itis intuitive that video models should benefit from incorporating scene structure, e g., the objects thatappear in a video, their attributes and the way they interact.

Original: Indeed, several works [5, 25, 26, 33, 34,41, 57, 59, 62, 74, 79, 80, 82, 88] have demonstrated that incorporating structured representationsinto models improves both performance and sample efﬁciency.

Original: In particular, [33, 80, 88] showed howto incorporate structured representations by utilizing static image object annotations in videos.

Original: Recently, vision transformers (ViT) [20] have emerged as the leading model for many vision applications [4, 21, 12]. This begs the question, how can structured representations be used to leveragevideo transformers for video tasks? Past works[4, 21] have proposed video transformer modelsbut these approaches were not focused on learning structured representations. A recent line ofworks [33, 80, 88] utilized objects for structured approaches.
Revised: In fact, several works have shown thatsuch “object-centric” models perform well on tasks such as action recognition [30, 68].

Original: [82] leveraged the decoder with CNNsregion crop features from SlowFast [22], and [33] proposed a new block that incorporates objectrepresentations.

Original: However, these approaches are limited because they use external pretrained detectorsfor the frames, which require additional processing time (also during inference), they ignore thepotential beneﬁts of the connection between the detection and the other tasks, and it is not clear howto use them with more detailed structured annotations.

Original: Finally, [88] proposed a structured modelfor action detection that explicitly utilizes image annotations, but not in a transformer architecture.

Original: Our approach, described next, builds on these prior works and proposes an effective way to leveragescene structure in video transformers, using static image supervision.

Original: Most structured transformer-based modelsfor videos require  structured annotations  during training.
Revised: However, mostof these models require ground-truth structured annotations of video during training.

Original: For example, frame-level object bounding boxes are considered available in [5, 7, 33, 34, 58, 80]. [Figure]
Revised: For example,frame-level ground-truth object bounding boxes are considered available in [5, 7, 30, 31, 50, 68].

Original: [Figure]
Revised: This is clearly very costly, time-consuming, and not scalable. This begs the question, could we still benefit from scene structure in a less costly way? A possibledirection is to explore methods that would only require sparsely annotated frames [28, 60] withina downstream video domain.

Original: When it comes to structured annotations of static images, there are numerous datasets with annotationssuch as boxes, visual relationships, attributes, and segmentations [10, 48, 58].
Revised: Moreover, when it comes to structured annotations of static images,there are numerous datasets with annotations such as boxes, visual relationships, attributes, andsegmentations [10, 42, 50].

Original: Since we cannot alwaysexpect to have annotated images that perfectly align with our downstream video task, how can theseresources be used to build bettertransformer-based video models?
Revised: Since we can not always expect to have annotated images that perfect alignwith our downstream video task, how can these resources be used to build better video models?

Original: In this work, we propose suchan approach, which is particularly designed for transformer architectures and offers an effectivemechanism to leverage image structure to improve video transformers.
Revised: In thiswork, we propose such an approach, which is particularly well suited for transformer architecturesand offers an effective mechanism to leverage image structure to improve video transformers.

Original: A natural ﬁrst step towards image-video knowledge sharing is to use the same transformer model toprocess both: namely, to view images as “single frame videos”, and feed them to a video transformermodel.
Revised: A natural first step towards image-video knowledge sharing is to use the same transformer model toprocess both: namely, to view images as “single frame videos”, and feed them to a video transformermodel.

Original: However, this still leaves two key questions: how to model structured information, and howto account for domain mismatch between images and videos.
Revised: However, this still leaves two key questions: how to model structured information, and how [Figure] [Figure] to account for domain mismatch between images and videos.

Original: Towards this end, we introduce twokey concepts: (i) A set of transformer “object-tokens” – additional tokens initialized from learnedembeddings (we refer to these embeddings as “object prompts”) that are meant to capture objectcentric information in both still image and video.
Revised: Towards this end, we introduce twokey concepts: (i) A set of transformer “object-tokens” – additional tokens initialized from learnedembeddings (we refer to these embeddings as “object prompts”) that are meant to capture objectcentric information in both still image and video.

Original: These tokens can be supervised based on the scenestructure of images, and that information will also be propagated to videos.
Revised: These tokens can be supervised based on the scenestructure of images, and that information will also be propagated to videos.

Original: To formalize the imagestructure, we proposed the HAnd-Object Graph (HAOG), a simple representation of the interactionsbetween hands and objects in the scene.
Revised: To formalize the imagestructure, we proposed the HAnd-Object Graph (HAOG), a simple representation of the interactionsbetween hands and objects in the scene.

Original: (ii)
Revised: (ii)

Original: A novel “Frame-Clip Consistency loss” that ensuresconsistency between the “object-tokens” when they are part of a video vs. a still image.
Revised: A novel “Frame-Clip Consistency loss” that ensuresconsistency between the “object-tokens” when they are part of a video vs. a still image.

Original: Moreover, aswe show in our experiments, our approach does not require any alignment between object categoriesappearing in still images and the ones appearing in the videos of the target downstream task.
Revised: Moreover, aswe show in our experiments, our approach does not require any alignment between object categoriesappearing in still images and the ones appearing in the videos of the target downstream task.

Original: We nameour proposed approach StructureViT ( Bringing Scene Structure from Images to Vi deo via Frame-Clip
Revised: We nameour proposed approach StructureViT ( Bringing Scene Structure from Images to Vi deo via Frame-Clip

Original: Consistency of Object T okens , or SViT for short).
Revised: Consistency of Object T okens , or SViT for short).

Original: See Figure 1 for an overview.
Revised: See Figure 1 for an overview.

Original: Prior work has explored several other video & image models.
Revised: Prior work has explored several other video & image models.

Original: The I3D model [13] was one of the ﬁrstsuccessful attempts to leverage ImageNet architectures in video.
Revised: The I3D model [13] was one of the firstsuccessful attempts to leverage ImageNet architectures in video.

Original: They proposed to simply convertimage 2D classiﬁcation models into 3D ConvNets by inﬂating all the ﬁlters and pooling kernels.
Revised: They proposed to simply convertimage 2D classification models into 3D ConvNets by inflating all the filters and pooling kernels.

Original: Here, our focus is on learning the shared structured representations between the two domains in orderto transfer this knowledge into videos.
Revised: Here,our focus is on learning the shared structured representations between the two modalities in order totransfer this knowledge into videos.

Original: Several other methods have explored simultaneous image andvideo training in the context of multi-task [6] and multi-modal [27] learning.
Revised: Several other methods have explored simultaneous image andvideo training in the context of multi-task [6] and multi-modal [25] learning.

Original: However, these worksdid not utilize structural information, nor did they promote consistency between image and videopredictions – which, as we show, leads to signiﬁcant gains in downstream video task performance.
Revised: However, these worksdid not utilize structural information, nor did they promote consistency between image and videopredictions – which, as we show, leads to significant gains in downstream video task performance.

Original: To summarize, our main contributions are as follows: (i) we propose a new method for exploitingstructured information present in images in order to improve performance of video understandingtasks; (ii) we propose a novel concept of “object tokens” - a form of learned prompts to a video &image transformer, which during training are associated to the available structure representation ofthe still images (HAOG).
Revised: To summarize, our main contributions are as follows: (i) we propose a new method for exploitingstructured information present in images in order to improve performance of video understandingtasks; (ii) we propose a novel concept of “object tokens” - a form of learned prompts to a video &image transformer, which during training are associated to the available structure representation ofthe still images (HAOG).

Original: (iii) we introduce a new Frame-Clip Consistency loss, which promotesconsistency between image-level and video-level predictions made by a shared transformer backbone.
Revised: (iii) we introduce a new Frame-Clip Consistency loss, which promotesconsistency between image-level and video-level predictions made by a shared transformer backbone.

Original: We show how this loss helps to drive performance improvements in downstream video tasks even incases when image labels are unrelated to the video task; (iv) we demonstrate improved performanceon several video understanding benchmarks, highlighting the effectiveness of the proposed approach.
Revised: We show how this loss helps to drive performance improvements in downstream video tasks even incases when image labels are unrelated to the video task; (iv) we demonstrate improved performanceon several video understanding benchmarks, highlighting the effectiveness of the proposed approach.

Original:  The SViT Approach
Revised:  The SViT Approach

Original: Our approach SViT learns  structured  representation that is exhibited in both static images and video.
Revised: Our approach SViT learns a structured shared representation that arises from the interaction betweenthe two modalities of images and videos.

Original: We consider the setting where the main goal is to learn video understanding downstream taskswhile leveraging structured image data.
Revised: We consider the setting where the main goal is to learnvideo understanding downstream tasks while leveraging structured image data.

Original: In this paper, we focus on the following video tasks: actionrecognition and object state classiﬁcation & localization.
Revised: In this paper, wefocus on the following video tasks: action recognition and object state classification & localization.

Original: In training, we assume that we have accessto task-speciﬁc video labels and structured scene annotations for images, Hand-Object Graph s. Based on the structured representations obtained by using the object tokens and regularized by the Frame-Clip Consistency loss , the inference is performed only for the downstream video task withoutexplicitly using the structured image annotations.
Revised: Intraining, we assume that we have access to task-specific video labels and structured scene annotationsfor images, Hand-Object Graph s. Based on the structured representations obtained by using theobject tokens and regularized by the Frame-Clip Consistency loss , the inference is performed only forthe downstream video task without explicitly using the structured image annotations.

Original: Further, we begin by describing the Hand-Object Graph (HAOG) (Section 2.1), then introduce our
Revised: Further, we begin by describing the Hand-Object Graph (HAOG) (Section 2.1), then introduce our

Original: SViT model (Section 2.2), and the Frame-Clip Consistency loss function (Section 2.3).
Revised: SViT model (Section 2.2), and the Frame-Clip Consistency loss function (Section 2.3).

Original: Our methodis schematically illustrated in Figure 2.
Revised: Our methodis schematically illustrated in Figure 2.

Original: 2.1 The Hand-Object Graph
Revised: 2.1 The Hand-Object Graph

Original: As mentioned above, our motivation is to bring scene structure from still images into video.
Revised: As mentioned above, our motivation is to bring scene structure from still images into video.

Original: Inorder to achieve this, one component of this work is the construction of a semantic representationof the interactions between the hands and objects in the scene.
Revised: Inorder to achieve this, one component of this work is the construction of a semantic representationof the interactions between the hands and objects in the scene.

Original: Speciﬁcally, we propose to usea graph-structure
Revised: Specifically, we propose to usea graph-structure

Original: we call Hand-Object Graph (HAOG), see Figure 1.
Revised: we call Hand-Object Graph (HAOG), see Figure 1.

Original: The nodes of the HAOGrepresent two hands and two objects with their locations, whereas the edges correspond to physicalproperties such as contact/no contact.
Revised: The nodes of the HAOGrepresent two hands and two objects with their locations, whereas the edges correspond to physicalproperties such as contact/no contact.

Original: Formally, an HAOG is a tuple ( O, E ) deﬁned as follows:
Revised: Formally, an HAOG is a tuple ( O, E ) defined as follows:

Original: Nodes O : A set O of n objects.
Revised: Nodes O : A set O of n objects.

Original: Human-object interaction scenes, which we focus on in this work,usually consist of up to two hands and objects.
Revised: Human-object interaction scenes, which we focus on in this work,usually consist of up to two hands and objects.

Original: Thus our node categories are “hand” and “object”;they may also contain attributes (e.g., spatial coordinates).
Revised: Thus our node categories are “hand” and “object”;they may also contain attributes (e.g., spatial coordinates).

Original: We assume that each image comes withthe bounding boxes describing the locations of the two hands, and the objects they interact with.
Revised: We assume that each image comes withthe bounding boxes describing the locations of the two hands, and the objects they interact with.

Original: Thebounding boxes for the left and right hand are denoted as b 1 , b 2 2 R 4 , respectively, while for the twocorresponding objects interacting with the hands are denoted as b 3 , b 4 .
Revised: Thebounding boxes for the left and right hand are denoted as b 1 , b 2 ∈ R 4 , respectively, while for the twocorresponding objects interacting with the hands are denoted as b 3 , b 4 .

Original: We also assume access tofour binary variables e 1 , . .
Revised: We also assume access tofour binary variables e 1 , . .

Original: , e 4 2 { 0 , 1 } that indicate whether the corresponding hands or objects infact exist in the image (e.g., if e 1 = 

Revised: , e 4 ∈ { 0 , 1 } that indicate whether the corresponding hands or objects infact exist in the image (e.g., if e 1 =

Original: the left hand is not in the image, and b 1 should be ignored).
Revised: the left hand is not in the image, and b 1 should be ignored).

Original: Edges E : The edges are represented as labeled directed edges between hand and object nodes.
Revised: Edges E : The edges are represented as labeled directed edges between hand and object nodes.

Original: Edgevocabulary describes the physical properties of the scene.
Revised: Edgevocabulary describes the physical properties of the scene.

Original: In our current modeling, each edge ischaracterized by a physical property of “contact” or “no-contact”.
Revised: In our current modeling, each edge ischaracterized by a physical property of “contact” or “no-contact”.

Original: We denote two binary variablesc 1 , c 2 2 { 0 , 1 } that specify if each of the hands is in direct contact with the corresponding object (theleft hand b 1 can only interact with the object b 3 , similar for the right hand).
Revised: We denote two binary variablesc 1 , c 2 ∈ { 0 , 1 } that specify if each of the hands is in direct contact with the corresponding object (theleft hand b 1 can only interact with the object b 3 , similar for the right hand).

Original: In principle, we can alsoinclude properties such as distances between objects, 3D hand poses, etc., where available.
Revised: In principle, we can alsoinclude properties such as distances between objects, 3D hand poses, etc., where available.

Original: We note that images annotated consistent with our deﬁned HAOG format can be found in severaldatasets [31, 58, 71], and we leverage these in our experiments.
Revised: We note that images annotated consistent with our defined HAOG format can be found in severaldatasets [28, 50, 60], and we leverage these in our experiments.

Original: 2.2 The SViT Transformer Model
Revised: 2.2 The SViT Transformer Model

Original: We next present the SViT model: a video transformer that employs object tokens as a means to capturestructure in both images and videos.
Revised: We next present the SViT model: a video transformer that employs object tokens as a means to capturestructure in both images and videos.

Original: We begin by reviewing the video transformer architecture, whichour model extends.
Revised: We begin by reviewing the video transformer architecture, whichour model extends.

Original: Next, we explain how we process both images and videos.
Revised: Next, we explain how we process both images and videos.

Original: Finally, we describeobject tokens and how they are used in videos and still images.
Revised: Finally, we describeobject tokens and how they are used in videos and still images.

Original: Shared Video & Image Transformer .
Revised: Shared Video & Image Transformer .

Original: Video transformers [4, 11, 21, 33] extend the Vision Transformer model to the temporal domain.
Revised: Video transformers [4, 11, 19, 30] extend the Vision Transformer model to the temporal domain.

Original: Similar to vision transformers, the input is ﬁrst “patchiﬁed” butwith temporally extended 3-D convolution (instead of 2-D) producing a down-sampled tensor X ofsize T ⇥ H ⇥ W ⇥ d .
Revised: Similar to vision transformers, the input is first “patchified” butwith temporally extended 3-D convolution (instead of 2-D) producing a down-sampled tensor X ofsize T × H × W × d .

Original: We refer to this as “patch tokens”.
Revised: We refer to this as “patch tokens”.

Original: Then, spatio-temporal position embeddingsare added for providing location and time information.
Revised: Then, spatio-temporal position embeddingsare added for providing location and time information.

Original: Next, multiple stacked self-attention blocksare applied repeatedly on the down-sampled tensor X to produce the ﬁnal vector representation usingmean pooling over the patch tokens.
Revised: Next, multiple stacked self-attention blocksare applied repeatedly on the down-sampled tensor X to produce the final vector representation usingmean pooling over the patch tokens.

Original: In our approach, we want to be able to process batches of images or videos.
Revised: In our approach, we want to be able to process batches of images or videos.

Original: A key desideratum inthis context is to be able to input both videos and still-images into the same model.
Revised: A key desideratum inthis context is to be able to input both videos and still-images into the same model.

Original: Towards this end,our ﬁrst change is to rely only on 2-D convolutions in the ﬁrst transformer step instead of 3-D. Thisallows single images to be used as inputs without padding, and does not hurt performance in practice.
Revised: Towards this end,our first change is to rely only on 2-D convolutions in the first transformer step instead of 3-D. Thisallows single images to be used as inputs without padding, and does not hurt performance in practice.

Original: [Figure]
Revised: [Figure]

Original: Object tokens .
Revised: Object tokens .

Original: Our goal is to learn a shared structured representation between the video and theimage domains.
Revised: Our goal is to learn a shared structured representation between the video and theimage domains.

Original: We do this by adding transformer tokens to represent objects.
Revised: We do this by adding transformer tokens to represent objects.

Original: These tokens arefunctionally similar to the patch tokens, with two exceptions.
Revised: These tokens arefunctionally similar to the patch tokens, with two exceptions.

Original: First, they are initialized with additionaltrained embeddings, which we referred to as “object prompts”.
Revised: First, they are initialized with additionaltrained embeddings, which we referred to as “object prompts”.

Original: We consider the “object prompts” asa bridge between the images and the videos since they are being used in prediction tasks across bothdomains.
Revised: We consider the “object prompts” asa bridge between the images and the videos since they are being used in prediction tasks across bothmodalities.

Original: Second, they are used to predict the structured representations when those are available(here, we use the Hand-Object Graphs structure for still images).
Revised: Second, they are used to predict the structured representations when those are available(here, we use the Hand-Object Graphs structure for still images).

Original: An object token is used to predictproperties of the corresponding node in the HAOG, and the concatenation of two object tokens isused to predict edge properties.
Revised: An object token is used to predictproperties of the corresponding node in the HAOG, and the concatenation of two object tokens isused to predict edge properties.

Original: Next, we describe the model more formally.
Revised: Next, we describe the model more formally.

Original: Let n be the maximum number of modeled objects perframe (or still image).
Revised: Let n be the maximum number of modeled objects perframe (or still image).

Original: We have n tokens for each frame, and thus a total of T ⇥ n tokens.
Revised: We have n tokens for each frame, and thus a total of T × n tokens.

Original: The tokenfor object i at frame t is initialized with the vector o i + r t where o i 2 R d is a learned object 
Revised: The tokenfor object i at frame t is initialized with the vector o i + r t where o i ∈ R d is a learned object promptand r

Original: promptand r t 2 R d is a learned temporal positional embedding (the same one used for initializing the patchtokens).
Revised: i ∈ R d is a learned temporal positional embedding (the same one used for initializing the patchtokens).

Original: With these new tokens, we have T ⇥ ( H ⇥ W +  n) tokens (i.e, vectors in R d ), and thesetogether will go through the standard self-attention layers.
Revised: With these new tokens, we have T × H × W + T × n tokens (i.e, vectors in R d ), and thesetogether will go through the standard self-attention layers.

Original: We denote the operator that outputs the ﬁnal representation of the object tokens by F O , where fora video V we have F O ( V ) 2 R T ⇥ n ⇥ d and for an image
Revised: We denote the operator that outputs the final representation of the object tokens by F O , where fora video V we have F O ( V ) ∈ R T × n × d and for an image

Original: I we have F O ( I ) 2 R n ⇥ d .
Revised: I we have F O ( I ) ∈ R n × d .

Original: The F O ( I )representation is used to predict structured representations for single images.
Revised: The F O ( I )representation is used to predict structured representations for single images.

Original: We also include a lossthat makes single frame representations align with those of clip representations (see Section 2.3).
Revised: We also include a lossthat makes single frame representations align with those of clip representations (see Section 2.3).

Original: Theoperator that outputs the ﬁnal representation vector used for the video downstream task is denoted by
Revised: Theoperator that outputs the final representation vector used for the video downstream task is denoted by

Original: F CLS , where for video V we have F CLS ( V ) 2 R d .
Revised: F CLS , where for video V we have F CLS ( V ) ∈ R d .

Original: Finally, our method can be used on top of the most common video transformers (MViT [21],
Revised: Finally, our method can be used on top of the most common video transformers (MViT [19],

Original: TimeSformer [11], Mformer [65], Video Swin [55]).
Revised: TimeSformer [11], Mformer [55], Video Swin [48]).

Original: For our experiments, we use the MViTv2 [52]model because it performs well empirically.
Revised: For our experiments, we use the MViTv2 [45]model because it performs well empirically.

Original: 2.3 Losses and Training
Revised: 2.3 Losses and Training

Original: During training we have a set of images annotated with HAOGs and videos with downstream tasklabels.
Revised: During training we have a set of images annotated with HAOGs and videos with downstream tasklabels.

Original: We use these as inputs to our model (Section 2.2) and optimize the losses described below.
Revised: We use these as inputs to our model (Section 2.2) and optimize the losses described below.

Original: Video loss .
Revised: Video loss .

Original: Each training video V has a corresponding ground-truth category Y 2 { 1 , . .
Revised: Each training video V has a corresponding ground-truth category Y ∈ { 1 , . .

Original: . , K } .
Revised: . , K } .

Original: As noted above, we have a vector representation for the entire video denoted by F CLS ( V ) .
Revised: As noted above, we have a vector representation for the entire video denoted by F CLS ( V ) .

Original: Thus,we simply use a neural network to predict logits for Y .
Revised: Thus,we simply use a neural network to predict logits for Y .

Original: Speciﬁcally, our predicted logits are:ˆ Y =
Revised: Specifically, our predicted logits are:ˆ Y =

Original: FC ( F CLS ( V ))
Revised: FC ( F CLS ( V ))

Original: 2 R K , where FC is a trainable fully connected network.
Revised: ∈ R K , where FC is a trainable fully connected network.

Original: We then consider astandard cross-entropy between the predicted logits ˆ Y and true labels Y .
Revised: We then consider a standard cross-entropy between the predicted logits ˆ Y and true labels Y .

Original: [Equation]
Revised: [Equation]

Original: HAOG loss .
Revised: HAOG loss .

Original: The transformer outputs a set of n object tokens for each image.
Revised: The transformer outputs a set of n object tokens for each image.

Original: Since we havesupervision for these objects, we predict structured representations from the tokens, and introduce aloss to optimize these predictions.
Revised: Since we havesupervision for these objects, we predict structured representations from the tokens, and introduce aloss to optimize these predictions.

Original: Recall our notation from Section 2.1.
Revised: Recall our notation from Section 2.1.

Original: We let the transformer usen = 4 object tokens which correspond to the two hands (bounding boxes b 1 , b 2 ) and two manipulated objects (bounding boxes b 3 , b 4 ).
Revised: We let the transformer usen = 4 object tokens which correspond to the two hands (bounding boxes b 1 , b 2 ) and two manipulatedobjects (bounding boxes b 3 , b 4 ).

Original: The j th token is used to predict a corresponding bounding box b j ,as well as the existence variable e j .
Revised: The j th token is used to predict a corresponding bounding box b j ,as well as the existence variable e j .

Original: Formally, we let:
Revised: Formally, we let:

Original: [Equation]
Revised: [Equation]

Original: where FC bb , FC e are fully connected networks with four and one output respectively, F jO ( I ) is thej th object token, and I is the input image.
Revised: where FC bb , FC e are fully connected networks with four and one output respectively, F jO ( I ) is thej th object token, and I is the input image.

Original: Formally, this deﬁne the loss of the nodes as following:
Revised: Formally, this define the loss of the nodes as following:

Original: [Equation]
Revised: [Equation]

Original: where L1 is the standard L1 distance, and GIoU is used as in [68].
Revised: where L1 is the standard L1 distance, and GIoU is used as in [58].

Original: Furthermore, we predict thecontact variable from the concatenation of the corresponding hand and object tokens.
Revised: Furthermore, we predict thecontact variable from the concatenation of the corresponding hand and object tokens.

Original: Namely forj 2 { 1 , 2 } :
Revised: Namely forj ∈ { 1 , 2 } :

Original: [Equation]
Revised: [Equation]

Original: where FC c is a fully connected with one output.
Revised: where FC c is a fully connected with one output.

Original: Thus, we deﬁne the loss of the edges as follows:
Revised: Thus, we define the loss of the edges as follows:

Original: [Equation]
Revised: [Equation]

Original: Finally, to obtain the loss, we compare the predictions and ground-truth labels as follows:
Revised: Finally, to obtain the loss, we compare the predictions and ground-truth labels as follows:

Original: [Equation]
Revised: [Equation]

Original: Frame-Clip Consistency Loss .
Revised: Frame-Clip Consistency Loss .

Original: Since we have different losses for images and video, the modelcould ﬁnd a way to minimize the image loss in a way that only applies to the images and does nottransfer to video.
Revised: Since we have different losses for images and video, the modelcould find a way to minimize the image loss in a way that only applies to the images and does nottransfer to video.

Original: 1 To avoid this, we need to make sure that the video representation contains thesame type of information as the still images, and in particular the structured information carried bythe object tokens.
Revised: 1 To avoid this, we need to make sure that the video representation contains thesame type of information as the still images, and in particular the structured information carried bythe object tokens.

Original: To achieve this, we explicitly enforce the object tokens to be consistent acrossstill images (frames) and videos (clips) using a frame-clip Consistency loss.
Revised: To achieve this, we explicitly enforce the object tokens to be consistent acrossstill images (frames) and videos (clips) using a frame-clip Consistency loss.

Original: To calculate the loss,we process each video twice: once as a clip, and once as a batch of T frames.
Revised: To calculate the loss,we process each video twice: once as a clip, and once as a batch of T frames.

Original: Namely, a clip Vconsisting of T frames I 1 , ..., I T , will be processed once as F O ( V ) and once as a list of images( F O ( I 1 ) , ..., F O ( I T )) .
Revised: Namely, a clip Vconsisting of T frames I 1 , ..., I T , will be processed once as F O ( V ) and once as a list of images( F O ( I 1 ) , ..., F O ( I T )) .

Original: This results in two groups of T ⇥ n object tokens, the ﬁrst are the clip objecttokens , denoted as, O clip :=
Revised: This results in two groups of T × n object tokens, the first are the clip objecttokens , denoted as, O clip :=

Original: F O ( V ) 2 R T ⇥ n ⇥ d , the second are the frame object tokens , denotedas O frames :=
Revised: F O ( V ) ∈ R T × n × d , the second are the frame object tokens , denotedas O frames :=

Original: [ F O ( v 1 ) , ..., F O ( v T )] 2 R T ⇥ n ⇥ d .
Revised: [ F O ( v 1 ) , ..., F O ( v T )] ∈ R T × n × d .

Original: Since both groups originate from the same input,each clip object token has a single corresponding frame object token.
Revised: Since both groups originate from the same input,each clip object token has a single corresponding frame object token.

Original: Namely, there is an alignmentbetween the clip object tokens and the frame object tokens.
Revised: Namely, there is an alignmentbetween the clip object tokens and the frame object tokens.

Original: Following the intuition that each clipobject token should contain the information from its corresponding frame object token and potentiallymore (the temporal context), we minimize the L 1 distance between each clip object token and frameobject token, after applying a fully-connected layer to each clip object token.
Revised: Following the intuition that each clipobject token should contain the information from its corresponding frame object token and potentiallymore (the temporal context), we minimize the L 1 distance between each clip object token and frameobject token, after applying a fully-connected layer to each clip object token.

Original: Namely:
Revised: Namely:

Original: [Equation]
Revised: [Equation]

Original: Overall loss .
Revised: Overall loss .

Original: The total loss consists of L Con , L HAOG and L V id , where L V id is the loss of the main 190 task.
Revised: The total loss consists of L Con , L HAOG and L V id , where L V id is the loss of the maintask.

Original: Each of the three terms in the loss is multiplied by a hyper-parameter (  Con ,  HAOG ,  V id ),and the total loss is the weighted combination of the three terms:
Revised: Each of the three terms in the loss is multiplied by a hyper-parameter ( λ Con , λ HAOG , λ V id ),and the total loss is the weighted combination of the three terms:

Original: [Equation]
Revised: [Equation]

Original:  Experiments and Results
Revised:  Experiments and Results

Original: We begin by describing the datasets, implementation details, and baselines and variants.
Revised: We begin by describing the datasets, implementation details, and baselines and variants.

Original: We thenevaluate our approach on several benchmarks for video understanding.
Revised: We thenevaluate our approach on several benchmarks for video understanding.

Original: Speciﬁcally, we consider thefollowing tasks: Compositional Action Recognition (Section 3.4), Object State Change Classiﬁcation& Localization (Section 3.5), and Action Recognition (Section 3.6).
Revised: Specifically, we consider thefollowing tasks: Compositional Action Recognition (Section 3.4), Object State Change Classification& Localization (Section 3.5), and Action Recognition (Section 3.6).

Original: 3.1 Datasets
Revised: 3.1 Datasets

Original: We ﬁrst describe the datasets used for the downstream video tasks, and those used as “auxiliary”datasets with annotated images.
Revised: We first describe the datasets used for the downstream video tasks, and those used as “auxiliary”datasets with annotated images.

Original: We use the following video datasets: (1) Something-Somethingv2 (SSv2) [30] is a dataset containing 174 action categories of common human-object interactions.(2) SomethingElse [58] which exploits the compositional structure of SSv2, where a combinationof a verb and a noun deﬁnes an action.
Revised: We use the following video datasets: (1) Something-Somethingv2 (SSv2) [27] is a dataset containing 174 action categories of common human-object interactions.(2) SomethingElse [50] which exploits the compositional structure of SSv2, where a combinationof a verb and a noun defines an action.

Original: We follow the ofﬁcial compositional split from [58], whichassumes the set of noun-verb pairs available for training is disjoint from the set given at test time.(3) Ego4D [31] is a new large-scale dataset of more than 3,670 hours of video data, capturing thedaily-life scenarios of more than 900 unique individuals from nine different countries around theworld.
Revised: We follow the official compositional split from [50], whichassumes the set of noun-verb pairs available for training is disjoint from the set given at test time.(4) Ego4D [28] is a new large-scale dataset of more than 3,670 hours of video data, capturing thedaily-life scenarios of more than 900 unique individuals from nine different countries around theworld.

Original: (4) Diving48 [53] contains 48 ﬁne-grained categories of diving activities.
Revised: (5) Diving48 [46] contains 48 fine-grained categories of diving activities.

Original: For “auxiliary”image datasets we use the Ego4D, and the 100 Days of Hands (100DOH) [71] datasets.
Revised: For “auxiliary”image datasets we use the Ego4D, and the 100 Days of Hands (100DOH) [60] datasets.

Original: We collected79,921 annotated images from 100DOH, and 57,213 annotated images from Ego4D. 2 The collectedobject boxes from SSv2, 100DOH, and Ego4D are purely human annotated. In SSv2 and Ego4D,contact relations between the object and hand are not annotated, so we assign the closest object to thehand for each hand. The contact relations for 100DOH are available in the dataset.{+
+}
Revised: We collected79,921 annotated images from 100DOH, and 57,213 annotated images from Ego4D. {+
+}

Original: For the image datasets, the image annotations are based on frames selected from videos, but these areoften sparsely selected from the videos, so it is natural to treat them as annotated still images.
Revised: For the image datasets, the image annotations are based on frames selected from videos, but these areoften sparsely selected from the videos, so it is natural treated them as annotated still images.

Original: 3.2 Implementation Details
Revised: 3.2 Implementation Details

Original: SViT is implemented in PyTorch, we include the code in the supplementary and will release it uponacceptance.
Revised: SViT is implemented in PyTorch, we include the code in the supplementary and will release it uponacceptance.

Original: Our training recipes and code are based on the MViTv2 model, and were taken fromhttps://github.com/facebookresearch/mvit .
Revised: Our training recipes and code are based on the MViTv2 model, and were taken fromhttps://github.com/facebookresearch/mvit .

Original: We pretrain the SViT model on the K400 [44]video dataset with our auxiliary image datasets.
Revised: We pretrain the SViT model on the K400 [40]video dataset with our auxiliary image datasets.

Original: Then, we ﬁnetune on the target video understandingtask (detailed in Section 3.1) together with the auxiliary image datasets and the SViT loss.
Revised: Then, we finetune on the target video understandingtask (detailed in Section 3.1) together with the auxiliary image datasets and the SViT loss.

Original: Eachtraining batch contains 64 images and 64 videos in order to minimize the overall loss in Equation 8.
Revised: Eachtraining batch contains 64 images and 64 videos in order to minimize the overall loss in Equation 8.

Original: 3.3 Baselines and SViT variants
Revised: 3.3 Baselines and SViT variants

Original: Baselines .
Revised: Baselines .

Original: In the experiments, we compare SViT to several models from prior work reported for thecorresponding datasets.
Revised: In the experiments, we compare SViT to several models from prior work reported for thecorresponding datasets.

Original: These include the following approaches: I3D [13], SlowFast [22], and thestate-of-the-art transformers – Mformer [65] and MViTv2 [52].
Revised: These include the following approaches: I3D [13], SlowFast [20], and thestate-of-the-art transformers – Mformer [55] and MViTv2 [45].

Original: In addition, we provide the MViTv2multi-task (MViTv2 MT) baseline for comparing a naive multi-tasking approach to our model.
Revised: In addition, we provide the MViTv2multi-task (MViTv2 MT) baseline for comparing a naive multi-tasking approach to our model.

Original: This isa version of MViTv2 which performs video classiﬁcation and HAOG prediction from images.
Revised: This isa version of MViTv2 which performs video classification and HAOG prediction from images.

Original: The
Revised: The

Original: MViTv2 model is enhanced with two heads: the ﬁrst head is composed of an MLP that predicts thevideo labels given a ﬁnal representation vector, the second head consists of MLPs that predict the
Revised: MViTv2 model is enhanced with two heads: the first head is composed of an MLP that predicts thevideo labels given a final representation vector, the second head consists of MLPs that predict the

Original: HAOG information (i.e., hands and object boxes, and contact information).
Revised: HAOG information (i.e., hands and object boxes, and contact information).

Original: SViT Variants .
Revised: SViT Variants .

Original: As mentioned above, we use several sources of auxiliary image datasets.
Revised: As mentioned above, we use several sources of auxiliary image datasets.

Original: In somecases, we have auxiliary images that are frames from video understanding datasets.
Revised: In somecases, we have auxiliary images that are frames from video understanding datasets.

Original: For example,in Ego4D, we have both videos and frames annotated from some of these videos.
Revised: For example,in Ego4D, we have both videos and frames annotated from some of these videos.

Original: In this sense, theauxiliary data is “in-domain” for some of the video tasks.
Revised: In this sense, theauxiliary data is “in-domain” for some of the video tasks.

Original: In order to evaluate the importance of thiseffect, we explore “Different Domain” ( SViT-DD ) training, which avoids using images from a givenvideo task as auxiliary during ﬁnetuning and pretraining.
Revised: In order to evaluate the importance of thiseffect, we explore “Different Domain” ( SViT-DD ) training, which avoids using images from a givenvideo task as auxiliary during finetuning and pretraining.

Original: Similarly, “In Domain” ( SViT-ID ) refers tothe case where we use only images from the video task during ﬁnetuning, but not necessarily duringpretraining .
Revised: Similarly, “In Domain” ( SViT-ID ) refers tothe case where we use only images from the video task during finetuning, but not necessarily duringpretraining .

Original: We consider another variant that does not use the auxiliary images during ﬁnetuning atall (but does use them during pre-training), and refer to it as SViT-SFT .
Revised: We consider another variant that does not use the auxiliary images during finetuning atall (but does use them during pre-training), and refer to it as SViT-SFT .

Original: Finally, to explore the effectof data-size in the In Domain setting, we evaluate a version SViT ⇥ X% that only uses X% of thein-domain ﬁnetuning data.
Revised: Finally, to explore the effectof data-size in the In Domain setting, we evaluate a version SViT × X% that only uses X% of thein-domain finetuning data.

Original: For MViTv2 MT, the auxiliary images correspond to the “In Domain”setting (i.e., same setting as for SViT-ID). 
Revised: For MViTv2 MT, the auxiliary images correspond to the “In Domain”setting (i.e., same setting as for SViT-ID). [Table]

Original: Auxiliary Images datasets .
Revised: Auxiliary Images datasets .

Original: For SViT-DD, we generally use 100DOH and Ego4D for pretraining andﬁnetuning, while for the Ego4D tasks, we only use 100DOH.
Revised: For SViT-DD, we generally use 100DOH and Ego4D for pretraining andfinetuning, while for the Ego4D tasks, we only use 100DOH.

Original: SViT-ID uses 100DOH and Ego4D forpretraining and in-domain downstream frames for ﬁnetuning.
Revised: SViT-ID uses 100DOH and Ego4D forpretraining and in-domain downstream frames for finetuning.

Original: The SViT-SFT pretraining setting issimilar to that of SViT-DD.
Revised: The SViT-SFT pretraining setting issimilar to that of SViT-DD.

Original: SViT ⇥ X% setting is similar to that of SViT-ID.[Table] [Table]
Revised: SViT × X% setting is similar to that of SViT-ID.

Original: 3.4 Compositional & Few-Shot Action Recognition
Revised: 3.4 Compositional & Few-Shot Action Recognition

Original: Several video datasets deﬁne an action as a combination of a verb and a noun.
Revised: Several video datasets define an action as a combination of a verb and a noun.

Original: One challenge then is torecognize combinations that were not seen during training.
Revised: One challenge then is torecognize combinations that were not seen during training.

Original: This “compositional” setting was exploredin the “SomethingElse” dataset [58], where verb-noun combinations in the test data do not occur inthe training data.
Revised: This “compositional” setting was exploredin the “SomethingElse” dataset [50], where verb-noun combinations in the test data do not occur inthe training data.

Original: The split contains 174 classes with 54,919/54,876 videos for training/validation.
Revised: The split contains 174 classes with 54,919/54,876 videos for training/validation.

Original: We also evaluate on the few-shot compositional action recognition task in [58] (See supplementary).
Revised: We also evaluate on the few-shot compositional action recognition task in [50] (See supplementary).

Original: Table 1 reports the results for these two tasks.
Revised: Table 1 reports the results for these two tasks.

Original: All of our variants outperform all prior works and the
Revised: All of our variants outperform all prior works and the

Original: MViTv2 MT on the Compositional and Few-shot task.
Revised: MViTv2 MT on the Compositional and Few-shot task.

Original: From the table, we can see that using 2% ofthe in-domain data as auxiliary images (SViT ⇥ 2%), is almost comparable to using all the availablein-domain images (SViT-ID).
Revised: From the table, we can see that using 2% ofthe in-domain data as auxiliary images (SViT × 2%), is almost comparable to using all the availablein-domain images (SViT-ID).

Original: Furthermore, the results are comparable to SViT-DD (0.7 difference)when trained on auxiliary images from different domains.
Revised: Furthermore, the results are comparable to SViT-DD (0.7 difference)when trained on auxiliary images from different domains.

Original: In this way, our approach optimizes theuse of structural information across domains.
Revised: In this way, our approach optimizes theuse of structural information across domains.

Original: Last, SViT-DD outperforms SViT-SFT when using thesame data, demonstrating the value of ﬁnetuning video and image together.
Revised: Last, SViT-DD outperforms SViT-SFT when using thesame data, demonstrating the value of finetuning video and image together.

Original: 3.5 Object State Change Classiﬁcation and Localization
Revised: 3.5 Object State Change Classification and Localization

Original: Hands and objects are key elements in human activity.
Revised: Hands and objects are key elements in human activity.

Original: Two tasks related to hand-object interactionhave recently been introduced in the Ego4D [31] dataset.
Revised: Two tasks related to hand-object interactionhave recently been introduced in the Ego4D [28] dataset.

Original: The ﬁrst is temporal localization, whichis deﬁned as ﬁnding key frames in a video clip that indicate a change in object state.
Revised: The first is temporal localization, whichis defined as finding key frames in a video clip that indicate a change in object state.

Original: The second isobject state change classiﬁcation, which indicates whether an object state has changed or not.
Revised: The second isobject state change classification, which indicates whether an object state has changed or not.

Original: Table 2a reports results on the temporal localization and object state change classiﬁcation tasks onthe Ego4D dataset.
Revised: Table 2a reports results on the temporal localization and object state change classification tasks onthe Ego4D dataset.

Original: We observe that SViT-ID and SViT-DD are very close, and that both SViT-ID and
Revised: We observe that SViT-ID and SViT-DD are very close, and that both SViT-ID and

Original: SViT-DD are better than SViT-SFT.
Revised: SViT-DD are better than SViT-SFT.

Original: These results are consistent with those presented in Section 3.4,which indicates that our method successfully leverages structural information across domains, andthat a joint image and video ﬁnetuning is beneﬁcial. [Table]
Revised: These results are consistent with those presented in Section 3.4,which indicates that our method successfully leverages structural information across domains, andthat a joint image and video finetuning is beneficial.

Original: 3.6 Action Recognition
Revised: 3.6 Action Recognition

Original: Table 2b and Table 2c report the results for standard action recognition task on the SSv2 and Diving48datasets respectively.
Revised: Table 2b and Table 2c report the results for standard action recognition task on the SSv2 and Diving48datasets respectively.

Original: It can be seen that in Diving48 dataset, our method improves over the MViTv2 baseline by +6.7%, outperforming all the methods.
Revised: It can be seen that in Diving48 dataset, our method improves over the MViTv2 baseline by +6.7%, outperforming all the methods.

Original: This demonstrates that our method hassome ability to generalize the learned structural information of hand-object interactions even toﬁne-grained “human-motion” tasks such as Diving48.
Revised: This demonstrates that our method hassome ability to generalize the learned structural information of hand-object interactions even tofine-grained “human-motion” tasks such as Diving48.

Original: We also note that Diving48 has no structuredannotations available, thus SViT-ID can not be provided.
Revised: We also note that Diving48 has no structuredannotations available, thus SViT-ID can not be provided.

Original: On SSv2, we overall observe a smallerimprovement.
Revised: On SSv2, we overall observe a smallerimprovement.

Original: SViT-ID improves the baseline by 0.8% while SViT-DD improves by 0.1%.
Revised: SViT-ID improves the baseline by 0.8% while SViT-DD improves by 0.1%.

Original: As weobserved signiﬁcant improvements on the more challenging SomethingElse setup, we hypothesizethat the ability to generalize to unseen combinations of objects and hands exhibited by SViT is lessneeded on SSv2, hence leading to relatively smaller improvements.
Revised: As weobserved significant improvements on the more challenging SomethingElse setup, we hypothesize [Table] [Table]

Original: 3.7 Ablations
Revised: 3.7 Ablations

Original: We perform a comprehensive ablation study on the compositional action recognition task [58] on the
Revised: We perform a comprehensive ablation study on the compositional action recognition task [50] on the

Original: SomethingElse dataset to measure the contribution of the different SViT components (Table 3).
Revised: SomethingElse dataset to measure the contribution of the different SViT components (Table 3).

Original: Components of the SViT model .
Revised: Components of the SViT model .

Original: In Table 3a, we ablate our model to show the signiﬁcance of theindividual components.
Revised: In Table 3a, we ablate our model to show the significance of theindividual components.

Original: First, we train a standard MViTv2 in a multi-task setting (MViTv2 MT),where we use videos for video action recognition, and images for HAOG prediction.
Revised: First, we train a standard MViTv2 in a multi-task setting (MViTv2 MT),where we use videos for video action recognition, and images for HAOG prediction.

Original: There is a smallimprovement (+0.5) over the baseline.
Revised: There is a smallimprovement (+0.5) over the baseline.

Original: We then take the MViTv2 and add the object tokens andcorresponding SViT loss; this leads to a more signiﬁcant improvement (+1.7).
Revised: We then take the MViTv2 and add the object tokens andcorresponding SViT loss; this leads to a more significant improvement (+1.7).

Original: Last, when addingboth the consistency loss and the object tokens, we observe the largest improvement (+2.5).
Revised: Last, when addingboth the consistency loss and the object tokens, we observe the largest improvement (+2.5).

Original: The amount of in-domain data .
Revised: The amount of in-domain data .

Original: In Table 3b, we demonstrate the effect of the amount of in-domainimage data used during training.
Revised: In Table 3b, we demonstrate the effect of the amount of in-domainimage data used during training.

Original: Speciﬁcally, since “SomethingElse” includes HAOG annotationsfor each frame, we train our method with different amount of images ( 2% , 25% , 50% , and 100% )from the “SomthingElse” dataset.
Revised: Specifically, since “SomethingElse” includes HAOG annotationsfor each frame, we train our method with different amount of images ( 2% , 25% , 50% , and 100% )from the “SomthingElse” dataset.

Original: It can be seen that our approach takes advantage of the structuredinformation within a small amount of data and, as a result, has enhanced sample-efﬁciency.
Revised: It can be seen that our approach takes advantage of the structuredinformation within a small amount of data and, as a result, has enhanced sample-efficiency.

Original: HAOG attributes .
Revised: HAOG attributes .

Original: We use the object tokens to predict several aspects of the HAOG.
Revised: We use the object tokens to predict several aspects of the HAOG.

Original: In Table 3c weconsider several variations on this prediction.
Revised: In Table 3c weconsider several variations on this prediction.

Original: The model “SViT-Contact” does not predict the contactinformation, resulting in  0 .
Revised: The model “SViT-Contact” does not predict the contactinformation, resulting in − 0 .

Original: 3% drop compared to SViT.
Revised: 3% drop compared to SViT.

Original: The model “SViT+Geometry” adds to SViTadditional heads that predict distances between all HAOG objects (to explicitly add geometric bias).
Revised: The model “SViT+Geometry” adds to SViTadditional heads that predict distances between all HAOG objects (to explicitly add geometric bias).

Original: This does not affect performance.
Revised: This does not affect performance.

Original: Finally, we explore what happens when we do not provide theinformation about the identity of bounding boxes in the training data.
Revised: Finally, we explore what happens when we do not provide theinformation about the identity of bounding boxes in the training data.

Original: Namely, we treat them as fourboxes, and ask the model to match those to the object tokens, via a matching losss as in [12].
Revised: Namely, we treat them as fourboxes, and ask the model to match those to the object tokens, via a matching losss as in [12].

Original: This ismodel “SViT-Contact -Corresp.”
Revised: This ismodel “SViT-Contact -Corresp.”

Original: which is quite close to “SViT-Contact”, indicating that SViT canperhaps be trained with weaker labels.
Revised: which is quite close to “SViT-Contact”, indicating that SViT canperhaps be trained with weaker labels.

Original:  Related Work
Revised:  Related Work

Original: Joint Training with Images and Videos .
Revised: Joint Training with Images and Videos .

Original: Vision Transformers [20] are versatile architecturesdesigned to be ﬂexible in terms of input sequence length.
Revised: Vision Transformers [18] are versatile architecturesdesigned to be flexible in terms of input sequence length.

Original: Aside from the data “‘patchiﬁcation”’ layer,the rest of the transformer is agnostic to the domain of the input.
Revised: Aside from the data “‘patchification”’ layer,the rest of the transformer is agnostic to the modality of the input.

Original: Consequently, an exciting directionof joint training image-video has emerged in computer vision, a direction previously less natural withconvolutional architectures.
Revised: Consequently, an exciting directionof joint training image-video has emerged in computer vision, a direction previously less natural withconvolutional architectures.

Original: Speciﬁcally, this allows for training  network simultaneously on imageand video, with most of the network parameters shared across both domains.
Revised: Specifically, this allows for training a multimodal network simultaneouslyon image and video, with most of the network parameters shared across both modalities.

Original: Recently, multi-modalmodels became widely popular, from shared backbones [15, 56, 60] to separate encoders for eachinput modality [73, 76], and even non-vision modalities [38, 39].
Revised: Recently, multi-modal models became widely popular, from shared backbones [15, 49, 51] to separateencoders for each input modality [62, 65], and even non-vision modalities [34, 35].

Original: In our case, we focus on video and image input, similar to [6, 27, 86].
Revised: In our case,we focus on video and image input, similar to [6, 25, 73].

Original: In contrast to these works, our training approach utilizes twodomains, while one of them (image) is used only during training and is supervised by a different task.
Revised: In contrast to these works, our trainingapproach utilizes two modalities, while one of them (image) is used only during training and issupervised by a different task.

Original: Additionally, as we demonstrate here, we are not greatly impacted by domain mismatches betweenimages and videos, as in previous works.
Revised: Additionally, as we demonstrate here, we are not greatly impacted bydomain mismatches between images and videos, as in previous works.

Original: In a related area of research, multi-task learning [14]involves developing models that predict output for multiple tasks using a common input.
Revised: In a related area of research,multi-task learning [14] involves developing models that predict output for multiple tasks using acommon input.

Original: Our work isdifferent since it uses two inputs from different domains, images and videos.
Revised: Our work is different since it uses two inputs from different modalities, images andvideos.

Original: Similarly, our workcan be viewed as a form of auxiliary task, as already shown in many works [29, 37, 72].
Revised: Similarly, our work can be viewed as a form of auxiliary task, as already shown in manyworks [26, 33, 61].

Original: However,it differs in that the shared object prompts are utilized to learn the shared structured representationbetween video and image domains, which enhances the self-attention layers with the structuredrepresentation for the main video  task.
Revised: However, it differs in that it uses a novel share structural component and a novelloss, which involves explicit interactions between the main video task and the auxiliary image task.

Original: Learning from video using other modalities .
Revised: Learning from video using other modalities .

Original: Researchers have proposed different approachesto learn from videos using audio and natural language [1, 3, 19, 61, 63, 64, 67, 81].
Revised: Researchers have proposed different approachesto learn from videos using audio and natural language [1, 3, 17, 52, 53, 54, 57, 69].

Original:  Vision taskslike Video Captioning [19, 24, 64, 81] and Visual Quesion Answering [2, 78] require understandingof both visual and natural language.Another line of vision and language works explored objectlabels, supervision of textual descriptions, or the use of region crops [49, 51], in contrast to SViT,which does not require any of these components.
Revised: Using bothaudio and video, it is possible to learn good video representations [3, 53], localize objects [1], andpredict human gestures [23]. Vision tasks like Video Captioning [17, 22, 54, 69] and Visual Quesion Answering [2, 67] require understanding of both visual and natural language.

Original: Additionally, Since high quality captions areexpensive, researchers have proposed self-supervised ways to learn from both modalities withoutlabeled data [61, 60, 67].
Revised: Since high qualitycaptions are expensive, researchers have proposed self-supervised ways to learn from both modalitieswithout labeled data [52, 51, 57].

Original: In these previous works, the two modalities must be consistent and aligned(e.g. , text, audio, and RGB should be consistent for same sample).
Revised: In these previous works, the two modalities must be consistent andaligned (e.g., text, audio, and RGB should be consistent for same sample).

Original: However, in our work, we do notrequire that the annotated images be precisely aligned with our downstream video task.
Revised: However, in our work, wedo not require that the annotated images be precisely aligned with our downstream video task.

Original: Last, othernew works [16, 69] explored different structured representations for videos, different from what wedo, which is to leverage structure from images to videos.

Original: Structured Models .
Revised: Structured models .

Original: Recently structured models have been successfully applied in many computervision applications: video relation detection [54, 70, 75], vision and language [17, 50, 51, 77],relational reasoning [8, 9, 35, 47, 40, 66, 84, 85], human-object interactions [23, 43, 83], and evenimage & video generation [7, 32, 42].A recent line of works [5, 25, 26, 33, 34, 41, 57, 59, 62, 74, 79,80, 82, 88] focuses on capturing spatio-temporal interactions for video understanding. Some of theseworks [33, 80, 82, 88] used static image object annotations to build the representations. However,these works differ from ours.
Revised: Recently structured models have been successfully applied in many computervision applications: video relation detection [47, 59, 64], vision and language [16, 43, 44, 66],relational reasoning [8, 9, 32, 41, 36, 56, 71, 72], human-object interactions [21, 39, 70], and evenimage & video generation [7, 29, 38].

Original: [82] proposed the Object Transformers (OT) forlong form videounderstanding, which model a video as a set of objects by extracting region crops from SlowFast [22]based on bounding boxes from an external detector. Contrary to this work, our transformer containsboth patch tokens and object tokens, whereas the OT contains only object tokens. Additionally, ourmodel learns the object-centric representation without the need for an external detector. Anotherwork [88], proposed a structured model for action detection that explicitly utilizes image annotations.
Revised: A recent line of work [5, 24, 30, 31, 37, 63, 68] focuses oncapturing spatial-temporal interactions for video understanding, which heavily relies on frame-levelannotations.

Original: They use a GCN [46] to reason about actors and objects interaction, on top ofan I3D network. Unlikethis work, we examine how image annotations can be exploited as part of a video transformer model.
Revised: Contrary to this line of work, we study how to leverage a small set of structuredannotations from images, both in-domain and outside the downstream task domain.

Original: Video Understanding Models .
Revised: Video Understanding Models .

Original: We work on several well-benchmarked down-stream video tasks suchas compositional action recognition on Something-Else dataset [58] and action recognition with focuson temporal cues de-emphasizing appearance such as in Something-Something V2 dataset [30] and
Revised: We work on several well-benchmarked down-stream video tasks suchas compositional action recognition on Something-Else dataset [50] and action recognition with focuson temporal cues de-emphasizing appearance such as in Something-Something V2 dataset [27] and

Original: Diving48 [53].
Revised: Diving48 [46].

Original: Several recent video transfromer works performed well on SSv2, such as ViViT [4],
Revised: Several recent video transfromer works performed well on SSv2, such as ViViT [4],

Original: MViT [21], MFormer [65], TimeSformer [11], MViTv2 [52],and Video Swin [55].
Revised: MViT [19], MFormer [55], TimeSformer [11], MViTv2 [45],and Video Swin [48].

Original: We choose towork with MViTv2, although our method can be used on top of any of these.
Revised: We choose towork with MViTv2, although our method can be used on top of any of these.

Original: Additionally, wealso adapt our proposed method to interesting new tasks on the Ego4D dataset [31].
Revised: Additionally, wealso adapt our proposed method to interesting new tasks on the Ego4D dataset [28].

Original: We believesuch ﬁne-grained video understanding tasks require richer scene and object semantic reasoning, arequirement well afforded by our proposed Hand-Object Graph.
Revised: We believesuch fine-grained video understanding tasks require richer scene and object semantic reasoning, arequirement well afforded by our proposed Hand-Object Graph.

Original:  Conclusion
Revised:  Conclusion

Original: Video understanding is a key element of human visual perception, but modeling remains a challenge for machine vision.
Revised: Video understanding is a key element of human visual perception, but modeling remains a challenge for machine vision.

Original: In this work, we demonstrated the importance of learning from a scenestructure from a small set of images to facilitate video learning within or outside the domain ofinterest.
Revised: In this work, we demonstrated the importance of learning from a scenestructure from a small set of images to facilitate video learning within or outside the domain ofinterest.

Original: According to our empirical study, our SViT approach improves performance on three videounderstanding tasks.
Revised: According to our empirical study, our SViT approach improves performance on three videounderstanding tasks.

Original: Note, that we did not prioritize making the HAOG annotations richer, i.e., itmay be possible to add other physical properties to improve structure modeling.
Revised: Note, that we did not prioritize making the HAOG annotations richer, i.e., itmay be possible to add other physical properties to improve structure modeling.

Original: In addition, wenoticed that leveraging the structure is effective even with 2% of the in-domain frames, as well aswhen applying it from outside the domain.
Revised: In addition, wenoticed that leveraging the structure is effective even with 2% of the in-domain frames, as well aswhen applying it from outside the domain.

Original: We leave for future work how different physical propertiescan be more effectively utilized in more complex structures.
Revised: We leave for future work how different physical propertiescan be more effectively utilized in more complex structures.

Original: We do not anticipate a speciﬁc negativeimpact, but, as with any Machine Learning method, we recommend to exercise caution.
Revised: We do not anticipate a specific negativeimpact, but, as with any Machine Learning method, we recommend to exercise caution.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.
Revised: [No] The code and the data are proprietary.

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes](c)
Revised: [Yes]

Original:  Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See line341-342.(d)
Revised: [Yes] See line

Original:  Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]2.
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b)
Revised: [N/A](b)

Original: Did you include complete proofs of all theoretical results?
Revised: Did you include complete proofs of all theoretical results?

Original: [N/A]3.
Revised: [N/A]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?

Original: [Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes](c)
Revised: [Yes]

Original:  Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?

Original: Yes
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] Sec Section 3.1 (theDataset Section).
Revised: [Yes] Sec Section 3.1 (the Dataset Section).

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See the Dataset Section in supplementary.
Revised: [Yes] See the Dataset Section in supplemen-

Original:  (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: tary. (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [Yes] See the Dataset Section in supplementary.
Revised: [Yes] See the Dataset Section in supplementary.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes] See the Dataset Section in supplementary.
Revised: [Yes] See the Dataset Section in supplementary.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]