Original: Bringing ImageScene Structure to Video  via Frame-Clip Consistency of Object Tokens
Revised: Bringing Image Structure to Video Transformers via Frame-Clip Consistency of Object Tokens

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Recent action recognition models have achieved impressive results by integratingobjects, their locations and interactions.
Revised: Recent action recognition models have achieved impressive results by integratingobjects, their locations and interactions.

Original: However, obtaining dense structuredannotations for each frame is tedious and time-consuming, making these methodsexpensive to train and less scalable.
Revised: However, obtaining dense structuredannotations for each frame is tedious and time-consuming, making these methodsexpensive to train and less scalable.

Original: At the same time, if a small set of annotatedimages is available, either within or outside the domain of interest, how could weleverage these for a video downstream task?
Revised: At the same time, if a small set of annotatedimages is available, either within or outside the domain of interest, how could weleverage these for a video downstream task?

Original: We propose a learning framework
Revised: We propose a learning framework

Original: StructureViT (SViT for short), which demonstrates how utilizing the structure of asmall number of images only available during training can improve a video model.
Revised: StructureViT (SViT for short), which demonstrates how utilizing the structure of asmall number of images only available during training can improve a video model.

Original: SViT relies on two key insights.
Revised: SViT relies on two key insights.

Original: First, as both images and videos contain structuredinformation, we enrich a transformer model with a set of object tokens that can beused across images and videos.
Revised: First, as both images and videos contain structuredinformation, we enrich a transformer model with a set of object tokens that can beused across images and videos.

Original: Second, the scene representations of individualframes in video should “align” with those of still images.
Revised: Second, the scene representations of individualframes in video should “align” with those of still images.

Original: This is achieved via a
Revised: This is achieved via a

Original: Frame-Clip Consistency loss, which ensures the flow of structured informationbetween images and videos.
Revised: Frame-Clip Consistency loss, which ensures the flow of structured informationbetween images and videos.

Original: We explore a particular instantiation of scene structure,namely a Hand-Object Graph , consisting of hands and objects with their locationsas nodes, and physical relations of contact/no-contact as edges.
Revised: We explore a particular instantiation of scene structure,namely a Hand-Object Graph , consisting of hands and objects with their locationsas nodes, and physical relations of contact/no-contact as edges.

Original: SViT shows strongperformance improvements on multiple video understanding tasks and datasets.
Revised: SViT shows strongperformance improvements on multiple video understanding tasks and datasets.

Original:  Introduction
Revised:  Introduction

Original: Semantic understanding of videos is a key challenge for machine vision and artificial intelligence.
Revised: Semantic understanding of videos is a key challenge for machine vision and artificial intelligence.

Original: Itis intuitive that video models should benefit from incorporating scene structure, e g., the objects thatappear in a video, their attributes and the way they interact.
Revised: Itis intuitive that video models should benefit from incorporating scene structure, e g., the objects thatappear in a video, their attributes and the way they interact.

Original: In fact, several works  have shown thatsuch “object-centric” models perform well on tasks such as action recognition [31, 71].
Revised: Indeed, several works [5, 25, 26, 33, 34,41, 57, 59, 62, 74, 79, 80, 82, 88] have demonstrated that incorporating structured representationsinto models improves both performance and sample efficiency.

Original: However, mostof these models require ground-truth structured  annotations of video during training.
Revised: In particular, [33, 80, 88] showed howto incorporate structured representations by utilizing static image object annotations in videos. Recently, vision transformers (ViT) [20] have emerged as the leading model for many vision applications [4, 21, 12]. This begs the question, how can structured representations be used to leveragevideo transformers for video tasks? Past works [4, 21] have proposed video transformer modelsbut these approaches were not focused on learning structured representations.

Original: For example,frame-level ground-truth object bounding boxes are considered available in [5, 7, 31, 32, 52, 71].
Revised: A recent line ofworks [33, 80, 88] utilized objects for structured approaches. [82] leveraged the decoder with CNNsregion crop features from SlowFast [22], and [33] proposed a new block that incorporates objectrepresentations.

Original: This is clearly very costly, time-consuming, and  not scalable.
Revised: However, these approaches are limited because they use external pretrained detectorsfor the frames, which require additional processing time (also during inference), they ignore thepotential benefits of the connection between the detection and the other tasks, and it is not clear howto use them with more detailed structured annotations.

Original: This begs the question, could we still benefit from scene structure in a less costly way? A possibledirection is to explore methods that would only require sparsely annotated frames [29, 63] within adownstream video domain.
Revised: Finally, [88] proposed a structured modelfor action detection that explicitly utilizes image annotations, but not in a transformer architecture. Our approach, described next, builds on these prior works and proposes an effective way to leveragescene structure in video transformers, using static image supervision. Most structured transformer-based models for videos require structured annotations during training.

Original: Moreover, when it comes to structured annotations of static images, thereare numerous datasets with annotations such as boxes, visual relationships, attributes, and segmentations [10, 43, 52].
Revised: For example, frame-level object bounding boxes are considered available in [5, 7, 33, 34, 58, 80]. [Figure] [Figure] When it comes to structured annotations of static images, there are numerous datasets with annotationssuch as boxes, visual relationships, attributes, and segmentations [10, 48, 58].

Original: Since we can not always expect to have annotated images that perfectly align withour downstream video task, how can these resources be used to build better  video models?
Revised: Since we cannot alwaysexpect to have annotated images that perfectly align with our downstream video task, how can theseresources be used to build better transformer-based video models?

Original: In thiswork, we propose such an approach, which is particularly well suited for transformer architecturesand offers an effective mechanism to leverage image structure to improve video transformers.
Revised: In this work, we propose suchan approach, which is particularly designed for transformer architectures and offers an effectivemechanism to leverage image structure to improve video transformers.

Original: A natural first step towards image-video knowledge sharing is to use the same transformer model toprocess both: namely, to view images as “single frame videos”, and feed them to a video transformermodel.
Revised: A natural first step towards image-video knowledge sharing is to use the same transformer model toprocess both: namely, to view images as “single frame videos”, and feed them to a video transformermodel.

Original: However, this still leaves two key questions: how to model structured information, and how Shared Video & Image Transformer [Figure] s
Revised: However, this still leaves two key questions: how to model structured information, and howto account for domain mismatch between images and videos.

Original: [Figure]

Original: Hand-Object Graph [Figure] to account for domain mismatch between images and videos. Towards this end, we introduce twokey concepts: (i) A set of transformer “object-tokens” – additional tokens initialized from learnedembeddings (we refer to these embeddings as “object prompts”) that are meant to capture objectcentric information in both still image and video.
Revised: Towards this end, we introduce twokey concepts: (i) A set of transformer “object-tokens” – additional tokens initialized from learnedembeddings (we refer to these embeddings as “object prompts”) that are meant to capture objectcentric information in both still image and video.

Original: These tokens can be supervised based on the scenestructure of images, and that information will also be propagated to videos.
Revised: These tokens can be supervised based on the scenestructure of images, and that information will also be propagated to videos.

Original: To formalize the imagestructure, we proposed the HAnd-Object Graph (HAOG), a simple representation of the interactionsbetween hands and objects in the scene.
Revised: To formalize the imagestructure, we proposed the HAnd-Object Graph (HAOG), a simple representation of the interactionsbetween hands and objects in the scene.

Original: (ii)
Revised: (ii)

Original: A novel “Frame-Clip Consistency loss” that ensuresconsistency between the “object-tokens” when they are part of a video vs. a still image.
Revised: A novel “Frame-Clip Consistency loss” that ensuresconsistency between the “object-tokens” when they are part of a video vs. a still image.

Original: Moreover, aswe show in our experiments, our approach does not require any alignment between object categoriesappearing in still images and the ones appearing in the videos of the target downstream task.
Revised: Moreover, aswe show in our experiments, our approach does not require any alignment between object categoriesappearing in still images and the ones appearing in the videos of the target downstream task.

Original: We nameour proposed approach StructureViT ( Bringing Scene Structure from Images to Vi deo via Frame-Clip
Revised: We nameour proposed approach StructureViT ( Bringing Scene Structure from Images to Vi deo via Frame-Clip

Original: Consistency of Object T okens , or SViT for short).
Revised: Consistency of Object T okens , or SViT for short).

Original: See Figure 1 for an overview.
Revised: See Figure 1 for an overview.

Original: Prior work has explored several other video & image models.
Revised: Prior work has explored several other video & image models.

Original: The I3D model [13] was one of the firstsuccessful attempts to leverage ImageNet architectures in video.
Revised: The I3D model [13] was one of the firstsuccessful attempts to leverage ImageNet architectures in video.

Original: They proposed to simply convertimage 2D classification models into 3D ConvNets by inflating all the filters and pooling kernels.
Revised: They proposed to simply convertimage 2D classification models into 3D ConvNets by inflating all the filters and pooling kernels.

Original: Here, our focus is on learning the shared structured representations between the two domains in orderto transfer this knowledge into videos.
Revised: Here, our focus is on learning the shared structured representations between the two domains in orderto transfer this knowledge into videos.

Original: Several other methods have explored simultaneous image andvideo training in the context of multi-task [6] and multi-modal [26] learning.
Revised: Several other methods have explored simultaneous image andvideo training in the context of multi-task [6] and multi-modal [27] learning.

Original: However, these worksdid not utilize structural information, nor did they promote consistency between image and videopredictions – which, as we show, leads to significant gains in downstream video task performance.
Revised: However, these worksdid not utilize structural information, nor did they promote consistency between image and videopredictions – which, as we show, leads to significant gains in downstream video task performance.

Original: To summarize, our main contributions are as follows: (i) we propose a new method for exploitingstructured information present in images in order to improve performance of video understandingtasks; (ii) we propose a novel concept of “object tokens” - a form of learned prompts to a video &image transformer, which during training are associated to the available structure representation ofthe still images (HAOG).
Revised: To summarize, our main contributions are as follows: (i) we propose a new method for exploitingstructured information present in images in order to improve performance of video understandingtasks; (ii) we propose a novel concept of “object tokens” - a form of learned prompts to a video &image transformer, which during training are associated to the available structure representation ofthe still images (HAOG).

Original: (iii) we introduce a new Frame-Clip Consistency loss, which promotesconsistency between image-level and video-level predictions made by a shared transformer backbone.
Revised: (iii) we introduce a new Frame-Clip Consistency loss, which promotesconsistency between image-level and video-level predictions made by a shared transformer backbone.

Original: We show how this loss helps to drive performance improvements in downstream video tasks even incases when image labels are unrelated to the video task; (iv) we demonstrate improved performanceon several video understanding benchmarks, highlighting the effectiveness of the proposed approach.
Revised: We show how this loss helps to drive performance improvements in downstream video tasks even incases when image labels are unrelated to the video task; (iv we demonstrate improved performance on several video understanding benchmarks highlighting the effectiveness of the proposed approach

Original:  The SViT Approach
Revised:  The SViT Approach

Original: Our approach SViT learnsa structuredshared representation that arises from the interaction betweenthe two domain of images and videos.
Revised: Our approach SViT learns structured representation that is exhibited in both static images and video.

Original: We consider the setting where the main goal is to learn video[Figure] understanding downstream tasks while leveraging structured image data.
Revised: We consider the setting where the main goal is to learn video understanding downstream taskswhile leveraging structured image data.

Original: In this paper, we focus onthe following video tasks: action recognition and object state classification & localization.
Revised: In this paper, we focus on the following video tasks: actionrecognition and object state classification & localization.

Original: In training,we assume that we have access to task-specific video labels and structured scene annotations forimages, Hand-Object Graph s. Based on the structured representations obtained by using the objecttokens and regularized by the Frame-Clip Consistency loss , the inference is performed only for thedownstream video task without explicitly using the structured image annotations.
Revised: In training, we assume that we have accessto task-specific video labels and structured scene annotations for images, Hand-Object Graph s. Based on the structured representations obtained by using the object tokens and regularized by the Frame-Clip Consistency loss , the inference is performed only for the downstream video task withoutexplicitly using the structured image annotations.

Original: Further, we begin by describing the Hand-Object Graph (HAOG) (Section 2.1), then introduce our
Revised: Further, we begin by describing the Hand-Object Graph (HAOG) (Section 2.1), then introduce our

Original: SViT model (Section 2.2), and the Frame-Clip Consistency loss function (Section 2.3).
Revised: SViT model (Section 2.2), and the Frame-Clip Consistency loss function (Section 2.3).

Original: Our methodis schematically illustrated in Figure 2.
Revised: Our methodis schematically illustrated in Figure 2.

Original: 2.1 The Hand-Object Graph
Revised: 2.1 The Hand-Object Graph

Original: As mentioned above, our motivation is to bring scene structure from still images into video.
Revised: As mentioned above, our motivation is to bring scene structure from still images into video.

Original: Inorder to achieve this, one component of this work is the construction of a semantic representationof the interactions between the hands and objects in the scene.
Revised: Inorder to achieve this, one component of this work is the construction of a semantic representationof the interactions between the hands and objects in the scene.

Original: Specifically, we propose to usea graph-structure
Revised: Specifically, we propose to usea graph-structure

Original: we call Hand-Object Graph (HAOG), see Figure 1.
Revised: we call Hand-Object Graph (HAOG), see Figure 1.

Original: The nodes of the HAOGrepresent two hands and two objects with their locations, whereas the edges correspond to physicalproperties such as contact/no contact.
Revised: The nodes of the HAOGrepresent two hands and two objects with their locations, whereas the edges correspond to physicalproperties such as contact/no contact.

Original: Formally, an HAOG is a tuple ( O, E ) defined as follows:
Revised: Formally, an HAOG is a tuple ( O, E ) defined as follows:

Original: Nodes O : A set O of n objects.
Revised: Nodes O : A set O of n objects.

Original: Human-object interaction scenes, which we focus on in this work,usually consist of up to two hands and objects.
Revised: Human-object interaction scenes, which we focus on in this work,usually consist of up to two hands and objects.

Original: Thus our node categories are “hand” and “object”;they may also contain attributes (e.g., spatial coordinates).
Revised: Thus our node categories are “hand” and “object”;they may also contain attributes (e.g., spatial coordinates).

Original: We assume that each image comes withthe bounding boxes describing the locations of the two hands, and the objects they interact with.
Revised: We assume that each image comes withthe bounding boxes describing the locations of the two hands, and the objects they interact with.

Original: Thebounding boxes for the left and right hand are denoted as b 1 , b 2 ∈ R 4 , respectively, while for the twocorresponding objects interacting with the hands are denoted as b 3 , b 4 .
Revised: Thebounding boxes for the left and right hand are denoted as b 1 , b 2 ∈ R 4 , respectively, while for the twocorresponding objects interacting with the hands are denoted as b 3 , b 4 .

Original: We also assume access tofour binary variables e 1 , . .
Revised: We also assume access tofour binary variables e 1 , . .

Original: , e 4 ∈ { 0 , 1 } that indicate whether the corresponding hands or objects infact exist in the image (e.g., if e 1 = 

Revised: , e 4 ∈ { 0 , 1 } that indicate whether the corresponding hands or objects infact exist in the image (e.g., if e 1 =

Original: the left hand is not in the image, and b 1 should be ignored).
Revised: the left hand is not in the image, and b 1 should be ignored).

Original: Edges E : The edges are represented as labeled directed edges between hand and object nodes.
Revised: Edges E : The edges are represented as labeled directed edges between hand and object nodes.

Original: Edgevocabulary describes the physical properties of the scene.
Revised: Edgevocabulary describes the physical properties of the scene.

Original: In our current modeling, each edge ischaracterized by a physical property of “contact” or “no-contact”.
Revised: In our current modeling, each edge ischaracterized by a physical property of “contact” or “no-contact”.

Original: We denote two binary variablesc 1 , c 2 ∈ { 0 , 1 } that specify if each of the hands is in direct contact with the corresponding object (theleft hand b 1 can only interact with the object b 3 , similar for the right hand).
Revised: We denote two binary variablesc 1 , c 2 ∈ { 0 , 1 } that specify if each of the hands is in direct contact with the corresponding object (theleft hand b 1 can only interact with the object b 3 , similar for the right hand).

Original: In principle, we can alsoinclude properties such as distances between objects, 3D hand poses, etc., where available.
Revised: In principle, we can alsoinclude properties such as distances between objects, 3D hand poses, etc., where available.

Original: We note that images annotated consistent with our defined HAOG format can be found in severaldatasets [29, 52, 63], and we leverage these in our experiments.
Revised: We note that images annotated consistent with our defined HAOG format can be found in severaldatasets [31, 58, 71], and we leverage these in our experiments.

Original: 2.2 The SViT Transformer Model
Revised: 2.2 The SViT Transformer Model

Original: We next present the SViT model: a video transformer that employs object tokens as a means to capturestructure in both images and videos.
Revised: We next present the SViT model: a video transformer that employs object tokens as a means to capturestructure in both images and videos.

Original: We begin by reviewing the video transformer architecture, whichour model extends.
Revised: We begin by reviewing the video transformer architecture, whichour model extends.

Original: Next, we explain how we process both images and videos.
Revised: Next, we explain how we process both images and videos.

Original: Finally, we describeobject tokens and how they are used in videos and still images.
Revised: Finally, we describeobject tokens and how they are used in videos and still images.

Original: Shared Video & Image Transformer .
Revised: Shared Video & Image Transformer .

Original: Video transformers [4, 11, 20, 31] extend the Vision Transformer model to the temporal domain.
Revised: Video transformers [4, 11, 21, 33] extend the Vision Transformer model to the temporal domain.

Original: Similar to vision transformers, the input is first “patchified” butwith temporally extended 3-D convolution (instead of 2-D) producing a down-sampled tensor X ofsize T × H × W × d .
Revised: Similar to vision transformers, the input is first “patchified” butwith temporally extended 3-D convolution (instead of 2-D) producing a down-sampled tensor X ofsize T × H × W × d .

Original: We refer to this as “patch tokens”.
Revised: We refer to this as “patch tokens”.

Original: Then, spatio-temporal position embeddingsare added for providing location and time information.
Revised: Then, spatio-temporal position embeddingsare added for providing location and time information.

Original: Next, multiple stacked self-attention blocksare applied repeatedly on the down-sampled tensor X to produce the final vector representation usingmean pooling over the patch tokens.
Revised: Next, multiple stacked self-attention blocksare applied repeatedly on the down-sampled tensor X to produce the final vector representation usingmean pooling over the patch tokens.

Original: In our approach, we want to be able to process batches of images or videos.
Revised: In our approach, we want to be able to process batches of images or videos.

Original: A key desideratum inthis context is to be able to input both videos and still-images into the same model.
Revised: A key desideratum inthis context is to be able to input both videos and still-images into the same model.

Original: Towards this end,our first change is to rely only on 2-D convolutions in the first transformer step instead of 3-D. Thisallows single images to be used as inputs without padding, and does not hurt performance in practice. 
Revised: Towards this end,our first change is to rely only on 2-D convolutions in the first transformer step instead of 3-D. Thisallows single images to be used as inputs without padding, and does not hurt performance in practice. [Figure]

Original: Object tokens .
Revised: Object tokens .

Original: Our goal is to learn a shared structured representation between the video and theimage domains.
Revised: Our goal is to learn a shared structured representation between the video and theimage domains.

Original: We do this by adding transformer tokens to represent objects.
Revised: We do this by adding transformer tokens to represent objects.

Original: These tokens arefunctionally similar to the patch tokens, with two exceptions.
Revised: These tokens arefunctionally similar to the patch tokens, with two exceptions.

Original: First, they are initialized with additionaltrained embeddings, which we referred to as “object prompts”.
Revised: First, they are initialized with additionaltrained embeddings, which we referred to as “object prompts”.

Original: We consider the “object prompts” asa bridge between the images and the videos since they are being used in prediction tasks across bothdomains.
Revised: We consider the “object prompts” asa bridge between the images and the videos since they are being used in prediction tasks across bothdomains.

Original: Second, they are used to predict the structured representations when those are available(here, we use the Hand-Object Graphs structure for still images).
Revised: Second, they are used to predict the structured representations when those are available(here, we use the Hand-Object Graphs structure for still images).

Original: An object token is used to predictproperties of the corresponding node in the HAOG, and the concatenation of two object tokens isused to predict edge properties.
Revised: An object token is used to predictproperties of the corresponding node in the HAOG, and the concatenation of two object tokens isused to predict edge properties.

Original: Next, we describe the model more formally.
Revised: Next, we describe the model more formally.

Original: Let n be the maximum number of modeled objects perframe (or still image).
Revised: Let n be the maximum number of modeled objects perframe (or still image).

Original: We have n tokens for each frame, and thus a total of T × n tokens.
Revised: We have n tokens for each frame, and thus a total of T × n tokens.

Original: The tokenfor object i at frame t is initialized with the vector o i + r t where o i ∈
Revised: The tokenfor object i at frame t is initialized with the vector o i + r t where o i ∈

Original: R
Revised: R

Original: d is a learned object promptand r t ∈ R d is a learned temporal positional embedding (the same one used for initializing the patchtokens).
Revised: d is a learned object promptand r t ∈ R d is a learned temporal positional embedding (the same one used for initializing the patchtokens).

Original: With these new tokens, we have T × ( H × W + n ) tokens (i.e, vectors in R d ), and thesetogether will go through the standard self-attention layers.
Revised: With these new tokens, we have T × ( H × W + n ) tokens (i.e, vectors in R d ), and thesetogether will go through the standard self-attention layers.

Original: We denote the operator that outputs the final representation of the object tokens by F O , where fora video V we have F O ( V ) ∈ R T × n × d and for an image
Revised: We denote the operator that outputs the final representation of the object tokens by F O , where fora video V we have F O ( V ) ∈ R T × n × d and for an image

Original: I we have F O ( I ) ∈ R n × d .
Revised: I we have F O ( I ) ∈ R n × d .

Original: The F O ( I )representation is used to predict structured representations for single images.
Revised: The F O ( I )representation is used to predict structured representations for single images.

Original: We also include a lossthat makes single frame representations align with those of clip representations (see Section 2.3).
Revised: We also include a lossthat makes single frame representations align with those of clip representations (see Section 2.3).

Original: Theoperator that outputs the final representation vector used for the video downstream task is denoted by
Revised: Theoperator that outputs the final representation vector used for the video downstream task is denoted by

Original: F CLS , where for video V we have F CLS ( V ) ∈ R d .
Revised: F CLS , where for video V we have F CLS ( V ) ∈ R d .

Original: Finally, our method can be used on top of the most common video transformers (MViT [20],
Revised: Finally, our method can be used on top of the most common video transformers (MViT [21],

Original: TimeSformer [11], Mformer [57], Video Swin [50]).
Revised: TimeSformer [11], Mformer [65], Video Swin [55]).

Original: For our experiments, we use the MViTv2 [47]model because it performs well empirically.
Revised: For our experiments, we use the MViTv2 [52]model because it performs well empirically.

Original: 2.3 Losses and Training
Revised: 2.3 Losses and Training

Original: During training we have a set of images annotated with HAOGs and videos with downstream tasklabels.
Revised: During training we have a set of images annotated with HAOGs and videos with downstream tasklabels.

Original: We use these as inputs to our model (Section 2.2) and optimize the losses described below.
Revised: We use these as inputs to our model (Section 2.2) and optimize the losses described below.

Original: Video loss .
Revised: Video loss .

Original: Each training video V has a corresponding ground-truth category Y ∈ { 1 , . .
Revised: Each training video V has a corresponding ground-truth category Y ∈ { 1 , . .

Original: . , K } .
Revised: . , K } .

Original: As noted above, we have a vector representation for the entire video denoted by F CLS ( V ) .
Revised: As noted above, we have a vector representation for the entire video denoted by F CLS ( V ) .

Original: Thus,we simply use a neural network to predict logits for Y .
Revised: Thus,we simply use a neural network to predict logits for Y .

Original: Specifically, our predicted logits are:ˆ Y =
Revised: Specifically, our predicted logits are:ˆ Y =

Original: FC ( F CLS ( V ))
Revised: FC ( F CLS ( V ))

Original: ∈ R K , where FC is a trainable fully connected network.
Revised: ∈ R K , where FC is a trainable fully connected network.

Original: We then consider astandard cross-entropy between the predicted logits ˆ Y and true labels Y .
Revised: We then consider astandard cross-entropy between the predicted logits ˆ Y and true labels Y .

Original: [Equation]
Revised: [Equation]

Original: HAOG loss .
Revised: HAOG loss .

Original: The transformer outputs a set of n object tokens for each image.
Revised: The transformer outputs a set of n object tokens for each image.

Original: Since we havesupervision for these objects, we predict structured representations from the tokens, and introduce a loss to optimize these predictions.
Revised: Since we havesupervision for these objects, we predict structured representations from the tokens, and introduce aloss to optimize these predictions.

Original: Recall our notation from Section 2.1.
Revised: Recall our notation from Section 2.1.

Original: We let the transformer usen = 4 object tokens which correspond to the two hands (bounding boxes b 1 , b 2 ) and two manipulatedobjects (bounding boxes b 3 , b 4 ).
Revised: We let the transformer usen = 4 object tokens which correspond to the two hands (bounding boxes b 1 , b 2 ) and two manipulated objects (bounding boxes b 3 , b 4 ).

Original: The j th token is used to predict a corresponding bounding box b j ,as well as the existence variable e j .
Revised: The j th token is used to predict a corresponding bounding box b j ,as well as the existence variable e j .

Original: Formally, we let:
Revised: Formally, we let:

Original: [Equation]
Revised: [Equation]

Original: where FC bb , FC e are fully connected networks with four and one output respectively, F jO ( I ) is thej th object token, and I is the input image.
Revised: where FC bb , FC e are fully connected networks with four and one output respectively, F jO ( I ) is thej th object token, and I is the input image.

Original: Formally, this define the loss of the nodes as following:
Revised: Formally, this define the loss of the nodes as following:

Original: [Equation]
Revised: [Equation]

Original: where L1 is the standard L1 distance, and GIoU is used as in [60].
Revised: where L1 is the standard L1 distance, and GIoU is used as in [68].

Original: Furthermore, we predict thecontact variable from the concatenation of the corresponding hand and object tokens.
Revised: Furthermore, we predict thecontact variable from the concatenation of the corresponding hand and object tokens.

Original: Namely forj ∈ { 1 , 2 } :
Revised: Namely forj ∈ { 1 , 2 } :

Original: [Equation]
Revised: [Equation]

Original: where FC c is a fully connected with one output.
Revised: where FC c is a fully connected with one output.

Original: Thus, we define the loss of the edges as follows:
Revised: Thus, we define the loss of the edges as follows:

Original: [Equation]
Revised: [Equation]

Original: Finally, to obtain the loss, we compare the predictions and ground-truth labels as follows:
Revised: Finally, to obtain the loss, we compare the predictions and ground-truth labels as follows:

Original: [Equation]
Revised: [Equation]

Original: Frame-Clip Consistency Loss .
Revised: Frame-Clip Consistency Loss .

Original: Since we have different losses for images and video, the modelcould find a way to minimize the image loss in a way that only applies to the images and does nottransfer to video.
Revised: Since we have different losses for images and video, the modelcould find a way to minimize the image loss in a way that only applies to the images and does nottransfer to video.

Original: 1 To avoid this, we need to make sure that the video representation contains thesame type of information as the still images, and in particular the structured information carried bythe object tokens.
Revised: 1 To avoid this, we need to make sure that the video representation contains thesame type of information as the still images, and in particular the structured information carried bythe object tokens.

Original: To achieve this, we explicitly enforce the object tokens to be consistent acrossstill images (frames) and videos (clips) using a frame-clip Consistency loss.
Revised: To achieve this, we explicitly enforce the object tokens to be consistent acrossstill images (frames) and videos (clips) using a frame-clip Consistency loss.

Original: To calculate the loss,we process each video twice: once as a clip, and once as a batch of T frames.
Revised: To calculate the loss,we process each video twice: once as a clip, and once as a batch of T frames.

Original: Namely, a clip Vconsisting of T frames I 1 , ..., I T , will be processed once as F O ( V ) and once as a list of images( F O ( I 1 ) , ..., F O ( I T )) .
Revised: Namely, a clip Vconsisting of T frames I 1 , ..., I T , will be processed once as F O ( V ) and once as a list of images( F O ( I 1 ) , ..., F O ( I T )) .

Original: This results in two groups of T × n object tokens, the first are the clip objecttokens , denoted as, O clip :=
Revised: This results in two groups of T × n object tokens, the first are the clip objecttokens , denoted as, O clip :=

Original: F O ( V ) ∈ R T × n × d , the second are the frame object tokens , denotedas O frames :=
Revised: F O ( V ) ∈ R T × n × d , the second are the frame object tokens , denotedas O frames :=

Original: [ F O ( v 1 ) , ..., F O ( v T )] ∈ R T × n × d .
Revised: [ F O ( v 1 ) , ..., F O ( v T )] ∈ R T × n × d .

Original: Since both groups originate from the same input,each clip object token has a single corresponding frame object token.
Revised: Since both groups originate from the same input,each clip object token has a single corresponding frame object token.

Original: Namely, there is an alignmentbetween the clip object tokens and the frame object tokens.
Revised: Namely, there is an alignmentbetween the clip object tokens and the frame object tokens.

Original: Following the intuition that each clipobject token should contain the information from its corresponding frame object token and potentiallymore (the temporal context), we minimize the L 1 distance between each clip object token and frameobject token, after applying a fully-connected layer to each clip object token.
Revised: Following the intuition that each clipobject token should contain the information from its corresponding frame object token and potentiallymore (the temporal context), we minimize the L 1 distance between each clip object token and frameobject token, after applying a fully-connected layer to each clip object token.

Original: Namely:
Revised: Namely:

Original: [Equation]
Revised: [Equation]

Original: Overall loss .
Revised: Overall loss .

Original: The total loss consists of L Con , L HAOG and L V id , where L V id is the loss of the maintask.
Revised: The total loss consists of L Con , L HAOG and L V id , where L V id is the loss of the maintask.

Original: Each of the three terms in the loss is multiplied by a hyper-parameter ( λ Con , λ HAOG , λ V id ),and the total loss is the weighted combination of the three terms:
Revised: Each of the three terms in the loss is multiplied by a hyper-parameter ( λ Con , λ HAOG , λ V id ),and the total loss is the weighted combination of the three terms:

Original: [Equation]
Revised: [Equation]

Original:  Experiments and Results
Revised:  Experiments and Results

Original: We begin by describing the datasets, implementation details, and baselines and variants.
Revised: We begin by describing the datasets, implementation details, and baselines and variants.

Original: We thenevaluate our approach on several benchmarks for video understanding.
Revised: We thenevaluate our approach on several benchmarks for video understanding.

Original: Specifically, we consider thefollowing tasks: Compositional Action Recognition (Section 3.4) Object State Change Classification Localization (Section 3.5) (Section 3.
Revised: Specifically, we consider thefollowing tasks: Compositional Action Recognition (Section 3.4), Object State Change Classification& Localization (Section 3.5), and Action Recognition (Section 3.6).

Original: 3.1 Datasets
Revised: 3.1 Datasets

Original: We first describe the datasets used for the downstream video tasks, and those used as “auxiliary”datasets with annotated images.
Revised: We first describe the datasets used for the downstream video tasks, and those used as “auxiliary”datasets with annotated images.

Original: We use the following video datasets: (1) Something-Somethingv2 (SSv2) [28] is a dataset containing 174 action categories of common human-object interactions.(2) SomethingElse [52] which exploits the compositional structure of SSv2, where a combinationof a verb and a noun defines an action.
Revised: We use the following video datasets: (1) Something-Somethingv2 (SSv2) [30] is a dataset containing 174 action categories of common human-object interactions.(2) SomethingElse [58] which exploits the compositional structure of SSv2, where a combinationof a verb and a noun defines an action.

Original: We follow the official compositional split from [52], whichassumes the set of noun-verb pairs available for training is disjoint from the set given at test time.(3) Ego4D [29] is a new large-scale dataset of more than 3,670 hours of video data, capturing thedaily-life scenarios of more than 900 unique individuals from nine different countries around theworld.
Revised: We follow the official compositional split from [58], whichassumes the set of noun-verb pairs available for training is disjoint from the set given at test time.(3) Ego4D [31] is a new large-scale dataset of more than 3,670 hours of video data, capturing thedaily-life scenarios of more than 900 unique individuals from nine different countries around theworld.

Original: (4) Diving48 [48] contains 48 fine-grained categories of diving activities.
Revised: (4) Diving48 [53] contains 48 fine-grained categories of diving activities.

Original: For “auxiliary”image datasets we use the Ego4D, and the 100 Days of Hands (100DOH) [63] datasets.
Revised: For “auxiliary”image datasets we use the Ego4D, and the 100 Days of Hands (100DOH) [71] datasets.

Original: We collected79,921 annotated images from 100DOH, and 57,213 annotated images from Ego4D. 2 The collectedobject boxes from SSv2, 100DOH, and Ego4D are purely human annotated.
Revised: We collected79,921 annotated images from 100DOH, and 57,213 annotated images from Ego4D. 2 The collectedobject boxes from SSv2, 100DOH, and Ego4D are purely human annotated.

Original: In SSv2 and Ego4D,contact relations between the object and hand are not annotated, so we assign the closest object to thehand for each hand.
Revised: In SSv2 and Ego4D,contact relations between the object and hand are not annotated, so we assign the closest object to thehand for each hand.

Original: The contact relations for 100DOH are available in the dataset.
Revised: The contact relations for 100DOH are available in the dataset.

Original: For the image datasets, the image annotations are based on frames selected from videos, but these areoften sparsely selected from the videos, so it is natural to treat them as annotated still images.
Revised: For the image datasets, the image annotations are based on frames selected from videos, but these areoften sparsely selected from the videos, so it is natural to treat them as annotated still images.

Original: 3.2 Implementation Details
Revised: 3.2 Implementation Details

Original: SViT is implemented in PyTorch, we include the code in the supplementary and will release it uponacceptance.
Revised: SViT is implemented in PyTorch, we include the code in the supplementary and will release it uponacceptance.

Original: Our training recipes and code are based on the MViTv2 model, and were taken fromhttps://github.com/facebookresearch/mvit .
Revised: Our training recipes and code are based on the MViTv2 model, and were taken fromhttps://github.com/facebookresearch/mvit .

Original: We pretrain the SViT model on the K400 [41]video dataset with our auxiliary image datasets.
Revised: We pretrain the SViT model on the K400 [44]video dataset with our auxiliary image datasets.

Original: Then, we finetune on the target video understandingtask (detailed in Section 3.1) together with the auxiliary image datasets and the SViT loss.
Revised: Then, we finetune on the target video understandingtask (detailed in Section 3.1) together with the auxiliary image datasets and the SViT loss.

Original: Eachtraining batch contains 64 images and 64 videos in order to minimize the overall loss in Equation 8.
Revised: Eachtraining batch contains 64 images and 64 videos in order to minimize the overall loss in Equation 8.

Original: 3.3 Baselines and SViT variants
Revised: 3.3 Baselines and SViT variants

Original: Baselines .
Revised: Baselines .

Original: In the experiments, we compare SViT to several models from prior work reported for thecorresponding datasets.
Revised: In the experiments, we compare SViT to several models from prior work reported for thecorresponding datasets.

Original: These include the following approaches: I3D [13], SlowFast [21], and thestate-of-the-art transformers – Mformer [57] and MViTv2 [47].
Revised: These include the following approaches: I3D [13], SlowFast [22], and thestate-of-the-art transformers – Mformer [65] and MViTv2 [52].

Original: In addition, we provide the MViTv2multi-task (MViTv2 MT) baseline for comparing a naive multi-tasking approach to our model.
Revised: In addition, we provide the MViTv2multi-task (MViTv2 MT) baseline for comparing a naive multi-tasking approach to our model.

Original: This isa version of MViTv2 which performs video classification and HAOG prediction from images.
Revised: This isa version of MViTv2 which performs video classification and HAOG prediction from images.

Original: The
Revised: The

Original: MViTv2 model is enhanced with two heads: the first head is composed of an MLP that predicts thevideo labels given a final representation vector, the second head consists of MLPs that predict the
Revised: MViTv2 model is enhanced with two heads: the first head is composed of an MLP that predicts thevideo labels given a final representation vector, the second head consists of MLPs that predict the

Original: HAOG information (i.e., hands and object boxes, and contact information).
Revised: HAOG information (i.e., hands and object boxes, and contact information).

Original: SViT Variants .
Revised: SViT Variants .

Original: As mentioned above, we use several sources of auxiliary image datasets.
Revised: As mentioned above, we use several sources of auxiliary image datasets.

Original: In somecases, we have auxiliary images that are frames from video understanding datasets.
Revised: In somecases, we have auxiliary images that are frames from video understanding datasets.

Original: For example,in Ego4D, we have both videos and frames annotated from some of these videos.
Revised: For example,in Ego4D, we have both videos and frames annotated from some of these videos.

Original: In this sense, theauxiliary data is “in-domain” for some of the video tasks.
Revised: In this sense, theauxiliary data is “in-domain” for some of the video tasks.

Original: In order to evaluate the importance of thiseffect, we explore “Different Domain” ( SViT-DD ) training, which avoids using images from a givenvideo task as auxiliary during finetuning and pretraining.
Revised: In order to evaluate the importance of thiseffect, we explore “Different Domain” ( SViT-DD ) training, which avoids using images from a givenvideo task as auxiliary during finetuning and pretraining.

Original: Similarly, “In Domain” ( SViT-ID ) refers tothe case where we use only images from the video task during finetuning, but not necessarily duringpretraining .
Revised: Similarly, “In Domain” ( SViT-ID ) refers tothe case where we use only images from the video task during finetuning, but not necessarily duringpretraining .

Original: We consider another variant that does not use the auxiliary images during finetuning atall (but does use them during pre-training), and refer to it as SViT-SFT .
Revised: We consider another variant that does not use the auxiliary images during finetuning atall (but does use them during pre-training), and refer to it as SViT-SFT .

Original: Finally, to explore the effectof data-size in the In Domain setting, we evaluate a version SViT ×
Revised: Finally, to explore the effectof data-size in the In Domain setting, we evaluate a version SViT ×

Original: X% that only uses X% of thein-domain finetuning data.
Revised: X% that only uses X% of thein-domain finetuning data.

Original: For MViTv2 MT, the auxiliary images correspond to the “In Domain”setting (i.e., same setting as for SViT-ID).
Revised: For MViTv2 MT, the auxiliary images correspond to the “In Domain”setting (i.e., same setting as for SViT-ID).

Original: Auxiliary Images datasets .
Revised: Auxiliary Images datasets .

Original: For SViT-DD, we generally use 100DOH and Ego4D for pretraining andfinetuning, while for the Ego4D tasks, we only use 100DOH.
Revised: For SViT-DD, we generally use 100DOH and Ego4D for pretraining andfinetuning, while for the Ego4D tasks, we only use 100DOH.

Original: SViT-ID uses 100DOH and Ego4D forpretraining and in-domain downstream frames for finetuning.
Revised: SViT-ID uses 100DOH and Ego4D forpretraining and in-domain downstream frames for finetuning.

Original: The SViT-SFT pretraining setting issimilar to that of SViT-DD.
Revised: The SViT-SFT pretraining setting issimilar to that of SViT-DD.

Original: SViT × X% setting is similar to that of SViT-ID. 
Revised: SViT × X% setting is similar to that of SViT-ID. [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 3.4 Compositional & Few-Shot Action Recognition
Revised: 3.4 Compositional & Few-Shot Action Recognition

Original: Several video datasets define an action as a combination of a verb and a noun.
Revised: Several video datasets define an action as a combination of a verb and a noun.

Original: One challenge then is torecognize combinations that were not seen during training.
Revised: One challenge then is torecognize combinations that were not seen during training.

Original: This “compositional” setting was exploredin the “SomethingElse” dataset [52], where verb-noun combinations in the test data do not occur inthe training data.
Revised: This “compositional” setting was exploredin the “SomethingElse” dataset [58], where verb-noun combinations in the test data do not occur inthe training data.

Original: The split contains 174 classes with 54,919/54,876 videos for training/validation.
Revised: The split contains 174 classes with 54,919/54,876 videos for training/validation.

Original: We also evaluate on the few-shot compositional action recognition task in [52] (See supplementary).
Revised: We also evaluate on the few-shot compositional action recognition task in [58] (See supplementary).

Original: Table 1 reports the results for these two tasks.
Revised: Table 1 reports the results for these two tasks.

Original: All of our variants outperform all prior works and the
Revised: All of our variants outperform all prior works and the

Original: MViTv2 MT on the Compositional and Few-shot task.
Revised: MViTv2 MT on the Compositional and Few-shot task.

Original: From the table, we can see that using 2% ofthe in-domain data as auxiliary images (SViT × 2%), is almost comparable to using all the availablein-domain images (SViT-ID).
Revised: From the table, we can see that using 2% ofthe in-domain data as auxiliary images (SViT × 2%), is almost comparable to using all the availablein-domain images (SViT-ID).

Original: Furthermore, the results are comparable to SViT-DD (0.7 difference)when trained on auxiliary images from different domains.
Revised: Furthermore, the results are comparable to SViT-DD (0.7 difference)when trained on auxiliary images from different domains.

Original: In this way, our approach optimizes theuse of structural information across domains.
Revised: In this way, our approach optimizes theuse of structural information across domains.

Original: Last, SViT-DD outperforms SViT-SFT when using thesame data, demonstrating the value of finetuning video and image together.
Revised: Last, SViT-DD outperforms SViT-SFT when using thesame data, demonstrating the value of finetuning video and image together.

Original: 3.5 Object State Change Classification and Localization
Revised: 3.5 Object State Change Classification and Localization

Original: Hands and objects are key elements in human activity.
Revised: Hands and objects are key elements in human activity.

Original: Two tasks related to hand-object interactionhave recently been introduced in the Ego4D [29] dataset.
Revised: Two tasks related to hand-object interactionhave recently been introduced in the Ego4D [31] dataset.

Original: The first is temporal localization, whichis defined as finding key frames in a video clip that indicate a change in object state.
Revised: The first is temporal localization, whichis defined as finding key frames in a video clip that indicate a change in object state.

Original: The second isobject state change classification, which indicates whether an object state has changed or not.
Revised: The second isobject state change classification, which indicates whether an object state has changed or not.

Original: Table 2a reports results on the temporal localization and object state change classification tasks onthe Ego4D dataset.
Revised: Table 2a reports results on the temporal localization and object state change classification tasks onthe Ego4D dataset.

Original: We observe that SViT-ID and SViT-DD are very close, and that both SViT-ID and
Revised: We observe that SViT-ID and SViT-DD are very close, and that both SViT-ID and

Original: SViT-DD are better than SViT-SFT.
Revised: SViT-DD are better than SViT-SFT.

Original: These results are consistent with those presented in Section 3.4,which indicates that our method successfully leverages structural information across domains, andthat a joint image and video finetuning is beneficial. 
Revised: These results are consistent with those presented in Section 3.4,which indicates that our method successfully leverages structural information across domains, andthat a joint image and video finetuning is beneficial. [Table]

Original: 3.6 Action Recognition
Revised: 3.6 Action Recognition

Original: Table 2b and Table 2c report the results for standard action recognition task on the SSv2 and Diving48datasets respectively.
Revised: Table 2b and Table 2c report the results for standard action recognition task on the SSv2 and Diving48datasets respectively.

Original: It can be seen that in Diving48 dataset, our method improves over the MViTv2 baseline by +6.7%, outperforming all the methods.
Revised: It can be seen that in Diving48 dataset, our method improves over the MViTv2 baseline by +6.7%, outperforming all the methods.

Original: This demonstrates that our method hassome ability to generalize the learned structural information of hand-object interactions even tofine-grained “human-motion” tasks such as Diving48.
Revised: This demonstrates that our method hassome ability to generalize the learned structural information of hand-object interactions even tofine-grained “human-motion” tasks such as Diving48.

Original: We also note that Diving48 has no structuredannotations available, thus SViT-ID can not be provided.
Revised: We also note that Diving48 has no structuredannotations available, thus SViT-ID can not be provided.

Original: On SSv2, we overall observe a smallerimprovement.
Revised: On SSv2, we overall observe a smallerimprovement.

Original: SViT-ID improves the baseline by 0.8% while SViT-DD improves by 0.1%.
Revised: SViT-ID improves the baseline by 0.8% while SViT-DD improves by 0.1%.

Original: As weobserved significant improvements on the more challenging SomethingElse setup, we hypothesizethat the ability to generalize to unseen combinations of objects and hands exhibited by SViT is lessneeded on SSv2, hence leading to relatively smaller improvements.
Revised: As weobserved significant improvements on the more challenging SomethingElse setup, we hypothesizethat the ability to generalize to unseen combinations of objects and hands exhibited by SViT is lessneeded on SSv2, hence leading to relatively smaller improvements.

Original: 3.7 Ablations
Revised: 3.7 Ablations

Original: We perform a comprehensive ablation study on the compositional action recognition task [52] on the
Revised: We perform a comprehensive ablation study on the compositional action recognition task [58] on the

Original: SomethingElse dataset to measure the contribution of the different SViT components (Table 3).[Table] [Table]
Revised: SomethingElse dataset to measure the contribution of the different SViT components (Table 3).

Original: Components of the SViT model .
Revised: Components of the SViT model .

Original: In Table 3a, we ablate our model to show the significance of theindividual components.
Revised: In Table 3a, we ablate our model to show the significance of theindividual components.

Original: First, we train a standard MViTv2 in a multi-task setting (MViTv2 MT),where we use videos for video action recognition, and images for HAOG prediction.
Revised: First, we train a standard MViTv2 in a multi-task setting (MViTv2 MT),where we use videos for video action recognition, and images for HAOG prediction.

Original: There is a smallimprovement (+0.5) over the baseline.
Revised: There is a smallimprovement (+0.5) over the baseline.

Original: We then take the MViTv2 and add the object tokens andcorresponding SViT loss; this leads to a more significant improvement (+1.7).
Revised: We then take the MViTv2 and add the object tokens andcorresponding SViT loss; this leads to a more significant improvement (+1.7).

Original: Last, when addingboth the consistency loss and the object tokens, we observe the largest improvement (+2.5).
Revised: Last, when addingboth the consistency loss and the object tokens, we observe the largest improvement (+2.5).

Original: The amount of in-domain data .
Revised: The amount of in-domain data .

Original: In Table 3b, we demonstrate the effect of the amount of in-domainimage data used during training.
Revised: In Table 3b, we demonstrate the effect of the amount of in-domainimage data used during training.

Original: Specifically, since “SomethingElse” includes HAOG annotationsfor each frame, we train our method with different amount of images ( 2% , 25% , 50% , and 100% )from the “SomthingElse” dataset.
Revised: Specifically, since “SomethingElse” includes HAOG annotationsfor each frame, we train our method with different amount of images ( 2% , 25% , 50% , and 100% )from the “SomthingElse” dataset.

Original: It can be seen that our approach takes advantage of the structuredinformation within a small amount of data and, as a result, has enhanced sample-efficiency.
Revised: It can be seen that our approach takes advantage of the structuredinformation within a small amount of data and, as a result, has enhanced sample-efficiency.

Original: HAOG attributes .
Revised: HAOG attributes .

Original: We use the object tokens to predict several aspects of the HAOG.
Revised: We use the object tokens to predict several aspects of the HAOG.

Original: In Table 3c weconsider several variations on this prediction.
Revised: In Table 3c weconsider several variations on this prediction.

Original: The model “SViT-Contact” does not predict the contactinformation, resulting in − 0 .
Revised: The model “SViT-Contact” does not predict the contactinformation, resulting in − 0 .

Original: 3% drop compared to SViT.
Revised: 3% drop compared to SViT.

Original: The model “SViT+Geometry” adds to SViTadditional heads that predict distances between all HAOG objects (to explicitly add geometric bias).
Revised: The model “SViT+Geometry” adds to SViTadditional heads that predict distances between all HAOG objects (to explicitly add geometric bias).

Original: This does not affect performance.
Revised: This does not affect performance.

Original: Finally, we explore what happens when we do not provide theinformation about the identity of bounding boxes in the training data.
Revised: Finally, we explore what happens when we do not provide theinformation about the identity of bounding boxes in the training data.

Original: Namely, we treat them as fourboxes, and ask the model to match those to the object tokens, via a matching losss as in [12].
Revised: Namely, we treat them as fourboxes, and ask the model to match those to the object tokens, via a matching losss as in [12].

Original: This ismodel “SViT-Contact -Corresp.”
Revised: This ismodel “SViT-Contact -Corresp.”

Original: which is quite close to “SViT-Contact”, indicating that SViT canperhaps be trained with weaker labels.
Revised: which is quite close to “SViT-Contact”, indicating that SViT canperhaps be trained with weaker labels.

Original:  Related Work
Revised:  Related Work

Original: Joint Training with Images and Videos .
Revised: Joint Training with Images and Videos .

Original: Vision Transformers [19] are versatile architecturesdesigned to be flexible in terms of input sequence length.
Revised: Vision Transformers [20] are versatile architecturesdesigned to be flexible in terms of input sequence length.

Original: Aside from the data “‘patchification”’ layer,the rest of the transformer is agnostic to the domain of the input.
Revised: Aside from the data “‘patchification”’ layer,the rest of the transformer is agnostic to the domain of the input.

Original: Consequently, an exciting directionof joint training image-video has emerged in computer vision, a direction previously less natural withconvolutional architectures.
Revised: Consequently, an exciting directionof joint training image-video has emerged in computer vision, a direction previously less natural withconvolutional architectures.

Original: Specifically, this allows for training network simultaneously on imageand video, with most of the network parameters shared across both domains.
Revised: Specifically, this allows for training network simultaneously on imageand video, with most of the network parameters shared across both domains.

Original: Recently, multi-modalmodels became widely popular, from shared backbones [15, 51, 53] to separate encoders for each input modality [65, 68], and even non-vision modalities [35, 36].
Revised: Recently, multi-modalmodels became widely popular, from shared backbones [15, 56, 60] to separate encoders for eachinput modality [73, 76], and even non-vision modalities [38, 39].

Original: In our case, we focus on videoand image input, similar to [6, 26, 76].
Revised: In our case, we focus on video and image input, similar to [6, 27, 86].

Original: In contrast to these works, our training approach utilizes twodomains, while one of them (image) is used only during training and is supervised by a different task.
Revised: In contrast to these works, our training approach utilizes twodomains, while one of them (image) is used only during training and is supervised by a different task.

Original: Additionally, as we demonstrate here, we are not greatly impacted by domain mismatches betweenimages and videos, as in previous works.
Revised: Additionally, as we demonstrate here, we are not greatly impacted by domain mismatches betweenimages and videos, as in previous works.

Original: In a related area of research, multi-task learning [14]involves developing models that predict output for multiple tasks using a common input.
Revised: In a related area of research, multi-task learning [14]involves developing models that predict output for multiple tasks using a common input.

Original: Our work isdifferent since it uses two inputs from different domains, images and videos.
Revised: Our work isdifferent since it uses two inputs from different domains, images and videos.

Original: Similarly, our workcan be viewed as a form of auxiliary task, as already shown in many works [27, 34, 64].
Revised: Similarly, our workcan be viewed as a form of auxiliary task, as already shown in many works [29, 37, 72].

Original: However,it differs in that the shared object prompts are utilized to learn the shared structured representationbetween video and image domains, which enhances the self-attention layers with the structuredrepresentation for the main video task.
Revised: However,it differs in that the shared object prompts are utilized to learn the shared structured representationbetween video and image domains, which enhances the self-attention layers with the structuredrepresentation for the main video task.

Original: Learning from video using other modalities .
Revised: Learning from video using other modalities .

Original: Researchers have proposed different approachesto learn from videos using audio and natural language [1, 3, 18, 54, 55, 56, 59, 72].
Revised: Researchers have proposed different approachesto learn from videos using audio and natural language [1, 3, 19, 61, 63, 64, 67, 81].

Original: Using bothaudio and video, it is possible to learn good video representations [3, 55], localize objects [1], andpredict human gestures [24].

Original: Vision tasks like Video Captioning [18, 23, 56, 72] and Visual Quesion Answering [2, 70] require understanding of both visual and natural language.
Revised: Vision taskslike Video Captioning [19, 24, 64, 81] and Visual Quesion Answering [2, 78] require understandingof both visual and natural language.

Original: Another line visionand language works explored object labels, supervision of textual descriptions, or the use of regioncrops [44, 46], in contrast to SViT, which does not require any of these components.
Revised: Another line of vision and language works explored objectlabels, supervision of textual descriptions, or the use of region crops [49, 51], in contrast to SViT,which does not require any of these components.

Original: Additionally, Since high quality captions are expensive, researchers have proposed self-supervised ways to learnfrom both modalities without labeled data [54, 53, 59].
Revised: Additionally, Since high quality captions areexpensive, researchers have proposed self-supervised ways to learn from both modalities withoutlabeled data [61, 60, 67].

Original: In these previous works, the two modalitiesmust be consistent and aligned (e.g., text, audio, and RGB should be consistent for same sample).
Revised: In these previous works, the two modalities must be consistent and aligned(e.g. , text, audio, and RGB should be consistent for same sample).

Original: However, in our work, we do not require that the annotated images be precisely aligned with ourdownstream video task.
Revised: However, in our work, we do notrequire that the annotated images be precisely aligned with our downstream video task.

Original: Last, other new works [16, 61] explored different structured representationsfor videos, different from what we do, which is to leverage structure from images to videos.
Revised: Last, othernew works [16, 69] explored different structured representations for videos, different from what wedo, which is to leverage structure from images to videos.

Original: Structured Models .
Revised: Structured Models .

Original: Recently structured models have been successfully applied in many computervision applications: video relation detection [49, 62, 67], vision and language [17, 45, 46, 69],relational reasoning [8, 9, 33, 42, 37, 58, 74, 75], human-object interactions [22, 40, 73], and evenimage & video generation [7, 30, 39].A recent line of work [5, 25, 31, 32, 38, 66, 71] focuses oncapturing spatial-temporal interactions for video understanding, which heavily relies on frame-levelannotations.
Revised: Recently structured models have been successfully applied in many computervision applications: video relation detection [54, 70, 75], vision and language [17, 50, 51, 77],relational reasoning [8, 9, 35, 47, 40, 66, 84, 85], human-object interactions [23, 43, 83], and evenimage & video generation [7, 32, 42].A recent line of works [5, 25, 26, 33, 34, 41, 57, 59, 62, 74, 79,80, 82, 88] focuses on capturing spatio-temporal interactions for video understanding. Some of theseworks [33, 80, 82, 88] used static image object annotations to build the representations.

Original: STRG [71] and ORN [8] used spatio-temporal interactions with two consecutive frameinteractions, while STAG [32] considered long-range temporal interaction. Unified [5] tried togeneralize all these models and propose long spatio-temporal object interactions.
Revised: However,these works differ from ours. [82] proposed the Object Transformers (OT) for long form videounderstanding, which model a video as a set of objects by extracting region crops from SlowFast [22]based on bounding boxes from an external detector. Contrary to this work, our transformer containsboth patch tokens and object tokens, whereas the OT contains only object tokens.

Original: Last, [77] proposeda structured model for action detection that explicitly models long-term temporal behavior as well asobject manipulation and human interaction.
Revised: Additionally, ourmodel learns the object-centric representation without the need for an external detector. Anotherwork [88], proposed a structured model for action detection that explicitly utilizes image annotations.

Original: Contrary to this line of  work, we study how distill andpropagate the information from images through the transformer object prompts (learned from thegraph prediction task only in training) to the network for predicting the video downstream task.
Revised: They use a GCN [46] to reason about actors and objects interaction, on top of an I3D network. Unlikethis work, we examine how image annotations can be exploited as part of a video transformer model.

Original: Video Understanding Models .
Revised: Video Understanding Models .

Original: We work on several well-benchmarked down-stream video tasks suchas compositional action recognition on Something-Else dataset [52] and action recognition with focuson temporal cues de-emphasizing appearance such as in Something-Something V2 dataset [28] and
Revised: We work on several well-benchmarked down-stream video tasks suchas compositional action recognition on Something-Else dataset [58] and action recognition with focuson temporal cues de-emphasizing appearance such as in Something-Something V2 dataset [30] and

Original: Diving48 [48].
Revised: Diving48 [53].

Original: Several recent video transfromer works performed well on SSv2, such as ViViT [4],
Revised: Several recent video transfromer works performed well on SSv2, such as ViViT [4],

Original: MViT [20], MFormer [57], TimeSformer [11], MViTv2 [47],and Video Swin [50].
Revised: MViT [21], MFormer [65], TimeSformer [11], MViTv2 [52],and Video Swin [55].

Original: We choose towork with MViTv2, although our method can be used on top of any of these.
Revised: We choose towork with MViTv2, although our method can be used on top of any of these.

Original: Additionally, wealso adapt our proposed method to interesting new tasks on the Ego4D dataset [29].
Revised: Additionally, wealso adapt our proposed method to interesting new tasks on the Ego4D dataset [31].

Original: We believesuch fine-grained video understanding tasks require richer scene and object semantic reasoning, arequirement well afforded by our proposed Hand-Object Graph.
Revised: We believesuch fine-grained video understanding tasks require richer scene and object semantic reasoning, arequirement well afforded by our proposed Hand-Object Graph.

Original:  Conclusion
Revised:  Conclusion

Original: Video understanding is a key element of human visual perception, but modeling remains a challenge for machine vision.
Revised: Video understanding is a key element of human visual perception, but modeling remains a challenge for machine vision.

Original: In this work, we demonstrated the importance of learning from a scenestructure from a small set of images to facilitate video learning within or outside the domain ofinterest.
Revised: In this work, we demonstrated the importance of learning from a scenestructure from a small set of images to facilitate video learning within or outside the domain ofinterest.

Original: According to our empirical study, our SViT approach improves performance on three videounderstanding tasks.
Revised: According to our empirical study, our SViT approach improves performance on three videounderstanding tasks.

Original: Note, that we did not prioritize making the HAOG annotations richer, i.e., itmay be possible to add other physical properties to improve structure modeling.
Revised: Note, that we did not prioritize making the HAOG annotations richer, i.e., itmay be possible to add other physical properties to improve structure modeling.

Original: In addition, wenoticed that leveraging the structure is effective even with 2% of the in-domain frames, as well aswhen applying it from outside the domain.
Revised: In addition, wenoticed that leveraging the structure is effective even with 2% of the in-domain frames, as well aswhen applying it from outside the domain.

Original: We leave for future work how different physical propertiescan be more effectively utilized in more complex structures.
Revised: We leave for future work how different physical propertiescan be more effectively utilized in more complex structures.

Original: We do not anticipate a specific negativeimpact, but, as with any Machine Learning method, we recommend to exercise caution.
Revised: We do not anticipate a specific negativeimpact, but, as with any Machine Learning method, we recommend to exercise caution.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data areproprietary.

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See line341-342.
Revised: [Yes] See line341-342.(d)

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A](b)

Original: (b) Did you include complete proofs of all theoretical results?
Revised: Did you include complete proofs of all theoretical results?

Original: [N/A]3.
Revised: [N/A]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: [Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes](c)
Revised: [Yes](c)

Original: Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [Yes](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: [Table]
Revised: [Table]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a)
Revised: (a)

Original: If your work uses existing assets, did you cite the creators?
Revised: If your work uses existing assets, did you cite the creators?

Original: [Yes] Sec Section 3.1 (theDataset Section).(b) Did you mention the license of the assets?
Revised: [Yes] Sec Section 3.1 (theDataset Section). (b) Did you mention the license of the assets?

Original: [Yes] See the Dataset Section in supplementary.(c)
Revised: [Yes] See the Dataset Section in supplementary.

Original:  Did you include any new assets either in the supplemental material or as a URL? 
Revised: (c) Did you include any new assets either in the supplemental material or as a URL? [No]

Original: [No](d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table] using/curating?

Original: [Yes] See the Dataset Section in supplementary.(e)
Revised: [Yes] See the Dataset Section in supplementary.

Original:  Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: [Table]
Revised: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes] See the Dataset Section in supplementary.5.
Revised: [Yes] See the Dataset Section in supplementary. [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original:  (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: [Table] (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A](b)
Revised: [N/A]

Original:  Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A](c)
Revised: [N/A]

Original:  Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]
Revised: [N/A]

Revised: Supplementary Material for “Bringing Image Scene Structure to

Revised: Video via Frame-Clip Consistency of Object Tokens”

Revised: In this supplementary file, we provide additional information about our experimental results, qualitative examples, implementation details, and datasets.

Revised: Specifically, Section A provides more experimentresults, Section B provides qualitative visualizations to illustrate our approach, Section C providesadditional implementation details, and Section D provides additional datasets details.

Revised: Finally, wehave also included our code with this supplementary material.

Revised: A Additional Experiment Results

Revised: First, we discuss the pretraining and finetuning of the SViT model variants (Section A.1).

Revised: Next,we present additional ablations (Section A.2)

Revised: we performed in order to test the contribution of thedifferent SViT components.

Revised: A.1 SViT Variants

Revised: As explained in Section 3.3, we use several SViT variants, see Table 4 for details.

Revised: Additionally,

Revised: Table 7 provides a detailed listing of the datasets we used for each SViT variant and task.

Revised: [Table]

Revised: [Table]

Revised: Next, we provide additional ablations in Table 5.

Revised: Pretraining and finetuning using in-domain auxiliary images .

Revised: As part of finetuning, SViT-IDmakes use of in-domain auxiliary image data, whereas for pretraining, it uses a different domain andin-domain of image data (see Table 4).

Revised: It is interesting to see what happens if one uses only in-domaindata for both pretraining and finetuning, and no data from external datasets.

Revised: Towards this end, weconsider a SViT training setup where only the in-domain images are used for both pre-training andfine-tuning.

Revised: We do this both for SmthElse and Ego4D. Results are shown in Tables 5a and 5b.

Revised: Itcan be seen that results with only in-domain image data (both pretraining and finetuning) are quiteclose to those of using both different and in-domain data.

Revised: Thus, we conclude that the performanceimprovement of the in-domain and different domain are similar, demonstrating that our method iscapable of benefiting both from in-domain and different-domain data.

Revised: Pertraining and Finetuning with or without auxiliary images .

Revised: In Table 5c, we check the effect ofusing the auxiliary image data during the pretraining and finetuning for the standard SViT-ID model.

Revised: Namely, during the pretraining and fine tuning processes, we examine each combination of usingor not using auxiliary image data.

Revised: We can see that doing finetuning and pretraining with auxiliaryimages yields the best results (65.8).

Revised: Alternatively, only pretraining with auxiliary images (64.2) oronly finetuning with auxiliary images (64.4) are advantageous compared to not using any auxiliaryimages.

Revised: In fact, not using images at all is equivalent to MViT-v2 with additional tokens, and has nobenefit over MViT-v2.

Revised: Consistency Loss .

Revised: In Table 5d we consider two different types of frame-clip consistencies.

Revised: The firstis patch token consistency, which is simply replacing the object tokens with the patch tokens in theframe-clip consistency loss and the second is object token consistency (as described in Equation 7in the main paper).

Revised: Comparing the results to not using any frame-clip consistency (65.0), we note that object token consistency is beneficial (+0.8), but patch consistency decreases the performance(-6.3).

Revised: We hypothesize that patch consistency may reduce patch diversification within the frames,thus reducing performance.

Revised: A recent study [28] has demonstrated that vision transformers tendto map different patches into similar latent representations, which results in information loss andperformance degradation.

Revised: The minimum requirements to make the model work .

Revised: In Table 6a we aim to systematicallydemonstrate the minimum requirements necessary for the model to function reasonably well.

Revised: Weexplored the minimum requirements to make our model work.

Revised: We provide the following experiments:(i)

Revised: A ratio of 1 image to X videos (where X is 1,10, and 100): 65.6 (1-to-1), 64.8 (1-to-10), and 61.55(1-to-100).

Revised: See Table 6a.

Revised: The results indicate that using too few annotated images (1 image per 100videos) may result in degradation, but using a relatively small number of images (1 image per 10videos) is sufficient to achieve good improvement.

Revised: Robustness to multiple domains .

Revised: In Table 6b We performed experiments on SomethingElse (groundtruth annotations available) where hands or objects were not visible in order to verify the usefulnessof our approach: (i) After filtering the videos containing (annotated) hands, we tested our modeland the MViTv2 model on the filtered hand split.

Revised: MViTv2 achieved an accuracy of 64.6 while ourmodel achieved 66.5.

Revised: This implies an improvement of +1.9.

Revised: (ii)

Revised: After filtering the videos containing(annotated) objects in more than 40% of the frames, we tested our model and the MViTv2 model onthe filtered object split.

Revised: MViTv2 achieved an accuracy of 64.9 while our model achieved 67.0.

Revised: Thisimplies an improvement of +2.1.

Revised: We can observe that our model outperforms the baseline even whenthere are no objects or hands in the videos, demonstrating the robustness of our approach.

Revised: The totalimprovement (+1.9 and +2.1) is also a little bit lower than before the filtering (+2.5), which indicatesthat there has been a slight degradation.

Revised: Even so, it is still valuable to use the hand-object graphs.

Revised: Frame-clip consistency loss .

Revised: In order to validate frame-clip consistency loss, we evaluate thefollowing experiments: (i)

Revised: We perturb the image temporal position without consistency loss.

Revised: We referto this version as SViT-Perturb (and similarly SViT-Perturb-DD and SViT-Perturb-ID).

Revised: (ii)

Revised: The HAOGannotations are extrapolated from one random frame of a video.

Revised: This serves as additional supervision(without the consistency loss) since the HAOG annotations correspond to the video frames (we notethat SViT does not require such correspondence since it uses only HAOG annotations from singleimages).

Revised: (iii)

Revised: We predict the HAOG of a random frame in a video, and then duplicate it over thetemporal dimension and use it in the same manner as in the consistency loss.

Revised: We find that thesethree baselines lead to worse performance.

Revised: For the first experiment, we obtained SViT-Perturb-DDwith 64.1 (while SViT-DD obtained 65.1), and SViT-Perturb-ID with 65.2 (while SViT-ID withconsistency loss got 65.8).

Revised: For the second experiment, the proposed baseline achieved 65.0 comparedto our SViT-ID, which achieved 65.8 (and does not require correspondence).

Revised: Taken together, thisdemonstrates the importance of our frame-clip consistency loss.

Revised: SViT-2% for Ego4D, SSV2, and Diving48 datasets .

Revised: In Table 6c we evaluated SViT-2% on the

Revised: Ego4d, SSv, and Diving48 datasets.

Revised: The SViT-2% in Diving48 shows an improvement of +5.8compared to the MViTv2 baseline (while SViT shows an improvement of +6.7).

Revised: For SSv2, the SViT2% experiment results in an improvement of +0.6 compared to the MViTv2 baseline (while SViTshows an improvement of +0.8).

Original: {+For Ego4D, the SViT-2% experiment results in an improvement of+1.8 compared to the MViTv2 baseline for object state change classification and an improvement of0.
+}
Revised: {+For Ego4D, the SViT-2% experiment results in an improvement of+1.8 compared to the MViTv2 baseline for object state change classification and an improvement of0.
+}

Revised: compared to the MViTv2 baseline for object state change temporal localization.

Revised: It can be seenthat the SViT-2% also appears effective and provides similar improvements on other datasets to theresults shown in Table 1.

Revised: HAOG "distillation" .

Revised: In this experiment we first train a vision transformer to predict the HAOG taskalone

Revised: and then fine-tune its backbone network for the video-related task.

Revised: We evaluated and achieved62.3 compared to 63.3 of MViTv2.

Revised: This indicates that training HAOGs as a form of “distillation”, aswe do in SViT, is indeed important.

Revised: The object tokens representations .

Revised: To verify what the object tokens learned, we can evaluate theability of object tokens to be utilized explicitly for the auxiliary task as a simple detector of hands andobjects in images.

Revised: This is accomplished by predicting the detections on SomethingElse based on thelearned object tokens.

Revised: The learned object tokens were compared to the MViTv2 model extended withregression and detection heads.

Revised: Our model achieved an mAP of 16.8, while the proposed baselineachieved a similar result with an mAP of 15.5.

Revised: These results suggest that the object tokens learnmeaningful and useful representations.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: How important are the HAOGs .

Revised: To examine how important is the information provided by the

Revised: HAOG, we suggest learning HAOGs without any useful information.

Revised: Thus, we run an experimentin which the HAOGs are completely random.

Revised: This means that, for each image, a random HOAG isgenerated by sampling the boxes and their relationships uniformly.

Revised: We refer to this experiment as

Revised: SViT-Random-ID.

Revised: The SViT-Random-ID result on SomethingElse is 50.6 compared to 63.3 in the

Revised: SViT-ID baseline.

Revised: This demonstrates that predicting the actual HAOG attributes is important but thatnot many annotations are necessary for the auxiliary task.

Revised: One explanation may be that learning theshared structured representation between the domains is sample efficient and does not require manyannotated images, which we view as a big advantage of our approach.

Revised: Computational cost .

Revised: The cost computation of our approach is relatively small compared to MViTv2:+0.4% in Giga ( 10 9 ) FLOPs (MViTv2 with 64.5, while SViT with 64.7) and +5% in Mega ( 10 6 )parameters (MViTv2 with 34.4, while SViT with 36.1).

Revised: Hands in Diving48 .

Revised: To further analyze SViT performance on Diving48 we provide an experiment inwhich we remove the "hand" tokens from the HAOG annotation during training, and an experimentin which we remove the "object" tokens from the HAOG annotation during training.

Revised: The result forremoving the "hand" tokens is a degradation of 1.7 in top-1 accuracy, and for removing the "object"tokens is 0.6.

Revised: We hypothesize that the “human” boxes provide a prior for localization that helps inclassifying diving actions (diving actions could be classified based on appearance and pose).

Revised: A.3 Confidence Intervals

Revised: We provide confidence intervals for the SViT-ID and MViTv2 results on the SomethingElse dataset,as seen in Table 1.

Revised: We used n = 10 experiments and provide a 95% confidence intervals.

Revised: The resultsfor SViT-ID are 65 .

Revised: 8 ± 0 .

Revised: 44 and for MViTv2 are 63 .

Revised: 3 ± 0 .

Revised: We can see that the intervals do notintersect, thus our improvement is significant.

Revised: A.4 Additional Results

Revised: We provide more results on 3rd-person video datasets, such as Kinetics-400 and AVA.

Revised: Our results for

Revised: Kinetics-400 with SViT showed an improvement of +1.5 over MViTv2.

Revised: As for AVA, we evaluatedthe mAP and received an improvement of +0.7 over MViTv2.

Revised: [Table]

Revised: B Qualitative Visualizations

Revised: We present qualitative visualizations of the SViT-ID “object tokens” in videos for the followingdatasets: SomethingElse, Ego4D, and Diving48 in Figure 3.

Revised: As explained in the main paper, twotokens correspond to the “right hand” and “left hand”, whereas the other two tokens correspond tothe objects they are interacting with.

Revised: We can see that the “object tokens” are used to detect relevantobjects, even on Diving48, where there is no human-object interaction.

Revised: In Diving48, we observeda few interesting phenomena: (i) the diving persons are recognized as hands.

Revised: This may be explainby the fact that skin has shared visual features with hands.

Revised: (ii)

Revised: The objects recognized in low scoresindicate that the scene does not contain any objects.

Revised: Overall, despite the issues we raised above, ourmodel performed well on Diving48.

Revised: This is evidence that our method is robust to multiple domainsthat do not necessarily fit with human-object graph structure (HAOG), and that it still benefits fromthese learned object tokens through the attention.

Revised: C Additional Implementation Details

Revised: Our SViT model can be used on top of the most common video transformers (MViT [21], TimeSformer [11], Mformer [65], Video Swin [55]).

Revised: For our experiments, we choose the MViTv2 [52]model because it performs well empirically.

Revised: These are all implemented based on the MViTv2 [52]library (available at https://github.com/facebookresearch/mvit ), and we implement SViTbased on this repository.

Revised: Next, we elaborate on the additional implementation details for each dataset,including details about the dataset description, optimization, and training and inference.

Revised: C.1 Ego4D

Revised: Dataset . Ego4D [31] is a new large-scale dataset of more than 3,670 hours of video data, capturingthe daily-life scenarios of more than 900 unique individuals from nine different countries around theworld.

Revised: The videos contain audio, 3D meshes of the environment, eye gaze, stereo and/or synchronizedvideos from multiple egocentric cameras.

Revised: Metrics .

Revised: In the Object State Change Temporal Localization task, the absolute error (in seconds) isused for evaluation.

Revised: In the Object State Change Classification task, the top-1 accuracy is used forevaluation.

Revised: Optimization details .

Revised: We train using 16 frames with sample rate 4 and batch-size 128 (comprising64 videos and 64 images) on 8 RTX 3090 GPUs.

Revised: We train our network for 10 epochs with Adamoptimizer [45] with a momentum of 9 e − 1 and Gamma 1 e − 1 .

Revised: Following [52], we use lr = 1 e − 4 withhalf-period cosine decay.

Revised: Additionally, we used Automatic Mixed Precision, which is implementedby PyTorch.

Revised: Training details .

Revised: We use a standard crop size of 224 , and we jitter the scales from 256 to 320 .

Revised: Additionally, we set λ

Revised: Con = 10 , λ HAOG = 5 , λ V id = 1 .

Revised: Inference details .

Revised: We follow the official evaluation, both for the state change temporal localization and the state change classification tasks, available at https://github.com/EGO4D/hands-and-objects .

Revised: [Figure]

Revised: [Figure]

Revised: C.2 Diving48

Revised: Dataset .

Revised: Diving48 [53] contains 16K training and 3K testing videos spanning 48 fine-grained divingcategories of diving activities.

Revised: For all of these datasets, we use standard classification accuracy as ourmain performance metric.

Revised: Optimization details .

Revised: We train using 16 frames with sample rate 4 and batch-size 128 (comprising64 videos and 64 images) on 8 RTX 3090 GPUs.

Revised: We train our network for 10 epochs with Adamoptimizer [45] with a momentum of 9 e − 1 and Gamma 1 e − 1 .

Revised: Following [52], we use lr = 1 e − 4with half-period cosine decay.

Revised: Training details .

Revised: We use a standard crop size of 224 for the standard model and jitter the scales from256 to 320 .

Revised: Additionally, we use RandomFlip augmentation.

Revised: Finally, we sample the T frames fromthe start and end diving annotation time, following [87].

Revised: We set λ

Revised: Con = 10 , λ HAOG = 5 , λ V id = 1 .

Revised: Inference details .

Revised: We take 3 spatial crops per single clip to form predictions over a single video intesting, as in [11].

Revised: C.3 SomethingElse

Revised: Dataset .

Revised: The SomethingElse dataset [58] contains 174 action categories with 54,919 training and57,876 validation samples.

Revised: The proposed compositional [58] split in this dataset provides disjointcombinations of a verb (action) and noun (object) in the training and testing sets.

Revised: This split definestwo disjoint groups of nouns {A , B} and verbs { 1 , 2 } .

Revised: Given the splits of groups, they combine thetraining set as 1 A

Revised: + 2 B , while the validation set is constructed by flipping the combination into1 B + 2 A .

Revised: In this way, different combinations of verbs and nouns are divided into training or testingsplits.

Revised: Few Shot Compositional Action Recognition .

Revised: As mentioned in Section 3.4 of the main paper,we also evaluate on the few-shot compositional action recognition task in [58].

Revised: For this setting,we have 88 base action categories and 86 novel action categories.

Revised: We train on the base categories(113K/12K for training/validation) and finetune on few-shot samples from the novel categories (for5-shot, 430/50K for training/validation; for 10-shot, 860/44K for training/validation).

Revised: Optimization details .

Revised: We train using 16 frames with sample rate 4 and batch-size 128 (comprising64 videos and 64 images) on 8 RTX 3090 GPUs.

Revised: We train our network for 100 epochs with Adamoptimizer [45] with a momentum of 9 e − 1 and Gamma 1 e − 1 .

Revised: Following [52], we use lr = 1 e − 4 withhalf-period cosine decay.

Revised: Additionally, we used Automatic Mixed Precision, which is implementedby PyTorch.

Revised: Regularization details .

Revised: We use weight decay of 1 e − 4 , and a dropout [36] of 5 e − 1 before the finalclassification.

Revised: Training details .

Revised: We use a standard crop size of 224 , and we jitter the scales from 256 to 320 .

Revised: Additionally, we set λ

Revised: Con = 10 , λ HAOG = 5 , λ V id = 1 .

Revised: Inference details .

Revised: We take 3 spatial crops per single clip to form predictions over a single video intesting.

Revised: C.4 Something-Something v2

Revised: Dataset .

Revised: The SSv2 [58] contains 174 action categories of common human-object interactions.

Revised: Optimization details .

Revised: For the standard SSv2 [58] dataset, we train using 16 frames with samplerate 4 and batch-size 128 (comprising 64 videos and 64 images) on 8 RTX 3090 GPUs.

Revised: We train ournetwork for 100 epochs with Adam optimizer [45] with a momentum of 9 e − 1 and Gamma 1 e − 1 .

Revised: Following [52], we use lr = 1 e − 4 with half-period cosine decay.

Revised: Additionally, we used Automatic

Revised: Mixed Precision, which is implemented by PyTorch.

Revised: Regularization details .

Revised: We use weight decay of 1 e − 4 , and a dropout [36] of 5 e − 1 before the finalclassification.

Revised: Inference details .

Revised: We take 3 spatial crops per single clip to form predictions over a single video intesting as done in [52].

Revised: D Additional Datasets Details

Revised: Here provide detailed information about the “auxiliary image datasets” (Section D.1) as well as thelicenses and privacy policies for these datasets (Section D.2)

Revised: [Table]

Revised: D.1 Auxiliary Image Datasets

Revised: [Table]

Revised: Recognition, AR refers to Action Recognition,

Revised: SCTL to State Change Temporal Localization, SCC to State Change Classification

Revised: In Table 7 we explicitly describe the auxiliary image datasets used in each experiment.

Revised: As an example,in row 1, we describe the data used for training the SViT-SFT model for the Compositional Action

Revised: Recognition (CAR) task on the SomethingElse dataset.

Revised: We note that we began by pretraining on

Revised: Ego4D and 100DOH.

Revised: Image/Video annotation naming conventions .

Revised: The auxiliary image data we used is originated invideo frames.

Revised: The main difference between video frames and a batch of images is the temporalinformation.

Revised: Since we do not use the temporal order of the annotated video frames, and practicallyuse them as images, we refer to them in the paper as “image annotations”.

Revised: D.2 Licenses and Privacy

Revised: The license, PII, and consent details of each dataset are in the respective papers.

Revised: In addition, we wishto emphasize that the datasets we use do not contain any harmful or offensive content, as many otherpapers in the field also use them.

Revised: Thus, we do not anticipate a specific negative impact, but, as withany Machine Learning method, we recommend to exercise caution.