Video-based Human-Object Interaction Detection from Tubelet Tokens
Anonymous Author(s)
Abstract
We present a novel vision Transforme r , named TUTOR, which is able to learntu belet to kens, served as highly-abstracted spatiotemporal representations, forvideo-based human-object interaction (V-HOI) detection.
The tubelet tokens structurize videos by agglomerating and linking semantically-related patch tokens alongspatial and temporal domains, which enjoy two benefits: 1) Compactness: eachtubelet token is learned by a selective attention mechanism to reduce redundantspatial dependencies from others;
2) Expressiveness: each tubelet token is enabledto align with a semantic instance, i.e. , an object or a human, across frames, thanksto agglomeration and linking.
The effectiveness and efficiency of TUTOR areverified by extensive experiments.
Results show our method outperforms existingworks by large margins, with a relative mAP gain of 16 .
14% on VidHOI and a 2points gain on CAD-120 as well as a 4 × speedup.
Introduction
Human-object interaction (HOI) detection is a detailed scene understanding task, which requires bothlocalization of interacted human-object pairs and recognition of interaction labels.
Existing methodsmostly investigated detecting HOIs in static images without capturing temporal information (Figure 1(a)), thus lack the ability to detect time-related interactions ( e.g. , shoot or pass a basketball).
However, interactions are more of time-related in practical scenario, leading to a strong demand to detect
HOIs from videos, a more challenging problem built on spatiotemporal semantic representations.
Transformer, originated from natural language processing (NLP), is an intuitive choice for its eminentcapability of reasoning long-range dependencies, in which one of the most crucial components is thetoken.
A token serves as an element of data representations, which is usually a word in language.
However, unlike language that naturally has such a discrete signal space for building tokenizeddictionaries, images lie in a continuous and high-dimensional space.
To address this issue, vision
Transformer (ViT) [7] provided a solution that divides each image into several local patch tokens (“visual words”) to structurize the entire image as a “visual sentence”(Figure 1(b)).
This solution has become a de facto tokenization standard followed by most existing Transformer-based methods, which has achieved excellent performance for various vision tasks, especially image classification.
Nevertheless, this patch based tokenization strategy might not be proper for video-based HOI (V-HOI) detection (as the performance degradation of the ViT-like framework shown in Table.
1a).
We find that the reason is the patch tokens generated by regular splitting are difficult to exactly capture semantics (an instance is an object or a human, e g. , the basketball shooter in Figure 1), which yet is crucial for V-HOI detection to reason the interaction labels.
These patch tokens inevitably suffer from redundancy due to an information mixture from different instances as well as insufficiency due to only a part occupancy of an instance, which limit their representation ability
[Figure] In this paper, we present TUTOR, a new Transforme R for V-HOI detection built on TU belet TO kens,handling aforementioned limitations favorably.
The tokenization of the tubelet tokens is not basedon fixed regular splitting but is jointly performed with the learning of the Transformer encoder.
This enables the tubelet tokens to progressively emerge and represent high-level visual semantics.
Concretely, first, along the spatial domain, we alternatively update the representation for each patchtoken by a selective attention mechanism and agglomerate semantically-related patch tokens intoinstance tokens.
The selective attention mechanism ensures that attention is performed among tokensexpected to belong to the same instance, which reduces redundant spatial dependencies from others.
Then, along the temporal domain, we link instance tokens across frames to form the tubelet tokens.
Figure 1(c) illustrates the process of tubelet token generation.
Experimental results show that TUTORoutperforms existing sota methods by large margins.
Specifically, we achieve a relative mAP gain of16 .
4% on VidHOI [5] and a 2 points F1 score gain on CAD-120 [22], with a 4 × inference speedup.
Related Work
HOI detection.
Most previous works are devoted to detecting HOIs in static images [3, 10, 11,13, 14, 16, 19, 21, 23, 24, 27, 29, 35, 38, 40, 41, 43, 45, 46, 49, 50, 51, 37, 53, 20, 4, 47].
Withoutconsidering temporal information, these methods fail to detect time-related interactions, restrictingtheir value in practical applications.
In contrast, video-based HOI detection is a more practicalproblem, which however is less explored [35, 33, 34, 36, 5, 42, 18].
[35, 36, 42] detected HOIsin videos by building graph neural networks to capture spatiotemporal information.
In [33], HOI“hotspots” can be directly learned from videos by jointly training a video-based action recognitionnetwork as well as an anticipation model.
Inspired by image-based methods, [5] introduced atwo-stage framework where the frame-wise human/object features are firstly extracted by usingtrajectories, and then HOIs are detected by processing the instance features as well as auxiliaryfeatures, including spatial configurations and human poses.
However, these methods lack the abilityto model long range contextural information, resulting in poor performance when the interactedhuman and object are far apart.
[18] proposed a spatiotemporal Transformer to reason human-objectrelationships in videos, which detects human/object proposals firstly and then captures spatial andtemporal information by using two dense-connected Transformers, respectively.
However, suchdense-connected manner introduces extra computation and ambiguity in token representation.
Transformer in video analysis.
Transformer [39] has shown a great potential in video analysis, e.g. ,action recognition [28, 48], video restoration [25], video question answering [12], video instancesegmentation [44] and etc.
However, most spatiotemporal Transformer follow the de facto schemeof ViT [7], i.e. , simply dividing an image into local patches and stacking global attention, whichlacks sufficient exploration of the properties of visual signal, thus suffering from insufficiency tokenrepresentation and explosive computation.
Token Abstraction
Backbone
TokenLinking
MLP
Hold Hold
Shoot
GlobalContextRefining
Decoder
IWP
FC
Concat
Regular Window Partition
Irregular Window Partition
Offset filed
Conv.
S t a g e ×
W-MSA
Add&Norm
FFN
IWP ...
: Spatial position : Temporal position : Token distillation Selective attetion based S-block
Similarities ... ...
Tubelet token ...
Patch token
I n s t a n ce t ok e n × N :  Methodology
The main idea of TUTOR is to structurize a video into a few tubelet tokens, which serve as highlyabstracted spatiotemporal representations.
To this end, we propose a reinforced tokenization strategy,which jointly performs tokenization and optimization of the Transformer encoder, as illustrated in
Figure 2.
The process of tubelet token generation consists of two steps: 1) Token abstraction alongthe spatial domain, where patch tokens are alternatively updated by a selective attention mechanismand agglomerated into instance tokens;
2) Token linking along the temporal domain, where instancetokens across frames are linked to form tubelet tokens.
We describe these two steps in details below.
3.1 Backbone
Taking a video clip x ∈ R T × H × W × 3 that consists of T frames with size H × W as the input, we usea ResNet [15] followed by a feature pyramid network (FPN)
[26] as the backbone on t -th frame togenerate a feature map z ( t,b )
∈ R H 4 × W 4 × C 0 , where t = 1 , 2 , .., T and C
= 32 is the channel numberof the initial feature map.
3.2 Token Abstraction
Token abstraction is organized in 3 Stage s through a hierarchy of Transformer layers.
Each stageperforms token representation learning by a selective attention mechanism and merges semanticallyrelated patch tokens into instance tokens by an agglomeration layer.
Here, we denote the feature mapof t -th frame inputted into s -th stage as z ( t,s ) ∈
R H s × W s × C s .
Specifically, z ( t, 1) =
z ( t,b ) .
Selective attention.
To eliminate the redundancy caused by information mixture from differentinstances, we are motivated to selectively calculate attention weights among related tokens, i.e. ,tokens belong to the same instances.
To this end, we propose an irregular window partition (IWP)mechanism (orange rectangle in the right of Figure 2), a simple yet effective strategy that samplesa group of related tokens into a local window.
IWP is inspired from regular window partition [30],where the tokens are grouped by sliding a regular rectangle R with size of S w × S w over the featuremap z ( t,s ) in s -th stage.
For instance, R = { [0 , 0] , [0 , 1] , ..., [3 , 4] , [4 , 4] } defines a regular windowwith size of 5 × 5 .
Then, for the i -th regular window, we have
[Equation]
where z ( t,s )
([ x, y ]) ∈ R 1 × C s denotes the feature vector at spatial location [ x, y ] and [ x iw , y iw ] is thelocation of the top-left point of i -th window.
However, as shown in Figure 2, regular windows caneasily divide an instance into several parts due to the limitation of a fixed shape, leads to unrelatedtokens within a window, i.e. , belonging to different instances.
Inspired by deformable DETR [52]and deformalbe convolution [6], IWP makes a simple change by augmenting regular grid R withlearned offsets, which allows the generated irregular windows to be aligned with humans/objects witharbitrary shapes.
With offsets { ∆ p n | n = 1 , 2 , ..., N } and N = | R | , for the tokens in i -th irregularwindow, we have
[Equation]
Specifically, ∆ p n are learned by performing a convolutional layer with kernel size of 3 × 3 over theinput feature map z ( t,s ) .
As the offsets are typically fractional, the right part of Eq.
2 is implementedpractically via bilinear interpolation as
[Equation]
where p = p n +[ x iw , y iw ]+∆ p n denotes an arbitrary location, q enumerates all neighboring integrallocations, and B is the bilinear interpolation kernel.
On this basis, we alternatively update the token representation by stacking several S-blocks (rectanglein white solid line in Figure 2) that are built on selective attention, i.e. , performing attention mechanismwithin irregular windows.
Specifically, each block is computed as
[Equation]
where z l is updated representation for all tokens , p le is sine-based spatial position encoding at lth S-block , and ˆ z denotes various intermediate features.
Here, we factorize the conventional 3Dposition encoding into a 2D spatial position encoding and a 1D temporal one since the spatial andtemporal information are separately extracted.
In detail, “W-MSA” denotes window-based multi-headself-attention, “LN” is layer normalization and “MLP” refers to multi-layer perceptron.
Since theconvolution layer in IWP is operated on 2D feature map yet attention is calculated on sequentialfeatures, we use a “Flatten” ( 2 D → 1 D ) operation to collapse the spatial dimension and a “Reshape”( 1 D → 2 D ) operation to restore it.
The computational complexity of a global MSA (G-MSA) blockand an irregular-window-based block (IW-MSA) for total T frames at s -th stage are respectively:
[Equation]
where K = 3 is the kernel size of convolutional layer and S w is fixed as 7 .
In comparison, IW-MSAcan effectively reduce the computational complexity.
In our experiment, the number of S-block for(1-3)-th stage is set to 1, 1, 3, respectively.
Token agglomeration.
We perform token agglomeration at the end of each stage to merge semantically similar tokens.
Specifically, we first perform IWP with a window size of 2 × 2 to dynamicallysample every 4 related tokens into a window.
Then, we concatenate the tokens within each windowand apply a fully-connected (FC) layer on the concatenated 4 C s -dimensional features.
We set theoutput dimension to 2 C s .
It reduces the number of tokens by a multiple of 2 × [-
-]2 = 4 after each stage.
To sum up, token abstraction totally reduces the number of tokens by a factor of 4 3 = 64 and increasesthe dimension by a factor of 2 3 = 8 .
It structures each frame into a few instance tokens on the basisof selective attention and token agglomeration, which reduces the visual redundancy and also enjoysthe advantage of Transformer with an affordable computational costs.
3.3 Token Linking
Assuming that after token abstraction, each frame is structured as J instance tokens.
Then a video clipof T frames can be denoted as Z ins =
{ z jt | j = 1 , 2 , ..., J ; t = 1 , 2 , ..., T } , where z jt refers to the j -thtoken in the t -th frame.
Here, a sine-based 1D temporal position encoding is additionally added to Z ins .
The goal of token linking is to link T instance tokens with the same semantic across T frames, so that avideo clip can be structurized as J spatiotemporal tubelet tokens.
To this end, we propose an exemplarbased between-frame one-to-one matching strategy.
We first chose Z q = { z j q r | j q = 1 , 2 , ..., J } asthe exemplar frame, where r = (cid:4) T 2 (cid:5) is the index of the middle frame in the video clip.
We denote thetokens in the exemplar frame and those in rest frames Z k
= { z j k t | z j k t ∈ Z ins , t ̸ = r } as query tokensand key tokens, respectively.
Then, we compute a similarity matrix A between the query tokens andthe key tokens via a Gumbel-Softmax [17] operation computed over the query tokens as
[Equation]
where W j q q and W j k k are the weights of the learned linear projections for the j q -th query tokens andthe j k -th key tokens, respectively, and γ s are i.i.d random samples drawn from the Gumbel(0,1)distribution that enables the Gumbel-Softmax distribution to be close with the real categoricaldistribution.
Then, we introduce a modified nms-one-hot operation to determine the one-to-onecorrespondence between the query tokens and the key tokens of each frame.
Specifically, for j kth key token z j k t in t -th frame, the norm one-hot assignment is performed by taking the valueof argmax { A ( j q ,j k ) t | j q = 1 , 2 , ..., J } .
However, such an operation cannot ensure a one-to-onecorrespondence, i.e. , more than one key tokens in the same frame could be assigned to the same querytoken.
To address this issue, in nms-one-hot scheme, for example, when m 1 -th and m 2 -th keytoken in t -th frame are assigned to j q -th query token simultaneously, we assign the one with a highersimilarity, i.e. , max ( A ( j q ,m 1 ) t , A ( j q ,m 2 ) t ) , to the j q -th query token.
Then, if z m 1 t has been assignedto the j q -th query token, we manually set A ( j q ,m 2 ) t as 0 and continue to conduct the assignmentoperation.
Since the nms-one-hot operation is not differentiable, we adopt the straight throughstrategy in [8] to compute the assignment matrix:
[Equation]
where sg ( · ) is the stop gradient operator.
ˆ A is numerically equal to nms-one-hot assignments andits gradient is equal to the gradient of A , which makes the token linking module differentiable andend-to-end trainable.
Finally, we link the tokens corresponding to the same query token to form thetubelet tokens Z tube =
{ z j tube | j = 1 , 2 , ..., J } , which is computed as
[Equation]
where W o and W v are the learned weights of projectors, and ϕ ( j, t ) is the index of token in t -th frameand being assigned to j -th query token.
3.4 Global Context Refining
After token agglomeration and linking, a spatiotemporal video representation is structurized as afew tubelet tokens.
On this basis, we perform an additional global attention layer to model globalcontextual information.
Our intuition is two-fold: 1)
Global context can significantly boost theperformance of interaction recognition, e.g. , if grassland is detected, a person is more likely tobe playing soccer than basketball.
Different interactions can be co-occurring, e.g. , a person isholding a fork could be eating something.
3.5 Decoder & Prediction Head
Decoder.
Following the standard architecture in [2], the decoder transforms N q embeddings bystacking 6 layers consisting of self-attention and cross-attention mechanisms.
These embeddings arelearned position encodings which are initialized to constants and we refer them to as HOI queries .
Being added to the input of each attention layer, the N q queries are transformed as output embeddingsby the decoder, which performs global reasoning by using the entire video clip as context.
Prediction head.
Following [37], the prediction head is composed of four feed-forward networks(FFNs): human-bounding-box FFNs f h , object-bounding-box FFNs f o , object-class FFNs f co , and action-class FFNs f ca .
Specifically, f h and f o are both a 3-layer perceptron followed by a sigmoidfunction, which output normalized human- and object-bounding box ˆ b h ∈ [0 , 1] 4 , ˆ b o ∈ [0 , 1] 4 ,respectively. f co is a linear layer followed by a softmax function, predicting the probability of objectclasses ˆ c o ∈ 0 , 1] N obj +1 , where N obj is the number of object classes and the ( N obj +1) -th element inˆ c
o indicates the query has no corresponding human-object pair.
Since actions could be co-occurring,f ca is a linear layer followed by a sigmoid function rather than the softmax function.
It outputs theprobability of action classes ˆ c a ∈ [0 , 1] N act , which has no an additional element to indicate no-action .
Here, N act is the number of action classes.
3.6 Loss Function
We follow the loss calculation scheme in [37], including bipartite matching and loss calculation.
Wedescribe the detailed calculating process in supplementary material.
Experiments
4.1 Datasets & Metrics
We conduct experiments on VidHOI [5] and CAD-120 [22] benchmarks to evaluate the proposedmethods by following the standard scheme.
VidHOI is a large-scale dataset for V-HOI detection,comprising 6,366 videos for training and 756 videos for validation.
In VidHOI, 50 relation categoriesare annotated, of which half are time-related ones.
Mean AP ( mAP ) is calculated as the evaluationmetric for VidHOI, which is reported over three sets: 1) Full: all 557 categories are evaluated; 2) Rare:315 categories with less than 25 instances and 3) Non-rare: 242 categories with more than 25instances.
CAD-120 is a relatively smaller dataset that consists of 120 RGB-D videos.
Here, weonly use the RGB images and the 2D bounding boxes annotations of humans and objects.
Followingstandard scheme, we calculate the sub-activity F1 score as metrics.
4.2 Implementation Details
The dimension of HOI query is set to 256, which is the same as the tubelet tokens ( 32 × 2 3 ).
Thenumber of queries is set to 100 for VidHOI and 50 for CAD-120.
To save computational resources, thebackbone is initialized by the backbone weights of QPIC [37], and then frozen without being updated.
We employed an AdamW [31] optimizer for 150 epochs.
A batch size of 16 on 8 RTX-2080Ti GPUs,and learning rate l r = 2 .
5 e − 4 for Transformer and 1 e − 5 for FPN are used.
The l r decayed by half at50-th, 90-th and 120-th epoch, respectively.
We use a l r = 10 − 6 to warm up the training for the first5 epochs, and then go back to 2 .
5 e − 4 and continue training.
4.3 Analysis of CNN-based & Transformer-based Methods
We compare CNN-based and Transformer-based methods in terms of: 1) long-range dependencymodeling, 2) robustness to time discontinuity and 3) contextual relation reasoning.
Long-range dependency modeling.
We split HOI instances into bins of size 0.1 according to thenormalized spatial distances, and report the APs of each bin.
As shown in Figure 3a, our Transformerbased method outperforms existing CNN-based methods in all cases, which becomes increasinglyevident as the spatial distance grows.
It indicates that Transformer has better long-range dependencymodeling capability compared to CNN-based methods that relay on limited receptive field.
With thisability, Transformer can dynamically aggregate important information from global context.
Robustness to time discontinuity.
we randomly sample one frame every t seconds from the originalvideo to generate a new video as inputs, and report the relative performance compared to the baseline(sampling 1 frame per second).
As shown in Figure 3b, the performance of CNN-based methodsdrop dramatically in contrast to Transformer.
The main reason is that the ROI features from differentframes are likely to be inconsistent due to the discontinuity of temporal domain.
For Transformer, itcan be partly solved by learning a variable attention weights to selectively process different featuresof different frames. [Figure] [Figure]
Contextual relations reasoning.
we randomly pick 5 static interaction types (represented in blue)and 5 dynamic ones (green), each with over 10,000 images.
Then, we calculate the average weightsof self-attention in the last decoder layer on all pictures where two interactions are co-predicted.
Asshown in Figure 3c, Transformer can mine the interrelations among different HOI instances, e.g. ,watch and feed, which two are likely to co-occur, get a relatively high attention weights (0.71).
4.4 Analysis of Token abstraction and Linking
Token abstraction.
Table.
1a shows the influence of token abstraction, which is proposed to captureinstance level representation.
In comparison, CNN-based methods process cropped proposal features(Figure 1(a)), which suffers from temporal inconsistencies and the lack of contextual information,leading to the worst performance.
In terms of Transformer, performance is significantly improved (over 25% ), but varies under different strategies.
Interestingly, adding a simple token fusion module to ViT-like framework (ViT-like † ), i.e. , fusing every 4 neighboring patch tokens after each Transformerlayer, can achieve a 4% relative mAP improvement.
It implies that visual redundancy is an obstaclefor Transformer to achieve better performance.
Moreover, our irregular-window-based (IR-win) tokenabstraction mechanism achieves the optimal performance.
Nevertheless, when replacing all irregularwindows in TUTOR with regular windows (R-win), the performance is unexpectedly surpassed by ViT-like † .
It indicates that regular windows can reduce the computational complexity, but cannoteliminate visual redundancy thoroughly.
Token linking.
Table.
1b shows the influence of token linking.
Here, the inputs for all methods areidentical, which are the instance tokens generated by token abstraction module.
Although computingglobal attentions along temporal domain without token linking achieves a competitive performance inthe detection of time-related interactions, its performance is relatively poor for detecting static HOIs.
We conjecture that the instance tokens in different frames are semantically similar, which introducesredundant information to static interaction detection.
Moreover, the mAP decreases severely whendirectly use the value of Gumbel-Softmax as assignment weights, i.e. , replace the ˆ A in Eq.11 with
A in Eq.9.
One possible reason is the redundancy arises within token representation due to theabsence of zero value in A .
In contrast, one-hot assignment is sparse but cannot ensure an one-to-oneassignment among frames, which can also cause ambiguity.
In comparison, nms-one-hot assignmentenforces every T tokens (one per frame) to be linked, which minimizes the ambiguity and redundancyin token representation, thus achieving the optimal performance.
We further investigate the effectof video length on these two assignment approaches.
As shown in Figure 1c, one-hot assignmentsurpasses nms-one-hot when a video is longer than 16 seconds, which is caused by the simple way ofchoosing exemplar frame, i.e. , intuitively selecting the middle frame.
When a video clip is long, themiddle frame is semantically inconsistent with the frames that are temporally far away.
We solve thisproblem by splitting a long video into uniform short clips and performing nms-one-hot assignment ineach clip (nms-one-hot*) respectively, which yet introduces more computation.
4.5 Analysis of Effectiveness and Efficiency
Effectiveness.
We verify TUTOR’s effectiveness of capturing spatial and temporal semantic byobserving the performance of detecting static HOI and dynamic HOI when using a quite simple [Table] [Table] [Table]
decoder. For former, we use a 1-layer Transformer decoder on patch tokens in ViT-like methodand instance tokens in TUTOR, respectively.
the mAP is reported only on static HOI detection.
Asthe Table 2a shows, instance tokens generated by token abstraction can stupendously improve theperformance by 70% , compared with patch tokens.
It demonstrates that token abstraction mechanismcan significantly extract highly-abstracted instance level semantic, which can be easily capturedeven the decoder is simple.
For dynamic HOI detection, we use a 4-layer perception with RELUin between as decoder.
Next, we perform a global average pooling on tubelet tokens and instancetokens, which are then fed to the simple decoder to predict the dynamic interactions, respectively.
Interestingly, tubelet tokens generated by token linking boost performance by 4 × , showing us theimportance to reduce the temporal redundancy.
Efficiency.
Computing attention weights accounts for most of computational overhead in Transformer.
Compared to the quadratic computational costs in global attention, we achieve a linear one.
As shownin Table 2b, TUTOR achieves a 4 × speedup, greatly improving its usability in practical applications.
Here, “FPS” is reported in terms of video, i.e. , the number of videos being processed per second.
4.6 Ablation Study
Token agglomeration.
Token agglomeration is proposed to distill the token representation byselectively merging and projecting the semantically related tokens.
We first intuitively try the kmean [32], an excellent classical clustering algorithm, but obtain a unexpectedly poor performance,as shown in Table 3a.
The reason is two-fold:1) it is difficult to integrate the k-mean with the mainnetwork into an end-to-end pipeline; 2) it is hard to determine the value of K.
Then we replaceirregular winodws in TUTOR with regular windows to merge every 4 neighboring tokens, an averagepooling operation essentially, which reduces the feature redundancy to some extent.
In comparison,irregular-window is more of an operation to selectively merge the semantically similar tokens tomodel highly abstracted features.
It is worth emphasizing that gradually increase the dimension ofagglomerated token is interestingly important, which achieves a gain of more than 6% on mAP.
Weguess that the features are richer with increasing dimensions, as extensively adopted in CNNs.
Window size.
Table 3b varies the window size.
The instances in an image could have variant sizes.
Therefore, a small-sized window is hard to overlap different instances while a large-sized one couldcause information mixture as unrelated tokens may be included.
We find 7 to be optimal.
Small tricks.
We represent some small tricks for key module design in Table 3c.
For nms-one-hotassignment, another commonly used strategy for merging assigned tokens is to concatenate them and [Table] [Table] then project them with a fully-connected layer.
Compared with weighted-sum, it can slightly improveperformance, but introduces more computation.
For position encoding, we factorize the normallyused 3D position encoding for spatiotemporal Transformer into a 2D spatial position encoding and a1D temporal one, which two are added in token agglomeration and linking module, respectively.
Itis an experiential operation since spatial and temporal information are separately extracted.
Globalcontext refining, which models global contextual information, can achieve almost 1 point gain.
4.7 Comparison with State-of-the-art
Unlike the popularity of image-based HOI detection, relatively less works investigate video-basedone as a more practical yet challenging problem.
Interestingly, the ability of image-based methodsto detect dynamic HOI can be partly improved by replacing the original 2D backbone with a 3Done, but it weaken the ability of detecting static HOIs.
With aforementioned strategies, our methodsoutperforms existing sota methods by a large margins.
It is our belief that detecting HOI from videois more reasonable and practical since most interactions are time-related.
Therefore, we hope ourwork will be useful for video-based human activity understanding research.
Discussion & Conclusion
Limitation.
Our Transformer-based method suffers from the problem of overfitting when handlingwith small-scale datasets.
In our experiments, we have to use the pertrained weights on VidHOI toinitialize the model for CAD-120 (small scale), or performance would be severely degraded.
Broader impacts.
We know some applications that illegally analyze user behavior by video monitoring.
Therefore, strict ethical review is essential to avoid our model being used for such applications.
Conclusion.
In this paper, we present TUTOR, a novel spatiotemporal Transformer for video-based
HOI detection, which structurizes a video into a few tubelet tokens.
To generate compact andexpressive tubelet tokens, we propose a token abstraction scheme built on selective attention andtoken agglomeration, along with token linking strategy to link semantically-related tokens acrossframes.
Our methods outperforms existing works by large margins.
Going further, visual redundancyis one of the biggest obstacles for vision Transformer to achieve the same excellent performance aslanguage Transformer, and we will devote more exploration on this in the future works.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes]
• Did you include the license to the code and datasets?
[No] The code and the data are
proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 5. (d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The code isprovided in supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects..
(a) Did you include the full text of instructions given to participants and screenshots ifapplicable
[N/A]
(b) Did you describe any potential participant risks , with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant
compensation?
[N/A]
A Appendix
Optionally include extra information (complete proofs, additional experiments and plots) in theappendix.
This section will often be part of the supplemental material.