Video-based Human-Object Interaction Detection from Tubelet Tokens
Anonymous Author(s)
Abstract
We present a novel vision Transforme r , named TUTOR, which is able to learntu belet to kens, served as highly-abstracted spatiotemporal representations, forvideo-based human-object interaction (V-HOI) detection.
The tubelet tokens structurize videos by agglomerating and linking semantically-related patch tokens alongspatial and temporal domains, which enjoy two benefits: 1) Compactness: eachtubelet token is learned by a selective attention mechanism to reduce redundantspatial dependencies from others;
2) Expressiveness: each tubelet token is enabledto align with a semantic instance, i.e. , an object or a human, across frames, thanksto agglomeration and linking.
The effectiveness and efficiency of TUTOR areverified by extensive experiments.
Results shows our method outperforms existingworks by large margins, with a relative mAP gain of 16 .
14% on VidHOI and a 2points gain on CAD-120 as well as a 4 × speedup.
Introduction
Human-object interaction (HOI) detection is a detailed scene understanding task, which requires bothlocalization of interacted human-object pairs and recognition of interaction labels.
Existing methodsmostly investigated detecting HOIs in static images without capturing temporal information (Figure 1(a)), thus lack the ability to detect time-related interactions ( e.g. , shoot or pass a basketball).
However, interactions are more of time-related in practical scenario, leading to a strong demand to detect
HOIs from videos, a more challenging problem built on spatiotemporal semantic representations.
Transformer, originated from natural language processing (NLP), is an intuitive choice for its eminentcapability of reasoning long-range dependencies, in which one of the most crucial components is thetoken.
A token serves as an element of data representations, which is usually a word in language.
However, unlike language that naturally has such a discrete signal space for building tokenizeddictionaries, images lie in a continuous and high-dimensional space.
To address this issue, vision
Transformer (ViT) [5] provided a solution that divides each image into several local patch tokens(“visual words”) to structurize the entire image as a “visual sentence”(Figure 1(b)).
This solutionhas become a de facto tokenization standard followed by most existing Transformer-based methods,which has achieved excellent performance for various vision tasks, especially image classification.
Nevertheless, this patch based tokenization strategy might not be proper for video-based HOI (V-HOI)detection (as the performance degradation of the ViT-like framework shown in Table.
1a).
We findthat the reason is the patch tokens generated by regular splitting are difficult to exactly captureinstance-level semantics (an instance is an object or a human, e.g. , the basketball shooter in Figure 1),which yet is crucial for V-HOI detection to reason the interaction labels.
These patch tokens inevitablysuffer from redundancy due to an information mixture from different instances as well as insufficiencydue to only a part occupancy of an instance, which limit their representation ability.
In this paper, we present TUTOR, a new Transforme R for V-HOI detection built on TU belet TO kens,handling aforementioned limitations favorably.
The tokenization of the tubelet tokens is not basedon fixed regular splitting but is jointly performed with the learning of the Transformer encoder.
This enables the tubelet tokens to progressively emerge and represent high-level visual semantics. [Figure]
Concretely, first, along the spatial domain, we alternatively update the representation for each patchtoken by a selective attention mechanism and agglomerate semantically-related patch tokens intoinstance tokens.
The selective attention mechanism ensures that attention is performed among tokensexpected to belong to the same instance, which reduce redundant spatial dependencies from others.
Then, along the temporal domain, we link instance tokens across frames to form the tubelet tokens.
Figure 1(c) illustrates the process of tubelet token generation.
Experimental results show that TUTORoutperforms existing sota methods by large margins.
Specifically, we achieve a relative mAP gain of16 .
4% on VidHOI [4] and a 2 points F1 score gain CAD-120 [21], with a 4 × inference speedup.
Related Work
HOI detection.
HOI detection has attracted considerable research interests recently for its greatpotential in high-level visual understanding tasks, such as captioning [14], visual question answering [10].
Most previous works are devoted to detecting HOIs in static images [2, 8, 9, 11, 12, 15,18, 20, 23, 24, 27, 29, 35, 38, 40, 41, 43, 45, 46, 49, 50, 51, 37, 53, 19, 3, 47].
Without considering temporal information, these methods fail to detect time-related interactions, restricting theirvalue in practical applications.
In contrast, video-based HOI detection is a more practical problem,which however is less explored [35, 33, 34, 36, 4, 42, 17].
[35, 36, 42] detected HOIs in videosby building graph neural networks to capture spatiotemporal information.
In [33], HOI “hotspots”can be directly learned from videos by jointly training a video-based action recognition networkas well as an anticipation model.
Inspired by image-based methods, [4] introduced a two-stageframework where the frame-wise human/object features are firstly extracted by using trajectories, andthen HOIs are detected by processing the instance features as well as auxiliary features, includingspatial configurations and human poses.
However, these methods lack the ability to model longrange contextural information, resulting in poor performance when the interacted human and objectare far apart.
Inspired by the strong ability of Transformer in modeling long-range dependencies,[17] proposed a spatiotemporal Transformer to reason human-object relationships in videos, whichdetects human/object proposals firstly and then captures spatial and temporal information by usingtwo dense-connected Transformers, respectively.
However, such dense-connected manner introducesextra computation and ambiguity in token representation as mentioned above.
Transformer in video analysis.
Transformer [39] originated from natural language processing(NLP), and has been widely explored in vision tasks recently.
Specifically, Transformer has also showa great potential in video analysis, e.g. , action recognition [28, 48], video restoration [25], videoquestion answering [10], video instance segmentation [44] and etc.
However, most spatiotemporal Transformer follow the de facto scheme of ViT [5], i.e. , simply dividing an image into local patchesand stacking global attention, which lacks sufficient exploration of the properties of visual signal,thus suffering from insufficiency token representation and explosive computation.
Methodology
The main idea of TUTOR is to structurize a video into a few tubelet tokens, which serve as highlyabstracted spatiotemporal representations.
Towards this end, we propose a reinforced tokenizationstrategy, which jointly performs tokenization and optimization of the Transformer encoder, as
Token Abstraction
Backbone
TokenLinking
MLP
Hold Hold
Shoot
GlobalContextRefining
Decoder
IWP
FC
Concat
Regular Window Partition
Irregular Window Partition
Offset filed
Conv.
S t a g e ×
W-MSA
Add&Norm
FFN
IWP ...
: Spatial position : Temporal position : Token distillation Selective attetion based S-block
Similarities ... ...
Tubelet token ...
Patch token
I n s t a n ce t ok e n × N : illustrated in Figure 2.
The process of tubelet token generation consists of two steps: 1) Tokenabstraction along the spatial domain, where patch tokens are alternatively updated by a selectiveattention mechanism and agglomerated into instance tokens; 2)
Token linking along the temporaldomain, where instance tokens across frames are linked to form tubelet tokens.
In this section, wedescribe these two steps in details.
3.1 Backbone
Taking a video clip x ∈ R T × H × W × 3 that consists of T frames with size H × W as the input, we usea ResNet [13] followed by a feature pyramid network (FPN)
[26] as the backbone on t -th frame togenerate a feature map z ( t,b )
∈ R H 4 × W 4 × C 0 , where t = 1 , 2 , .., T and C
= 32 is the channel numberof the initial feature map.
3.2 Token Abstraction
Token abstraction along the spatial domain is organized in 3 Stage s through a hierarchy of Transformerlayers.
Each stage perform token representation learning by a selective attention mechanism andmerge semantically-related patch tokens into instance tokens by an agglomeration layer.
Here, wedenote the feature map of t -th frame inputted into s -th stage as z ( t,s ) ∈
R H s × W s × C s .
Specifically,z ( t, 1) =
z ( t,b ) .
Selective attention.
To eliminate the redundancy caused by information mixture from differentinstances, we are motivated to selectively calculate attention weights among related tokens, i.e. ,tokens belong to the same instances.
To this end, we propose an irregular window partition (IWP)mechanism (orange rectangle in the right of Figure 2), a simple yet effective strategy that samplesa group of related tokens into a local window.
IWP is inspired from regular window partition ( e.g. , Swin [30]), where the tokens are grouped by sliding a regular rectangle R with size of S w × S w overthe feature map z ( t,s ) in s -th stage.
For instance, R = { [0 , 0] , [0 , 1] , ..., [3 , 4] , [4 , 4] } defines a regularwindow with size of 5 × 5 .
Then, for the i -th regular window, we have
[Equation]
where z ( t,s )
([ i, j ]) ∈ R 1 × C i denotes the feature vector at spatial location [ i, j ] , which is also therepresentation of ( i · j ) -th token, unless specified below.
However, as shown in Figure 2, regularwindows can easily divide an instance into several parts due to the limitation of a fixed shape, leads tounrelated tokens within a window, i.e. , belonging to different instances.
IWP makes a simple changeby augmenting regular grid R with learned offsets, which allows the generated irregular windowsto be aligned with humans/objects with arbitrary shapes.
With offsets { ∆ p n | n = 1 , 2 , ..., N } and N = | R | , for the tokens in i -th irregular window, we have Z i
[Equation]
z [Equation] Specifically, ∆ p n are learned by performing a convolutional layer with kernel size of 3 × 3 over theinput feature map z ( t,s ) .
As the offsets are typically fractional, the right part of Eq.
2 is implementedpractically via bilinear interpolation as
[Equation]
t [Equation] where p = p n + [( i − 1) S w , ( i − 1) S w ] + ∆ p n denotes an arbitrary location, q enumerates allintegral locations, and B is the bilinear interpolation kernel.
On this basis, we alternatively update the token representation by stacking several S-blocks (rectanglein white solid line in Figure 2) that are built on selective attention, i.e. , performing attention mechanismwithin irregular windows.
Specifically, each block is computed as
[Equation]
where z l is updated representation for all tokens, p ls is sine-based spatial position encoding at l -th S-block , and ˆ z denotes various intermediate variables.
The detailed calculation process can befound in supplementary file .
Here, we factorize the conventional 3D position encoding into a 2Dspatial position encoding and a 1D temporal one since the spatial and temporal information areseparately extracted.
In detail, “W-MSA” denotes window-based multi-head self-attention, “LN”is layer normalization and “MLP” refers to multi-layer perceptron.
Since the convolution layerin IWP is operated on 2D feature map yet attention is calculated on sequential features, we use a“Flatten” ( 2 D → 1 D ) operation to collapse the spatial dimension as well as a “Reshape” ( 1 D → 2 D )operation to restore it.
The computational complexity of a global MSA (G-MSA) block and airregular-window-based block (IW-MSA) for total T frames at s -th stage are respectively:
[Equation]
where K = 3 is the kernel size of convolutional layer and S w is fixed as 7 .
In comparison, IW-MSAcan effectively reduce the computational complexity.
In our experiment, the number of S-block for(1-3)-th stage is set to 1, 1, 3, respectively.
Token agglomeration.
We perform token agglomeration at the end of each stage to merge semantically similar tokens.
Specifically, we first perform IWP with a window size of 2 × 2 to dynamicallysample every 4 related tokens into a window.
Then, we concatenate the tokens within each windowand apply a fully-connected (FC) layer on the concatenated 4 C s -dimensional features (each token isrepresented by a feature vector with C s channels at s -th stage).
We set the output dimension to 2 C s .
This process reduces the number of tokens by a multiple of 2 × {+
+} = 4 after each stage.
To sum up, token abstraction totally reduces the number of tokens by a factor of 4 3 = 64 and increasesthe dimension by a factor of 2 3 = 8 .
It structures each frame into a few instance tokens on the basisof selective attention and token agglomeration, which reduces the visual redundancy and also enjoysthe advantage of Transformer with a affordable computational costs.
3.3 Token Linking
Token linking aims at forming compact and expressive tubelet tokens by linking instance tokens withthe same semantic along the temporal domain.
Specifically, assuming that after token agglomeration,each frame is structured as N instance tokens.
Then a video clip of T frames can be denoted as 144 Z ins =
{ z it | i = 1 , 2 , ..., N ; t = 1 , 2 , ..., T } , where z it refers to the i -th token in the t -th frame.
Here,a sine-based 1D temporal position encoding is additionally added to Z ins .
The goal of token linking isto link T instance tokens with the same semantic across T frames (one per frame), so that the videoclip can be structurized as N spatiotemporal tubelet tokens.
To this end, we propose an exemplarbased between-frame one-to-one matching strategy.
We first chose Z q = { z iq | i = 1 , 2 , ..., N } as theexemplar frame, where q = (cid:4) T 2 (cid:5) is the index of the middle frame in the video clip.
We denote thetokens in the exemplar frame and those in rest frames Z k
= { z it | z it ∈ z s , t ̸ = r } as query tokens and key tokens, respectively.
Then, we compute a similarity matrix A between the query tokens and thekey tokens via a Gumbel-Softmax [16] operation computed over the query tokens as
[Equation]
where W q and W k are the weights of the learned linear projections for the query tokens and the keytokens, respectively, and γ i s are i.i.d random samples drawn from the Gumbel(0,1) distribution thatenables the Gumbel-Softmax distribution to be close with the real categorical distribution.
Then, weintroduce a modified nms-one-hot operation to determine the one-to-one correspondence betweenthe query tokens and the key tokens of each frame.
Specifically, for n -th key token z nt in t -th frame,the norm one-hot assignment is performed by taking the value of argmax { A ( i,n ) t | i = 1 , 2 , ..., N } .
However, such an operation cannot ensure a one-to-one correspondence, i.e. , more than one keytokens in a frame could be assigned to the same query token.
To address this issue, in nms-one-hotscheme, for example, when n -th and m -th key token in t -th frame are assigned to i -th query tokensimultaneously, we assign the the one with a higher similarity, i.e. , max ( A ( i,n ) t , A ( i,m ) t ) , to the i -thquery token.
Then, if z nt has been assigned to the i -th query token, we manually set A ( i,m ) t as 0 andcontinue to conduct the assignment operation.
Since the nms-one-hot operation is not differentiable,we adopt the straight through strategy in [6] to compute the assignment matrix:
[Equation]
where sg ( · ) is the stop gradient operator.
ˆ A is numerically equal to nms-one-hot assignments andits gradient is equal to the gradient of A , which makes the token linking module differentiable andend-to-end trainable.
Finally, we link the tokens corresponding to the same query token to form thetubelet tokens Z tube =
{ z i tube | i = 1 , 2 , ..., N } , which is computed as [Equation] i
[Equation]
where W o and W v are the learned weights of projectors, and ϕ ( i, t ) is the index of token in t -th frameand being assigned to i -th query token.
3.4 Global Context Refining
After token agglomeration and linking, a spatiotemporal video representation is structurized as afew tubelet tokens.
On this basis, we perform an additional global attention layer to model globalcontextual information.
Our intuition is two-fold: 1)
Global context can significantly boost theperformance of interaction recognition, e.g. , if grassland is detected, a person is more likely tobe playing soccer than basketball.
Different interactions can be co-occurring, e.g. , a person isholding a fork could be eating something.
3.5 Decoder & Prediction Head
Decoder.
The decoder follows the standard architecture of the Transformer, which transforms N qembeddings by stacking 6 layers consisting of self-attention and cross-attention mechanisms.
Theseembeddings are learned position encodings which are initialized to constants and we refer them toas HOI queries .
Being added to the input of each attention layer, the N q queries are transformed asoutput embeddings by the decoder, which performs global reasoning by using the entire video clip ascontext.
Prediction head.
Following [37], the prediction head is composed of four feed-forward networks(FFNs): human-bounding-box FFNs f h , object-bounding-box f o , object-class FFNs f co , and actionclass FFNs f ca .
Specifically, f h and f o are both a 3-layer perceptron followed by a sigmoid function,which output normalized human- and object-bounding box ˆ b h ∈ [0 , 1] 4 , ˆ b o ∈ [0 , 1] 4 , respectively.f co is a linear layer followed by a softmax function, predicting the probability of object classesˆ c o ∈ [0 , 1] N obj +1 , where N obj is the number of object classes and the ( N obj + 1) -th
element in ˆ c oindicates the query has no corresponding human-object pair.
Since actions could be co-occurring,f ca is a linear layer followed by a sigmoid function rather than the softmax function.
It outputs theprobability of action classes ˆ c a ∈ [0 , 1] N act , which has no an additional element to indicate no-action .
Here, N act is the number of action classes.
3.6 Loss Function
We follow the loss calculation scheme in [37], including bipartite matching and loss calculation. Bipartite matching.
The model outputs a fixed-size set of N q predictions, which is denoted as
O =
{ o i } N q i
=1 .
We use the G = { g i | i = 1 , 2 , ..., M, ∅ 1 , ∅ 2 , ..., ∅ N q − M } to represent the paddedgroundtruths, where M is the real number of HOI instances in a video clip and ∅ refers to no-instance . Then, the matching process can be formulated as an injective function: ω T → O , being computed as [Equation]
12) where ω ( i ) is the index of predicted HOI instance assigned the i -th groundtruth, and L match ( t i , o ω ( i ) )is the matching cost between the i -th groundtruth and ( ω ( i ))
-th prediction, being calculated as
[Equation] L b is regression cost between groundtruths b and predictions ˆ b for human and object boxes: [Equation] [Equation]
L o is recognition cost between groundtruths c
o (one-hot vector) and predictions ˆ c o for objects:
[Equation]
[Equation]
L a is recognition cost between groundtruths c a and predictions ˆ c a for actions:
[Equation]
which takes both positive and negative action classes into account by using the weighted average ofthe two with the inverse number of nonzero elements as weight rather than the vanilla one.
Finally,we follow DETR [52] to use Hungarian algorithm [22] to determine the optimal assignment ˆ ω amongthe set of all possible permutations of N q elements Ω N q , which is formulated as
[Equation]
[Equation]
Loss calculation.
After the optimal one-to-one matching between the groundtruths and the predictionsis found, the loss to be minimized in the training phase is calculated as L = η 1 L b + η 2 L o + η 3 L a ,where L b is defined as the same as Eq.
14 and L o is the CE loss and L a is the focal loss.
Experiments4.1 Datasets & Metrics
We conduct experiments on VidHOI [4] and CAD-120 [21] benchmarks to evaluate the proposedmethods by following the standard scheme.
VidHOI is a large-scale dataset for V-HOI detection,comprising 6,366 videos for training set and 756 videos for validation.
In VidHOI, 50 relationcategories are annotated, of which half are time-related ones.
As same as the widely used imagebased HOI detection dataset HICO-DET [2], mean AP ( mAP ) is calculated as the evaluation metricfor VidHOI, which is reported over three sets: 1)Full: all 557 categories are evaluated; 2)Rare:315categories with less than 25 instances and 3)Non-rare: 242 categories with more than 25 instances.
CAD-120 is a relatively smaller dataset that consists of 120 RGB-D videos.
Here, we only use the RGB images and the 2D bounding boxes annotations of humans and objects.
Following standardscheme, we calculate the sub-activity F1 score as metrics.
4.2 Implementation Details
The dimension of HOI query is set to 256, which is the same as the tubelet tokens ( 32 × 2 3 ).
Thenumber of queries is set to 100 for VidHOI and 50 for CAD-120.
Following [37], we set α 1 , α 2 ,α 3 to 1, 1, 1, and β 1 , β 2 to 2.5, 1, respectively.
For loss calculation, η (1 , 2 , 3) are all set to 1.
To savecomputational resources, the backbone is initialized by the backbone weights of QPIC [37], and thenfrozen without being updated.
We employed a AdamW [31] optimizer for 150 epochs.
A batch sizeof 16 on 8 RTX-2080Ti GPUs, and learning rate l r = 2 .
5 e − 4 for Transformer and 1 e − 5 for FPN areused.
The l r decayed by half at 50-th, 90-th and 120-th epoch, respectively.
We use a l r = 10 − 6 towarm up the training for the first 5 epochs, and then go back to 2 .
5 e − 4 and continue training.
4.3 Analysis of CNN-based & Transformer-based Methods
We compare CNN-based and Transformer-based methods in terms of: 1) long-range dependencymodeling, 2) robustness to time discontinuity and 3) contextual relation reasoning. [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Table] [Figure] Long-range dependency modeling. We split HOI instances into bins of size 0.1 according to thenormalized spatial distances, and report the APs of each bin.
As shown in Figure 3a, our Transformerbased method outperforms existing CNN-based methods in all cases, which becomes increasinglyevident as the spatial distance grows.
It indicates that Transformer has better long-range dependencymodeling capability compared to CNN-based methods that relay on limited receptive field.
With thisability, Transformer can dynamically aggregate important information from global context.
Robustness to time discontinuity.
we randomly sample one frame every t seconds from the originalvideo to generate a new video as inputs, and report the relative performance compared to the baseline(sampling 1 frame per second).
As shown in Figure 3b, the performance of CNN-based methodsdrop dramatically in contrast to Transformer.
The main reason is that the ROI features from differentframes are likely to be inconsistent due to the discontinuity of temporal domain.
For Transformer, itcan be partly solved by learning a variable attention weights to selectively process different featuresof different frames.
Contextual relations reasoning.
we randomly pick 5 static interaction types (represented in blue)and 5 dynamic ones (green), each with over 10,000 images.
Then, we calculate the average weightsof self-attention in the last decoder layer on all pictures where two interactions are co-predicted.
Asshown in Figure 3c, Transformer can mine the interrelations among different HOI instances, e.g. ,watch and feed, which two are likely to co-occur, get a relatively high attention weights (0.71).
4.4 Analysis of Token abstraction and Linking
Token abstraction.
Table.
1a shows the influence of token abstraction, which is proposed to captureinstance level representation.
In comparison, CNN-based methods process cropped proposal features(Figure 1(a)), which suffers from temporal inconsistencies and the lack of contextual information,leading to the worst performance.
In terms of Transformer, performance is significantly improved(over 25% improvement), but varies under different strategies.
Interestingly, adding a simple tokenfusion module to ViT-like framework (ViT-like † ), i.e. , fusing every 4 neighboring patch tokensafter each Transformer layer, can achieve a 4% relative mAP improvement.
It implies that visualredundancy is an obstacle for Transformer to achieve better performance.
Moreover, our irregularwindow-based (IR-win) token abstraction mechanism achieves the optimal performance.
Nevertheless,when replacing all irregular windows in TUTOR with regular windows (R-win), the performance isunexpectedly surpassed by ViT-like † .
It indicates that regular windows can reduce the computationalcomplexity, but cannot eliminate visual redundancy thoroughly.
Token linking.
Table.
1b shows the influence of token linking.
Here, the inputs for all methods areidentical, which are the instance tokens generated by token abstraction module.
Although computingglobal attentions along temporal domain without token linking achieves a competitive performance inthe detection of time-related interactions, its performance is relatively poor for detecting static HOIs.
We conjecture that the instance tokens in different frames are semantically similar, which introducesredundant information to static interaction detection.
Moreover, the mAP decreases severely whendirectly use the value of Gumbel-Softmax as assignment weights, i.e. , replace the ˆ A in Eq.11 with
A in Eq.9.
One possible reason is the redundancy arises within token representation due to theabsence of zero value in A .
In contrast, one-hot assignment is sparse but cannot ensure an one-to-oneassignment among frames, which can also cause ambiguity.
In comparison, nms-one-hot assignmentenforces every T tokens (one per frame) to be linked, which minimizes the ambiguity and redundancyin token representation, thus achieving the optimal performance.
We further investigate the effectof video length on these two assignment approaches.
As shown in Figure 1c, one-hot assignment [Table] [Table] nms-one-hot when a video is longer than 16 seconds, which is caused by the simple way ofchoosing exemplar frame, i.e. , intuitively selecting the middle frame.
when a video clip is long, themiddle frame is semantically inconsistent with the frames that are temporally far away.
We solve thisproblem by splitting a long video into uniform short clips and performing nms-one-hot assignment ineach clip (nms-one-hot*), respectively, which yet introduces more computation.
4.5 Analysis of Effectiveness and Efficiency
Effectiveness.
We verify TUTOR’s effectiveness of capturing spatial and temporal semantic byobserving the performance of detecting static HOI and dynamic HOI when using a quite simpledecoder.
For former, we use a 1-layer Transformer decoder on patch tokens in ViT-like methodand instance tokens in TUTOR, respectively.
the mAP is reported only on static HOI detection.
Asthe Table 2a shows, instance tokens generated by token abstraction can stupendously improve theperformance by 70% , compared with patch tokens.
It demonstrates that token abstraction mechanismcan significantly extract highly-abstracted instance level semantic, which can be easily capturedeven the decoder is simple.
For dynamic HOI detection, we use a 4-layer perception with RELUin between as decoder.
Next, we perform a global average pooling on tubelet tokens and instancetokens, which are then fed to the simple decoder to predict the dynamic interactions, respectively.
Interestingly, tubelet tokens generated by token linking boost performance by 4 × , showing us theimportance to reduce the temporal redundancy.
Efficiency.
Computing attention weights accounts for most of computational overhead in Transformer.
Compared to the quadratic computational costs in global attention, we achieve a linear one.
As shownin Table 2b, TUTOR achieves a 4 × speedup, greatly improving its usability in practical applications.
Here, “FPS” is reported in terms of video, i.e. , the number of videos being processed per second.
4.6 Ablation Study
Token agglomeration.
Token agglomeration is proposed to distill the token representation byselectively merging and projecting the semantically related tokens.
We first intuitively try the kmean [32], an excellent classical clustering algorithm, but obtain a unexpectedly poor performance,as shown in Table 3a.
The reason is two-fold:1) it is difficult to integrate the k-mean with the mainnetwork into an end-to-end pipeline; 2) it is hard to determine the value of K.
Then we replaceirregular winodws in TUTOR with regular windows to merge every 4 neighboring tokens, an averagepooling operation essentially, which reduces the feature redundancy to some extent.
In comparison,irregular-window is more of an operation to selectively merge the semantically similar tokens tomodel highly abstracted features.
It is worth emphasizing that gradually increase the dimension ofagglomerated token is interestingly important, which achieves a gain of more than 6% on mAP.
Weguess that the features are richer with increasing dimensions, as extensively adopted in CNNs. [Table]
[Table]
Window size.
Table 3b varies the window size.
The instances in an image could have variant sizes.
Therefore, a small-sized window is hard to overlap different instances while a large-sized one couldcause information mixture as unrelated tokens may be included.
We find 7 to be optimal.
Small tricks.
We represent some small tricks for key module design in Table 3c.
For nms-one-hotassignment, another commonly used strategy for merging assigned tokens is to concatenate them andthen project them with a fully-connected layer.
Compared with weighted-sum, it can slightly improveperformance, but introduces more computation.
For position encoding, we factorize the normallyused 3D position encoding for spatiotemporal Transformer into a 2D spatial position encoding and a1D temporal one, which two are added in token agglomeration and linking module, respectively.
It isan experiential operation since spatial and temporal information are separately extracted.
4.7 Comparison with State-of-the-art
Unlike the popularity of image-based HOI detection, relatively less works investigate video-basedone as a more practical yet challenging problem.
Interestingly, the ability of image-based methodsto detect dynamic HOI can be partly improved by replacing the original 2D backbone with a 3Done, but it weaken the ability of detecting static HOIs.
With aforementioned strategies, our methodsoutperforms existing sota methods by a large margins.
It is our belief that detecting HOI from videois more reasonable and practical since most interactions are time-related.
Therefore, we hope ourwork will be useful for video-based human activity understanding research.
Discussion & Conclusion
[Figure]
Limitation.
Our Transformer-based method suffers from the problem of overfitting when handlingwith small-scale datasets.
In our experiments, we have to use the pertrained weights on VidHOI toinitialize the model for CAD-120 (small scale), or performance would be severely degraded.
Broader impacts.
We know some applications that illegally analyze user behavior by video monitoring.
Therefore, strict ethical review is essential to avoid our model being used for such applications.
Conclusion.
In this paper, we represent TUTOR, a novel spatiotemporal Transformer for video-based
HOI detection, which structurizes a video into a few tubelet tokens.
To generate compact andexpressive tubelet tokens, we propose a token abstraction scheme built on selective attention andtoken agglomeration, along with token linking strategy to link semantically-related tokens acrossframes.
Our methods outperforms existing works by large margins.
Going further, visual redundancyis one of the biggest obstacles for vision Transformer to achieve the same excellent performance aslanguage Transformer, and we will devote more exploration on this in the future works.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes]
• Did you include the license to the code and datasets?
[No] The code and the data are
proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that theChecklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 5.(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The code isprovided in supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Appendix
Optionally include extra information (complete proofs, additional experiments and plots) in theappendix.
This section will often be part of the supplemental material.