PointTAD: Multi-Label Temporal Action Detection with Learnable Query Points
Abstract
Traditional temporal action detection (TAD) usually handles untrimmed videos with small number of action instances from a single label (e.g., ActivityNet, THUMOS).
However, this setting might be unrealistic as different classes of actions often co-occur in practice.
In this paper, we focus on the task of multi-label temporal action detection that aims to localize all action instances from a multi-label untrimmed video.
Multi-label TAD is more challenging as it requires for finegrained class discrimination within a single video and precise localization of the co-occurring instances.
To mitigate this issue, we extend the sparse query-based detection paradigm from the traditional TAD and propose the multi-label TAD framework of PointTAD.
Specifically, our PointTAD introduces a small set of learnable query points to represent the important frames of each action instance.
This point-based representation provides a flexible mechanism to localize the discriminative frames at boundaries and as well the important frames inside the action.
Moreover, we perform the action decoding process with the Multi-level Interactive Module to capture both point-level and instance-level action semantics.
Finally, our PointTAD employs an end-to-end trainable framework simply based on RGB input for easy deployment.
We evaluate our proposed method on two popular benchmarks and introduce the new metric of detection-mAP for multi-label TAD.
Our model outperforms all previous methods by a large margin under the detectionmAP metric, and also achieves promising results under the segmentation-mAP metric. Code is available at https://github.com/MCG-NJU/PointTAD.
Introduction
With the increasing amount of video resources on the Internet, video understanding is becoming one of the most important topics in computer vision.
Temporal action detection (TAD) [52, 23, 21, 4, 10, 19, 37] has been formally studied on traditional benchmarks such as THUMOS [15], ActivityNet [14], and HACS [50].
However, the task seems impractical because their videos almost contain nonoverlapping actions from a single category: 85% videos in THUMOS are annotated with single action category.
As a result, most TAD methods [23, 21, 45, 5, 35] simply cast this TAD problem into sub-problems of action proposal generation and global video classification [42].
In this paper, we shift our playground to the more complex setup of multi-label temporal action detection, which aims to detect all action instances from multi-labeled untrimmed videos.
Existing works [9, 16, 40, 8] in this field formulate the problem as a dense prediction task and perform multi-label classification in a frame-wise manner.
Consequently, these methods are weak in localization and fail to provide the instance-level detection results (i.e., the starting time and ending time of each instance).
In analogy to image instance segmentation [22], we argue that it is necessary to redefine multi-label TAD as a [Figure] instance-level detection problem rather than a frame-wise segmentation task.
In this sense, multi-label
TAD results not only provide the action labels, but also the exact temporal extent of each instance.
Direct adaptation of action detectors is insufficient to deal with the challenges of concurrent instances and complex action relations in multi-label TAD.
The convention of extracting action features from action segments [4, 31, 49, 43] lacks the flexibility of handling both important semantic frames inside the instance as well as discriminative boundary frames.
Consider the groundtruth action of “Jump” in Fig.
1, segment-based action detectors mainly produce two kinds of error predictions, as in type
S 1 and type S 2 .
S 1 successfully predicts the correct action category with an incomplete segment of action highlights, whereas S 2 does a better job in locating the boundaries yet get misclassified as “Sit” due to the inclusion of confusing frames.
In addition, most action detectors [49, 31, 25] are inadequate in processing sampled frames and classifying fine-grained action classes.
They often exploit temporal context modeling at a single level and ignore the exploration of channel semantics.
To address the above issues, we present PointTAD, a sparse query-based action detector that leverages learnable query points to flexibly attend important frames for action instances.
Inspired by RepPoints [47], the query points are directly supervised by regression loss.
Given specific regression targets, the query points learn to localize at discriminative frames at action boundaries as well as semantic key frames within actions.
Hence, concurrent actions of different categories can yield distinctive features through the specific arrangement of query points.
Moreover, we improve the action localization and semantics decoding by proposing the Multi-level Interactive Module with dynamic kernels based on query vector.
Direct interpolation or pooling at each query point lacks temporal reasoning over consecutive frames.
Following deformable DETR [55], we extract point-level features with deformable convolution [6, 54] from a local snippet to capture the temporal cues of action change or important movement.
At instance-level, both temporal context and channel semantics are captured with frame-wise and channel-wise dynamic mixing [36, 11] to further decode the distinctive features of simultaneous actions.
PointTAD streamlines end-to-end TAD with joint optimization of the backbone network and action decoder without any post-processing technique.
We validate our model on two challenging multilabel TAD benchmarks.
Our model achieves the state-of-the-art detection-mAP performance and competitive segmentation-mAP performance to previous methods with RGB input.
Related Work
Multi-label temporal action detection.
Multi-label temporal action detection has been studied as a multi-label frame-wise classification problem in the previous literature.
Early methods [29, 30] paid a lot of attention on modeling the temporal relations between frames with the help of Gaussian filters in temporal dimension.
Other works deployed dilated temporal kernels [9] or iterative temporal convolution and self attention blocks [8] to integrate features at different temporal scales.
Recently, attention has shifted beyond temporal modeling.
Coarse-Fine [16] handled different temporal resolutions in the slow-fast fashion and performed spatial-temporal attention during fusion.
MLAD [40] used multi-head self-attention blocks at both spatial and class dimension to model class relations at each timestamp.
In our proposed method, we study the temporal context at different semantic levels and model the channel dynamics within each instance.
In addition, we employ point-based action representation and query-based action detection framework for more accurate and precise action localization.
Segment-based representation.
Following the prevailing practice of bounding boxes [20, 13, 32, 1] in object detection, existing temporal action detectors incorporated action segments heavily with three kinds of usage: as anchors, as intermediate proposals, and as final predictions.
Segments as anchors are explored mainly in anchor-based frameworks.
These methods [28, 26, 49, 31] used sliding windows or proposal generation results as anchors.
Most TAD methods [49, 19, 45, 4, 52, 51] use segments as intermediate proposals.
Uniform sampling or pooling are commonly used to extract features from these segments.
P-GCN [49] applied max-pooling within local segments for proposal features.
G-TAD [45] uniformly divided segments into bins and average-pooled each bin to obtain proposal features.
AFSD [19] proposed boundary pooling in boundary region to refine action feature.
Segments as final predictions are employed among all three types of TAD frameworks, because segments generally facilitate the computation of action overlaps and loss calculation.
Instead, in this paper, we do not need segments as anchors and directly employ learnable query points as intermediate proposals with iterative refinement.
The learnable query points represent the important frames within action and action feature is extracted only from these keyframes rather than using RoI pooling.
Point-based representation.
Several existing works have used point representations to describe keyframes [12, 38], objects [47, 11], tracks [53], and actions [18].
[12, 38] tackled keyframe selection by operating greedy algorithm on spatial SIFT keypoints [12] or clustering on local extremes of image color/intensity [38].
These methods followed a bottom-up strategy to choose keyframes based on local cues.
In contrast, PointTAD represents action as a set of temporal points (keyframes).
We follow
RepPoints [47] to handle the important frames of actions with point representations and refine these points by action feature iteratively.
Our method directly regresses keyframes from query vectors in a top-down manner for more flexible temporal action detection.
Note that PointTAD tackles different tasks from RepPoints [47].
We also built PointTAD upon a query-based detector, where a small set of action queries is employed to sparsely attend the frame sequence for potential actions, resulting in an efficient detection framework.
Temporal context in videos.
Context aggregation at different levels of semantics is crucial for temporal action modeling [41] and has been discussed in previous TAD methods.
G-TAD [45] treated each snippet input as graph node and applied graph convolution networks to enhance snippet-level features with global context.
ContextLoc [56] handled action semantics in hierarchy: it updated snippet features with global context, obtained proposal features with frame-wise dynamic modeling within each proposal and modeled the inter-proposal relations with GCNs.
Although we considered the same levels of semantic modeling, our method is different from ContextLoc.
PointTAD focuses on aggregating temporal cues at multiple levels, with deformable convolution as well as frame and channel attentions. We also apply multi-head self-attention for inter-query relation modeling.
PointTAD
We formulate the task of multi-label temporal action detection (TAD) as a set prediction problem.
Formally, given a video clip with T consecutive frames, we predict a set of action instances Ψ = { ψ n = ( t sn , t en , c n ) } N q n =1 , N q is the number of learnable queries, t sn , t en are the starting and ending timestamp of the n -th detected instance, c n is its action category.
The groundtruth action set to detect are denoted ˆΨ =
{ ˆ ψ n = ( ˆ t sn , ˆ t en , ˆ c n ) } N g n =1 , where ˆ t sn , ˆ t en are the starting and ending timestamp of the n -th action, ˆ c n is the groundtruth action category, N g is the number of groundtruth actions.
The overall architecture of PointTAD is depicted in Fig.
PointTAD consists of a video encoder and an action decoder .
The model takes three inputs for each sample: RGB frame sequence of length T , a set of learnable query points P = {P i } N q i =1 , and query vectors q ∈ R N q × D .
Learnable query points explicitly describe the action locations by positioning themselves around action boundaries and semantic key frames as well, and the query vectors decode action semantics and locations from the sampled features.
The video encoder extracts video features X ∈ R T × D from RGB frames.
The action decoder contains L stacked decoder layers and takes query points P , query vectors q and video features X as input.
Each decoder layer contains two parts: 1) the multi-head self-attention block models the pair-wise relationship of query vectors and establishes inter-proposal modeling for action detection; 2) the Multi-level Interactive Module models the point-level and instance-level [Figure] semantics with dynamic weights based on query vector.
Overall, the action decoder accomplishes context modeling at point-wise , intra-proposal and inter-proposal levels.
Finally, we use two linear projection heads to decode action labels from query vectors, and transform query points to detected segments for loss computation and prediction output.
3.1 Video Encoder
We use the I3D backbone [3] as the video encoder in our framework.
The video encoder is trained end-to-end with the action decoder and optimized by action detection loss to bridge the domain gap between action recognition and detection.
For easy deployment of our framework in practice, we avoid the usage of optical flow due to its cumbersome preparation procedure.
In order to achieve good performance on par with two-stream features by only using RGB input, we follow [46] to remove the temporal pooling at Mixed _ 5c and fuse the features from Mixed _ 5c with features from Mixed _ 4f as in [24]. As a result, the temporal stride of encoded video features is 4. Spatial average pooling is performed to squeeze the spatiotemporal representations from backbone to temporal features. 3.2 Learnable Query Points
Segment-based action representation (i.e., representing each action instance simply with a starting and ending time) is limited in describing its boundary and content at the same time.
To increase the representation flexibility, we present a novel point-based representation to automatically learn the positions of action boundary as well as its semantic key frames inside the instance.
Specifically, the point-based representation is denoted by P = { t j } N s j =1 for each query, where t j is the temporal location of j th query point, and the point quantity per query is N s and set to 21 empirically.
We explain the updating strategy and the learning of query points below.
Iterative point refinement.
During training, the query points are initially placed at the midpoint of the input video clip.
Then, they are refined by query vectors q through iterations of decoder layers to reach final positions.
To be specific, at each decoder layer, the query point offsets are predicted from updated query vector (see Sec. 3.3) by linear projection.
We design a self-paced updating strategy with adaptive scaling for each query at each layer to stabilize the training process.
At decoder layer l , the query points for one query are represented by P l = { t lj } N s j =1 .
The N s offsets are denoted { ∆ t lj } N s j =1 .
The refinement can be summarized as:
[Equation]
where s l = max ( { t lj } ) − min ( { t lj } ) is the scaling parameter and describes the span of query points at layer l .
As a result, the updated step size gets smaller for shorter action, which helps with the localization of short actions.
Updated query points from previous layer are inputs to the next layer.
Learning query points.
The training of query points is directly supervised by regression loss at both intermediate and final stages.
We follow [47] to transform query points to pseudo segments for regression loss calculation.
The resulted pseudo segments participate in the calculation of L1-loss and tIoU loss with groundtruth action segments in both label assignment and loss computation.
Linear w/ params
D channels
Channel Mixing
Linear w/ params
Frame Mixing
Linear
LN & ReLU
T
Linear
Concat
T
Flatten & Linear
Query Vector: 1 x D
LN & ReLU
Residual connection
D e f o r m a b l e
C o n v .
Q u e r y
P o i n t s : N s x
Point-Level
Instance-Level
N s po i n t s
Linear
The transformation function is denoted by T : P → S
= ( t s , t e ) .
We experiment with two kinds of functions: Min-max T 1 and Partial min-max T 2 .
Min-max is to take the minimum and maximum location from the set of query points as the starting and ending timestamp of the pseudo segment, T 1 : P → min ( { t j } j ∈P ) , max ( { t j } j ∈P ) . With Min-max transformation, the query points are strictly bounded within the local segment of the target action instance. Partial min-max function is to select a subset of query points P local and perform the min-max function on them to determine a pseudo segment, T 2 : P → min ( { t j } j ∈P local ) , max ( { t j } j ∈P local ) .
It allows several query points to aggregate information from outside the action region.
Empirically, we choose partial min-max by default.
For each query, we randomly take 23 N s points from the query point set to form P local .
3.3 Multi-level Interactive Module
Apart from the limitation in the segment-based representation, previous temporal action detectors are also insufficient in decoding the sampled frames.
These methods seldom consider semantic aggregation from different levels and at multiple aspects.
Accordingly, we present a multi-level interactive module that consider both local temporal cues at point-level and intra-proposal relation modeling at instance-level , depicted in Fig. 3.
To achieve distinct representations for each query, these parameters at both levels are dynamically generated based on the query vector.
Point-level local deformation. Paired with the refined point representation, we employ deformable convolutions to extract point features within the local neighborhood.
For the j th query point, we predict four temporal offsets { o k } 4 k =1 from the point location and corresponding weights { w k } 4 k =1 .
The query point at frame t j acts as the center point and is added with temporal offsets to achieve four deformable sub-points.
These sub-points characterize the local neighborhood of center points.
The features at sub-points are extracted by bilinear interpolation, then multiplied with weights and combined together to get point-level feature x ( j ) .
This process can be expressed as:
[Equation]
Both the offsets and weights are generated from query vectors q by linear projection,
[Equation]
where the weights are additionally normalized by softmax for each query.
Instance-level adaptive mixing.
Faced with the challenge of simultaneous actions, with only temporal modeling, actions with large overlap may result in similar representation and harms the classification.
To tackle this problem, we propose adaptive mixing at both frames and channels by using dynamic convolutions.
Specifically, the stacked features of query points is denoted by x ∈ R N s × D .
Given the query vector q , we generate dynamic parameters for frame mixing and channel mixing: [Equation]
Frame mixing is carried out with dynamic projection followed by LayerNorm and ReLU activation on N s points to explore intra-proposal frame relations:
[Equation]
Similar to frame mixing, channel mixing uses two bottle-necked layers of dynamic projection on the channel dimension to enhance action semantics:
[Equation]
These two mixed features are concatenated along channel and squeezed by linear operations to the size of query vector.
The query vector q l at the l th layer is then updated with residual connection:
[Equation]
Finally, the query point offsets and the action labels are decoded from the updated vector by two linear projection heads to produce detection results at each layer.
3.4 Training and Inference
Label assignment.
Similar to all query-based detectors [2, 55, 11, 37], we apply hungarian matcher on (detected) pseudo segments to search for the optimal permutation σ ( · ) for label assignment.
The groundtruth set ˆΨ on each video clip is extended with no action ∅ to the size of N q .
The matching cost is formulated as:
[Equation]
By minimizing the matching cost, the bipartite matching algorithm finds the optimal permutation σ ∗ ( · ) that assigns each prediction with a target.
α L 1 , α iou , α cls is set to 1 , 5 , 5 , respectively.
Loss functions.
PointTAD is jointly optimized by localization loss and classification loss.
We use Lloss and iou loss as localization loss:
[Equation]
The cross-entropy loss between query labels and target labels is used as classification loss.
In addition, to improve the performance under segmentation-mAP, we generate dense classification scores S ∈ R T × C by linear projection from video features X .
Cross-entropy loss is enforced on
S with dense groundtruth ˆ S ∈ R T × C .
Therefore, the classification loss is composed of the set prediction loss and the dense per-frame loss:
[Equation]
[Equation]
The overall loss function is formulated as follows:
[Equation]
[Equation]
λ cls , λ loc and λ seg are hyper-parameters that are set to 5 , 10 , 0 . 5 , respectively.
Inference.
During inference, our PointTAD uses a single linear projection layer followed by LayerNorm and ReLU activation to predict class labels from query vectors.
As for localization, we use pseudo segments transformed from query points as the final predictions.
These sparse predictions are then evaluated under the detection-mAP metric.
Additional dense scores S could be generated at video features for the segmentation-mAP calculation.
The sparse predictions are filtered with threshold γ and processed with Gaussian kernels to approximate dense scores at each frame.
Then, the approximated scores are added with weight β to the predicted dense scores for segmentation-mAP calculation.
The final dense scores are generated as:
[Equation]
Note that the sparse predictions are adjusted by dense scores only for segmentation-mAP. [Table]
Experiments 4.1 Datasets and Setup
Datasets.
We conduct experiments on two popular multi-label action detection benchmarks: MultiTHUMOS [48] and Charades [34].
MultiTHUMOS is a densely labeled dataset extended from
THUMOS14.
It includes 413 sports videos of 65 classes.
The dataset has an average number of distinctive action categories per video of 10.5, compared with 1.1 in THUMOS14.
Charades is a large Multi-label TAD dataset that contains 9848 videos of daily in-door activities.
The annotations are spread over 157 action classes, with an average of 6.8 instances per video.
Implementation details.
With I3D backbone network, we encode videos at 10 fps for MultiTHUMOS and 12 fps for Charades.
The spatial resolution is set to 192 × 192 for both datasets.
The video sequence is pre-processed with the widely-used sliding window mechanism.
To accommodate most of the actions, the window size is set to 256 frames for MultiTHUMOS (99.1% actions included), and 400 frames for Charades (97.3% actions included).
Short videos in Charades are padded with the last input frame.
Its overlap ratio is 0.75 at training, and 0 at inference.
The number of queries N q is set to 48 for both benchmarks.
The number of query points per query N s is 21.
The number of deformable sub-points is set to 4 according to the number of sampling points in TadTR [25].
Appropriate initialization is required for backbone, query points and point-level deformable convolutions for stable training.
Following common practice, the I3D backbone are initialized with Kinetics400 [17] pre-trained weights for MultiTHUMOS and Charades pre-trained weights for Charades.
Query points are initialized with constant 0.5 in training and with learned weights in inference.
Other possible initializations are explored in ablations.
The linear layer to produce deformable offsets are initialized as follows: zeroing for weights and [1 , 2 , 3 , 4] for biases.
The weights and biases to generate deformable weights are initialized as zero.
We adopt AdamW as optimizer with 1e-4 weight decay.
The network is trained on a server with V100 GPUs.
The batch size is 3 per GPU.
The learning rate is set to 2e-4 and drops by half at every 12 epochs.
Backbone learning rate is additionally multiplied with 0.1 for stable training.
Evaluation metrics.
The default evaluation metric for multi-label TAD is segmentation-mAP, which is the frame-wise mAP.
In addition, we extend the detection-mAP metric from traditional TAD to further evaluate the completeness of predicted action instances.
The detection-mAP is the instancewise mAP of action predictions under different tIoU thresholds.
We report the average mAP as well as mAPs at tIoU threshold set { 0 . 1 , 0 . 2 , 0 . 3 } for both datasets.
The average detection-mAP is calculated with tIoU thresholds set [0 . 1 : 0 .
We argue that detection-mAP is more reasonable for an instance detection task. [Table] [Table] [Table]
[Table] Comparison with the State-of-the-Art Methods
In Tab. 1, we compare the performance of PointTAD with previous multi-label TAD methods under both segmentation-mAP and detection-mAP.
The sparse prediction tuples of PointTAD are converted to dense segmentation scores with Eq.
(14) for segmentation-mAP.
In order to compare the detectionmAP, we reproduce several previous multi-lable temporal action localization methods and convert their dense segmentation results to sparse predictions following [48].
The prediction confidence score of an action with L f consecutive frames is calculated as:
[Equation]
where p i is the probability for class c at frame i , µ C and σ C is the mean and standard derivation of action duration of class C in the training set.
Our PointTAD surpasses all previous multi-label TAD methods by a large margin under detectionmAP, indicating our ability to predict complete actions is beyond previous dense segmentation models.
As for segmentation-mAP, we achieve encouraging and comparable results to the previous methods on both benchmarks with a sparse detection framework.
[Figure]
In Fig.
4, we show the qualitative results of PointTAD on the MultiTHUMOS test set compared to the segmentbased baseline, PDAN [9] and MS-TCT [8].
PointTAD detects more instances at harder categories, such as “Fall” and “SoccerPenalty”.
MS-TCT and PDAN perform better at “NoHuman” category. We argue that this is because “NoHuman” class is not a well-defined action category with precise paired action boundaries, whereas PointTAD have to leverage pair relations of boundaries for localization.
4.3 Ablation Study
Segment-based representation vs point-based representation.
In Tab.
2a, we compare the performance between segment-based representation and point-based representation.
In the segment-based baseline, actions are represented by segments of paired start-end timestamps.
We apply temporal RoI align to the segments as in [43, 51] to retrieve action features.
Temporal RoI align divides segments into uniform bins and apply average pooling on each bin to transform frame features for query vector mixing.
The number of bins for RoI align is set to N s , the same as query point quantity.
The parallel dynamic mixing is also carried out in implementations to ablate only frame sampling strategies with segments and points.
Results show that point-based representation significantly outperforms segment-based representation, demonstrating the strength of adaptive sampling over segment grid alignment of frame features. [Table] [Table] [Table]
Study on query point initialization.
We consider three different initialization for query points in training: uniform distribution in [0 , 1] , normal distribution with mean= 0 .
5 and std = 0 .
3 , and initialization with constant 0 .
Results in Tab. 2b indicates that all three initialization methods are beneficial for the training, with little performance gap in avg-mAP.
In addition, normal distribution and uniform distribution achieve higher detection-mAP with lower tIoU threshold, which shows these two initialization methods are weaker at accurate localization under higher tIoU threshold.
Empirically, we set constant initialization by default.
Study on pseudo segment transformation functions.
Tab. 2c shows two alternatives of transformation function T .
Min-max and Partial min-max achieve similar performance under average detection-mAP.
We choose Partial min-max as the default function because it offers a more relaxed constraint for query points with higher flexibility.
Study on the number of query points.
We ablate with different numbers of query points N s in
Tab. 2d.
In general, the performance is in proportion to the point quantity, although the increase of N s only benefits the performance a little.
N s reaches the sweet spot at 21 , where the performance no longer increases with larger N s .
To balance the model complexity and performance, we set N s to 21.
Study on point-level deformation.
In Tab. 2e, we compare point-level deformable convolution and direct interpolation for point-level feature extraction.
Using deformable operator achieves higher performance than direct interpolation at query point location, demonstrating that it is beneficial to consider local context with dynamic modeling in action detection.
Study on instance-level mixing.
The effect of frame-wise and channel-wise mixing is studied in Tab. 2f.
We first ablate with the single application of each mixing, i.e. frame-only or channelonly.
Results show that the performance degrades more without frame mixing, demonstrating the importance of frame mixing over channel mixing in multi-label TAD.
Furthermore, we explore different combinations of frame and channel mixing with two cascaded alternatives, i.e. frame mixing → channel mixing and channel mixing → frame mixing.
Tab.
2f shows that compared with frame-only and channel-only performances respectively, cascaded designs backfire at the performance with a decrease of 1% avg-mAP at frame → channel and 3% avg-mAP at channel → frame.
To tackle action detection in video domain, intra-proposal channel mixing does not help with subsequent frame mixing and vice versa.
Study on the result fusion parameter β . Combining sparse detection results with dense segmentation scores provides smoother frame-level scores for segmentation-mAP. We ablate with choices of β on both datasets in Tab. 3a. β is set to 0.2 for MultiTHUMOS and 0.96 for Charades based on empirical results.
Study on the offset scaling parameter s . This scaling parameter is conventional in box object detectors (Faster R-CNN [32], Cascade R-CNN [1] and Sparse R-CNN [36], which is to scale regression offsets with respect to the box size instead of the image size. We extend this design to our PointTAD.
In Tab.
3b, we compare the regression offsets predicted with respect to action duration (a.k.a offset scaled by duration) and with respect to window size (offset without scaling) on
MultiTHUMOS.
It demonstrates the effectiveness of this scaling strategy in multi-label TAD. 4.4 Query Points Visualization
Fig. 5 illustrates the learned query points and the corresponding action with two extreme points and semantic key frames with interior points.
Global points sparsely distribute over the entire video for context integration.
Specifically, we observe that the interior local points effectively capture the most distinguished characteristic of target action “Run” , by capturing rapid forward movement of both legs and neglecting frames of similar yet disruptive movements: hopping and scoring.
Moreover, the iterative refining process of query points are depicted from initial positions to the final positions in the last decoder layer.
Query points are learned to kick off from the median position in the window.
After the first decoder layer, the query points are already able to coarsely locate the action target.
Then, these query points automatically converge to finer localization along the decoder layers.  Limitations and Future Work
PointTAD is proposed to solve the complex problem of multi-label TAD, by leveraging learnable query points for flexible and distinct action representation.
Currently, we validate our model on two popular multi-label TAD benchmarks which include sport events and daily indoor activities.
PointTAD achieves superior performance to all previous multi-label TAD methods as well as the state-of-the-art single-label TAD methods under detection-mAP metric.
We have not demonstrated our model’s ability in general action detection in more diverse scenarios, including important tasks such as action spotting, sentence grounding, and so on. In the future, we would continue to explore the advantages of point-based action representation in broader scope for video understanding. Meanwhile, our PointTAD training still relies on the intermediate supervision and we hope to design more effective training paradigm for the query-based detection pipeline.  Conclusion
In this paper, we have studied the complex multi-label TAD that aims to detect all actions from a multi-label untrimmed video.
We formulate this problem as a sparse detection task and extend the traditional query-based action detection framework from single-label TAD.
Faced with the challenge of concurrent instances and complex action relations in multi-label TAD, we introduce a set of learnable query points to effectively capture action boundaries and characterize action semantics for fine-grained action modeling Moreover, to facilitate the decoding process, we propose the Multilevel Interactive Module that integrates action semantics at both point level and instance level, by using dynamic kernels based on query vector.
Finally, PointTAD yields an end-to-end trainable architecture by using only RGB inputs for easy deployment in practice.
Our PointTAD surpasses all previous methods by a large margin under the detection-mAP and achieves promising results under the segmentation-mAP on two popular multi-label TAD benchmarks. Acknowledgments and Disclosure of Funding
This work is supported by National Natural Science Foundation of China (No. 62076119, No.
61921006), the Fundamental Research Funds for the Central Universities (No. 020214380091), and Collaborative Innovation Center of Novel Software Technology and Industrialization.
A Appendix A.1 Comparison with the State-of-the-Art Single-label TAD Methods
In this sub-section, we report the detection-mAP results of some state-of-the-art single-label TAD methods (i.e. P-GCN [49], AFSD [19], ContextLoc [56]) reproduced on multi-label TAD task.
Since
MultiTHUMOS share similar data preparation with THUMOS14, the reproductions are conducted on MultiTHUMOS to get the best performance out of these models without additional hyper-parameter tuning. P-GCN [49] and ContextLoc [56] are both two-staged action detector that take in proposal generation results (e.g. BSN [23] proposals) and perform relation modeling based on coarse proposals to refine and classify each action candidate.
AFSD [19] follows FCOS [39] to employ anchor-free architecture for TAD task.
It is able to generate action detection results directly from the network, without pre-defined anchors or external video-level class labels.
The results in Tab.
4 show that direct application of single-label TAD models on multi-label TAD is deficient to achieve good detection performance, hence it is non-trivial to extend action detectors to multi-label TAD.
Our PointTAD surpasses all of these state-of-the-art single-label TAD methods by a large margin, demonstrating the advance of our model to deal with concurrent instances and complex action relations as an action detector.
Moreover, the performance of state-of-the-art single-label
TAD methods are significantly worse at multi-label TAD benchmarks, indicating clear room for improvement on these benchmarks.
In addition, we follow the practice in [33] to compute the error bar over 3 runs, and report the mean and standard derivation in Tab. 4. The error bar for Charades results is as follows: mAP@0.1 = 16.72% ± 0.26%, mAP@0.2= 16.11% ± 0.28%, mAP@0.3= 15.17% ± 0.26%, mAP@0.4= 14.00% ± 0.26%, mAP@0.5= 12.68% ± 0.29%, avg
[Table] [Table]
A.2 Evaluation on THUMOS
Following RTD (query-based TAD method)
[Table]
we use the same feature representation and place our
PointTAD head on top to build a direct TAD detector.
Note that our TAD detector does not reply on the video-level classifier for action recognition and directly produce the action labels with our own
PointTAD head.
The result on the THUMOS14 dataset is reported in the Tab. 5. We obtain better performance on this single-label TAD dataset, demonstrating the generalization ability of PointTAD to various TAD datasets.
[Table] A.3 Comparison with DETR-based Baselines
In Table 2a of main paper
[Table]
we have shown the comparison between PointTAD and a Sparse-RCNN based baseline (segment-based variant), which proves the effectiveness of point representation. We have implemented another DETR based baseline on the MultiTHUMOS dataset. The performance comparison is reported in Tab. 6, and our PointTAD obtains better results thanks to our more flexible point-based representation.
[Table]
[Table]
In this sub-section
[Table] we share some of our observations in building an end-to-end trainable architecture for multi-label TAD via ablations on input image resolution and the number of decoder layers L . Study on input image resolution. In Tab. 7a
[Table]
we show the detection performance with different input image resolution.
According to [24], cropping images to 128 2 is adequate to tackle single-label
TAD.
However, we observe from the experiments that handling multi-label TAD requires more spatial information to distinguish concurrent instances from different categories and 128 2 image resolution is far less sufficient to solve the task. Our detection performance improves greatly by the increase of image resolution: from 96 2 to 128 2 the detection-mAP improves by absolute 6% at mAP@0.1, from 128 2 to 160 2 the performance improves by absolute 2% at mAP@0.1. The performance gain slows down with the increase of spatial resolution: from 160 2 to 192 2 , the performance increases by 0.7% at mAP@0.1. We settled at 192 2 to balance memory consumption and model performance.
Study on the number of decoder layers.
The choice of L is influenced by end-to-end training due to memory consumption by backbone.
We carefully decrease L from the common setting of L = 6 , as shown in Tab. 7b.
Results indicate that 4-layer and 5-layer designs are quite similar in performance, yet from 4-layer design down, the decrease of L leads to obvious performance degrade.
Hence, L is set to 4 empirically. A.5 Visualization To further exhibit our ability in building distinct representations for concurrent action instances, we take two concurrent actions as example and illustrate their corresponding query point predictions in Fig. 6.
According to the figure, action Run and SoccerPenalty happen almost simultaneously in the video.
To learn the distinct semantics for each action, two sets of query points Pred1 and Pred2 attend at different frames within the highly overlapping time ranges.
Pred1 aims at action Run and places the local points on frames that include iterative forward movements of both legs. Pred2 detects action SoccerPenalty and positions the local points on frames that describe the movement of passing the ball as well as the movement of a kick. With query points, PointTAD successfully learns the authentic action semantics based on actor movements, making big process from previous action detectors that sometimes reply on background information to decide action category.
This paper proposes PointTAD, a solution with learnable query points to tackle multi-label TAD. PointTAD is the first to introduce points/keyframes for segment-level video representation.
Such
[Figure]
practice addresses the non-uniform temporal structure of videos well and could potentially drive the development of general video understanding systems for finer point-based representation. The potential applications includes video editing, anomaly event detection, etc.
PointTAD enables end-toend inference with raw video input, which benefits the deployment of automated online services for batch video processing, saving lots of human effort from offline, video-per-video handling. As the model is data-driven, any bias in training data could be captured in the algorithm.
Apart from this aspect, there are no known ethical issues in the real-world applications of this technology. A.7 Code and License
Our codebase is built upon RTD-Net ‡ [37] protected by Apache-2.0 License and Sparse R-CNN § [36] protected by MIT license. MultiTHUMOS [48] dataset and Charades [34] dataset are restricted to non-commercial use only.