PointTAD: Multi-Label Temporal Action Detection with Learnable Query Points
Anonymous Author(s)
AffiliationAddress email
Abstract
Traditional temporal action detection (TAD) usually handles untrimmed videos withsmall number of action instances from a single label (e.g., ActivityNet, THUMOS).
However, this setting might be unrealistic as different classes of actions often cooccur in practice.
In this paper, we focus on the complex multi-label temporal actiondetection that aims to localize all action instances from a multi-label untrimmedvideo.
Multi-label TAD is more challenging as it requires for fine-grained classdiscrimination within a single video and dedicated module to precisely localize theco-occurring instances.
Specifically, we extend the sparse query-based detectionparadigm from the traditional TAD and propose the multi-label TAD framework of
PointTAD.
Existing query-based action detectors employs a segment to representan action instance, which is insufficient to handle the concurrent instances andtheir richer relations.
To mitigate this issue, our PointTAD introduces a small setof learnable query points to represent important frames of each action instance.
Our PointTAD provides a flexible mechanism to localize the discriminative framesat boundaries and as well the important frames inside the action.
Moreover, weimprove action decoding with the Multi-level Interactive Module to integrate actionsemantics at point-level and instance-level.
Finally, our PointTAD employs an endto-end trainable framework based on RGB input for easy deployment.
We evaluateour proposed method on two popular benchmarks for multi-label TAD.
Our modeloutperforms all previous methods by a large margin under the detection-mAPmetric and achieves promising results under the segmentation-mAP metric.
Introduction
With the increasing amount of video resources on the Internet, video understanding is becomingone of the most important topics in computer vision.
Temporal action detection (TAD) [47, 21,19, 3, 9, 17, 34] has been formally explored in traditional benchmarks such as THUMOS [14],
ActivityNet [13] and HACS [45].
However, the task seems impractical because their videos almostcontain non-overlapping actions from a single category: 85% videos in THUMOS are annotatedwith single action category.
As a result, most TAD methods [21, 19, 40, 4, 32] simply cast this TADproblem into sub-problems of action proposal generation and global video classification.
In this paper,we shift our playground to the more complex setup of multi-label temporal action detection, whichaims to detect all action instances from multi-labeled, untrimmed videos.
Existing works [8, 15, 36, 7]in this field formulate the problem as a dense segmentation task and perform multi-label classificationin a frame-wise manner.
Consequently, these methods are weak in localization and fails to provide theinstance-level detection results (i.e., the starting time and ending time of each instance).
In analogyto image instance segmentation [20], we argue that it is necessary to redefine multi-label TAD as ainstance-level detection problem rather than a dense segmentation task.
In this sense, multi-label
TAD results not only provide the action labels, but also the exact temporal extent of each instance.
[Figure]
[Figure] [Figure]
Direct adaptation of action detectors is insufficient to deal with the challenges of concurrent instancesand complex action relations in multi-label TAD.
The convention of extracting action features fromaction segments [3, 29, 44, 38] lacks the flexibility of handling both important semantic frames insidethe instance as well as discriminative boundary frames.
Consider the groundtruth action of "Jump"in Fig.
1, segment-based action detectors mainly produce two kinds of error predictions, as in type
S 1 and type S 2 .
S 1 successfully predicts the correct action category with an incomplete segmentof action highlights, whereas S 2 does a better job in locating the boundaries yet get misclassifiedas "Sit" due to the distracting frames of landing.
In addition, most action detectors [44, 29, 23] areinadequate in processing sampled frames and classifying fine-grained action classes.
They oftenexploit temporal context modeling at single level and ignore the exploration of channel semantics.
To address the above issues, we present PointTAD, a sparse query-based action detector that leverageslearnable query points to flexibly attend important frames for action instances.
Inspired by RepPoints [42], query points are directly supervised by regression loss.
Given specific regression targets,the query points learn to localize at discriminative frames at action boundaries as well as semantickey frames within actions.
Hence, concurrent actions of different categories can achieve distinctivefeatures through the specific arrangement of query points.
Moreover, we improve the action locationand semantics decoding by proposing the Multi-level Interactive Module with dynamic kernelsbased on query vector.
Direct interpolation or pooling at each query point lacks temporal reasoningover consecutive frames.
Following deformable DETR [50], we extract point-level features withdeformable convolution [5, 49] from a local snippet to capture the temporal cues of action change orimportant movement.
At instance-level, both temporal context and channel semantics are capturedwith frame-wise and channel-wise dynamic mixing [33, 10] to further describe the distinctive featuresof simultaneous actions.
PointTAD streamlines end-to-end trainable temporal action detection with joint optimization of thebackbone network and action decoder.
We validate our model on two popular benchmarks of thechallenging multi-label TAD.
Our model achieves the state-of-the-art detection-mAP performanceand competitive segmentation-mAP performance to previous methods with RGB input only.
Related work
Multi-label temporal action detection.
Multi-label temporal action detection has been studied asa multi-label frame classification problem in the previous literature.
Early methods [27, 28] paida lot of attention to model the temporal relations between frames with the help of Gaussian filtersin temporal dimension.
Other works deployed dilated temporal kernels [8] or iterative temporalconvolution and self attention blocks [7] to integrate features at different temporal scales.
Recently,attention has shifted beyond temporal modeling.
Coarse-Fine[15] employed X3D backbone to handledifferent temporal resolutions in the slow-fast fashion and perform spatial-temporal attention duringfusion.
MLAD [36] used multi-head self-attention blocks at both spatial and class dimension tomodel class relations at each timestamp.
In our proposed method, we study the temporal context atdifferent semantic levels and model the channel dynamics within instances.
In addition, we take intoaccount the paired relations between action starts and ends, and employ query-based action detectionframework for more accurate and precise action localization.
Segment-based representation.
Following the prevailing practice of bounding boxes [18, 12, 30, 1]in object detection, existing temporal action detectors incorporated action segments heavily withthree kinds of usage: as anchors, as intermediate proposals, and as final predictions.
Segments asanchors are explored mainly in anchor-based frameworks.
These methods [26, 24, 44, 29] usedsliding windows or proposal generation results as anchors. Most TAD methods [44, 17, 40, 3, 47, 46]
[Figure]
use segments as intermediate proposals.
Uniform sampling or pooling are commonly used to extractfeatures from these segments.
P-GCN [44] applied max-pooling within local segments for proposalfeatures.
G-TAD [40] uniformly divided segments into bins and average-pooled each bin to achieveproposal features.
AFSD [17] proposed boundary pooling in boundary region to refine action feature.
Segments as final predictions are employed among all three types of TAD frameworks, becausesegments generally facilitate the computation of action overlaps and loss calculation.
Instead, in thispaper, we do not need segments as anchors and directly employ learnable query points as intermediateproposals. The learnable query points handle the important frames within action. Therefore, actionfeature is extracted only from learned keyframes, instead of pooling from uniform bins. Point-based representation.
Several existing works have used point representations to describekeyframes [11, 35], objects [42, 10], tracks [48], and actions [16]. [11, 35] tackled keyframe selectionby operating greedy algorithm on spatial SIFT keypoints [11] or clustering on local extremes ofimage color/intensity [35]. These methods followed a bottom-up strategy to choose keyframes basedon local cues.
In contrast, PointTAD represents action as a set of temporal points (keyframes). Wefollow RepPoints [42] to handle the important frames of actions with point representations and refinethese points by action feature iteratively. Our method directly regresses keyframes from query vectorsin a top-down manner for more flexible temporal action detection. Moreover, we improve over thedense anchor positions in RepPoints [42] and built PointTAD upon query detector, where a smallset of action queries use query points to sparsely search the frame sequence for potential actions,resulting in an efficient and effective framework.
Temporal context in videos.
Context aggregation at different levels of semantics is crucial fortemporal action reasoning and has been discussed in previous TAD methods.
G-TAD [40] treatedeach snippet input as graph node and applied graph convolution networks to enhance snippet-levelfeatures with global context.
ContextLoc [51] handled action semantics in hierarchy: it updatedsnippet features with global context, obtained proposal features with frame-wise dynamic modelingwithin each proposal and modeled the inter-proposal relations with GCNs.
Although we consideredthe same levels of semantic modeling, our method is different from ContextLoc.
On the contrary,
PointTAD focuses on aggregating local temporal cues at snippet-level, attaining informative queryfeature by frame and channel attention and applying multi-head self-attention for inter-query relationmodeling.
PointTAD
3.1 Overview
We formulate the task of multi-label TAD as a set prediction problem.
Formally, given a video segmentwith T consecutive frames, we predict a set of action instances Ψ
= { ψ n = ( t sn , t en , c n ) }
N q n =1 , N qis the number of learnable queries.
The groundtruth action set to detect are denoted ˆΨ =
{ ˆ ψ =( ˆ t sn , ˆ t en , ˆ c n ) } N g n =1 , where ˆ t sn , ˆ t en are the starting and ending timestamp of the n -th action, ˆ c n is thegroundtruth action category, N g is the number of groundtruth actions.
The overall architecture of PointTAD is depicted in Fig.
PointTAD consists of a backbone videoencoder and an action decoder .
The model takes three inputs for each sample: RGB frame sequenceof length T , a set of learnable query points p ∈ R N q × N s and query vectors q ∈ R N q × D .
Learnablequery points describe the action locations by positioning themselves on action transits as well assemantic key frames, and the query vectors encode action semantics.
The backbone video encoderextracts video features X ∈ R T × D from RGB frames.
The action decoder contains L stackeddecoder layers and takes query points p , query vectors q and video features X as input.
Each decoderlayer contains two parts: 1) the multi-head self-attention [37] block models the pair-wise relationshipof query vectors; 2) the Multi-level Interactive Module models the point-level and instance-levelsemantics with dynamic weights based on query vector.
From local to global, the action decoderaccomplishes context modeling at point-wise , intra-proposal and inter-proposal levels.
Finally,we decode action labels from query vectors and transform query points to pseudo segments for losscomputation and prediction output.
3.2 Learnable Query Points
Segment representation (i.e., representing each instance simply with a starting and ending time) islimited in describing boundary and action characteristics at the same time without redundancy.
Wepresent a novel query point representation to automatically learn the positions of action boundary aswell as semantic key frames.
The query point representation is denoted P = { t i } N s i =1 for each query,where t i is the temporal location of i th query point, and the point quantity per query N s is set to 21empirically.
We explain the updating strategy and the learning of query points below.
Iterative Point Refinement.
The query points are refined by query vectors through iterations ofdecoder layers to reach final positions.
To be specific, at each layer the query point offsets ∈ R N q × N sare predicted from query vector q by linear projection.
We design a self-paced updating strategy withadaptive scaling for each query at each layer to stabilize the training process.
At decoder layer l , thequery points for one query are represented by P l = { t li } N s i =1 .
The N s offsets are denoted { ∆ t li } N s i =1 .
The refinement can be summarized as:
[Equation]
where s l = max ( { t li } ) − min ( { t li } ) is the scaling parameter and describes the span of query pointsat layer l .
As a result, the updated step size gets smaller for shorter action, which helps with thelocalization of short actions.
Updated query points from previous layer are inputs to the next layer.
Learning Query Points.
The training of query points is directly supervised by regression loss.
We follow [42] to transform query points to pseudo segments for regression loss calculation.
Theconverted pseudo segments participate in the calculation of L1-loss and tIoU loss with groundtruthaction segments in both label assignment and loss computation.
The transformation function is denoted by T : P → S
= ( t s , t e ) .
We experiment with two kindsof functions: Min-max T 1 and Partial min-max T 2 .
Min-max is to take the minimum and maximumlocation from the set of query points as the starting and ending timestamp of the pseudo segment,
T 1 : P → min ( { t i } i ∈P ) , max ( { t i } i ∈P ) .
With Min-max transformation, the query points are strictlybounded within the local segment of the target action instance.
Partial min-max function is toselect a subset of query points P local and perform the min-max function on them to determine apseudo segment, T 2 : P → min ( { t i } i ∈P local ) , max ( { t i } i ∈P local ) .
It allows several query points toaggregate information from outside the action region.
Empirically, we choose partial min-max bydefault.
For each query, we randomly take 23 N s points from the query point set to form P local .
3.3 Multi-level Interactive Module
Apart from the limited segment representation, previous temporal action detectors are also insufficientin decoding sampled frames.
These methods seldom consider semantic aggregation from differentlevels and at multiple aspects.
Accordingly, we present a multi-level interactive module that considerboth local temporal cues at point-level and intra-proposal relation modeling at instance-level ,depicted in Fig. 3.
To achieve distinct representations for each query, these parameters at both levelsare dynamically generated based on the query vector.
Linear w/ params
D channels
Channel Mixing
Linear w/ params
Frame Mixing
Linear
LN & ReLU
T
Linear
Concat
T
Flatten & Linear
Query Vector: 1 x D
LN & ReLU
Residual connection
D e f o r m a b l e
C o n v .
Q u e r y
P o i n t s : N s x
Point-Level
Instance-Level
N s po i n t s
Linear
Point-level Locality Preservation.
Paired with the refined point representation, we employ deformable convolutions to extract point features within the local neighborhood.
For each query point,we predict four temporal offsets { o j } from the point location and corresponding weights { w j } .
Thequery point t i act as the center point and is added with temporal offsets to achieve four deformablesub-points.
These sub-points characterize the local neighborhood of center points.
The features atsub-points are extracted by bilinear interpolation, then multiplied with weights and combined togetherto get point-level feature x
( t i ) .
This process can be expressed as:
[Equation]
Both the offsets and weights are generated from query vectors q by linear projection,
[Equation]
[Equation]
) where the weights are additionally normalized by softmax for each query.
Instance-level Adaptive Mixing.
Faced with the challenge of simultaneous actions, with onlytemporal modeling, actions with large overlap may result in similar representation and harms theclassification.
To tackle this problem, we propose adaptive mixing at both frames and channels withdynamic temporal convolution of kernel size 1.
The stacked features of query points is denotedby x ∈ R N s × D .
Given the query vector q , we generate dynamic parameters for frame mixing andchannel mixing.
Frame mixing is carried out with dynamic projection followed by LayerNorm and
ReLU activation on N s points to explore intra-proposal frame relations:
[Equation]
Similar to frame mixing, channel mixing uses two bottle-necked layers of dynamic projection on thechannel dimension to enhance action semantics:
[Equation]
These Two mixed features are concatenated along channel and get squeezed by flatten and linearoperations to the size of query vector.
The query vector is then updated with residual connection.
3.4 Video Encoder
We exploit the commonly used I3D backbone [2] as the video encoder in our framework.
The videoencoder is trained end-to-end with the action decoder and optimized by action detection loss to bridgethe domain gap between action recognition and detection.
For easy deployment of our frameworkon online platforms, we avoid the usage of optical flow due to its dreadful preparation procedure.
In order to achieve good performance on par with two-stream features using only RGB input, wefollow [41] to remove the temporal pooling at Mixed _ 5c and fuse the features from Mixed _ 5c withfeatures from Mixed _ 4f as in [22].
As a result, the temporal stride of encoded video features is4.
Average pooling is performed to squeeze the spatiotemporal representations from backbone totemporal features.
3.5 Training and Inference
Label Assignment.
Similar to all query-based detectors, we apply hungarian matcher on pseudosegments to search for the optimal permutation σ ( · ) for label assignment.
The groundtruth set ˆΨ oneach video segment is extended with no action ∅ to the size of N q .
The matching cost is formulatedas:
[Equation]
By minimizing the matching cost, the bipartite matching algorithm finds the desired permutationσ ∗ ( · ) that assigns each prediction with a target.
α L 1 , α iou , α cls is set to 1 , 5 , 5 respectively.
Loss Functions.
PointTAD is jointly optimized by localization loss and classification loss.
We use
L1 loss and iou loss as localization loss:
[Equation]
The cross-entropy loss between query labels and target labels is used as classification loss.
Inaddition, to facilitate the performance under segmentation-mAP, we generate dense classificationscores S ∈ R T × C by linear projection from video features X .
Cross-entropy loss is enforced on
S with dense groundtruth ˆ S ∈ R T × C .
Therefore, the classification loss is composed of the setprediction loss and the dense per-frame loss:
[Equation]
[Equation]
The overall loss function is formulated as follows:
[Equation]
[Equation]
where λ cls , λ loc and λ seg are hyper-parameters that are set to 5 , 10 , 0 .
5 respectively.
Inference.
During inference, the network uses a single linear projection layer followed by LayerNormand ReLU activation to predict class labels from query vectors.
As for localization, we use pseudosegments transformed from query points as the final predictions.
These sparse predictions are thentested under the detection-mAP metric.
Additional dense scores S could be generated at videofeatures for the segmentation-mAP calculation.
The sparse predictions are filtered with thresholdγ and processed with Gaussian kernels to approximate dense scores at each frame.
Then, theapproximated scores are added with weight β to the predicted dense scores for segmentation-mAPcalculation.
The final dense scores are generated as:
[Equation]
Note that the sparse predictions are adjusted by dense scores only for segmentation-mAP.
Experiments4.1 Dataset and Setup
Dataset.
We conduct experiments on two popular multi-label action detection benchmarks: MultiTHUMOS [43] and Charades [31].
MultiTHUMOS is a densely labeled dataset extended from
THUMOS’14.
It includes 413 sports videos of 65 classes.
The dataset has an average number ofdistinctive action categories per video of 10.5, compared with 1.1 in THUMOS14.
Charades is alarge Multi-label TAD dataset that contains 9848 videos of daily in-door activities.
The annotationsare spread over 157 action classes, with an average of 6.8 instances per video.
Implementation Details.
With I3D backbone network, we encode videos at 10 fps for MultiTHUMOS and 12 fps for Charades.
The spatial resolution is set to 192 × 192 for both datasets.
The video
[Table]
sequence is processed with the widely-used sliding window mechanism.
To accommodate most ofthe actions, the window size is set to 256 frames for MultiTHUMOS (99.1% actions included) [Table] and400 frames for Charades (97.3% actions included).
Short videos in Charades are padded with the lastinput frame.
The overlap ratio is 0.75 at training, and 0 at inference.
The number of queries N q is setto 48 for both benchmarks.
The number of query points per query N s is 21.
Appropriate initialization is required for backbone, query points and point-level deformable convolutions for stable training.
Following convention, the I3D backbone are initialized with K400pre-trained weights for MultiTHUMOS and Charades pre-trained weights for Charades.
Query pointsare initialized with constant 0.5.
Other possible initialization are explored in ablations.
The linearlayer to produce deformable offsets are initialized as follows: zeroing for weights and [1 , 2 , 3 , 4] forbiases.
The weights and biases to generate deformable weights are initialized as zero.
We adopt AdamW as optimizer with 1e-4 weight decay.
The network is trained on 8-card V100 GPU.
The batch size is 3 per GPU.
The learning rate is set to 2e-4 and drops by half at every 12 epochs.
Backbone learning rate is additionally multiplied with 0.1 for stable training.
Evaluation Metrics.
The default evaluation metric for multi-label TAD is segmentation-mAP, whichis the per-frame mean average precision.
In addition, we extend the detection-mAP metric fromtraditional TAD to further evaluate the completeness of predicted action instances.
The detection-mAPis the mean average precision of action predictions under different tIoU thresholds.
We report theaverage mAP as well as mAP with tIoU threshold set { 0 . 1 , 0 . 2 , 0 . 3 } for both datasets.
The averagedetection-mAP is calculated with tIoU thresholds set [0 . 1 : 0 .
We argue that detection-mAPis more reasonable for an instance detection algorithm.
4.2 Comparison with state-of-the-art methods
In Tab. 1, we compare the performance of PointTAD with previous multi-label TAD methods underboth segmentation-mAP and detection-mAP.
The sparse prediction tuples of PointTAD are convertedto dense segmentation scores with Eq.
(13) for segmentation-mAP.
In order to compare the detectionmAP, we reproduce some of the multi-lable TAD methods and convert their dense segmentationresults to sparse predictions following [43].
The prediction confidence of an action with l consecutiveframes is calculated as:
[Equation]
where p i is the probability for class c at frame i , µ c and σ c is the mean and standard derivation ofaction duration of class c in the training set.
Our method surpasses all previous multi-label TAD methods by large under detection-mAP,indicating our ability to predict complete actions is beyond previous dense segmentation models.
Asfor segmentation-mAP, we achieve comparable results to previous methods on both benchmarks,
[Table]
which is encouraging for a sparse detection framework [Table] [Figure]
In Fig.
4, we show the qualitative results of PointTADon the MultiTHUMOS test set compared to the segmentbased baseline, PDAN and MS-TCT.
PointTAD detectsmore instances at harder categories, such as "Fall" and"SoccerPenalty".
MS-TCT and PDAN perform better at"NoHuman" category, we argue that this is because "NoHuman" class is not a well-defined action category thathas paired action boundaries, whereas PointTAD leveragespair relations of boundaries for localization.
4.3 Ablation Study
Segment representation vs Point representation.
In Tab.
2a, we compare the performance betweensegment representation and query point representation.
In segment implementation, we apply temporalroi align to segments as in [38].
Temporal roi align divides the segment into uniform bins and applyaverage pooling on each bin to arrange frame features for query vector.
The pooler resolution for
RoI Align is set to N s , the same as query point quantity.
Parallel dynamic mixing is carried outin both implementations.
Results show that point representation significantly outperforms segmentrepresentation, demonstrating the strength of adaptive sampling over grid-like alignment upon staticsegments.
Study on query point initialization.
We consider three different initialization for query pointsat training: uniform distribution in [0 , 1] , normal distribution with mean= 0 .
5 and std = 0 .
3 , andinitialization with constant 0 .
Results in Tab. 2b indicates that all three initialization methodsare beneficial for the training, with little performance different in avg-mAP.
In addition, normaldistribution and uniform distribution achieve higher detection-mAP with smaller tIoU threshold,which shows these two initialization methods are weaker at accurate localization.
Empirically, we setconstant initialization as default.
Study on pseudo segment transformation functions.
Tab. 2c shows two alternatives of transformation function T .
Min-max and Partial min-max achieve similar performance under averagedetection-mAP.
We choose Partial min-max as the default function because it offers a more relaxedconstraint for query points.
Study on the number of query points.
We ablate with different numbers of query points N s in
Tab. 2d.
In general, the performance is in proportion to the point quantity, although the increase of N sonly benefits the performance by little.
N s reaches the sweet spot at 21 , where the performance nolonger increases with larger N s .
To balance the model complexity and performance, we set N s to 21. [Figure]
Study on point-level deformable operator.
In Tab. 2e, we compare point-level deformable convolution and direct interpolation for point-level feature extraction.
Using deformable operator achieveshigher performance than direct interpolation at query point location, proving that it’s beneficial toconsider local context with dynamic parameters in action detection.
Study on instance-level dynamic mixing.
The effect of frame-wise and channel-wise mixing isstudied in Tab. 2f.
We first ablate with the single application of each mixing, i.e. frame-only andchannel-only.
Results show that the performance degrades more without frame mixing, demonstratingthe importance of frame mixing over channel mixing in multi-label TAD.
Furthermore, we exploredifferent combinations of frame and channel mixing with two cascaded alternatives, i.e. framemixing → channel mixing and channel mixing → frame mixing.
Tab.
2f shows that compared withframe-only and channel-only performances respectively, cascaded designs backfire at the performancewith a decrease of 1% avg-mAP at frame → channel and 3% avg-mAP at channel → frame.
To tackleaction detection in video domain, intra-proposal channel mixing does not help with subsequent framemixing and vice versa.
4.4 Query points Visualization
[Figure]
Fig.
5 illustrates the learnt query points and the corresponding action target on a sample from
MultiTHUMOS.
Partial mix-max transformation divides the query points into confined local points(blue) and open-ended global points (orange).
Local points are learned to acknowledge actionboundaries with two extreme points and semantic key frames with interior points.
Global pointssparsely distribute all over the video for open-ended context integration.
Specifically, we observethat the interior local points effectively capture the most distinguishing characteristic of target action"Run" , by capturing rapid forward movement of both legs and neglecting frames of similar yetdisruptive movements: hopping and scoring.
Moreover, the iterative refining process of query pointsare depicted from initial positions to the final positions in the last decoder layer.
Query points arelearnt to kick off from the median position in the window.
After the first decoder layer, the querypoints are already able to coarsely locate the action target.
Then, these query points automaticallyadjust their locations for finer localization throughout the first three layers and stabilize at the fourthlayer.
Conclusion
In this paper, we have studied the complex multi-label TAD that aims to detect all actions from amulti-label untrimmed video.
We formulate this problem as sparse detection task and extend sparsequery-based action detect framework from traditional TAD.
Faced with the challenge of concurrentinstances and complex action relations, we introduce a set of learnable query points to effectivelycapture action boundaries as well as characterize action semantics for fine-grained action features.
Moreover, to facilitate the decoding process, we propose the Multi-level Interactive Module thatintegrates action semantics at point-level and instance-level, with dynamic kernels based on queryvector.
Finally, PointTAD streamlines end-to-end trainable architecture using only RGB inputs foreasy deployment.
Our proposed method surpasses all previous methods by a large margin underdetection-mAP and achieves promising results under segmentation-mAP on two popular multi-label
TAD benchmarks.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See the supplementary material.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] No potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A] No theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A] No theoretical results.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] The code anddata are proprietary.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Sec.
4.1 and the supplementary material
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] See the supplementary material
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Sec. 4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Sec. 4.
(b) Did you mention the license of the assets?
[Yes] See the supplementary material.(c)
Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See the supplementary material for details.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
The datasets we use are open datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]