PointTAD: Multi-Label Temporal Action Detection with Learnable Query Points
Anonymous Author(s)
AffiliationAddress email
Abstract
Traditional temporal action detection (TAD) usually handles untrimmed videos withsmall number of action instances from a single label (e.g., ActivityNet, THUMOS).
However, this setting might be unrealistic as different classes of actions often cooccur in practice.
In this paper, we focus on the complex multi-label temporal actiondetection that aims to localize all action instances from a multi-label untrimmedvideo.
Multi-label TAD is more challenging as it requires for fine-grained classdiscrimination within a single video and dedicated module to precisely localize theco-occurring instances.
Specifically, we extend the sparse query-based detectionparadigm from the traditional TAD and propose the multi-label TAD framework of
PointTAD.
Existing query-based action detectors employs a segment to representan action instance, which is insufficient to handle the concurrent instances andtheir richer relations.
To mitigate this issue, our PointTAD introduces a small setof learnable query points to represent important frames of each action instance.
Our PointTAD provides a flexible mechanism to localize the discriminative framesat boundaries and as well the important frames inside the action.
Moreover, weimprove action decoding with the Multi-level Interactive Module to integrate actionsemantics at point-level and instance-level.
Finally, our PointTAD employs an endto-end trainable framework based on RGB input for easy deployment.
We evaluateour proposed method on two popular benchmarks for multi-label TAD.
Our modeloutperforms all previous methods by a large margin under the detection-mAPmetric and achieves promising results under the segmentation-mAP metric.
Introduction
With the increasing amount of video resources on the Internet, video understanding is becomingone of the most important topics in computer vision.
Temporal action detection (TAD) [42, 19,17, 4, 9, 15, 31] has been formally explored in traditional benchmarks such as THUMOS [13],
ActivityNet [12] and HACS [41].
However, the task seems impractical because their videos almostcontain non-overlapping actions from a single category: 85% videos in THUMOS are annotatedwith single action category.
As a result, most TAD methods [19, 17, 36, 5, 29] simply cast this TADproblem into sub-problems of action proposal generation and global video classification.
In this paper,we shift our playground to the more complex setup of multi-label temporal action detection, whichaims to detect all action instances from multi-labeled, untrimmed videos.
Existing works [8, 14, 32, 7]in this field formulate the problem as a dense segmentation task and perform multi-label classificationin a frame-wise manner.
Consequently, these methods are weak in localization and fails to provide theinstance-level detection results (i.e., the starting time and ending time of each instance).
In analogyto image instance segmentation [18], we argue that it is necessary to redefine multi-label TAD as ainstance-level detection problem rather than a dense segmentation task.
In this sense, multi-label
TAD results not only provide the action labels, but also the exact temporal extent of each instance.
[Figure]
[Figure] JUMP]
Direct adaptation of action detectors is insufficient to deal with the challenges of concurrent instancesand complex action relations in multi-label TAD.
The convention of extracting action features fromaction segments [4, 26, 40, 34] lacks the flexibility of handling both important semantic framesinside the instance as well as discriminative boundary frames.
Consider the groundtruth action of"Jump" in Fig.
1, segment-based action detectors mainly produce two kinds of predictions, as in type
S 1 and type S 2 .
S 1 successfully predicts the correct action category with an incomplete segmentof action highlights, whereas S 2 does a better job in locating the boundaries yet get misclassifiedas "Sit" due to the distracting frames of landing.
In addition, most action detectors [40, 26, 21] areinadequate in processing sampled frames and classifying fine-grained action classes.
They oftenexploit temporal context modeling at single level and ignore the exploration of channel semantics.
To address the above issues, we present PointTAD, a sparse query-based action detector that leverageslearnable query points to flexibly attend important frames for action instances.
Inspired by RepPoints [38], query points are directly supervised by regression loss.
Given specific regression targets,the query points learn to localize at discriminative frames at action boundaries as well as semantickey frames within actions.
Hence, concurrent actions of different categories can achieve distinctivefeatures through the specific arrangement of query points.
Moreover, we improve the action locationand semantics decoding by proposing the Multi-level Interactive Module with dynamic kernels basedon query vector.
Direct interpolation or pooling at each query point lacks temporal reasoning overconsecutive frames.
We extract point-level features with deformable convolution [6, 43] from a localsnippet to capture the temporal cues of action change or important movement.
At instance-level, bothtemporal context and channel semantics are captured with frame-wise and channel-wise dynamicmixing [30, 10] to further describe the distinctive features of simultaneous actions.
PointTAD streamlines end-to-end trainable temporal action detection with joint optimization of thebackbone network and action decoder.
We validate our model on two popular benchmarks of thechallenging multi-label TAD.
Our model achieves the state-of-the-art detection-mAP performanceand competitive segmentation-mAP performance to previous methods with RGB input only.
Related work
Multi-label temporal action detection.
Multi-label temporal action detection has been studied asa multi-label frame classification problem in the previous literature.
Early methods [24, 25] paida lot of attention to model the temporal relations between frames with the help of Gaussian filtersin temporal dimension.
Other works deployed dilated temporal kernels [8] or iterative temporalconvolution and self attention blocks [7] to integrate features at different temporal scales.
Recently,attention has shifted beyond temporal modeling.
Coarse-Fine[14] employed X3D backbone to handledifferent temporal resolutions in the slow-fast fashion and perform spatial-temporal attention duringfusion.
MLAD [32] used multi-head self-attention blocks at both spatial and class dimension tomodel class relations at each timestamp.
In our proposed method, we study the temporal context atdifferent semantic levels and model the channel dynamics within instances.
In addition, we take intoaccount the paired relations between action starts and ends, and employ query-based action detectionframework for more accurate and precise action localization.
Segment representation in TAD.
Following the prevailing practice of bounding boxes [16, 11, 27, 1]in object detection, existing temporal action detectors incorporated action segments heavily in theirframeworks with three kinds of usage: as anchors to estimate action location, as proposals forsubsequent recognition and as final predictions.
We divide temporal action detectors mainly into threekinds to explain their usage of action segments: anchor-based, anchor-free and query-based methods.
Segments as anchors are explored mainly in anchor-based frameworks.
Early methods [23, 22]
[Figure]
used sliding windows to generate pre-defined anchors and methods to date [40, 26] took in proposalgeneration results as anchors. TAL-Net [4] followed Faster R-CNN [27] to learn its own anchorsegments. Segments as intermediate proposals are exploited in both anchor-based and anchor-freemethods.
Anchor-based methods commonly adopt uniform sampling within proposal segments togenerate action features.
P-GCN [40] applied max-pooling over all frames within local segments forproposal features.
G-TAD [36] uniformly divided segments into bins and average the frame featuresin each bin to achieve proposal features.
Anchor-free methods, such as AFSD[15], converted centerpoints and boundary offsets to segment proposals and proposed novel boundary pooling module toboost boundary information in refined action features.
Segments as final predictions are employedamong all three types of TAD frameworks, because segments generally facilitate the computationof action overlaps and loss calculation.
In this paper, we replace temporal segments in the firstand second usage with learnable query points.
Compared with anchor-free method, we liberatethe extraction of action features from uniform bins and handcrafted boundary regions with flexibleframe sampling strategy that characterize the action semantics as well as boundaries.
In contrastto query-based methods that followed DETR and its variant style [2, 44], PointTAD replaces theglobal, dense attention with sparse learnable query points under direct regression supervision to betterencode action locations and semantics
Temporal context in videos.
Context aggregation at different levels of semantics is crucial fortemporal action reasoning and has been discussed in previous TAD methods.
G-TAD [36] treatedeach snippet input as graph node and applied graph convolution networks to enhance snippet-levelfeatures with global context.
ContextLoc [45] handled action semantics in hierarchy: it updatedsnippet features with global context, obtained proposal features with frame-wise dynamic modelingwithin each proposal and modeled the inter-proposal relations with GCNs.
Although we consideredthe same levels of semantic modeling, our method is different from ContextLoc.
On the contrary,
PointTAD focuses on aggregating local temporal cues at snippet-level, attaining informative queryfeature by frame and channel attention and applying multi-head self-attention for inter-query relationmodeling.
PointTAD
3.1 Overview
We formulate the task of multi-label TAD as a set prediction problem.
Formally, given a video segmentwith T consecutive frames, we predict a set of action instances Ψ
= { ψ n = ( t sn , t en , c n ) }
N q n =1 , N qis the number of learnable queries.
The groundtruth action set to detect are denoted ˆΨ =
{ ˆ ψ =( ˆ t sn , ˆ t en , ˆ c n ) } N g n =1 , where ˆ t sn , ˆ t en are the starting and ending timestamp of the n -th action, ˆ c n is thegroundtruth action category, N g is the number of groundtruth actions.
The overall architecture of PointTAD is depicted in Fig.
PointTAD consists of a backbone videoencoder and an action decoder .
The model takes three inputs: RGB frame sequence of length T ,a set of learnable query points p ∈ R N q × N s and one query vector q ∈ R N q × D .
Learnable querypoints describe the action locations by positioning themselves on action transits as well as semantickey frames, and the query vector encodes action semantics.
The backbone video encoder extractsvideo features X ∈ R T × D from RGB frames.
The action decoder contains L stacked decoder layersand takes query points, query vector q and video features X as input.
Each decoder layer containstwo parts: 1) the multi-head self-attention [33] block models the pair-wise relationship of queryvectors; 2) the Multi-level Interactive Module models the point-level and instance-level semanticswith dynamic weights based on query vector.
From local to global, the action decoder accomplishescontext modeling at point-wise , intra-proposal and inter-proposal levels.
Finally, we decode actionlabels from query vectors and transform query points to pseudo segments for loss computation andprediction output.
3.2 Learnable Query Points
Segment representation (i.e., representing each instance simply with a starting and ending time) islimited in describing boundary and action characteristics at the same time without redundancy.
Wepresent a novel query point representation to automatically learn the positions of action boundary aswell as semantic key frames.
The query point representation is denoted P = { t i } N s i =1 for each query,where t i is the temporal location of i th query point, and the point quantity per query N s is set to 21empirically.
We explain the updating strategy and the learning of query points below.
Iterative Point Refinement.
The query points are refined by a query vector through iterations ofdecoder layers to reach final positions.
We design a self-paced updating strategy with adaptive scalingfor each query at each layer to stabilize the training process.
At decoder layer l , the query pointsare represented by P l = { t li } N s i =1 .
The query vector predicts N s offsets, denoted { ∆ t li } N s i =1 , for eachquery point.
The refinement can be summarized as:
[Equation]
where s l = max ( { t li } ) − min ( { t li } ) is the scaling parameter and describes the span of query pointsat layer l .
As a result, the updated step size gets smaller for shorter action, which helps with thelocalization of short actions.
Updated query points from previous layer are inputs to the next layer.
Learning Query Points.
The training of query points is directly supervised by regression loss.
We follow [38] to transform query points to pseudo segments for regression loss calculation.
Theconverted pseudo segments participate in the calculation of L1-loss and tIoU loss with groundtruthaction segments in both label assignment and loss computation.
The transformation function is denoted by T : P → S
= ( t s , t e ) .
We experiment with two kindsof functions: Min-max T 1 and Partial min-max T 2 .
Min-max is to take the minimum and maximumlocation from the set of query points as the starting and ending timestamp of the pseudo segment,
T 1 : P → min ( { t i } i ∈P ) , max ( { t i } i ∈P ) .
With Min-max transformation, the query points are strictlybounded within the local segment of the target action instance.
Partial min-max function is toselect a subset of query points P local and perform the min-max function on them to determine apseudo segment, T 2 : P → min ( { t i } i ∈P local ) , max ( { t i } i ∈P local ) .
It allows several query points toaggregate information from outside the action region.
Empirically, we choose partial min-max bydefault.
3.3 Multi-level Interactive Module
Apart from the limited segment representation, previous temporal action detectors are also insufficientin decoding sampled frames.
These methods seldom consider semantic aggregation from differentlevels and at multiple aspects.
Accordingly, we present a multi-level interactive module that considerboth local temporal cues at point-level and intra-proposal relation modeling at instance-level ,depicted in Fig. 3.
To achieve distinct representations for each query, these parameters at both levelsare dynamically generated based on the query vector.
Point-level Locality Preservation.
Paired with the refined point representation, we employ deformable convolutions to extract point features within the local neighborhood.
For each query point,
Linear w/ params
D channels
Channel Mixing
Linear w/ params
Frame Mixing
Linear
LN & ReLU
T
Linear
Concat
T
Flatten & Linear
Query Vector: 1 x D
LN & ReLU
Residual connection
D e f o r m a b l e
C o n v .
Q u e r y
P o i n t s : N s x
Point-Level
Instance-Level
N s po i n t s
Linear we predict four temporal offsets { o j } from the point location and corresponding weights { w j } .
Thequery point t i act as the center point and is added with temporal offsets to achieve four deformablesub-points.
These sub-points characterize the local neighborhood of center points.
The featuresat sub-points are extracted by a bilinear interpolation, then multiplied with weights and combinedtogether to get point-level feature x
( t i ) .
This process can be expressed as:
[Equation]
Both the offsets and weights are generated from query vector q by linear projection,
[Equation]
where the weights are additionally normalized by softmax.
Instance-level Adaptive Mixing.
Faced with the challenge of simultaneous actions, with onlytemporal modeling, actions with large overlap may result in similar representation and harms theclassification.
To tackle this problem, we propose adaptive mixing at both frames and channels withdynamic temporal convolution of kernel size 1.
The stacked features of query points is denotedby x ∈ R N s × D .
Given the query vector q , we generate dynamic parameters for frame mixing andchannel mixing.
Frame mixing is carried out with dynamic projection followed by LayerNorm and
ReLU activation on N s points to explore intra-proposal frame relations:
[Equation]
Similar to frame mixing, channel mixing uses two bottle-necked layers of dynamic projection on thechannel dimension to enhance action semantics:
[Equation]
These Two mixed features are concatenated along channel and get squeezed by flatten and linearoperations to the size of query vector.
The query vector is then updated with residual connection.
3.4 Video Encoder
We exploit the commonly used I3D backbone [3] as the video encoder in our framework.
The videoencoder is trained end-to-end with the action decoder and optimized by action detection loss to bridgethe domain gap between action recognition and detection.
For easy deployment of our frameworkon online platforms, we avoid the usage of optical flow due to its dreadful preparation procedure.
In order to achieve good performance on par with two-stream features using only RGB input, wefollow [37] to remove the temporal pooling at Mixed _ 5c and fuse the features from Mixed _ 5c withfeatures from Mixed _ 4f as in [20].
As a result, the temporal stride of encoded video features is4.
Average pooling is performed to squeeze the spatiotemporal representations from backbone totemporal features.
3.5 Training and Inference
Label Assignment.
Similar to all query-based detectors, we apply hungarian matcher on pseudosegments to search for the optimal permutation σ ( · ) for label assignment.
The groundtruth set ˆΨ oneach video segment is extended with no action ∅ to the size of N q .
The matching cost is formulatedas:
[Equation]
By minimizing the matching cost, the bipartite matching algorithm finds the desired permutation σ ( · )that assigns each prediction with a target.
α L 1 , α iou , α cls is set to 1 , 5 , 5 respectively.
Loss Functions.
PointTAD is jointly optimized by localization loss and classification loss.
We use
L1 loss and iou loss as localization loss:
[Equation]
The cross-entropy loss between query labels and target labels is used as classification loss.
Inaddition, to facilitate the performance under segmentation-mAP, we generate dense classificationscores S ∈ R T × C by linear projection from video features X .
Cross-entropy loss is enforced on
S with dense groundtruth ˆ S ∈ R T × C .
Therefore, the classification loss is composed of the setprediction loss and the dense per-frame loss:
[Equation]
[Equation]
The overall loss function is formulated as follows:
[Equation]
where λ cls , λ loc and λ seg are hyper-parameters that are set to 5 , 10 , 0 .
5 respectively.
Inference.
During inference, the network uses a single linear projection layer followed by LayerNormand ReLU activation to predict class labels from query vectors.
As for localization, we use pseudosegments transformed from query points as the final predictions.
These sparse predictions are thentested under the detection-mAP metric.
Additional dense scores S could be generated at videofeatures for the segmentation-mAP calculation.
The sparse predictions are filtered with thresholdγ and processed with Gaussian kernels to approximate dense scores at each frame.
Then, theapproximated scores are added with weight β to the predicted dense scores for segmentation-mAPcalculation.
The final dense scores are generated as:
[Equation]
Note that the sparse predictions are adjusted by dense scores only for segmentation-mAP.
Experiments
4.1 Dataset and Setup
Dataset.
We conduct experiments on two popular multi-label action detection benchmarks: MultiTHUMOS [39] and Charades [28].
MultiTHUMOS is a densely labeled dataset extended from
THUMOS’14.
It includes 413 sports videos of 65 classes.
The dataset has an average number ofdistinctive action categories per video of 10.5, compared with 1.1 in THUMOS14.
Charades is alarge Multi-label TAD dataset that contains 9848 videos of daily in-door activities.
The annotationsare spread over 157 action classes, with an average of 6.8 instances per video.
Implementation Details.
With I3D backbone network, we encode videos at 10 fps for MultiTHUMOS and 12 fps for Charades.
The spatial resolution is set to 192 × 192 for both datasets.
The video
[Table]
sequence is processed with the widely-used sliding window mechanism.
To accommodate most ofthe actions, the window size is set to 256 frames for MultiTHUMOS (99.1% actions included), and400 frames for Charades (97.3% actions included).
Short videos in Charades are padded with the lastinput frame.
The overlap ratio is 0.75 at training, and 0 at inference.
Appropriate initialization is required for backbone, query points and point-level deformable convolutions for stable training.
Following convention, the I3D backbone are initialized with K400pre-trained weights for MultiTHUMOS and Charades pre-trained weights for Charades.
Query pointsare initialized with constant 0.5.
Other possible initialization are explored in ablations.
The linearlayer to produce deformable offsets are initialized as follows: zeroing for weights and [1 , 2 , 3 , 4] forbiases.
The weights and biases to generate deformable weights are initialized as zero.
We adopt AdamW as optimizer with 1e-4 weight decay.
The network is trained on 8-card V100 GPU.
The batch size is 3 per GPU.
The learning rate is set to 2e-4 and drops by half at every 12 epochs.
Backbone learning rate is additionally multiplied with 0.1 for stable training.
Evaluation Metrics.
The default evaluation metric for multi-label TAD is segmentation-mAP, whichis the per-frame mean average precision.
In addition, we extend the detection-mAP metric fromtraditional TAD to further evaluate the completeness of predicted action instances.
The detection-mAPis the mean average precision of action predictions under different tIoU thresholds.
We report theaverage mAP as well as mAP with tIoU threshold set { 0 . 1 , 0 . 2 , 0 . 3 } for both datasets.
The averagedetection-mAP is calculated with tIoU thresholds set [0 . 1 : 0 .
We argue that detection-mAPis more reasonable for an instance detection algorithm.
4.2 Comparison with state-of-the-art methods
[Figure] In Tab. 1, we compare the performance of PointTADwith previous multi-label TAD methods under bothsegmentation-mAP and detection-mAP.
The sparseprediction tuples of PointTAD are converted to densesegmentation scores with Eq.
(13) for segmentationmAP.
In order to compare the detection-mAP, wereproduce some of the multi-lable TAD methods andconvert their dense segmentation results to sparsepredictions following [39].
The prediction confidenceof an action with l consecutive frames is calculatedas:
[Equation]
where p i is the probability for class c at frame i , µ c and σ c is the mean and standard derivation ofaction duration of class c in the training set.
Our method surpasses all previous multi-label TAD methods by large under detection-mAP, indicatingour ability to predict complete actions is beyond previous dense segmentation models.
As for
[Table]
segmentation-mAP, we achieve comparable results to previous methods on both benchmarks, whichis encouraging for a sparse detection framework.
In Fig.
4, we show the qualitative results of PointTAD on the MultiTHUMOS test set compared to PDAN and MS-TCT.
PointTAD detects more instances with finer localization especially at hardercategories, such as "Fall" and "SoccerPenalty".
MS-TCT and PDAN perform better at "NoHuman"category, we argue that this is because "NoHuman" class is not a well-defined action category that haspaired action boundaries, whereas PointTAD leverages pair relations of boundaries for localization.
4.3 Ablation Study
Segment representation vs Point representation.
In Tab.
2a, we compare the performance betweensegment representation and query point representation.
In segment implementation, we apply temporalroi align to segments as in [34].
Temporal roi align divides the segment into uniform bins and applyaverage pooling on each bin to arrange frame features for query vector.
The pooler resolution for
RoI Align is set to N s , the same as query point quantity.
Parallel dynamic mixing is carried outin both implementations.
Results show that point representation significantly outperforms segmentrepresentation, demonstrating the strength of adaptive sampling over grid-like alignment upon staticsegments.
Study on query point initialization.
We consider three different initialization for query pointsat training: uniform distribution in [0 , 1] , normal distribution with mean= 0 .
5 and std = 0 .
3 , andinitialization with constant 0 .
Results in Tab. 2b indicates that all three initialization methodsare beneficial for the training, with little performance different in avg-mAP.
In addition, normaldistribution and uniform distribution achieve higher detection-mAP with smaller tIoU threshold,which shows these two initialization methods are weaker at accurate localization.
Empirically, we setconstant initialization as default.
Study on pseudo segment transformation functions.
Tab. 2c shows two alternatives of transformation function T .
Min-max and Partial min-max achieve similar performance under averagedetection-mAP.
We choose Partial min-max as the default function because it offers a more relaxedconstraint for query points.
Study on the number of query points.
We ablate with different numbers of query points N s in
Tab. 2d.
In general, the performance is in proportion to the point quantity, although the increase of N sonly benefits the performance by little.
N s reaches the sweet spot at 21 , where the performance nolonger increases with larger N s .
To balance the model complexity and performance, we set N s to 21.
Study on point-level deformable operator.
In Tab. 2e, we compare point-level deformable convolution and direct interpolation for point-level feature extraction.
Using deformable operator achieves [Figure] higher performance than direct interpolation at query point location, proving that it’s beneficial toconsider local context with dynamic parameters in action detection. [Figure]
Study on instance-level dynamic mixing.
The effect of frame-wise and channel-wise mixing isstudied in Tab. 2f.
We first ablate with the single application of each mixing, i.e. frame-only andchannel-only.
Results show that the performance degrades more without frame mixing, demonstratingthe importance of frame mixing over channel mixing in multi-label TAD.
Furthermore, we exploredifferent combinations of frame and channel mixing with two cascaded alternatives, i.e. framemixing → channel mixing and channel mixing → frame mixing.
Tab.
2f shows that compared withframe-only and channel-only performances respectively, cascaded designs backfire at the performancewith a decrease of 1% avg-mAP at frame → channel and 3% avg-mAP at channel → frame.
To tackleaction detection in video domain, intra-proposal channel mixing does not help with subsequent framemixing and vice versa.
4.4 Query points Visualization
Fig.
5 illustrates the learnt query points and the corresponding action target on a sample from
MultiTHUMOS.
Partial mix-max transformation divides the query points into confined local points(blue) and open-ended global points (orange).
Local points are learned to acknowledge actionboundaries with two extreme points and semantic key frames with interior points.
Global pointssparsely distribute all over the video for open-ended context integration.
Specifically, we observethat the interior local points effectively capture the most distinguishing characteristic of target action"Run" , by capturing rapid forward movement of both legs and neglecting frames of similar yetdisruptive movements: hopping and scoring.
Moreover, the iterative refining process of query pointsare depicted from initial positions to the final positions in the last decoder layer.
Query points arelearnt to start around the median position in the window.
After the first decoder layer, the query pointsare already able to coarsely locate the action target.
Then, these query points automatically adjusttheir locations for finer localization throughout the first three layers and stabilize at the fourth layer.
Conclusion
In this paper, we have studied the complex multi-label TAD that aims to detect all actions from amulti-label untrimmed video.
We formulate this problem as sparse detection task and extend sparsequery-based action detect framework from traditional TAD.
Faced with the challenge of concurrentinstances and complex action relations, we introduce a set of learnable query points to effectivelycapture action boundaries as well as characterize action semantics for fine-grained action features.
Moreover, to facilitate the decoding process, we propose the Multi-level Interactive Module thatintegrates action semantics at point-level and instance-level, with dynamic kernels based on queryvector.
Finally, PointTAD streamlines end-to-end trainable architecture using only RGB inputs foreasy deployment.
Our proposed method surpasses all previous methods by a large margin underdetection-mAP and achieves promising results under segmentation-mAP on two popular multi-label
TAD benchmarks.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See the supplementary material.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] Nopotential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A] No theoreticalresults.
(b) Did you include complete proofs of all theoretical results?
[N/A] No theoretical results.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] The code anddata are proprietary.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Sec.
4.1 and the supplementary material
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] See the supplementary material
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Sec. 4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Sec. 4.
(b) Did you mention the license of the assets?
[Yes] See the supplementary material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See the supplementary material for details.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
The datasets we use are open datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]