InsPro: Propagating Instance Query and Proposal for Online Video Instance Segmentation
Anonymous Author(s)
AffiliationAddress email
Abstract
Video instance segmentation (VIS) aims at segmenting and tracking objects invideos.
Prior methods typically first generate frame-level or clip-level objectinstances and then associate them by either additional tracking heads or complexinstance matching algorithms.
This explicit instance association approach increasessystem complexity and fails to fully exploit temporal cues in videos.
In this paper,we design a simple, fast and yet effective query-based framework for online VIS,where we propose an instance query and proposal propagation mechanism toperform instance association implicitly.
Specifically, we generate frame-levelobject instances based on a set of instance query-proposal pairs propagated fromprevious frames.
This instance query-proposal pair is learned to represent onespecific instance across frames through several developed strategies.
When usingsuch a query-proposal pair to predict an object instance on the current frame,not only the generated instance is automatically associated with its precursors onprevious frames, but the model gets a good prior for predicting the same object.
In this way, we naturally achieve implicit instance association in parallel withsegmentation and elegantly take advantage of temporal clues in videos.
To showthe effectiveness of our method which is dubbed as InsPro, we evaluate it ontwo popular VIS benchmarks, i.e. , YouTube-VIS 2019 and YouTube-VIS 2021.
Without bells-and-whistles, our InsPro with ResNet-50 backbone achieves 43.2 APand 37.6 AP on these two benchmarks respectively, outperforming all other online VIS methods.
Code will be made publicly available.
Introduction
Video instance segmentation (VIS) [1] is a challenging but important computer vision task.
It requiresnot only segmenting object instances on each video frame but also associating them across all frames.
Due to its fine-grained object representation form, it has got a wide range of applications such asautonomous driving and video editing.
Existing VIS methods can be categorized into two groups: frame-level methods and clip-levelmethods.
Frame-level methods [1, 2, 3, 4] generally follow a ‘tracking-by-detection’ paradigm,which first generate per-frame object instances by existing instance segmentation models [5, 6],and then associate them across frames via additional tracking heads (as shown in Figure 1 (a)).
Incomparison, clip-level methods [7, 8, 9, 10] take a ‘clip-matching’ paradigm, which divide a videointo multiple overlapped clips and generate instance predictions of each clip, and then associate these clip-level predictions by some hand-crafted instance matching algorithms.
Whether frame-level or
[Figure]
assign
[Figure]
Previous
Current
SegNet
SegNet temporal propagation Previous
Current
Ins Ins
Ins Ins
OurSegNet
OurSegNet query-proposal pair query [Figure] proposal pair clip-level methods, both of them inevitably need an explicit instance association step to fulfill objecttracking.
This generally requires to design a complicated association strategy to achieve good trackingperformance, which is not trivial.
More importantly, the explicit association step increases modelcomplexity and slows inference speed.
Furthermore, this extra step also indicates that the temporalclues intrinsic in videos are not well utilized, as the instance prediction is performed separately oneach frame or each clip.
In this work, inspired by the recent success of query-based object detectors [11, 12], we propose asimple, fast and yet effective query-based framework for online VIS.
Our system, dubbed as InsPro,segments and tracks objects in one shot through a designed instance query and proposal propagationstrategy (Figure 1 (b)), which eliminates the explicit instance association step.
Specifically, ourapproach generates frame-level object instances based on a set of instance query-proposal pairspropagated from previous frames.
In the learning process, we develop several techniques to makesure that the generated instance query-proposal pair corresponds to one specific object across frames.
Thus, when an object instance is generated using such a query-proposal pair on the current frame, itis automatically associated with its precursors on all previous frames.
In this way, we achieve theimplicit object association without a linking step.
Meanwhile, the proposed instance query-proposalpropagation mechanism also enables our VIS system to perform better in terms of object instanceaccuracy (see Table 1).
This is because the instance query-proposal pair implicitly encodes oneobject’s temporal and spatial information across all previous frames, which provides a very goodprior for the model to infer the same object on the current frame.
In this sense, our query-based VISmethod actually implements an efficient way to exploit the intrinsic temporal clues in videos.
To fulfill the advantages of our VIS system, learning exclusive and expressive instance query-proposalpairs is the key.
We develop several strategies to ensure learning effectiveness.
First, we proposean intra-query attention module that enhances instance query with its predecessors encoding thesame object.
This explicitly aggregates long-range object information into the query, augmenting itsrepresentation capacity.
Second, we design a temporally consistent matching mechanism to enforcethe one-to-one correspondence between the instance query-proposal pair and a specific ground truthobject across frames during training.
Third, we propose a box deduplication loss to enlarge thedistance between instance proposals.
This helps suppress duplicate proposals on the same object andincrease the exclusivity of the generated instance query-proposal pair.
To validate the effectiveness and efficiency of our InsPro, we conduct extensive experiments on twopopular VIS benchmarks [1], i.e. , YouTube-VIS 2019 and YouTube-VIS 2021.
Without bells-andwhistles, our InsPro with ResNet-50 [13] backbone achieves 43.2 AP on YouTube-VIS 2019 and 37.
AP on YouTube-VIS 2021 respectively, outperforming all other online VIS models.
Moreover, ourlite variant, InsPro-lite, reaches 38.7 AP at impressive 45.7 FPS on YouTube-VIS 2019 on a Nvidia
RTX2080Ti GPU.
Related Work
Frame-level VIS Methods mainly adopt a ‘tracking-by-detection’ paradigm and can run in anonline fashion.
They first generate instance predictions frame by frame and then perform explicitinstance association.
MaskTrack R-CNN [1] proposes the VIS task for the first time and provides anexample VIS method. It simply adds an additional tracking head to Mask R-CNN [5] for instanceassociation.
Follow-up works [2, 3, 4] improve either the image instance segmentation model or thetracking algorithm to achieve better performance.
On the other hand, some works [8, 14, 15, 16, 17]attempt to perform temporal feature fusion to improve instance segmentation and association.
Allthese methods require additional modules to achieve explicit instance association, which expandsmodel complexity and reduces inference speed.
Additionally, they do instance segmentation oneach frame independently, ignoring the temporal and spatial consistency between video frames.
Bycontrast, our method performs the instance association implicitly through an instance query andproposal propagation mechanism, which is simpler and naturally exploits the temporal and spatialconsistency in videos.
Clip-level VIS Methods take a ‘clip-matching’ paradigm, which process multiple frames withina clip simultaneously and then perform instance matching between clips to complete VIS.
Whilesome methods [7, 18, 9] propagate instance information within a clip with well-designed propagationmodules to model temporal context, recent works [8, 10] utilize transformer [19] to model temporalcontext in an end-to-end manner.
These methods normally need hand-crafted matching algorithms tocomplete instance association after getting the instances of each clip.
Although they usually achievehigh performance, they can only run in an offline mode, which restricts their application in limitedareas.
In contrast, our method achieves comparable performance but can run online.
Query-based Methods have attracted increasing attention in recent years.
DETR [11] first introduces a query-based architecture with transformer [19] to reformulate object detection as a setprediction problem. It uses a set of learned queries interacting with image features to encodeobjects, and then directly outputs detection predictions by decoding the transformed queries.
Following works [20, 21, 22, 23, 24] improve DETR in terms of either training efficiency or detectionperformance.
Similarly, Sparse R-CNN [12] builds a query-based detector on top of R-CNN architecture [25, 26].
Its follow-up works [3, 27] extend it to instance segmentation and video object detection.
TrackFormer [28], TransTrack [29] and MOTR [30] utilize DETR to build query-based multipleobject tracking methods.
Besides, Max-DeepLab [31] proposes a box-free panoptic segmentationmethod with external query.
DAFL [32] uses a set of queries to encode pedestrian information forpedestrian attribute recognition.
Query-based methods have shown great potential in computer visiondue to their flexibility and simplicity.
In this work, we develop a new query-based paradigm for VISto achieve implicit tracking and video temporal information exploitation.
InsPro
We aim to design a simple and fast VIS system that performs instance association implicitly andexploits video temporal clues elegantly.
To this end, we take a query-based VIS approach thatpredicts object instances on each frame based on a set of instance query-proposal pairs propagatedfrom previous frames.
In this section, we introduce our VIS system, InsPro, including an instancequery and proposal propagation mechanism and an instance segmentation head. Meanwhile, we alsodescribe those proposed techniques that make our propagation mechanism work well.
3.1 Instance Query and Proposal Propagation
The instance query and proposal propagation mechanism enables our VIS system to perform objectinstance association implicitly in parallel with instance segmentation.
Since it is inspired by therecent query-based object detector Sparse R-CNN [12], we first review Sparse R-CNN.
Sparse R-CNN [12] formulates object detection as a set prediction problem and achieves state-of-theart performance.
It simplifies the detection pipeline and removes heuristic components like NMS. [Figure]
Temporal propagation
[Figure]
The former transforms instance queries with RoI features of corresponding proposals and produces object features, while the latter predicts object instances based on the object features and conditional convolution [33].
Specifically, it first initializes a fixed set of learnable instance queries ( N × C , N denotes the numberof queries and C the query dimension) paired with learnable instance proposals ( N × 4 ) to describeobjects in an image.
As illustrated in Figure 2 (b), each instance query is then convolved with the RoIfeature of the corresponding proposal to output a more discriminate feature o t [12].
After multi-stageiterative updating, the instance query encodes more accurate object appearance information while theproposal captures more precise location information.
Finally, through decoding the object feature o tproduced based on the instance query-proposal pairs, we get the detection results.
Inspired by the instance query-proposal representation of an object, we design a query-proposaltemporal propagation mechanism (as shown in Figure 2 (a)) to achieve implicit object instance association and temporal cue utilization in VIS.
Our key insight is that there is a one-to-one correspondencebetween the learned instance query-proposal pair and a specific object.
If we manage to preserve thiscorrespondence from the first frame to the one where the object disappears, then we realize objecttracking and object information propagation spontaneously.
To this end, we first initialize a set of instance queries q init ∈ R N × C and proposals p init
∈
R N × 4 onthe first video frame
I 0 , where q init and p init are learnable parameters and arranged in pairs.
Afterlearning, they are able to encode objects on the first frame.
Decoding them with the first frame imagefeature inside the SegHead, we obtain instance results r 0 as well as a new set of updated pairs ( q 0 ,p 0 ).
Then we propagate this pair set ( q 0 , p 0 ) to the next frame as input to the SegHead.
Similarly,we get the instance results r 1 and another new set of ( q 1 , p 1 ) on this frame.
Among them, the objectinstance produced on this frame shares the same ID with the one on the previous frame if they areboth decoded by the same slice of the instance queries .
In this way, we automatically link objectinstances belonging to an identical object across frames and elegantly make use of object priors fromthe past.
Repeating the above process until the last video frame, we then accomplish the VIS task onthis video.
Intra-query Attention Since frame-by-frame temporal propagation encodes only short-rangetemporal cues, the instance query from just the last frame shows limitations in dealing with toughscenarios, e.g. , occlusion and motion blur.
To boost the representation capacity of instance query,we augment it in practice with instance features from previous T frames.
Specifically, we build afeature bank that caches instance features from previous T frames and perform intra-query attentioninside this bank to aggregate long-range temporal cues into the current instance query, as shown inthe upper part of Figure 2 (b).
Formally, at frame I t , instance features o from previous T frames are put together to form a feature bank fb =
{ o t − T +1 , . . .
, o t } .
Then, the enhanced instance query iscomputed as:
[Equation]
where i denotes the i -th query and ε is a linear transformation function.
The enhanced q t is basicallya weighted sum of instance features inside the feature bank, and the weights are learned upon thequality of the queries.
Experiments (Table 2 (b)) show that this augmentation improves the queryrepresentation capacity greatly.
3.2 Segmentation Head
The segmentation head transforms the instance query and performs instance segmentation on eachframe.
As illustrated in Figure 2 (b), it is a multi-stage network and has two main parts: a dynamicinstance interaction module and an instance segmentation module.
Dynamic Instance Interaction Module transforms the instance query with the proposal RoIfeature and yields object features.
It has M stages to form an iterative structure.
At the first stage,given a pair of instance queries q t − 1 ∈ R N × C and proposals p t − 1 ∈ R N × 4 that propagated fromthe previous frame I t − 1 , it first augments the instance queries by a self-attention module [19], whichmodels the inter-query relations.
At the same time, it extracts the RoI feature of each proposal on thefeature map by RoIAlign [5].
Then, each enhanced instance query convolves with its corresponding
RoI feature through dynamic convolution [34] to get the object feature o t ∈ R N × C .
Since the objectfeature absorbs temporal cues encoded in instance queries, it has better representation ability than thesingle-frame RoI feature (see Table 2 (d)).
The object feature is used to predict object instances in thefollowing instance segmentation module, and the newly generated object boxes together with theobject features proceed as input to the next stage in the iterative process.
At the final stage, the objectfeature is processed together with its precursors from previous frames through the aforementionedintra-query attention module, to produce instance queries for the next frame.
Instance Segmentation Module decodes the object feature
o t and produces VIS predictions.
It hasthree main heads.
While the classification head predicts object classes, the regression one generatesobject boxes and the predicted boxes at the final stage are passed to the next frame.
Another maskhead is responsible for producing instance masks through a conditional convolution [33] approach.
Specifically, it first uses o t to generate conditional convolution weights in weight generator.
Asshown in Figure 2 (b), it inputs o it that represents the i -th object instance feature to the weightgenerator and outputs a set of convolution parameters ω i .
Meanwhile, it produces mask feature mapsby transforming FPN [35] feature maps through a mask branch.
Note that the output mask featuremaps have 8 channels and a 18 resolution of the input image, and are boosted by concatenating a2-channel relative coordinates map.
This relative coordinates map is computed using the center ofpredicted object boxes and provides strong location cues for predicting instance masks.
Finally, wefeed the combined feature maps f maskt ∈ R 10 × H 8 × W 8 and convolution parameters ω i to a mask FCNhead, predicting the i -th object instance mask via a conditional convolution:
[Equation]
For more details about the instance segmentation, please refer to [33].
3.3 Temporally Consistent Matching
The key to the success of our InsPro is to make sure the evolving instance query-proposal paircorresponds to the same object across frames in a video.
To ensure this, one technique we propose istemporally consistent matching.
This technique matches predictions and ground truth during training,assigns each ground truth object a proper prediction, and propagates the matching made on previousframes to subsequent frames.
Specifically, given a training batch consisting of multiple consecutive frames, following [11, 12], wefirst compute the matching cost L match between predictions and ground truth objects on the first
[Figure]
[Figure]
[Equation]
where L cls is the focal loss [36] between predicted classifications and ground-truth labels, L L 1and L giou are L1 loss and the generalized IoU loss [37] between predicted boxes and ground-truthboxes, respectively.
λ cls , λ L 1 and λ giou are loss weights and set as 2, 5, and 2, respectively.
Wesearch for the best bipartite matching that minimizes the matching cost L match with the Hungarianalgorithm [38].
After finding the best matching on the first frame, we propagate this matching toother frames.
Concretely, on subsequent frames, if one ground truth object still exists, it will bematched to the prediction that is generated by the same instance query on the first frame.
If there arenew ground truth objects emerging, new matching will be made between the new objects and yetunmatched predictions.
If a ground truth object disappears, its corresponding predictions will notparticipate in a new matching process.
Through this temporally consistent matching mechanism, webind one ground truth object to a single instance query during training.
3.4 Loss Function
Box Deduplication Loss
Although the self-attention mechanism between queries has driven themodel to generate fewer duplication predictions [11], we still observe multiple overlapped proposalboxes on the same object across many frames, as displayed in Figure 3 (a).
We conjecture this iscaused by those unmatched queries that cannot be pushed away from those matched queries due tolack of supervision.
To address this problem, we propose a new box deduplication loss to push awayprediction boxes in terms of the center-to-center distance between them.
It is defined as:
[Equation]
where b i is a ground truth box, ˆ b ineg is a negative box that has the largest IoU with b i among thoseunmatched predicted boxes, c ( · ) is the center distance, and d ( · ) is the diagonal length.
β is set as 0.1.
This loss penalizes the short distance between b i and ˆ b ineg , and drags all other duplicate boxes awayfrom b i [39].
With this new loss, our final box loss function is formed as:
[Equation]
where λ L 1 and λ giou have the same values in Equation 3, and λ dedup is set as 1.
Overall Loss Given the one-to-one matching results, the final loss on each training frame is a sumof classification, box and mask losses:
[Equation]
where L dice and L focal are dice loss [40] and focal loss [36] for foreground mask prediction,respectively.
We set λ box = 1 , λ dice = 5 and λ focal = 5 .
Finally, the losses of all training framesinside a batch are summed together and normalized by the number of frames.
Experiments
4.1 Datasets and Evaluation Metrics
We evaluate our method on YouTube-VIS 2019 and 2021 benchmarks [1].
YouTube-VIS 2019consists of 2,238 training videos and 302 validation videos, and labels 40 object categories.
YouTubeVIS 2021 is an extended version, which comprises 2,985 training videos and 421 validation videos,and labels improved 40 categories.
All videos in these two datasets are annotated every 5 frames withobject bounding box, object category, instance mask and instance ID.
Following [1], we report thevideo-level average precision (AP) and average recall (AR) on the validation sets as the evaluationmetrics, where both accurate instance segmentation and instance association are necessary to achievehigh performance.
4.2 Implementation Details
We implement our InsPro with detectron2 [41], and most hyperparameters are set following Sparse
R-CNN [12] and CondInst [33] unless otherwise specified.
Training Details We employ AdamW [42] with an initial learning rate of 2 .
5 × 10 − 5 and weightdecay 0.0001 as our model optimizer.
We initialize our model with parameters pre-trained on
COCO [43], and train it for 32k iterations where the learning rate is divided by 10 at iterations 28kand 24k, respectively.
The training is performed end-to-end on 8 Nvidia RTX2080Ti GPUs and each
GPU holds one mini-batch which contains three frame images randomly sampled from the samevideo.
Data augmentation includes only random horizontal flip and multi-scale training where thetraining image is resized with the shortest side at least 288 and at most 512.
Unless otherwise noted,our InsPro adopts 100 instance queries and ResNet-50 [13] as backbone in our experiments.
Inference Details In inference, we resize the frame image size to 640 × 360 , which follows
MaskTrack R-CNN [1].
The length of the feature bank is set to 18 by default.
If the generatedproposal box size exceeds the frame’s, it will be reset to the frame size.
No multi-scale testing isadopted in our experiments.
InsPro-lite we also build a lite version of our method, named InsPro-lite.
In this variant, inspiredby [27], we divide video frames into key frames and non-key frames, i.e. , we select one key frameper K frames in a video and treat other frames as non-key ones.
K is 10 by default.
On key frames,we conduct the dynamic instance interaction 6 times while only once on non-key frames.
This takesadvantage of the redundancy of videos and helps reduce inference computation time.
Our InsPro-litereaches a high inference speed of 45.7 FPS at a small accuracy loss (Table 1).
4.3 Main Results
We perform a thorough comparison of our InsPro to state-of-the-art VIS methods on YouTube-VIS2019 and 2021.
Existing VIS methods can be divided into two categories according to whether theyrun online or offline [46].
Since some methods [7, 9] also use 80k transformed COCO images [43] asextra training data to prevent overfitting to YouTube-VIS, for a fair comparison, we also report ourresults with and without extra COCO training data.
Table 1 present all the results obtained with a
ResNet-50 backbone on a Nvidia RTX2080Ti GPU.
YouTube-VIS 2019 Table 1 (a) shows the comparison between our InsPro and other state-of-the-artmethods on YouTube-VIS 2019 validation set.
We can see that, in the online group, our InsProoutperforms all other popular methods under the same data setting.
Specifically, our InsPro achieves40.2 AP without COCO data and 43.2 AP with COCO data respectively, surpassing other online VIS methods by a large margin.
Even our lite version, InsPro-lite, performs better than all otheronline methods trained without COCO data, reaching 38.7 AP at an impressive speed of 45.7 FPS.
Compared to those offline methods, our InsPro beats most of them and is only inferior to IFC [10]which takes a transformer-based approach to VIS. However, IFC is a clip-level method and can onlyrun offline, which is not suitable in real-time video processing settings. [Table]
[Table] [Table] [Table] MaskTrack R
[Table]
CNN [1]
[Table] [Table]
[Table] [Table] [Table] [Table]
YouTube-VIS 2021 Table 1 (b) displays results on YouTube-VIS 2021 validation set.
It showsa similar comparison pattern to YouTube-VIS 2019 and our InsPro achieves the state-of-the-artperformance once again.
4.4 Ablation Study
We conduct extensive experiments on YouTube-2019 to study the effectiveness and individualperformance contribution of our proposed modules.
Temporal Propagation and Matching Our InsPro is built on the proposed mechanism of instancequery and proposal temporal propagation.
Table 2 (a) shows how this mechanism contributes toour high performance.
In this table, method A represents the image instance segmentation baseline,where each frame is processed individually without any temporal propagation, and object instancesgenerated on each frame are linked if they are produced from the same instance query slice.
Sincethis method lacks the mechanism to ensure the instance query-proposal pair corresponds to thesame object across frames, it only achieves 24.0 AP due to inaccurate instance association.
Bycontrast, when we add the temporal propagation (method B), we can easily improve the performancesignificantly to 36.3 AP.
This evidences the importance and effectiveness of the proposed temporalpropagation technique in a query-based VIS framework.
If we further adopts the temporally consistentmatching strategy during training (method C), we achieve an even better performance of 37.4 AP.
We also analyze the separate performance of propagating only instance query (method D) or instanceproposal (method E).
The results show that these two settings achieve a similar performance boost(36.7 AP vs 36.6 AP).
Applying them together yields 37.4 AP, bringing further performance gain.
Intra-query Attention We perform intra-query attention inside a feature bank to augment theinstance query so that it can capture long-range temporal cues.
As we can see in Table 2 (b), thissimple method works well and improves the performance considerably.
In particular, T = 1 indicatesno intra-query attention is used and 38.4 AP is achieved.
When we increase the volume T of thefeature bank, the performance also rises and saturates at 40.2 AP with T = 18 .
It is worthwhile tonote that this lightweight yet effective intra-query attention operation brings almost no speed drop.
Box Deduplication Loss
We propose a box deduplication loss to suppress the duplicate proposalboxes on the same object across frames.
While the qualitative results can be found in Figure 3, weshow the quantitative comparison in Table 2 (c).
We can see that supervising the learning with thisloss during training can lead 1.0 AP improvement (38.4 AP vs 37.4 AP).
This performance gain isbrought by less false positives.
Track-by-Detect vs. Temporal Propagation Despite the fact that our InsPro does not performexplicit instance association, it still outperforms all other online methods implementing explicittracking or matching.
To verify that our superior performance comes from the temporal propagationmechanism rather than the image instance segmentation model design, we compare our temporalpropagation VIS approach to the typical ‘track-by-detect’ paradigm with the same instance segmentation baseline.
We implement a ‘track-by-detect’ VIS system by replacing the Mask R-CNN part in
MaskTrack R-CNN [1] with our instance segmentation model.
In this case, the only independentvariable is the object tracking method.
As shown in Table 2 (d), our InsPro surpasses the ‘track-by-detect’ model by a large margin even ifour design is simpler and faster, which soundly proves the effectiveness of our method.
We argueagain this is because the evolving instance query-proposal pair in propagation encodes object temporaland spatial cues intrinsic in videos, whereas ‘track-by-detect’ methods are generally incapable ofexploiting this advantage.
Conclusion
In this paper, we propose a simple, fast and yet effective query-based framework for online VIS.
Inthis framework, we rely on a novel instance query and proposal propagation mechanism to undertake
VIS, where we generate object instances based on a set of evolving instance query-proposal pairspropagated from previous frames.
This mechanism enables our model not only to associate objectinstances implicitly, but to utilize video temporal cues elegantly.
To make this propagation mechanismwork well, we develop several modules to ensure that the learned instance query-proposal pairkeeps being bound to one object, These modules include an intra-query attention unit, a temporallyconsistent matching mechanism and a box deduplication loss.
Extensive experiments on YouTubeVIS 2019 and 2021 verify the effectiveness of our designs, and show that our InsPro achieves superior
VIS performance, outperforming all other online VIS methods.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the
paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We discuss limitations and future
work in Appendix.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We discuss
the potential negative societal impacts in Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
Yes
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We listedthe data and instructions in Sec. 4 and Appendix, and the code will be released uponacceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] All training details are listed in Sec. 4 and Appendix.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]