InsPro: Propagating Instance Query and Proposal for Online Video Instance Segmentation
Abstract
Video instance segmentation (VIS) aims at segmenting and tracking objects in videos.
Prior methods typically generate frame-level or clip-level object instances first and then associate them by either additional tracking heads or complex instance matching algorithms.
This explicit instance association approach increases system complexity and fails to fully exploit temporal cues in videos.
In this paper, we design a simple, fast and yet effective query-based framework for online VIS.
Relying on an instance query and proposal propagation mechanism with several specially developed components, this framework can perform accurate instance association implicitly.
Specifically, we generate frame-level object instances based on a set of instance query-proposal pairs propagated from previous frames.
This instance query-proposal pair is learned to bind with one specific object across frames through conscientiously developed strategies.
When using such a pair to predict an object instance on the current frame, not only the generated instance is automatically associated with its precursors on previous frames, but the model gets a good prior for predicting the same object.
In this way, we naturally achieve implicit instance association in parallel with segmentation and elegantly take advantage of temporal clues in videos.
To show the effectiveness of our method InsPro, we evaluate it on two popular VIS benchmarks, i.e. , YouTube-VIS 2019 and YouTube-VIS 2021.
Without bells-and-whistles, our InsPro with ResNet-50 backbone achieves 43.2 AP and 37.6 AP on these two benchmarks respectively, outperforming all other online VIS
Introduction
Video instance segmentation (VIS) [1] is a challenging but important computer vision task.
It requires not only segmenting object instances on each video frame but also associating them across all frames.
Due to its fine-grained object representation form, it has got a wide range of applications in various areas such as autonomous driving and video editing.
Existing VIS methods can be categorized into two groups: frame-level methods and clip-level methods.
Frame-level methods [1, 2, 3, 4] generally follow a ‘tracking-by-detection’ paradigm, which first generate per-frame object instances by existing instance segmentation models [5, 6], and then associate them across frames via additional tracking heads (as shown in Figure 1 (a)).
In comparison, clip-level methods [7, 8, 9, 10] take a ‘clip-matching’ paradigm, which divide a video
[Figure]
assign ID
Previous
Current
SegNet
SegNet temporal propagation Previous
Current
Ins Ins
Ins Ins
OurSegNet
OurSegNet query-proposal pair query-proposal pair
In this work, inspired by the recent success of query-based object detectors [11, 12], we propose a simple, fast and yet effective query-based framework for online VIS.
Our system, dubbed as InsPro, segments and tracks objects in one shot through an instance query and proposal propagation strategy with carefully designed modules (Figure 1 (b)) , which eliminates the explicit instance association step.
Specifically, our approach generates frame-level object instances based on a set of instance query-proposal pairs propagated from previous frames.
In the learning process, we develop several techniques to make sure that the generated instance query-proposal pair corresponds to one specific object across frames.
Thus, when an object instance is generated using such a query-proposal pair on the current frame, it is automatically associated with its precursors on all previous frames.
In this way, we achieve implicit object association without a linking step.
Meanwhile, this instance query-proposal propagation mechanism also enables our VIS system to achieve a better prediction accuracy (see Table 1).
This benefits from the instance query-proposal pair’s encoding one object’s temporal and spatial information across all previous frames, which provides a very good prior for the model to infer the same object on the current frame.
In this sense, our query-based VIS method actually implements an efficient way to exploit the intrinsic temporal clues in videos.
To fulfill the advantages of our VIS system, learning exclusive and expressive instance query-proposal pairs is the key.
In this work, we develop several strategies to ensure the learning effectiveness.
First, we design a temporally consistent matching mechanism to enforce the one-to-one correspondence between the instance query-proposal pair and a specific ground truth object across frames during training.
Second, we propose a box deduplication loss to enlarge the distance between instance proposals.
This helps suppress duplicate proposals on the same object and increase the exclusivity of the generated instance query-proposal pair.
At the same time, the sparsely distributed unoccupied query-proposal pairs can serve as candidates in the next frame to detect new objects, allowing our system to achieve new object detection and tracking effortlessly.
Third, we propose an intra-query attention module that enhances instance query with its predecessors encoding the same object.
This explicitly aggregates long-range object information into the query, augmenting its representation capacity, which helps handle occlusion and motion blur.
To validate the effectiveness and efficiency of our InsPro, we conduct extensive experiments on two popular VIS benchmarks [1], i.e. , YouTube-VIS 2019 and YouTube-VIS 2021.
Without bells-andwhistles, our InsPro with ResNet-50 [13] backbone achieves 43.2 AP on YouTube-VIS 2019 and 37.
AP on YouTube-VIS 2021 respectively, outperforming all other online VIS models.
Moreover, our lite variant, InsPro-lite, reaches 38.7 AP on YouTube-VIS 2019 at impressive 45.7 FPS on a Nvidia
RTX2080Ti GPU.
In summary, we make the following contributions in this paper.
We propose a simple, fast and yet effective query-based framework for online VIS.
We develop several techniques to make the query-proposal pair propagation mechanism work smoothly.
These techniques distinguish our work from other query propagation-based object association methods [14, 15, 16], and make our work simpler, more elegant and more effective than them.
Our VIS system achieves the state-of-the-art performances on two popular VIS benchmarks.
Related Work
Frame-level VIS Methods mainly adopt a ‘tracking-by-detection’ paradigm and can run in an online fashion.
They first generate instance predictions frame by frame and then perform explicit instance association.
MaskTrack R-CNN [1] first proposes the VIS task and simply adds an additional tracking head to Mask R-CNN [5] for instance association.
Follow-up works [2, 3, 4, 17] improve either the segmentation or the tracking algorithm to achieve better performance.
On the other hand, some works [8, 18, 19, 20, 21, 22] attempt to perform temporal feature fusion to improve instance segmentation and association.
For example, PCAN [22] proposes frame- and instance-level prototypical cross-attention modules to leverage rich spatio-temporal information to facilitate better segmentation.
All these methods require additional modules to achieve explicit instance association, which expands model complexity and reduces inference speed.
By contrast, our method performs instance association implicitly through an instance query and proposal propagation mechanism, which is simpler and naturally exploits the temporal and spatial consistency in videos.
Clip-level VIS Methods take a ‘clip-matching’ paradigm, which process multiple frames within a clip simultaneously and then perform instance matching between clips to complete VIS.
While some methods [7, 23, 9] propagate instance information within a clip with well-designed propagation modules to model temporal context, recent works [8, 10] utilize transformer [24] to model temporal context in an end-to-end manner.
These methods normally need hand-crafted matching algorithms to complete instance association between clips.
Although they usually achieve high performance, they can only run in an offline mode, which restricts their application to limited areas.
In contrast, our method achieves comparable performance but can run online.
Query-based Methods have attracted increasing attention in recent years due to their flexibility and simplicity.
DETR [11] first uses a set of learned queries interacting with image features to encode objects, and then directly outputs detections by decoding the transformed queries.
Following works [25, 26, 27, 28, 29] improve DETR in terms of either training efficiency or detection performance.
Sparse R-CNN [12] builds a query-based detector on top of R-CNN architecture [30, 31].
Its follow-up works [3, 32] extend it to instance segmentation and video object detection.
Besides, MaxDeepLab [33] proposes a box-free panoptic segmentation method with external query.
DAFL [34] uses a set of queries to encode pedestrian information for pedestrian attribute recognition.
The success of DETR has also inspired query-based VIS methods.
VisTR [8] adapts DETR to the
VIS task.
It takes a video clip as input and directly outputs the sequence of masks for each instance orderly.
IFC [10] proposes inter-frame communication transformers to reduce the heavy computation and memory usage of VisTR-like VIS methods.
Similar to VisTR, Mask2Former [35] applies masked attention to a video clip and directly predicts a 3D instance volume.
To learn a powerful video-level instance query, SeqFormer [36] aggregates temporal information from each frame to the instance query.
These methods work on clips rather than frames, and achieve object association through sharing of the queries within a clip rather than query propagation.
Thus, they still need instance matching between clips.
Instead, our method applies to frames, and can propagate query-proposal pairs through the entire video and thereby can associate object instances over any frame length.
Query Propagation-based Object Association Methods have been recently explored in several works, such as TransTrack [14], TrackFormer [15], MOTR [16] and EfficientVIS [37], which are also inspired by query-based methods [11, 12].
This shows the effectiveness and potential of such a new object linking approach.
The differences between our InsPro and them are as follows.
[Figure]
Temporal propagation
[Figure]
The former transforms instance queries with RoI features of corresponding proposals and produces object features, while the latter predicts object instances based on the object features and conditional convolution [39].
First, our InsPro is different from them in the way of either tracking seen objects or detecting new objects.
TransTrack is basically a ‘tracking-by-detection’ method, because it still needs to explicitly match detection boxes to tracked boxes in each frame, while our InsPro performs implicit association.
More importantly, TransTrack, TrackFormer and MOTR adopt a track query subset to track seen objects and an extra object query subset to detect new objects.
This requires additional heuristic rules to combine two type queries, and may miss occluded or blurred objects with low scores, which can result in object trajectory break [38].
Our InsPro simply propagates all object queries produced in the previous frame to the current frame, and keeps using this set to track seen objects and detect new objects, which is much simpler and more elegant.
As for EfficientVIS, our concurrent work, it does not consider this new object detection problem, and its performance will probably be impacted greatly if there are new objects in the next clip.
Furthermore, we design a more intelligent strategy to suppress duplicates.
TransTrack and TrackFormer employ score filtering or NMS to reduce duplicate predictions.
MOTR builds a temporal aggregation network to learn more discriminative features to address this problem, while EfficientVIS does not discuss this problem.
By contrast, we design a Box Deduplication Loss to suppress duplicates and an Inter-query attention module to enhance queries with their predecessors.
Our solution avoids heuristic rules and post-processing steps, and is more effective according to the experimental results (see Table 2 (e)).
InsPro
We aim to design a simple and fast VIS system that performs instance association implicitly and exploits video temporal clues elegantly.
To this end, we take a query-based VIS approach that predicts object instances on each frame based on a set of instance query-proposal pairs propagated from previous frames.
In this section, we introduce our VIS system, InsPro, including an instance query and proposal propagation mechanism and an instance segmentation head.
Meanwhile, we also describe those proposed techniques that make our propagation mechanism work well.
3.1 Instance Query and Proposal Propagation
The instance query and proposal propagation mechanism enables our VIS system to perform object instance association implicitly in parallel with instance segmentation.
Since it is inspired by the recent query-based object detector Sparse R-CNN [12], we first briefly review Sparse R-CNN.
Sparse R-CNN [12] formulates object detection as a set prediction problem and achieves state-of-theart performance.
It simplifies the detection pipeline and removes heuristic components like NMS.
Specifically, it first initializes a fixed set of learnable instance queries ( N × C , N denotes the number of queries and C the query dimension) paired with learnable instance proposals ( N × 4 ) to describe objects in an image.
As illustrated in Figure 2 (b), each instance query is convolved with the RoI feature of the corresponding proposal to output a more discriminate feature o t [12].
After multi-stage iterative updating, the instance query encodes more accurate object appearance information while the proposal captures more precise location.
Finally, decoding the object feature o t produced using the instance query-proposal pairs, we get the detection results.
Inspired by this instance query-proposal representation of an object, we design a query-proposal temporal propagation mechanism (as shown in Figure 2 (a)) to achieve implicit object instance association and temporal cue utilization in VIS.
Our key insight is that there is a one-to-one correspondence between the learned instance query-proposal pair and a specific object.
If we manage to preserve this correspondence from the first frame to the one where the object finally disappears, then we realize object tracking and object information propagation spontaneously.
To this end, we first initialize a set of instance queries q init ∈ R N × C and proposals p init
∈
R N × 4 on the first video frame I 0 , where q init and p init are learnable parameters and arranged in pairs.
After learning, they are able to encode objects on the first frame.
Decoding them with the first frame image feature inside the SegHead, we obtain instance results r 0 as well as a new set of updated pairs ( q 0 , p 0 ).
Then we propagate this pair set ( q 0 , p 0 ) to the next frame as input to the SegHead.
Similarly, we get the instance results r 1 and another new set of ( q 1 , p 1 ) on this frame.
Among them, the object instance produced on this frame shares the same ID with the one on the previous frame if they are both decoded by the same slice of the instance queries .
In this way, we automatically link object instances belonging to an identical object across frames and elegantly make use of object priors from the past.
Repeating the above process until the last video frame, we then accomplish the VIS task on this video.
Please note that our InsPro simply propagates all object queries produced in the previous frame to the current frame, and keeps using this set to track seen objects and detect new objects.
Instead, recent works [14, 15, 16] that take a similar query-propagation mechanism use a track query set to track seen objects and a new object query set to detect new objects respectively, which requires additional heuristic rules to combine these two type queries.
Moreover, they rely on hand-crafted rules like a score threshold to select a subset of track queries, and occluded objects with low prediction scores are probably filtered out, which results in non-negligible true object missing and fragmented trajectories [38].
In comparison, our method is obviously simpler, more elegant and more effective (see Table 2 (e)).
Intra-query Attention Since frame-by-frame temporal propagation encodes only short-range temporal cues, the instance query from just the last frame shows limitations in dealing with tough scenarios, e.g. , occlusion and motion blur.
To boost the representation capacity of instance query, we augment it in practice with instance features from previous T frames [40, 41].
Specifically, we build a feature bank that caches instance features from previous T frames and perform intra-query attention inside this bank to aggregate long-range temporal cues into the current instance query, as shown in the upper part of Figure 2 (b).
Formally, at frame I t , instance features o from previous T frames are put together to form a feature bank fb =
{ o t − T +1 , . . .
, o t } .
Then, the enhanced instance query is computed as:
[Equation]
where i denotes the i -th query and ε ( · ) is a linear transformation function.
The enhanced q t is basically a weighted sum of instance features inside the feature bank, and the weights are learned upon the quality of the queries.
Experiments (Table 6 (c)) show that this augmentation improves the query representation capacity greatly.
3.2 Segmentation Head
The segmentation head transforms the instance query and performs instance segmentation on each frame.
As illustrated in Figure 2 (b), it is a multi-stage network and has two main parts: a dynamic instance interaction module and an instance segmentation module.
Dynamic Instance Interaction Module transforms the instance query with the proposal RoI feature and yields object features.
It has M stages and forms an iterative structure.
At the first stage, given a pair of instance queries q t − 1 ∈ R N × C and proposals p t − 1 ∈ R N × 4 that propagated from the previous frame I t − 1 , it first augments the instance queries by a self-attention module [24], which models the inter-query relations.
At the same time, it extracts the RoI feature of each proposal on the feature map by RoIAlign [5].
Then, each enhanced instance query convolves with its corresponding
RoI feature through dynamic convolution [42] to get the object feature o t ∈ R N × C .
Since the object feature absorbs temporal cues encoded in instance queries, it has better representation ability than the single-frame RoI feature (see Table 2 (d)).
The object feature is used to predict object instances in the following instance segmentation module, and the newly generated object boxes together with the object features proceed as input to the next stage in the iterative process.
At the final stage, the object feature is processed together with its precursors from previous frames through the aforementioned intra-query attention module, to produce instance queries for the next frame.
Instance Segmentation Module decodes the object feature
o t and produces VIS predictions.
It has three main heads.
While the classification head predicts object classes, the regression one generates object boxes.
Another mask head is responsible for producing instance masks through a conditional convolution [39] approach.
Specifically, it first uses o t to generate conditional convolution weights in weight generator.
As shown in Figure 2 (b), it inputs o it that represents the i -th object instance feature to the weight generator and outputs a set of convolution parameters ω i .
Meanwhile, it produces mask feature maps by transforming FPN [43] feature maps through a mask branch.
Note that the output mask feature maps have 8 channels and a 18 resolution of the input image, and are boosted by concatenating a 2-channel relative coordinates map to it.
This relative coordinates map is computed using the center of predicted object boxes and provides strong location cues for predicting instance masks.
Finally, we feed the combined feature maps f maskt ∈ R 10 × H 8 × W 8 and convolution parameters ω i to a mask FCN head, predicting the i -th object instance mask via a conditional convolution:
[Equation]
For more details about the instance segmentation module, please refer to [39].
3.3 Temporally Consistent Matching
The key to the success of our InsPro is to make sure that the evolving instance query-proposal pair corresponds to the same object across frames in a video.
To ensure this, one technique we propose is temporally consistent matching.
This technique matches predictions and ground truth during training, assigns each ground truth object a proper prediction, and propagates the matching made on previous frames to subsequent frames.
Specifically, given a training batch consisting of multiple consecutive frames, we first compute the matching cost L match between predictions and ground truth objects on the first frame:
[Equation]
where L cls is the focal loss [44] between predicted classifications and ground-truth labels, L L 1 and
L giou are L1 loss and the generalized IoU loss [45] between predicted boxes and ground-truth boxes, respectively.
λ cls , λ L 1 and λ giou are loss weights and set as 2, 5, and 2, respectively.
We search for the best bipartite matching that minimizes the matching cost L match with the Hungarian algorithm [46].
After finding the best matching on the first frame, we propagate this matching to other frames.
Concretely, if one ground truth object still exists on subsequent frames, it will be matched to the prediction that is generated by the same instance query on the first frame.
If there are new objects emerging, new matching will be made between the new objects and yet unmatched predictions.
If a ground truth object disappears, its corresponding predictions will not participate in a new matching process.
Through this temporally consistent matching mechanism, we bind one ground truth object to a single instance query during training.
3.4 Loss Function
Box Deduplication Loss
Although the self-attention mechanism between queries has driven the model to generate fewer duplication predictions [11], we still observe multiple overlapped proposal
[Figure]
boxes on the same object across many frames, as displayed in Figure 3 (a).
We conjecture this is caused by those unmatched queries which cannot be pushed away from those matched queries due to lack of supervision.
To address this problem, we propose a box deduplication loss to push away prediction boxes in terms of the center-to-center distance between them.
As a result, not only the duplication problem is alleviated, but the sparsely distributed unmatched query-proposal pairs can serve as candidates in the next frame to detect and track new objects (see Figure 7 in Appendix).
The loss is defined as:
[Equation]
where b is a ground truth box, ˆ b i neg is a negative box that has the i -th largest IoU with b among those unmatched predicted boxes, C ( · ) is the center distance calculation function, and D ( · ) is the diagonal length calculation function.
β is set as 0.1 and k as 5.
This loss penalizes the short distance between b and ˆ b i neg , and drags all other duplicate boxes away from b [47].
With this new loss, our final box loss function is formed as:
[Equation]
[Equation]
where λ L 1 and λ giou have the same values as in Equation 3, and λ dedup is set as 1.
Overall Loss Given the one-to-one matching results, the final loss on each training frame is a sum of classification, box and mask losses:
[Equation]
[Equation]
where L dice and L focal are dice loss [48] and focal loss [44] for foreground mask prediction, respectively.
We set λ box = 1 , λ dice = 5 and λ focal = 5 .
Finally, the losses of all training frames inside a batch are summed together and normalized by the number of frames.
Experiments 4.1 Datasets and Evaluation Metrics
We evaluate our method on YouTube-VIS 2019 and 2021 benchmarks [1].
YouTube-VISconsists of 2,238 training videos and 302 validation videos, and labels 40 object categories.
YouTubeVIS 2021 is an extended version, which comprises 2,985 training videos and 421 validation videos, and labels improved 40 categories.
All videos in these two datasets are annotated every 5 frames with object bounding box, object category, instance mask and instance ID.
Following [1], we report the video-level average precision (AP) and average recall (AR) on the validation sets as the evaluation metrics, where both accurate instance segmentation and instance association are necessary to achieve high performance.
[Table]
4.2 Implementation Details
We implement our InsPro with Detectron2 [49], and most hyperparameters are set following Sparse
R-CNN [12] and CondInst [39] unless otherwise specified.
More implementation details can be found in Appendix A.
[Table]
Training Details We employ AdamW [50] with an initial learning rate of 2 .
5 × 10 − 5 and weight decay 0.0001 as our model optimizer.
We initialize our model with parameters pre-trained on
COCO [51]
[Table]
and train it for 32k iterations where the learning rate is divided by 10 at iterations 24k and 28k, respectively.
The training is performed end-to-end on 8 Nvidia RTX2080Ti GPUs and each
GPU holds one mini-batch which contains three frame images randomly sampled from the same video.
Data augmentation includes only random horizontal flip and multi-scale training where the training image is resized so that the length of its shortest side is at least 288 and at most 512.
Unless otherwise noted, our InsPro adopts ResNet-50 [13] as backbone and uses 100 instance queries in our experiments.
Inference Details In inference, we resize the frame image size to 640 × 360 , which follows
MaskTrack R-CNN [1].
The size of the feature bank is set to 18 by default.
If the generated proposal box exceeds the frame’s boundaries, it will be clipped to corresponding boundaries.
No multi-scale testing is adopted in our experiments.
InsPro-lite We also build a lite version of our method, named InsPro-lite.
In this variant, inspired by [32], we divide video frames into key frames and non-key frames, i.e. , we select one key frame per K frames in a video and treat other frames as non-key ones.
K is 10 by default.
On key frames, we conduct the dynamic instance interaction 6 times while only once on non-key frames.
This takes advantage of the redundancy of videos and helps reduce inference computation time.
Our InsPro-lite reaches a high inference speed of 45.7 FPS at a small accuracy loss (Table 1).
4.3 Main Results
We perform a thorough comparison of our InsPro to state-of-the-art VIS methods on YouTube-VIS 2019 and 2021.
Existing VIS methods can be divided into two categories according to whether they run online or offline [53].
Since some methods [7, 9] use 80k transformed COCO images [51] as extra training data to prevent overfitting to YouTube-VIS, for a fair comparison, we also report our
[Table] [Table]
[Table]
results with and without extra COCO training data.
Table 1 presents all the results obtained with a
ResNet-50 backbone on a Nvidia RTX2080Ti GPU
[Table]
YouTube-VIS 2019 Table 1 ( left ) shows the comparison between our InsPro and other state-of-theart methods on YouTube-VIS 2019 validation set.
We can see that, in the online group, our InsPro outperforms all other popular methods under the same data setting.
Specifically, our InsPro achieves 40.2 AP without COCO data and 43.2 AP with COCO data respectively, surpassing other online VIS methods by a large margin.
Even our lite version, InsPro-lite, performs better than all other online methods trained without COCO data, reaching 38.7 AP at an impressive speed of 45.7 FPS.
YouTube-VIS
Table 1 ( right ) displays results on YouTube-VIS 2021 validation set.
It shows a similar comparison pattern to YouTube-VIS 2019 and our InsPro achieves the state-of-the-art performance once again.
4.4 Ablation Study
We conduct extensive experiments on YouTube-VIS 2019 to study the effectiveness and individual performance contribution of our proposed modules.
Temporal Propagation and Matching Our InsPro is built on the proposed instance query and proposal temporal propagation mechanism.
Table 2 (a) shows how this mechanism contributes to our high performance.
In this table, method A represents the video instance segmentation baseline, where each frame is processed individually without any temporal propagation, and object instances generated on each frame are linked if they are produced from the same instance query slice.
Since this method lacks the mechanism to ensure the instance query-proposal pair corresponds to the same object across frames, it only achieves 24.0 AP due to inaccurate instance association.
By contrast, when we add the temporal propagation (method B), we can easily improve the performance significantly to 36.3 AP.
This evidences the importance and effectiveness of the proposed temporal propagation technique in a query-based VIS framework.
If we further adopt the temporally consistent matching strategy during training (method C), we achieve an even better performance of 37.4 AP.
We also analyze the separate performance of propagating only instance query (method D) or instance proposal (method E).
The results show that these two settings achieve a similar performance boost (36.7 AP vs 36.6 AP).
Applying them together yields 37.4 AP, bringing further performance gain.
Box Deduplication Loss
We propose a box deduplication loss to suppress the duplicate proposal boxes on the same object across frames.
With the qualitative results shown in Figure 3, we show the quantitative comparison in Table 2 (b).
We can see that supervising the learning with this loss during training can lead 1.0 AP improvement (38.4 AP vs 37.4 AP).
This performance gain is brought by fewer duplicate boxes and fewer missed detections.
Intra-query Attention We perform intra-query attention inside a feature bank to augment the instance query so that it can capture long-range temporal cues.
As we can see in Table 2 (c), this simple method works well and improves the performance considerably.
In particular, T = 1 indicates no intra-query attention is used and 38.4 AP is achieved.
When we increase the volume T of the feature bank, the performance rises and saturates at 40.2 AP with T = 18 .
It is worthwhile to note that this lightweight yet effective intra-query attention module brings almost no speed drop.
Temporal Propagation vs. Tracking-by-Detection Despite the fact that our InsPro does not perform explicit instance association, it still outperforms all other online methods implementing explicit tracking or matching.
To verify that our superior performance comes from the temporal propagation mechanism rather than the instance segmentation model design, we compare our temporal propagation VIS approach to the typical ‘tracking-by-detection’ paradigm with the same instance segmentation baseline.
We implement a ‘tracking-by-detection’ VIS system by replacing the Mask
R-CNN part in MaskTrack R-CNN [1] with our instance segmentation model.
In this case, the only independent variable is the object tracking method.
As shown in Table 2 (d), our InsPro surpasses the ‘tracking-by-detection’ model by a large margin even if our design is simpler and faster, which soundly proves the effectiveness of our method.
We argue again that this is because the evolving instance query-proposal pair in propagation encodes object temporal and spatial cues intrinsic in videos, whereas ‘tracking-by-detection’ methods are generally incapable of exploiting this advantage.
Our United Query vs. Track-and-Detect Query We further compare our method to those MOT methods that adopt a similar query-propagation method for object tracking.
These methods rely on two different query sets, i.e. , a track query set and an object query set, to track seen objects and detect new objects respectively, while we only maintain one united query set.
They need heuristic rules to combine these two type queries.
Meanwhile, they manually select track queries with high scores from the previous frame to build the track query set.
This makes them complex and less effective in tracking since occluded objects with low scores probably have broken trajectories because of the filtering.
To show the superiority of our method, we compare the ‘track-and-detect query’ paradigm adopted in the most recent MOTR [16] to ours using the same instance segmentation baseline.
We follow
MOTR [16] exactly to set up the model and experiment settings.
To exclude the influence of other factors, we do not use temporal feature aggregation in both methods.
Table 2 (e) shows the comparisons on YouTube-VIS 2019.
It can be seen that our InsPro achieves a higher performance even using a simpler query design.
We attribute this advantage to our conscientiously designed modules described in Sec 3.
Conclusion
In this paper, we propose a simple, fast and yet effective query-based framework for online VIS.
In this framework, we rely on a novel instance query and proposal propagation mechanism to undertake
VIS, where we generate object instances based on a set of evolving instance query-proposal pairs propagated from previous frames.
This mechanism enables our model not only to associate object instances implicitly, but to utilize video temporal cues elegantly.
To make this propagation mechanism work well, we develop several modules to ensure that the learned instance query-proposal pair keeps being bound to one object, These modules include an intra-query attention unit, a temporally consistent matching mechanism and a box deduplication loss.
Extensive experiments on YouTubeVIS 2019 and 2021 verify the effectiveness of our designs, and show that our InsPro achieves superior
VIS performance, outperforming all other online VIS methods.
Acknowledgments
This work is supported in part by the National Natural Science Foundation of China (Grant No.
61721004), the Projects of Chinese Academy of Science (Grant No. QYZDB-SSW-JSC006), the
Strategic Priority Research Program of Chinese Academy of Sciences (Grant No. XDA27000000), and the Youth Innovation Promotion Association CAS.
Overview
Figure 4 displays the overview of our InsPro-lite.
In this framework, the video frames are divided into key frames and non-key frames, and we process them differently.
On the key frame I k , the segmentation head consists of 6 dynamic instance interaction modules (DIIM) and one intra-query attention module for generating VIS results r k , and both the instance query and proposal are updated.
On the non-key frame I k +1 , the segmentation head contains only one dynamic instance interaction module to generate VIS results r k +1 , and only the instance proposal is updated while the propagated instance query keeps unchanged.
For more details, please refer to [32].
Training Details Two-phase training is performed for InsPro-lite.
In the first phase, we train the segmentation head processing key frames as described in Sec. 4.2.
In the second phase, We train the segmentation head processing non-key frames while other parts in the network are fixed.
Each training batch contains five frame images randomly sampled from the same video, with three as key frames and two as non-key frames.
Effects of key frame interval Table 3 lists the performances of InsPro-lite using different key frame intervals.
When K = 1 , it represents the original InsPro model.
[Table]
A.2 Additional Comparison on OVIS
We additionally evaluate our InsPro on the occluded video instance segmentation dataset OVIS [54].
OVIS is a very challenging dataset that contains many occlusion scenes.
It consists of 296k highquality instance masks (about 2 × of YouTube-VIS 2019 [1]), 5.8 instances per video (about 3.4 × of
YouTube-VIS 2019), 25 semantic categories, 607 training videos and 140 validation videos.
Table 4 shows the comparison between our InsPro and other state-of-the-art methods on the OVIS validation set.
Due to the severe occlusion and crowded scenes, the performances of all methods
[Table] [Table]
[Table] [Table]
[Table]
on OVIS drop significantly compared to those on YouTube VIS [1].
To tackle occlusion, previous methods [1, 4, 18] apply temporal feature calibration [54] to align adjacent frames and complement the mission objective cues in severe occlusion, which increases the overhead of these systems.
Our InsPro employs flexible instance query-proposal pair propagation and intra-query attention, outperforming all other popular methods without feature calibration.
A.3 Object Instance Segmentation Baselines Comparison
The performance of the object instance segmentation baseline is also one of the important factors affecting the VIS results.
Table 5 lists the performance comparison between our InsPro, Mask
R-CNN [5] and QueryInst [3] on the COCO instance segmentation validation set.
Except that they use different segmentation head structures, both of them adopt the same ResNet-50 backbone, same training time of 36 epochs, and the same data augmentation with ours.
QueryInst [3] and InsPro both use 100 queries.
It can be seen that our base model performs a bit poorer than QueryInst.
A.4 Qualitative Results
We show qualitative results under some challenging scenarios on YouTube-VIS 2019 [1] and
OVIS [54] in Figure 5 and Figure 6, respectively.
Our InsPro is robust to many challenging situations, including fast motion, occlusion, crowd, similar objects, and motion blur, etc .
For example, our InsPro can easy handle new objects with similar appearance, as shown in line 6 and 8 of Figure 5.
This is because our InsPro propagates not only object queries but their corresponding proposals.
Since those proposals have tracked objects positional prior encoded, when using such a query-proposal pair to predict objects, it is easy for the network to distinguish objects of similar appearance.
However, InsPro may fail to segment some small objects since it has no specific design for processing small objects.
A.5 Additional Experiments on ImageNet VID
To further verify the effectiveness and generality of our InsPro, we conduct additional experiments on ImageNet VID [55].
ImageNet VID [55] is a large-scale video dataset where the object instances on every frame are fully annotated with object bounding box, object category and instance ID.
It consists of 3,862 training videos and 555 validation videos from 30 object categories.
The average length of videos in ImageNet VID is 317 (about × 11.5 of YouTube-VIS 2019 [1]).
We evaluate the joint detection and tracking performance of our InsPro on ImageNet VID.
Following [1], we use video-level average precision (AP) as the evaluation metric and the video-level IoU is computed using box sequences instead of mask sequences.
[Figure]
A.5.1 Implementation Details
We make minor modifications to InsPro to adapt it for joint detection and tracking on ImageNet
VID.
We remove the segmentation part in the instance segmentation module (Sec. 3.2), and the rest remain unchanged.
During training, the segmentation losses L dice and L focal are removed from the overall loss L (Equation 6) either.
The modified InsPro, called InsPro-VID, is trained on both the
ImageNet DET training set and the ImageNet VID training set
Each training batch contains three frame images
On VID, the three images are randomly sampled from the same video, while on DET, the three images are the same because DET contains only images
InsPro-VID is trained for 90k iterations and the initial learning rate is set to 2 .
5 × 10 − 5 , is divided by 10 at
65k
[Table] [Table]
[Table] [Table]
A.5.2 Main Results
We compare our InsPro-VID with other methods in Table 6 (a).
Other methods mainly use tracking or post-processing matching (e
[Table]
g., Viterbi [56] algorithm) techniques to associate detection results across frames, which are more complex and unable to fully utilize temporal cues.
Our InsPro-VID achieves instance association in parallel with detection and elegantly takes advantage of temporal clues in videos through the proposed instance query-proposal propagation mechanism.
As shown in
Table 6 (a), our InsPro-VID achieves superior performance compared over other methods.
A.5.3 Ablation Study
We conduct extensive experiments on ImageNet VID to study the effectiveness and individual performance contribution of our proposed modules.
All experiments use ResNet-50 [13] as backbone.
Temporal Propagation and Matching Table 6 (b) shows how the proposed instance query and proposal temporal propagation mechanism and the temporally consistent matching strategy contribute to our high performance.
[Figure]
Box Deduplication Loss Table 6 (c) shows the effectiveness of the proposed box deduplication loss.
β is a hyperparameter in Equation 4, which controls the center distance between the duplicate predicted boxes and the ground truth box during training.
β = 0 indicates no box deduplication loss is during and
2 AP is achieved
When increase the β the when β = 0 .
We also provide some qualitative results in Figure 7 to show the effect of the box deduplication loss.
We display all predicted boxes on each frame without a score threshold, with 100 detections by default.
The first row is the visualization results when β = 0 .
We can see that all predicted boxes in the first three frames are clustered around a few instances.
Since in our temporal propagation mechanism the prediction of the next frame is generated based on the instance proposals propagated from the last frame, if all predicted boxes in the last frame are clustered around the existing instances, it will be difficult for the model to detect new objects in the next frame.
As shown in Figure 7 (a), we can see that the newly emerging instances in red box on the last two frames are indeed not detected.
Figure 7 (b) is the visualization results when β
= 0 .
It shows that each instance has only one accurate predicted box and the new emerging objects are detected.
Therefore, the proposed box deduplication loss enables our InsPro not only to alleviate the duplicate problem, but to detect new objects easily with the sparsely distributed unmatched proposal.
Intra-query Attention Table 6 (d) shows the effectiveness of the intra-query attention.
When increasing the size T of the feature bank, the instance query can capture more temporal cues and the performance improves considerably.
A.6 Broader Impact and Future Work
Our InsPro introduces a novel instance query and proposal propagation mechanism to the VIS system, and constructs a simple, fast and yet effective online framework to achieve one-shot video instance segmentation.
Our InsPro-lite can achieve promising accuracy while running in real-time.
Due to its fine-grained object representation result and efficiency, we believe our VIS system InsPro can positively impact many applications such as autonomous driving and video editing, etc .
For future work, we plan to verify the generality of our method in other query-based frameworks [11, 25, 59], since our proposed temporal propagation mechanism is only applied in Sparse R-CNN [12] at present.
Furthermore, we also intend to implement the simple yet effective temporal propagation mechanism in tasks that require instance association, such as multi-object tracking [60], panoptic segmentation [61, 62] in video, and semi-supervised video object segmentation [63].