InsPro: Propagating Instance Query and Proposal for Online Video Instance Segmentation
Anonymous Author(s)
AffiliationAddress email
Abstract
Video instance segmentation (VIS) aims at segmenting and tracking objects invideos.
Prior methods typically first generate frame-level or clip-level objectinstances and then associate them by either additional tracking heads or complexinstance matching algorithms.
This explicit instance association approach increasessystem complexity and fails to fully exploit temporal cues in videos.
In this paper,we design a simple, fast and yet effective query-based framework for online VIS. Relying on an instance query and proposal propagation mechanism with severalspecially developed components, this framework can perform accurate instanceassociation implicitly.
Specifically, we generate frame-level object instances basedon a set of instance query-proposal pairs propagated from previous frames.
Thisinstance query-proposal pair is learned to bind with one specific object acrossframes through conscientiously developed strategies.
When using such a pair topredict an object instance on the current frame, not only the generated instanceis automatically associated with its precursors on previous frames, but the modelgets a good prior for predicting the same object.
In this way, we naturally achieveimplicit instance association in parallel with segmentation and elegantly takeadvantage of temporal clues in videos.
To show the effectiveness of our method InsPro, we evaluate it on two popular VIS benchmarks, i.e. , YouTube-VIS 2019and YouTube-VIS 2021.
Without bells-and-whistles, our InsPro with ResNet-50backbone achieves 43.2 AP and 37.6 AP on these two benchmarks respectively,outperforming all other online VIS methods.
Code will be made publicly available.
Introduction
Video instance segmentation (VIS) [1] is a challenging but important computer vision task.
It requiresnot only segmenting object instances on each video frame but also associating them across all frames.
Due to its fine-grained object representation form, it has got a wide range of applications in variousareas such as autonomous driving and video editing.
Existing VIS methods can be categorized into two groups: frame-level methods and clip-levelmethods.
Frame-level methods [1, 2, 3, 4] generally follow a ‘tracking-by-detection’ paradigm,which first generate per-frame object instances by existing instance segmentation models [5, 6],and then associate them across frames via additional tracking heads (as shown in Figure 1 (a)).
Incomparison, clip-level methods [7, 8, 9, 10] take a ‘clip-matching’ paradigm, which divide a videointo multiple overlapped clips, generate instance predictions for each clip, and then associate theseclip-level predictions by some hand-crafted instance matching algorithms.
Whether frame-level or
[Figure]
assign ID
Previous
Current
SegNet
SegNet temporal propagation Previous
Current
Ins Ins
Ins Ins
OurSegNet
OurSegNet query-proposal pair query-proposal pair clip-level methods, both of them inevitably need an explicit instance association step to fulfill objecttracking.
This generally requires to design a complicated association strategy to achieve good trackingperformance, which is not trivial.
More importantly, the explicit association step increases modelcomplexity and slows inference speed.
Furthermore, this extra step also indicates that the temporalclues intrinsic in videos are not well utilized, as the instance prediction is performed separately oneach frame or each clip.
In this work, inspired by the recent success of query-based object detectors [11, 12], we propose asimple, fast and yet effective query-based framework for online VIS.
Our system, dubbed as InsPro,segments and tracks objects in one shot through an instance query and proposal propagation strategywith carefully designed modules (Figure 1 (b)) , which eliminates the explicit instance associationstep.
Specifically, our approach generates frame-level object instances based on a set of instancequery-proposal pairs propagated from previous frames.
In the learning process, we develop severaltechniques to make sure that the generated instance query-proposal pair corresponds to one specificobject across frames.
Thus, when an object instance is generated using such a query-proposal pairon the current frame, it is automatically associated with its precursors on all previous frames.
Inthis way, we achieve the implicit object association without a linking step.
Meanwhile, this instancequery-proposal propagation mechanism also enables our VIS system to perform better in terms ofprediction accuracy (see Table 1).
This is because the instance query-proposal pair implicitly encodesone object’s temporal and spatial information across all previous frames, which provides a very goodprior for the model to infer the same object on the current frame.
In this sense, our query-based VISmethod actually implements an efficient way to exploit the intrinsic temporal clues in videos.
To fulfill the advantages of our VIS system, learning exclusive and expressive instance query-proposalpairs is the key.
In this work, we develop several strategies to ensure the learning effectiveness.
First,we design a temporally consistent matching mechanism to enforce the one-to-one correspondence between the instance query-proposal pair and a specific ground truth object across framesduring training.
Second, we propose a box deduplication loss to enlarge the distance between instanceproposals.
This helps suppress duplicate proposals on the same object and increase the exclusivity ofthe generated instance query-proposal pair.
At the same time, the sparsely distributed unoccupiedquery-proposal pairs can serve as candidates of the next frame to detect new objects, allowing oursystem to achieve new object detection and tracking effortlessly.
Third, we propose an intra-queryattention module that enhances instance query with its predecessors encoding the same object.
Thisexplicitly aggregates long-range object information into the query, augmenting its representationcapacity, which help handle occlusion and motion blur.
To validate the effectiveness and efficiency of our InsPro, we conduct extensive experiments on twopopular VIS benchmarks [1], i.e. , YouTube-VIS 2019 and YouTube-VIS 2021.
Without bells-andwhistles, our InsPro with ResNet-50 [13] backbone achieves 43.2 AP on YouTube-VIS 2019 and 37.
AP on YouTube-VIS 2021 respectively, outperforming all other online VIS models.
Moreover, ourlite variant, InsPro-lite, reaches 38.7 AP at impressive 45.7 FPS on YouTube-VIS 2019 on a Nvidia
RTX2080Ti GPU.
In summary, we make the following contributions in this paper.
We propose a simple, fast andyet effective query-based framework for online VIS.
We develop several techniques to make thequery-proposal pair propagation mechanism work smoothly.
These techniques distinguish our workfrom other query propagation-based object association methods [14, 15, 16], and make our worksimpler, more elegant and more effective than them.
Our VIS system achieves the state-of-the-artperformances on two popular VIS benchmarks.
Related Work
Frame-level VIS Methods mainly adopt a ‘tracking-by-detection’ paradigm and can run in anonline fashion.
They first generate instance predictions frame by frame and then perform explicitinstance association.
MaskTrack R-CNN [1] proposes the VIS task for the first time and simply addsan additional tracking head to Mask R-CNN [5] for instance association.
Follow-up works [2, 3, 4]improve either the image instance segmentation model or the tracking algorithm to achieve betterperformance.
On the other hand, some works [8, 17, 18, 19, 20, 21] attempt to perform temporalfeature fusion to improve instance segmentation and association.
For example, PCAN [21] proposesframe- and instance-level prototypical cross-attention modules to leverage rich spatio-temporalinformation to facilitate better segmentation.
All these methods require additional modules to achieveexplicit instance association, which expands model complexity and reduces inference speed.
Bycontrast, our method performs the instance association implicitly through an instance query andproposal propagation mechanism, which is simpler and naturally exploits the temporal and spatialconsistency in videos.
Clip-level VIS Methods take a ‘clip-matching’ paradigm, which process multiple frames withina clip simultaneously and then perform instance matching between clips to complete VIS.
Whilesome methods [7, 22, 9] propagate instance information within a clip with well-designed propagationmodules to model temporal context, recent works [8, 10] utilize transformer [23] to model temporalcontext in an end-to-end manner.
These methods normally need hand-crafted matching algorithms tocomplete instance association between clips.
Although they usually achieve high performance, theycan only run in an offline mode, which restricts their application to limited areas.
In contrast, ourmethod achieves comparable performance but can run online.
Query-based Methods have attracted increasing attention in recent years due to their flexibilityand simplicity.
DETR [11] first uses a set of learned queries interacting with image features toencode objects, and then directly outputs detections by decoding the transformed queries.
Followingworks [24, 25, 26, 27, 28] improve DETR in terms of either training efficiency or detection performance.
Sparse R-CNN [12] builds a query-based detector on top of R-CNN architecture [29, 30]. The success of DETR has also inspired query-based VIS methods. VisTR [8] adapts DETR to the VIS task.
It takes a video clip as input and directly outputs the sequence of masks for each instanceorderly.
IFC [10] proposes inter-frame communication transformers to reduce the heavy computationand memory usage of VisTR-like VIS methods.
Similar to VisTR, Mask2Former [31] applies maskedattention to a video clip and directly predicts a 3D instance volume. To learn a powerful video-levelinstance query, SeqFormer [32] aggregates temporal information from each frame to the instancequery. These methods work on clips rather than frames, and achieve object association throughsharing of the queries within a clip rather than query propagation. Thus, they still need instancematching between clips.
Instead, our method applies to frames, and can propagate query-proposalpairs through the entire video and thereby can associate object instances over any frame length. Query Propagation-based Object Association Methods inspired by query-based methods [11, 12]too, have been recently explored in several works, such as TransTrack [14], TrackFormer [15], MOTR [16] and EfficientVIS [33]. This shows the effectiveness and potential of such a new objectlinking approach.
The differences between our work and them are as follows. First, our InsPro is different from them in the way of either tracking seen objects or detecting newobjects. TransTrack is basically a ‘tracking-by-detection’ method, because it still needs to explicitlymatch detection boxes to tracking [Figure]
Temporal propagation [Figure] The former transforms instance queries with RoI features of corresponding proposals and produces object features, while the latter predicts object instances based on the object features and conditional convolution [35]. More importantly, TransTrack [14], TrackFormer [15] and MOTR [16] adopt a track query subset totrack seen objects and an object query subset to detect new objects.
This requires additional heuristicrules to combine two type queries, and may miss occluded or blurred objects with low scores whichresults in object trajectory break [34]. Our InsPro simply propagates all object queries produced inthe previous frame to the current frame, and keeps using this set to track seen objects and detectnew objects, which is much simpler and more elegant. As for EfficientVIS, our concurrent work, itdoes not consider this new object detection problem, and its performance will probably be greatlyimpacted if there are new objects in the next clip. Furthermore, we design a more intelligent strategy to suppress duplicates.
TransTrack and TrackFormer employ score filtering or NMS to reduce duplicate predictions. MOTR builds a temporalaggregation network to learn more discriminative features to address this problem, while EfficientVISdoes not discuss this problem. By contrast, we design a Box Deduplication Loss to suppress duplicates and an Inter-query attention module to enhance queries with their predecessors. Our solutionavoids heuristic rules and post-processing steps, and is more effective according to the experimentalresults (see Table 2 (e)).
InsPro
We aim to design a simple and fast VIS system that performs instance association implicitly andexploits video temporal clues elegantly.
To this end, we take a query-based VIS approach that predictsobject instances on each frame based on a set of instance query-proposal pairs propagated fromprevious frames.
In this section, we introduce our VIS system, InsPro, including an instance queryand proposal propagation mechanism and those proposed techniques that make the propagationmechanism work well.
3.1 Instance Query and Proposal Propagation
The instance query and proposal propagation mechanism enables our VIS system to perform objectinstance association implicitly in parallel with instance segmentation.
Since it is inspired by therecent query-based object detector Sparse R-CNN [12], we first review Sparse R-CNN.
Sparse R-CNN [12] formulates object detection as a set prediction problem and achieves state-of-theart performance.
It simplifies the detection pipeline and removes heuristic components like NMS.
Specifically, it first initializes a fixed set of learnable instance queries ( N × C , N denotes the numberof queries and C the query dimension) paired with learnable instance proposals ( N × 4 ) to describe objects in an image.
As illustrated in Figure 2 (b), each instance query is convolved with the RoIfeature of the corresponding proposal to output a more discriminate feature o t [12].
After multi-stageiterative updating, the instance query encodes more accurate object appearance information while theproposal captures more precise location information.
Finally, decoding the object feature o t producedbased on the instance query-proposal pairs, we get the detection results.
Inspired by this instance query-proposal representation of an object, we design a query-proposaltemporal propagation mechanism (as shown in Figure 2 (a)) to achieve implicit object instance association and temporal cue utilization in VIS.
Our key insight is that there is a one-to-one correspondencebetween the learned instance query-proposal pair and a specific object.
If we manage to preserve thiscorrespondence from the first frame to the one where the object disappears, then we realize objecttracking and object information propagation spontaneously.
To this end, we first initialize a set of instance queries q init ∈ R N × C and proposals p init
∈
R N × 4 onthe first video frame
I 0 , where q init and p init are learnable parameters and arranged in pairs.
Afterlearning, they are able to encode objects on the first frame.
Decoding them with the first frame imagefeature inside the SegHead, we obtain instance results r 0 as well as a new set of updated pairs ( q 0 ,p 0 ).
Then we propagate this pair set ( q 0 , p 0 ) to the next frame as input to the SegHead.
Similarly,we get the instance results r 1 and another new set of ( q 1 , p 1 ) on this frame.
Among them, the objectinstance produced on this frame shares the same ID with the one on the previous frame if they areboth decoded by the same slice of the instance queries .
In this way, we automatically link objectinstances belonging to an identical object across frames and elegantly make use of object priors fromthe past.
Repeating the above process until the last video frame, we then accomplish the VIS task onthis video.
The details of SegHead can be found in the supplementary material.
Please note that our InsPro simply propagates all object queries produced in the previous frame tothe current frame, and keeps using this set to track seen objects and detect new objects.
Instead,recent works [14, 15, 16] that take a similar query-propagation mechanism to use a track query setto track seen objects and a new object query set to detect new objects respectively, which requiresadditional heuristic rules to combine these two type queries.
Moreover, they rely on hand-crafted ruleslike a score threshold to select a subset of track queries, and occluded objects with low predictionscores are probably filtered out, which brings non-negligible true object missing and fragmentedtrajectories [34].
In comparison, our method is obviously simpler, more elegant and more effective(see Table 2 (e)).
Intra-query Attention Since frame-by-frame temporal propagation encodes only short-rangetemporal cues, the instance query from just the last frame shows limitations in dealing with toughscenarios, e.g. , occlusion and motion blur.
To boost the representation capacity of instance query,we augment it in practice with instance features from previous T frames.
Specifically, we build afeature bank that caches instance features from previous T frames and perform intra-query attentioninside this bank to aggregate long-range temporal cues into the current instance query, as shown inthe upper part of Figure 2 (b).
Formally, at frame I t , instance features o from previous T frames areput together to form a feature bank fb =
{ o t − T +1 , . . .
, o t } .
Then, the enhanced instance query iscomputed as:
[Equation]
where i denotes the i -th query and ε is a linear transformation function.
The enhanced q t is basicallya weighted sum of instance features inside the feature bank, and the weights are learned upon thequality of the queries.
Experiments (Table 2 (c)) show that this augmentation improves the queryrepresentation capacity greatly.
3.2 Temporally Consistent Matching
The key to the success of our InsPro is to make sure that the evolving instance query-proposal paircorresponds to the same object across frames in a video.
To ensure this, one technique we propose istemporally consistent matching.
This technique matches predictions and ground truth during training,assigns each ground truth object a proper prediction, and propagates the matching made on previousframes to subsequent frames.
[Figure]
Specifically, given a training batch consisting of multiple consecutive frames, we first compute thematching cost L match between predictions and ground truth objects on the first frame:
[Equation]
where L cls is the focal loss [36] between predicted classifications and ground-truth labels, L L 1and L giou are L1 loss and the generalized IoU loss [37] between predicted boxes and ground-truthboxes, respectively.
λ cls , λ L 1 and λ giou are loss weights and set as 2, 5, and 2, respectively.
Wesearch for the best bipartite matching that minimizes the matching cost L match with the Hungarianalgorithm [38].
After finding the best matching on the first frame, we propagate this matching to otherframes.
Concretely, if one ground truth object still exists on subsequent frames, it will be matched tothe prediction that is generated by the same instance query on the first frame.
If there are new groundtruth objects emerging, new matching will be made between the new objects and yet unmatchedpredictions.
If a ground truth object disappears, its corresponding predictions will not participate in anew matching process.
Through this temporally consistent matching mechanism, we bind one groundtruth object to a single instance query during training.
3.3 Loss Function
Box Deduplication Loss
Although the self-attention mechanism between queries has driven themodel to generate fewer duplication predictions [11], we still observe multiple overlapped proposalboxes on the same object across many frames, as displayed in Figure 3 (a).
We conjecture this iscaused by those unmatched queries that cannot be pushed away from those matched queries due tolack of supervision.
To address this problem, we propose a box deduplication loss to push awayprediction boxes in terms of the center-to-center distance between them.
As a result, not only theduplicate problem is alleviated, but the sparsely distributed unmatched query-proposal pairs can serveas candidates of the next frame to detect and track new objects (see Figure 5 in the supplementary).
The loss is defined as:
[Equation]
where b i is a ground truth box, ˆ b ineg is a negative box that has the largest IoU with b i among thoseunmatched predicted boxes, c ( · ) is the center distance, and d ( · ) is the diagonal length.
β is set as 0.1.
This loss penalizes the short distance between b i and ˆ b ineg , and drags all other duplicate boxes awayfrom b i [39].
With this new loss, our final box loss function is formed as:
[Equation]
where λ L 1 and λ giou have the same values as in Equation 2, and λ dedup is set as 1.
Overall Loss Given the one-to-one matching results, the final loss on each training frame is a sumof classification, box and mask losses:
[Equation]
where L dice and L focal are dice loss [40] and focal loss [36] for foreground mask prediction,respectively.
We set λ box = 1 , λ dice = 5 and λ focal = 5 .
Finally, the losses of all training framesinside a batch are summed together and normalized by the number of frames.
Experiments
4.1 Datasets and Evaluation Metrics
We evaluate our method on YouTube-VIS 2019 and 2021 benchmarks [1].
YouTube-VIS 2019consists of 2,238 training videos and 302 validation videos, and labels 40 object categories.
YouTubeVIS 2021 is an extended version, which comprises 2,985 training videos and 421 validation videos,and labels improved 40 categories.
All videos in these two datasets are annotated every 5 frames withobject bounding box, object category, instance mask and instance ID.
Following [1], we report thevideo-level average precision (AP) and average recall (AR) on the validation sets as the evaluationmetrics, where both accurate instance segmentation and instance association are necessary to achievehigh performance.
4.2 Implementation Details
We implement our InsPro with Detectron2 [41], and most hyperparameters are set following Sparse
R-CNN [12] and CondInst [35] unless otherwise specified.
More implementation details can be foundin the supplementary material.
Training Details We employ AdamW [42] with an initial learning rate of 2 .
5 × 10 − 5 and weightdecay 0.0001 as our model optimizer.
We initialize our model with parameters pre-trained on
COCO [43], and train it for 32k iterations where the learning rate is divided by 10 at iterations 28kand 24k, respectively.
The training is performed end-to-end on 8 Nvidia RTX2080Ti GPUs and each
GPU holds one mini-batch which contains three frame images randomly sampled from the samevideo.
Data augmentation includes only random horizontal flip and multi-scale training where thetraining image is resized with the shortest side being at least 288 and at most 512.
Unless otherwisenoted, our InsPro adopts 100 instance queries and ResNet-50 [13] as backbone in our experiments.
Inference Details In inference, we resize the frame image size to 640 × 360 , which follows
MaskTrack R-CNN [1].
The length of the feature bank is set to 18 by default.
If the generatedproposal box size exceeds the frame’s, it will be reset to the frame size.
No multi-scale testing isadopted in our experiments.
InsPro-lite we also build a lite version of our method, named InsPro-lite.
In this variant, inspiredby [44], we divide video frames into key frames and non-key frames, i.e. , we select one key frameper K frames in a video and treat other frames as non-key ones.
K is 10 by default.
On key frames,we conduct the dynamic instance interaction 6 times while only once on non-key frames.
This takesadvantage of the redundancy of videos and helps reduce inference computation time.
Our InsPro-litereaches a high inference speed of 45.7 FPS at a small accuracy loss (Table 1).
4.3 Main Results
We perform a thorough comparison of our InsPro to state-of-the-art VIS methods on YouTube-VIS2019 and 2021.
Existing VIS methods can be divided into two categories according to whether theyrun online or offline [45].
Since some methods [7, 9] also use 80k transformed COCO images [43] asextra training data to prevent overfitting to YouTube-VIS, for a fair comparison, we also report ourresults with and without extra COCO training data.
Table 1 presents all the results obtained with a
ResNet-50 backbone on a Nvidia RTX2080Ti GPU.
YouTube-VIS 2019 Table 1 left shows the comparison between our InsPro and other state-of-the-artmethods on YouTube-VIS 2019 validation set.
We can see that, in the online group, our InsProoutperforms all other popular methods under the same data setting.
Specifically, our InsPro achieves40.2 AP without COCO data and 43.2 AP with COCO data respectively, surpassing other online VISmethods by a large margin.
Even our lite version, InsPro-lite, performs better than all other onlinemethods trained without COCO data, reaching 38.7 AP at an impressive speed of 45.7 FPS.
YouTube-VIS 2021 Table 1 right displays results on YouTube-VIS 2021 validation set.
It showsa similar comparison pattern to YouTube-VIS 2019 and our InsPro achieves the state-of-the-artperformance once again.
[Table]
[Table]
[Table]
4.4 Ablation Study
We conduct extensive experiments on YouTube-2019 to study the effectiveness and individualperformance contribution of our proposed modules.
Temporal Propagation and Matching Our InsPro is built on the proposed mechanism of instancequery and proposal temporal propagation.
Table 2 (a) shows how this mechanism contributes toour high performance.
In this table, method A represents the video instance segmentation baseline,where each frame is processed individually without any temporal propagation, and object instancesgenerated on each frame are linked if they are produced from the same instance query slice.
Sincethis method lacks the mechanism to ensure the instance query-proposal pair corresponds to thesame object across frames, it only achieves 24.0 AP due to inaccurate instance association.
Bycontrast, when we add the temporal propagation (method B), we can easily improve the performancesignificantly to 36.3 AP.
This evidences the importance and effectiveness of the proposed temporalpropagation technique in a query-based VIS framework.
If we further adopts the temporally consistentmatching strategy during training (method C), we achieve an even better performance of 37.4 AP.
We also analyze the separate performance of propagating only instance query (method D) or instanceproposal (method E).
The results show that these two settings achieve a similar performance boost(36.7 AP vs 36.6 AP).
Applying them together yields 37.4 AP, bringing further performance gain.
Box Deduplication Loss
We propose a box deduplication loss to suppress the duplicate proposalboxes on the same object across frames.
As the qualitative results can be found in Figure 3, we showthe quantitative comparison in Table 2 (b).
We can see that supervising the learning with this lossduring training can lead 1.0 AP improvement (38.4 AP vs 37.4 AP).
This performance gain is broughtby less false positives.
Intra-query Attention We perform intra-query attention inside a feature bank to augment theinstance query so that it can capture long-range temporal cues.
As we can see in Table 2 (c), thissimple method works well and improves the performance considerably.
In particular, T = 1 indicatesno intra-query attention is used and 38.4 AP is achieved.
When we increase the volume T of thefeature bank, the performance rises and saturates at 40.2 AP with T = 18 .
It is worthwhile to notethat this lightweight yet effective intra-query attention operation brings almost no speed drop.
Track-by-Detect vs. Temporal Propagation Despite the fact that our InsPro does not performexplicit instance association, it still outperforms all other online methods implementing explicittracking or matching.
To verify that our superior performance comes from the temporal propagationmechanism rather than the image instance segmentation model design, we compare our temporalpropagation VIS approach to the typical ‘track-by-detect’ paradigm with the same instance segmentation baseline.
We implement a ‘track-by-detect’ VIS system by replacing the Mask R-CNN part in
MaskTrack R-CNN [1] with our instance segmentation model.
In this case, the only independentvariable is the object tracking method.
As shown in Table 2 (d), our InsPro surpasses the ‘track-by-detect’ model by a large margin even if ourdesign is simpler and faster, which soundly proves the effectiveness of our method.
We argue againthat this is because the evolving instance query-proposal pair in propagation encodes object temporaland spatial cues intrinsic in videos, whereas ‘track-by-detect’ methods are generally incapable ofexploiting this advantage.
Track-and-Detect query vs. Ours We further compare our method to those MOT methods thatadopt a similar query-propagation method for object tracking.
These methods rely on two differentquery sets, i.e. , a track query set and an object query set, to track seen objects and detect new objectsrespectively.
They need heuristic rules to combine these two type queries.
Meanwhile, they manuallyselect track queries with high scores from the previous frame to build the track query set.
This makesthem complex and less effective in tracking since occluded objects with low scores probably havebroken trajectories because of the filtering. To show the superiority of our method, we compare the ‘track-and-detect query’ paradigm adoptedin the most recent MOTR [16] to ours using the same instance segmentation baseline. We follow MOTR [16] exactly to set up the model and experiment settings.
To exclude the influence ofother factors, we do not use temporal feature aggregation in both methods.
Table 2 (e) shows thecomparisons on YouTube-VIS 2019.
It can be seen that our InsPro achieves a higher performance evenusing a simpler query propagation method.
We attribute this advantage to our those conscientiouslydesigned modules described in Sec 3.
Conclusion
In this paper, we propose a simple, fast and yet effective query-based framework for online VIS.
Inthis framework, we rely on a novel instance query and proposal propagation mechanism to undertake
VIS, where we generate object instances based on a set of evolving instance query-proposal pairspropagated from previous frames.
This mechanism enables our model not only to associate objectinstances implicitly, but to utilize video temporal cues elegantly.
To make this propagation mechanismwork well, we develop several modules to ensure that the learned instance query-proposal pairkeeps being bound to one object, These modules include an intra-query attention unit, a temporallyconsistent matching mechanism and a box deduplication loss.
Extensive experiments on YouTubeVIS 2019 and 2021 verify the effectiveness of our designs, and show that our InsPro achieves superior
VIS performance, outperforming all other online VIS methods.