Original: MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training
Revised: MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We propose MinVIS, a minimal video instance segmentation (VIS) framework thatachieves state-of-the-art VIS performance with neither video-based architecturesnor training procedures.
Revised: We propose MinVIS, a minimal video instance segmentation (VIS) framework thatachieves state-of-the-art VIS performance with neither video-based architecturesnor training procedures.

Original: By only training a query-based image instance segmentation model, MinVIS outperforms the previous best result on the challenging
Revised: By only training a query-based image instance segmentation model, MinVIS outperforms the previous best result on the challenging

Original: Occluded VIS dataset by over 10% AP.
Revised: Occluded VIS dataset by over 10% AP.

Original: Since MinVIS treats frames in trainingvideos as independent images, we can drastically sub-sample the annotated framesin training videos without any modifications.
Revised: Since MinVIS treats frames in trainingvideos as independent images, we can drastically sub-sample the annotated framesin training videos without any modifications.

Original: With only 1% of labeled frames, MinVIS outperforms or is comparable to fully-supervised state-of-the-art approacheson YouTube-VIS 2019/2021.
Revised: With only 1% of labeled frames, MinVIS outperforms or is comparable to fully-supervised state-of-the-art approacheson YouTube-VIS 2019/2021.

Original: Our key observation is that queries trained to be discriminative between intra-frame object instances are temporally consistent and canbe used to track instances without any manually designed heuristics.
Revised: Our key observation is that queries trained to be discriminative between intra-frame object instances are temporally consistent and canbe used to track instances without any manually designed heuristics.

Original: MinVIS thushas the following inference pipeline: we first apply the trained query-based imageinstance segmentation to video frames independently.
Revised: MinVIS thushas the following inference pipeline: we first apply the trained query-based imageinstance segmentation to video frames independently.

Original: The segmented instances arethen tracked by bipartite matching of the corresponding queries.
Revised: The segmented instances arethen tracked by bipartite matching of the corresponding queries.

Original: This inference isdone in an online fashion and does not need to process the whole video at once.
Revised: This inference isdone in an online fashion and does not need to process the whole video at once.

Original: MinVIS thus has the practical advantages of reducing both the labeling costs andthe memory requirements, while not sacrificing the VIS performance.
Revised: MinVIS thus has the practical advantages of reducing both the labeling costs andthe memory requirements, while not sacrificing the VIS performance.

Original:  Introduction
Revised:  Introduction

Original: Video instance segmentation (VIS) aims to simultaneously detect, segment, and track object instancesin videos [1].
Revised: Video instance segmentation (VIS) aims to simultaneously detect, segment, and track object instancesin videos [1].

Original: The requirement to accurately track object instances through an entire video makes
Revised: The requirement to accurately track object instances through an entire video makes

Original: VIS much more challenging than image instance segmentation.
Revised: VIS much more challenging than image instance segmentation.

Original: Most of the early approaches for VISbuild on image instance segmentation models and process videos on a per-frame basis [1, 2].
Revised: Most of the early approaches for VISbuild on image instance segmentation models and process videos on a per-frame basis [1, 2].

Original: Thesegmented object instances for each frame are then matched temporally with a post-processing step.
Revised: Thesegmented object instances for each frame are then matched temporally with a post-processing step.

Original: This post-processing step often involves manually designed heuristics that do not generalize well tochallenging scenarios like occlusions and large appearance deformations.
Revised: This post-processing step often involves manually designed heuristics that do not generalize well tochallenging scenarios like occlusions and large appearance deformations.

Original: Recent VIS works address this issue by taking a per-clip approach, where the spatial-temporalvolume of a video is processed as a whole to directly predict the spatial-temporal mask for eachobject instance [3–5].
Revised: Recent VIS works address this issue by taking a per-clip approach, where the spatial-temporalvolume of a video is processed as a whole to directly predict the spatial-temporal mask for eachobject instance [3–5].

Original: Many of these end-to-end VIS approaches are built upon the recent advancesof Transformers for end-to-end object detection [6].
Revised: Many of these end-to-end VIS approaches are built upon the recent advancesof Transformers for end-to-end object detection [6].

Original: Given learned embeddings called queries ,
Revised: Given learned embeddings called queries ,

Original: Transformers process the queries jointly with the input video using cross-attention, so that each of theprocessed queries can be used to predict the spatial-temporal mask for an object instance in the video.
Revised: Transformers process the queries jointly with the input video using cross-attention, so that each of theprocessed queries can be used to predict the spatial-temporal mask for an object instance in the video.

Original: While these per-clip methods have led to considerable improvements for VIS, using attention toprocess the whole video, especially longer ones, requires large memory and computation.
Revised: While these per-clip methods have led to considerable improvements for VIS, using attention toprocess the whole video, especially longer ones, requires large memory and computation.

Original: It isalso not straightforward to adapt per-clip methods from offline to online processing to reduce thecomputational requirements.
Revised: It isalso not straightforward to adapt per-clip methods from offline to online processing to reduce thecomputational requirements.

Original: This limits their practical application, and maintaining the effectiveness of these per-clip methods while improving their efficiency remains an active research direction [7, 8]
Revised: This limits their practical application, and maintaining the effectiveness of these per-clip methods while improving their efficiency remains an active research direction [7, 8]

Original: [Figure]
Revised: [Figure]

Original: Another limitation for existing VIS methods is the requirement on annotation.
Revised: Another limitation for existing VIS methods is the requirement on annotation.

Original: Annotating objectinstance masks for each video frame is prohibitively expensive at scale.
Revised: Annotating objectinstance masks for each video frame is prohibitively expensive at scale.

Original: While there have been worksthat alleviate this annotation requirement through weak supervision or image-based annotation, thereis still a significant performance gap compared to state-of-the-art fully-supervised methods [9, 10].
Revised: While there have been worksthat alleviate this annotation requirement through weak supervision or image-based annotation, thereis still a significant performance gap compared to state-of-the-art fully-supervised methods [9, 10].

Original: Our Approach.
Revised: Our Approach.

Original: We simultaneously address both of the aforementioned challenges of computationaland labeling costs by showing that we can achieve state-of-the-art VIS performance by only traininga query-based image instance segmentation model.
Revised: We simultaneously address both of the aforementioned challenges of computationaland labeling costs by showing that we can achieve state-of-the-art VIS performance by only traininga query-based image instance segmentation model.

Original: During inference, MinVIS first applies thequery-based image instance segmentation to video frames independently.
Revised: During inference, MinVIS first applies thequery-based image instance segmentation to video frames independently.

Original: The segmented instancesare then associated by bipartite matching of the corresponding queries.
Revised: The segmented instancesare then associated by bipartite matching of the corresponding queries.

Original: MinVIS processes each frameindependently in an online fashion and does not need to process the whole video at once.
Revised: MinVIS processes each frameindependently in an online fashion and does not need to process the whole video at once.

Original: MinVISdoes not use any video-based training procedure and thus does not need annotations for all the framesin a video.
Revised: MinVISdoes not use any video-based training procedure and thus does not need annotations for all the framesin a video.

Original: Our contributions are summarized below:
Revised: Our contributions are summarized below:

Original: We show that video-based architecture and training are not required for competitive VIS perfor-
Revised: We show that video-based architecture and training are not required for competitive VIS perfor-

Original: mances.
Revised: mances.

Original: MinVIS outperforms previous state-of-the-art on YouTube-VIS 2019 and 2021 datasets
Revised: MinVIS outperforms previous state-of-the-art on YouTube-VIS 2019 and 2021 datasets

Original: by 1% and 3% AP while only training an image instance segmentation model.
Revised: by 1% and 3% AP while only training an image instance segmentation model.

Original: We show that image instance segmentation models capable of segmenting occluded instances arealso well suited to track occluded instances in videos in our framework.
Revised: We show that image instance segmentation models capable of segmenting occluded instances arealso well suited to track occluded instances in videos in our framework.

Original: MinVIS outperforms itsper-clip counterpart by over 13% AP on the challenging Occluded VIS (OVIS) dataset, which is
Revised: MinVIS outperforms itsper-clip counterpart by over 13% AP on the challenging Occluded VIS (OVIS) dataset, which is

Original: over 10% improvement compared to the previous best performance on the dataset.
Revised: over 10% improvement compared to the previous best performance on the dataset.

Original: Our image-based approach allows us to significantly sub-sample the required segmentation
Revised: Our image-based approach allows us to significantly sub-sample the required segmentation

Original: annotations in training without any change to the model.
Revised: annotations in training without any change to the model.

Original: With only 1% of labeled frames, MinVIS
Revised: With only 1% of labeled frames, MinVIS

Original: outperforms or is comparable to fully-supervised state-of-the-art approaches on all three datasets.
Revised: outperforms or is comparable to fully-supervised state-of-the-art approaches on all three datasets.

Original: Our key observation is that queries trained to be discriminative between intra-frame object instancesare temporally consistent and can be used to track instances without being trained with video-basedloss functions.
Revised: Our key observation is that queries trained to be discriminative between intra-frame object instancesare temporally consistent and can be used to track instances without being trained with video-basedloss functions.

Original: MinVIS achieves this by requiring its image instance segmentation model to generatemasks by convolving query embeddings with features of the whole input image, including regionsof other object instances.
Revised: MinVIS achieves this by requiring its image instance segmentation model to generatemasks by convolving query embeddings with features of the whole input image, including regionsof other object instances.

Original: A query is thus trained to only have high responses on features of itscorresponding instance.
Revised: A query is thus trained to only have high responses on features of itscorresponding instance.

Original: Other query embeddings should instead have low responses on these featuresbecause instance masks are non-overlapping.
Revised: Other query embeddings should instead have low responses on these featuresbecause instance masks are non-overlapping.

Original: This design encourages the query embeddings fordifferent instances in a frame to be well-separated.
Revised: This design encourages the query embeddings fordifferent instances in a frame to be well-separated.

Original: On the other hand, the query embeddings thatsegment the same instance from two consecutive frames still need to be similar enough since theinstance’s image features to be convoluted do not change drastically between frames.
Revised: On the other hand, the query embeddings thatsegment the same instance from two consecutive frames still need to be similar enough since theinstance’s image features to be convoluted do not change drastically between frames.

Original: This leads totemporally consistent query embeddings for tracking without the need of video-based training.
Revised: This leads totemporally consistent query embeddings for tracking without the need of video-based training.

Original: MinVIS thus has the following design for inference: We first apply a query-based image instancesegmentation model on video frames independently.
Revised: MinVIS thus has the following design for inference: We first apply a query-based image instancesegmentation model on video frames independently.

Original: The segmented instances are then associatedbetween frames by bipartite matching of the corresponding query embeddings.
Revised: The segmented instances are then associatedbetween frames by bipartite matching of the corresponding query embeddings.

Original: This post-processingstep
Revised: This post-processingstep

Original: Since video frames are treated as independent images to train MinVIS, there is no requirementto annotate all the frames in a video for training.
Revised: Since video frames are treated as independent images to train MinVIS, there is no requirementto annotate all the frames in a video for training.

Original: This allows us to significantly sub-sample andreduce the annotation without any change to our model.
Revised: This allows us to significantly sub-sample andreduce the annotation without any change to our model.

Original: We find that on YouTube-VIS 2019/2021datasets [1], where there are less variations between video frames, using only 1% of labeled framesleads to less than 3% drop in AP for MinVIS.
Revised: We find that on YouTube-VIS 2019/2021datasets [1], where there are less variations between video frames, using only 1% of labeled framesleads to less than 3% drop in AP for MinVIS.

Original: We further evaluate MinVIS on the Occluded VIS (OVIS) dataset [12].
Revised: We further evaluate MinVIS on the Occluded VIS (OVIS) dataset [12].

Original: One common critique ofper-frame approaches is that their tracking heuristics based on mask overlaps would not work whenthere are heavy occlusions.
Revised: One common critique ofper-frame approaches is that their tracking heuristics based on mask overlaps would not work whenthere are heavy occlusions.

Original: This is not the case for MinVIS, as we do not use any manually designedheuristics.
Revised: This is not the case for MinVIS, as we do not use any manually designedheuristics.

Original: We show that our query-matching approach generalizes to occluded scenarios.
Revised: We show that our query-matching approach generalizes to occluded scenarios.

Original: MinVISwith
Revised: MinVISwith

Original: Swin Transformers backbone [13] achieves 39.4% AP on OVIS, which is over 10% improvementfrom the previous best result on the dataset [14].
Revised: Swin Transformers backbone [13] achieves 39.4% AP on OVIS, which is over 10% improvementfrom the previous best result on the dataset [14].

Original: We further show that our image-based strategy leadto easier and better learning on OVIS.
Revised: We further show that our image-based strategy leadsto easier and better learning on OVIS.

Original: MinVIS outperforms its per-clip counterpart by over 13% AP.
Revised: MinVIS outperforms its per-clip counterpart by over 13% AP.

Original:  Related Work
Revised:  Related Work

Original: Video Instance Segmentation.
Revised: Video Instance Segmentation.

Original: Per-frame approaches for VIS process each frame independentlyand later track instances by post-processing.
Revised: Per-frame approaches for VIS process each frame independentlyand later track instances by post-processing.

Original: MaskTrack R-CNN [1] adds a tracking head to Mask
Revised: MaskTrack R-CNN [1] adds a tracking head to Mask

Original: R-CNN [15] for VIS.
Revised: R-CNN [15] for VIS.

Original: MaskProp [16] instead adds a mask propagation head to propagate objectinstance masks.
Revised: MaskProp [16] instead adds a mask propagation head to propagate objectinstance masks.

Original: CrossVIS [2] uses crossover learning to improve instance representation across videoframes.
Revised: CrossVIS [2] uses crossover learning to improve instance representation across videoframes.

Original: QueryTrack [11] adds a contrastive tracking head to QueryInst [17] for instance association.
Revised: QueryTrack [11] adds a contrastive tracking head to QueryInst [17] for instance association.

Original: Our approach also builds on image instance segmentation models, but unlike previous approaches,we need neither additional parameters nor additional losses to apply to VIS.
Revised: Our approach also builds on image instance segmentation models, but unlike previous approaches,we need neither additional parameters nor additional losses to apply to VIS.

Original: Our query embeddingsfrom image instance segmentation can directly be used for tracking without video-based training.
Revised: Our query embeddingsfrom image instance segmentation can directly be used for tracking without video-based training.

Original: Recent per-clip approaches build on the success of Detection Transformer (DETR) [6].
Revised: Recent per-clip approaches build on the success of Detection Transformer (DETR) [6].

Original: VisTR [4]adopts the query-based approach of DETR to VIS, and there has been several follow-up works, suchas Mask2Former-VIS [3] and SeqFormer [5].
Revised: VisTR [4]adopts the query-based approach of DETR to VIS, and there has been several follow-up works, suchas Mask2Former-VIS [3] and SeqFormer [5].

Original: One limitation of these approaches is the need toprocess the whole video at once.
Revised: One limitation of these approaches is the need toprocess the whole video at once.

Original: IFC [7] reduces the overhead of temporal message passing byusing memory tokens.
Revised: IFC [7] reduces the overhead of temporal message passing byusing memory tokens.

Original: TeViT [8] uses a parameter-shared self attention to efficiently model temporalcontexts.
Revised: TeViT [8] uses a parameter-shared self attention to efficiently model temporalcontexts.

Original: We also use a query-based approach, but instead of using cross-attention to process thewhole video, we process each frame independently while not losing VIS performance.
Revised: We also use a query-based approach, but instead of using cross-attention to process thewhole video, we process each frame independently while not losing VIS performance.

Original: Reducing Supervision for Video Instance Segmentation.
Revised: Reducing Supervision for Video Instance Segmentation.

Original: Annotating instance masks for each videoframe can be prohibitively expensive.
Revised: Annotating instance masks for each videoframe can be prohibitively expensive.

Original: Compared to video object segmentation [18–20] and imageinstance segmentation [21–24], there have been less works on reducing supervision for VIS [25].
Revised: Compared to video object segmentation [18–20] and imageinstance segmentation [21–24], there have been less works on reducing supervision for VIS [25].

Original: FlowIRN [10] extends IRN [21] with motion and temporal consistency cues to have a weaklysupervised VIS framework that only requires classification labels.
Revised: FlowIRN [10] extends IRN [21] with motion and temporal consistency cues to have a weaklysupervised VIS framework that only requires classification labels.

Original: SOLO-Track [9] learns to trackinstances without video annotations.
Revised: SOLO-Track [9] learns to trackinstances without video annotations.

Original: It uses instance contrastive learning on SOLO [26] to learn gridcell embeddings for instance tracking.
Revised: It uses instance contrastive learning on SOLO [26] to learn gridcell embeddings for instance tracking.

Original: We make the same observation that disciminating betweeninstances within frames is beneficial or even sufficient for instance tracking.
Revised: We make the same observation that disciminating betweeninstances within frames is beneficial or even sufficient for instance tracking.

Original: However, unlike ourquery-based association, the grid cell embeddings still need threshold-based post-processing andadditional loss functions to better handle birth and death of objects.
Revised: However, unlike ourquery-based association, the grid cell embeddings still need threshold-based post-processing andadditional loss functions to better handle birth and death of objects.

Original:  Method
Revised:  Method

Original: MinVIS is a minimal VIS framework that does not require video-based training and thus can be easilyapplied to real-world applications that only have sparse image instance segmentation annotations.
Revised: MinVIS is a minimal VIS framework that does not require video-based training and thus can be easilyapplied to real-world applications that only have sparse image instance segmentation annotations.

Original: MinVIS is a two stage approach: (1) image instance segmentation on each frame independently,(2) associating instances between frames by matching queries.
Revised: MinVIS is a two stage approach: (1) image instance segmentation on each frame independently,(2) associating instances between frames by matching queries.

Original: We will first discuss the imageinstance segmentation architecture in MinVIS.
Revised: We will first discuss the imageinstance segmentation architecture in MinVIS.

Original: We will then discuss the temporal association of objectinstances.
Revised: We will then discuss the temporal association of objectinstances.

Original: Finally, we will discuss training and reducing supervision for MinVIS.
Revised: Finally, we will discuss training and reducing supervision for MinVIS.

Original: 3.1 Image Instance Segmentation Architecture for VIS
Revised: 3.1 Image Instance Segmentation Architecture for VIS

Original: MinVIS builds on the query-based transformer architectures for detection and segmentation [6, 17, 27,28], which has the following main components: (1) Image Encoder that learn to extract features frominput images.
Revised: MinVIS builds on the query-based transformer architectures for detection and segmentation [6, 17, 27,28], which has the following main components: (1) Image Encoder that learn to extract features frominput images.

Original: (2) Transformer Decoder that processes the outputs of Image Encoder to iterativelyupdate the query embeddings.
Revised: (2) Transformer Decoder that processes the outputs of Image Encoder to iterativelyupdate the query embeddings.

Original: (3) Prediction Heads that use the final query embeddings to predict
Revised: (3) Prediction Heads that use the final query embeddings to predict

Original: [Figure]
Revised: [Figure]

Original: Skateboard
Revised: Skateboard

Original: [Figure]
Revised: [Figure]

Original: SparseMask
Revised: SparseMask

Original: Annotation 𝑀 ) 𝑀
Revised: Annotation 𝑀 ) 𝑀

Original: [Figure]
Revised: [Figure]

Original: desired outputs ( e.g., segmentation masks, class labels).
Revised: desired outputs ( e.g., segmentation masks, class labels).

Original: The queries play an important role for thesuccess of such end-to-end pipeline for set prediction with unknown number of outputs.
Revised: The queries play an important role for thesuccess of such end-to-end pipeline for set prediction with unknown number of outputs.

Original: The numberof queries are selected as the maximum number of output instanes of the model.
Revised: The numberof queries are selected as the maximum number of output instanes of the model.

Original: During inference, asubset of queries predict ∅ outputs to dynamically adjust the number of valid outputs.
Revised: During inference, asubset of queries predict ∅ outputs to dynamically adjust the number of valid outputs.

Original: An overview of MinVIS’s image instance segmentation is shown in Figure 2(a).
Revised: An overview of MinVIS’s image instance segmentation is shown in Figure 2(a).

Original: Given an image
Revised: Given an image

Original: X ∈ R H,W , the Image Encoder E extracts a set of features F = E ( X ) from the image.
Revised: X ∈ R H,W , the Image Encoder E extracts a set of features F = E ( X ) from the image.

Original: F ={ F 0 . . .
Revised: F ={ F 0 . . .

Original: F − 1 } is a sequence of multi-scale feature maps
Revised: F − 1 } is a sequence of multi-scale feature maps

Original: F i ∈ R H
Revised: F i ∈ R H

Original: i ,W i ,C i .
Revised: i ,W i ,C i .

Original: F − 1 denotes the final outputof E .
Revised: F − 1 denotes the final outputof E .

Original: The N initial query embeddings ˆ Q ∈ R N,C are learnable parameters, where N is a largeenough number of outputs.
Revised: The N initial query embeddings ˆ Q ∈ R N,C are learnable parameters, where N is a largeenough number of outputs.

Original: The Transformer Decoder T then take both F and ˆ Q to iteratively obtain
Revised: The Transformer Decoder T then take both F and ˆ Q to iteratively obtain

Original: Q = T ( F , ˆ Q ) , Q ∈ R N,C .
Revised: Q = T ( F , ˆ Q ) , Q ∈ R N,C .

Original: While most recent works focus on the design of T to better process Ffor Q , MinVIS’s architectural constraints are on the Prediction Heads.
Revised: While most recent works focus on the design of T to better process Ffor Q , MinVIS’s architectural constraints are on the Prediction Heads.

Original: There are two outputs for eachinstance: classification and segmentation mask.
Revised: There are two outputs for eachinstance: classification and segmentation mask.

Original: The classification scores O = C ( Q ) , O ∈ R N,Kfor K classes are the output of Classification Head C , and Q should contain the class informationfor each instance.
Revised: The classification scores O = C ( Q ) , O ∈ R N,Kfor K classes are the output of Classification Head C , and Q should contain the class informationfor each instance.

Original: For segmentation masks M ∈ R N,H,W , MinVIS requires that M be generatedby convolving the query embeddings Q with the final feature map F − 1 .
Revised: For segmentation masks M ∈ R N,H,W , MinVIS requires that M be generatedby convolving the query embeddings Q with the final feature map F − 1 .

Original: The shape for F − 1 is thus
Revised: The shape for F − 1 is thus

Original: H, W, C .
Revised: H, W, C .

Original: We have M = σ ( Q ∗ F − 1 ), where σ ( · ) is the sigmoid function.
Revised: We have M = σ ( Q ∗ F − 1 ), where σ ( · ) is the sigmoid function.

Original: The constraint to have Q convolve with the whole feature map F − 1 is important for MinVIS.
Revised: The constraint to have Q convolve with the whole feature map F − 1 is important for MinVIS.

Original: Considertwo queries Q i and Q j that corresponds to two distinct object instances and thus non-overlappingmasks.
Revised: Considertwo queries Q i and Q j that corresponds to two distinct object instances and thus non-overlappingmasks.

Original: This formulation ensures that Q i should only have high inner products with features in
Revised: This formulation ensures that Q i should only have high inner products with features in

Original: F − 1 that are covered by the mask of instance i .
Revised: F − 1 that are covered by the mask of instance i .

Original: Since the instance masks are non-overlapping, Q jshould instead have low inner products with these features.
Revised: Since the instance masks are non-overlapping, Q jshould instead have low inner products with these features.

Original: This implicitly constrains the queryembeddings to be discriminative between each other.
Revised: This implicitly constrains the queryembeddings to be discriminative between each other.

Original: On the other hand, if we apply this pipelineto two consecutive frames X t and X t +1 .
Revised: On the other hand, if we apply this pipelineto two consecutive frames X t and X t +1 .

Original: Then Q ti should still have higher inner product with Q t +1 icompared to Q t +1 j .
Revised: Then Q ti should still have higher inner product with Q t +1 icompared to Q t +1 j .

Original: This is because Q t +1 i and Q t +1 j are also discriminative between each other, while
Revised: This is because Q t +1 i and Q t +1 j are also discriminative between each other, while

Original: Q ti and Q t +1 i both need to have high inner products with features of the same instance, which donot change drastically between consecutive frames.
Revised: Q ti and Q t +1 i both need to have high inner products with features of the same instance, which donot change drastically between consecutive frames.

Original: We visualize our learned query embeddingsin Figure 3.
Revised: We visualize our learned query embeddingsin Figure 3.

Original: Each plot is for a video.
Revised: Each plot is for a video.

Original: Query embeddings belonging to the same instance (fromdifferent frames) have the same color.
Revised: Query embeddings belonging to the same instance (fromdifferent frames) have the same color.

Original: These embeddings are already grouped by instances withoutany video-based training.
Revised: These embeddings are already grouped by instances withoutany video-based training.

Original: Further details are in Section 4.2.
Revised: Further details are in Section 4.2.

Original: While not all image instance segmentation models satisfy our architectural constraints ( e.g., ROIbased architectures), we believe these are rather flexible designs that are compatible with variousquery-based instance segmentation models.
Revised: While not all image instance segmentation models satisfy our architectural constraints ( e.g., ROIbased architectures), we believe these are rather flexible designs that are compatible with variousquery-based instance segmentation models.

Original: We use Mask2Former [27] in this work.
Revised: We use Mask2Former [27] in this work.

Original: The Image
Revised: The Image

Original: Encoder E includes both the backbone and the pixel decoder of Mask2Former.
Revised: Encoder E includes both the backbone and the pixel decoder of Mask2Former.

Original: We also find thathaving fully-connected layers to further process Q before convolution is beneficial to the performance.
Revised: We also find thathaving fully-connected layers to further process Q before convolution is beneficial to the performance.

Original: 3.2 Tracking by Query Matching
Revised: 3.2 Tracking by Query Matching

Original: MinVIS is a per-frame two-stage approach, which requires a post-processing step to temporallyassociate instances.
Revised: MinVIS is a per-frame two-stage approach, which requires a post-processing step to temporallyassociate instances.

Original: This post-processing often involves heuristics like mask overlaps, which does notgeneralize well to scenarios with heavy occlusions.
Revised: This post-processing often involves heuristics like mask overlaps, which does notgeneralize well to scenarios with heavy occlusions.

Original: Unlike previous two-stage approaches, MinVISassociate instances solely based on the query embeddings Q .
Revised: Unlike previous two-stage approaches, MinVISassociate instances solely based on the query embeddings Q .

Original: Given two consecutive frames X t
Revised: Given two consecutive frames X t

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Query embeddings are already grouped into clusters by instance without any video-based training.
Revised: Query embeddings are already grouped into clusters by instance without any video-based training.

Original: and X t +1 .
Revised: and X t +1 .

Original: We have Q t =
Revised: We have Q t =

Original: T ( F t , ˆ Q ) , where F t = E ( X t ) , and similarly for Q t +1 .
Revised: T ( F t , ˆ Q ) , where F t = E ( X t ) , and similarly for Q t +1 .

Original: Q ti is the queryembedding for instance i .
Revised: Q ti is the queryembedding for instance i .

Original: Tracking in MinVIS is done by the assignment of applying the Hungarianalgorithm on a score matrix S , where S ij = cos ( Q ti , Q t +1 j ) .
Revised: Tracking in MinVIS is done by the assignment of applying the Hungarianalgorithm on a score matrix S , where S ij = cos ( Q ti , Q t +1 j ) .

Original: cos ( · , · ) is the cosine similarity.
Revised: cos ( · , · ) is the cosine similarity.

Original: This approach is less affected by occlusions because each instance is represented by a query that doesnot have a spatial extent.
Revised: This approach is less affected by occlusions because each instance is represented by a query that doesnot have a spatial extent.

Original: In addition, we do not need heuristics to handle the birth and death of objectinstances in this framework.
Revised: In addition, we do not need heuristics to handle the birth and death of objectinstances in this framework.

Original: Since the number of queries is larger than the number of instances, thereare queries that produce empty masks.
Revised: Since the number of queries is larger than the number of instances, thereare queries that produce empty masks.

Original: The death of an object instance happens when its embeddingis matched to such a null query.
Revised: The death of an object instance happens when its embeddingis matched to such a null query.

Original: On the other hand, the birth of an instance is correctly handled if thematched query embeddings have been null before the actual birth of the object instance.
Revised: On the other hand, the birth of an instance is correctly handled if thematched query embeddings have been null before the actual birth of the object instance.

Original: 3.3 Training with Less Supervision for VIS
Revised: 3.3 Training with Less Supervision for VIS

Original: Since the matching process does not need training, only the image instance segmentation model needsto be trained.
Revised: Since the matching process does not need training, only the image instance segmentation model needsto be trained.

Original: There are two outputs of the model: classification scores O ∈ R N,K and segmentationmasks M ∈ R N,H,W for N queries, K object classes, and image size H , W .
Revised: There are two outputs of the model: classification scores O ∈ R N,K and segmentationmasks M ∈ R N,H,W for N queries, K object classes, and image size H , W .

Original: We can process thegroundtruth video instances to groundtruth image instances O ∗ ∈ R L,K and M ∗ ∈ R L,H,W , where
Revised: We can process thegroundtruth video instances to groundtruth image instances O ∗ ∈ R L,K and M ∗ ∈ R L,H,W , where

Original: L is the number of groundtruth instances ( N >> L ) and O ∗ j is a one-hot vector of groundtruth class.
Revised: L is the number of groundtruth instances ( N >> L ) and O ∗ j is a one-hot vector of groundtruth class.

Original: Given a loss function L ( O i , M i , O ∗ j , M ∗ j ) between predicted instance i and groundtruth instance j ,we first use bipartite matching to find the assignments between predicted and groundtruth instancesthat minimize the overall loss function, and train on those matched predictions with the loss function.
Revised: Given a loss function L ( O i , M i , O ∗ j , M ∗ j ) between predicted instance i and groundtruth instance j ,we first use bipartite matching to find the assignments between predicted and groundtruth instancesthat minimize the overall loss function, and train on those matched predictions with the loss function.

Original: More specifically, there are two terms in the loss function: L cls and L mask .
Revised: More specifically, there are two terms in the loss function: L cls and L mask .

Original: We use cross entropyloss for L cls and binary cross entropy plus dice loss [29] for L mask as in previous works [27].
Revised: We use cross entropyloss for L cls and binary cross entropy plus dice loss [29] for L mask as in previous works [27].

Original: Bothterms are purely image-based.
Revised: Bothterms are purely image-based.

Original: The groudtruth video instances are first processed to instances foreach frame independently.
Revised: The groudtruth video instances are first processed to instances foreach frame independently.

Original: Therefore, even if there are only sparse frames labeled with instance, wecan still train our model with the annotated frames.
Revised: Therefore, even if there are only sparse frames labeled with instance, wecan still train our model with the annotated frames.

Original: This provide a straightforward way to reduce thesupervision for VIS.
Revised: This provide a straightforward way to reduce thesupervision for VIS.

Original: Figure 2(b) shows the annotation sub-sampling to reduce supervision.
Revised: Figure 2(b) shows the annotation sub-sampling to reduce supervision.

Original:  Experiments
Revised:  Experiments

Original: Datasets.
Revised: Datasets.

Original: We evaluate MinVIS on three datasets: YouTube-VIS 2019/2021 [1] and Occluded VIS(OVIS)
Revised: We evaluate MinVIS on three datasets: YouTube-VIS 2019/2021 [1] and Occluded VIS(OVIS)

Original: The YouTube-VIS datasets contain 40 object classes.
Revised: The YouTube-VIS datasets contain 40 object classes.

Original: YouTube-VIS 2019 contains2238/302/343 videos for training/validation/testing, while YouTube-VIS 2021 expands the dataset to2985/421/453 videos for training/validation/testing, and includes higher quality annotations.
Revised: YouTube-VIS 2019 contains2238/302/343 videos for training/validation/testing, while YouTube-VIS 2021 expands the dataset to2985/421/453 videos for training/validation/testing, and includes higher quality annotations.

Original: OVIS
Revised: OVIS

Original: [Table] [Table]
Revised: [Table] [Table]

Original: has 25 object classes and contains 607/140/154 for training/validation/testing.
Revised: has 25 object classes and contains 607/140/154 for training/validation/testing.

Original: While the numberof videos is smaller, OVIS has more objects per frame, and the videos are also longer.
Revised: While the numberof videos is smaller, OVIS has more objects per frame, and the videos are also longer.

Original: This leadsto more annotated instance masks compared to the YouTube-VIS datasets.
Revised: This leadsto more annotated instance masks compared to the YouTube-VIS datasets.

Original: In addition, OVIS alsohas much higher Bounding-box Occlusion Rate (0.22 v.s. 0.06/0.07) compared to the YouTube-VISdatasets, which indicates heavier occlusions between object instances.
Revised: In addition, OVIS alsohas much higher Bounding-box Occlusion Rate (0.22 v.s. 0.06/0.07) compared to the YouTube-VISdatasets, which indicates heavier occlusions between object instances.

Original: Metrics.
Revised: Metrics.

Original: We follow previous works and use Average Precision (AP) and Average Recall (AR) asevaluation metrics [1].
Revised: We follow previous works and use Average Precision (AP) and Average Recall (AR) asevaluation metrics [1].

Original: AP is computed based on 10 intersection-over-union (IoU) thresholds from50% to 95% with 5% increment.
Revised: AP is computed based on 10 intersection-over-union (IoU) thresholds from50% to 95% with 5% increment.

Original: The reported AP and AR are first computed for each object classand then averaged over all classes.
Revised: The reported AP and AR are first computed for each object classand then averaged over all classes.

Original: All three datasets have public evaluation servers.
Revised: All three datasets have public evaluation servers.

Original: Baselines.
Revised: Baselines.

Original: We focus on results using ResNet50 and Swin-L backbones.
Revised: We focus on results using ResNet50 and Swin-L backbones.

Original: ResNet50 is still the mostwidely used backbone for VIS, while Swin-L gives the best performances.
Revised: ResNet50 is still the mostwidely used backbone for VIS, while Swin-L gives the best performances.

Original: Not all methods reportboth backbones on all three datasets.
Revised: Not all methods reportboth backbones on all three datasets.

Original: We include results that are available.
Revised: We include results that are available.

Original: For YouTube-VIS datasets,we include recent state-of-the-art results from SeqFormer [5], TeViT [8], and Mask2Former-VIS [3].
Revised: For YouTube-VIS datasets,we include recent state-of-the-art results from SeqFormer [5], TeViT [8], and Mask2Former-VIS [3].

Original: These are all Transformer-based per-clip approaches as this paradigm has been recently dominatingthe field.
Revised: These are all Transformer-based per-clip approaches as this paradigm has been recently dominatingthe field.

Original: On the other, out of of these methods, only TeViT is applied to OVIS.
Revised: On the other, out of of these methods, only TeViT is applied to OVIS.

Original: Therefore, we furthercompare to CMaskTrack R-CNN [12], CrossVIS [2], and STC [30].
Revised: Therefore, we furthercompare to CMaskTrack R-CNN [12], CrossVIS [2], and STC [30].

Original: These are all methods that allowonline processing.
Revised: These are all methods that allowonline processing.

Original: Even TeViT uses a near online inference for OVIS [31].
Revised: Even TeViT uses a near online inference for OVIS [31].

Original: This is because OVIS haslonger videos that would lead to out-of-memory for most of the per-clip approaches.
Revised: This is because OVIS haslonger videos that would lead to out-of-memory for most of the per-clip approaches.

Original: Out of all the baselines, Mask2Former-VIS [3] is the most related to MinVIS, as MinVIS is built on
Revised: Out of all the baselines, Mask2Former-VIS [3] is the most related to MinVIS, as MinVIS is built on

Original: Mask2Former in this work.
Revised: Mask2Former in this work.

Original: Mask2Former-VIS thus can be seen as the per-clip version of ours and isan important baseline for comparison.
Revised: Mask2Former-VIS thus can be seen as the per-clip version of ours and isan important baseline for comparison.

Original: Therefore, we further apply Mask2Former-VIS on the OVISdataset.
Revised: Therefore, we further apply Mask2Former-VIS on the OVISdataset.

Original: Due to memory constraints, the videos in OVIS are first split into clips of length 30.
Revised: Due to memory constraints, the videos in OVIS are first split into clips of length 30.

Original: We usethe same post-processing as MinVIS to merge the outputs from these clips.
Revised: We usethe same post-processing as MinVIS to merge the outputs from these clips.

Original: Implementation Details.
Revised: Implementation Details.

Original: Unless otherwise noted, our hyper-parameters follow Mask2FormerVIS [3].
Revised: Unless otherwise noted, our hyper-parameters follow Mask2FormerVIS [3].

Original: All models are pre-trained with COCO instance segmentation [32].
Revised: All models are pre-trained with COCO instance segmentation [32].

Original: For OVIS, we use thesame hyper-parameters as YouTube-VIS 2019 except training for 10 k iterations instead of 6 k .
Revised: For OVIS, we use thesame hyper-parameters as YouTube-VIS 2019 except training for 10 k iterations instead of 6 k .

Original: Fortraining losses, the weights are 5 . 

Revised: Fortraining losses, the weights are 5 .

Original: for L mask and 2 . 

Revised: for L mask and 2 .

Original: for L cls .
Revised: for L cls .

Original: All results of MinVIS are averagedover 3 random seeds.
Revised: All results of MinVIS are averagedover 3 random seeds.

Original: We sub-sample training to X% by uniformly sampling frames in the video.
Revised: We sub-sample training to X% by uniformly sampling frames in the video.

Original: We set a minimum of 1 frame per video.
Revised: We set a minimum of 1 frame per video.

Original: Since YouTube-VIS datasets often have videos less than ahundred frames.
Revised: Since YouTube-VIS datasets often have videos less than ahundred frames.

Original: Our 1% results are better seen as one frame per video results for YouTube-VIS.
Revised: Our 1% results are better seen as one frame per video results for YouTube-VIS.

Original: 4.1 Main Results
Revised: 4.1 Main Results

Original: YouTube-VIS 2019.
Revised: YouTube-VIS 2019.

Original: The results for YouTube-VIS 2019 are shown in Table 1.
Revised: The results for YouTube-VIS 2019 are shown in Table 1.

Original: MinVIS achieveshighest AP and most other metrics for both ResNet-50 and Swin-L backbones.
Revised: MinVIS achieveshighest AP and most other metrics for both ResNet-50 and Swin-L backbones.

Original: SeqFormer showsthat it is beneficial to jointly train with images from COCO [32] that contain YouTubeVIS categories(+C80k in table).
Revised: SeqFormer showsthat it is beneficial to jointly train with images from COCO [32] that contain YouTubeVIS categories(+C80k in table).

Original: TeViT proposes messenger shift transformer (MsgShifT) that are as efficient
Revised: TeViT proposes messenger shift transformer (MsgShifT) that are as efficient

Original: [Table] [Table]
Revised: [Table] [Table]

Original: as ResNet backbones, while improving the VIS performances.
Revised: as ResNet backbones, while improving the VIS performances.

Original: Our ResNet-50 results match oroutperform their results without further modifications.
Revised: Our ResNet-50 results match oroutperform their results without further modifications.

Original: Compared to the state-of-the-art Mask2FormerVIS, which can be seen as the per-clip approach to apply Mask2Former to VIS, MinVIS consistentlyoutperforms by around 1% for both backbones.
Revised: Compared to the state-of-the-art Mask2FormerVIS, which can be seen as the per-clip approach to apply Mask2Former to VIS, MinVIS consistentlyoutperforms by around 1% for both backbones.

Original: MinVIS with X% means sub-sampling the annotatedframes for each video in training.
Revised: MinVIS with X% means sub-sampling the annotatedframes for each video in training.

Original: Since there are less temporal variations for videos in YouTube-VIS2019, MinVIS with 1% of training frames only reduces AP by 2.6%.
Revised: Since there are less temporal variations for videos in YouTube-VIS2019, MinVIS with 1% of training frames only reduces AP by 2.6%.

Original: This significantly reduces theannotation effort while not sacrificing much performance.
Revised: This significantly reduces theannotation effort while not sacrificing much performance.

Original: YouTube-VIS 2021.
Revised: YouTube-VIS 2021.

Original: The results for YouTube-VIS 2021 are shown in Table 2.
Revised: The results for YouTube-VIS 2021 are shown in Table 2.

Original: On this morechallenging dataset, the performance improvements for MinVIS increase compared to YouTube-VIS2019.
Revised: On this morechallenging dataset, the performance improvements for MinVIS increase compared to YouTube-VIS2019.

Original: Without better backbone like TeViT and additional training data like SeqFormer, our ResNet50 results outperform by a large margin for all metrics.
Revised: Without better backbone like TeViT and additional training data like SeqFormer, our ResNet50 results outperform by a large margin for all metrics.

Original: This is the also case for Swin-L. MinVISoutperforms previous state-of-the-art Mask2Former-VIS by 2.7%.
Revised: This is the also case for Swin-L. MinVISoutperforms previous state-of-the-art Mask2Former-VIS by 2.7%.

Original: By using only 1% of trainingframes, MinVIS’s AP decrease by only 2.4%, which means that our 1% result still outperformsprevious state-of-the art.
Revised: By using only 1% of trainingframes, MinVIS’s AP decrease by only 2.4%, which means that our 1% result still outperformsprevious state-of-the art.

Original: We also see that on YouTube-VIS datasets, reducing the annotations by 10xdoes not significantly affect the performances (-0.6% AP for 2019 and -0.4% AP for 2021).
Revised: We also see that on YouTube-VIS datasets, reducing the annotations by 10xdoes not significantly affect the performances (-0.6% AP for 2019 and -0.4% AP for 2021).

Original: Occluded VIS (OVIS).
Revised: Occluded VIS (OVIS).

Original: The results for OVIS are shown in Table 3.
Revised: The results for OVIS are shown in Table 3.

Original: Mask2Former-VIS* denotesour application of Mask2Former-VIS to OVIS.
Revised: Mask2Former-VIS* denotesour application of Mask2Former-VIS to OVIS.

Original: Since videos in OVIS can have up to hundreds offrames, we apply Mask2Former-VIS to non-overlapping sliding windows of length 30.
Revised: Since videos in OVIS can have up to hundreds offrames, we apply Mask2Former-VIS to non-overlapping sliding windows of length 30.

Original: The outputsfrom these clips are then merged by our post-processing.
Revised: The outputsfrom these clips are then merged by our post-processing.

Original: MinVIS is an online method and does notneed modification to apply to OVIS.
Revised: MinVIS is an online method and does notneed modification to apply to OVIS.

Original: MinVIS shows significant improvement compared to existingworks on OVIS.
Revised: MinVIS shows significant improvement compared to existingworks on OVIS.

Original: With ResNet-50 backbone, MinVIS outperforms previous state-of-the-art TeViT with MsgShifT backbone by 7.6% AP.
Revised: With ResNet-50 backbone, MinVIS outperforms previous state-of-the-art TeViTwith MsgShifT backbone by 7.6% AP.

Original: With Swin-L backbone, MinVIS outperforms previous bestresult
Revised: With Swin-L backbone, MinVIS outperforms previous best

Original: M a s k 2 F o r m e r - V
Revised: M a s k 2 F o r m e r - V

Original: I S *
Revised: I S *

Original: H e u r
Revised: H e u r

Original: i s t
Revised: i s t

Original: i c s
Revised: i c s

Original: M i n V I S 𝑡 " 𝑡
Revised: M i n V I S 𝑡 " 𝑡

Original: [Table]
Revised: [Table]

Original:  MaskTrack R-CNN*+SWA by 10.5% AP, which is the winner of the 1st OVIS Challenge.
Revised: result MaskTrack R-CNN*+SWA by 10.5% AP, which is the winner of the 1st OVIS Challenge.

Original: Their keyobservation is that sampling frames that are far apart in OVIS leads to drastically different featuresand makes it hard to train MaskTrack R-CNN.
Revised: Their key observation is that sampling frames that are far apart in OVIS leads to drastically differentfeatures and makes it hard to train MaskTrack R-CNN.

Original: This is in contrast to YouTube-VIS datasets, in whichthe videos are shorter and there are less temporal variations within the video.
Revised: This is in contrast to YouTube-VIS datasets,in which the videos are shorter and there are less temporal variations within the video.

Original: We observe the samephenomenon when training Mask2Former-VIS*.
Revised: We observethe same phenomenon when training Mask2Former-VIS*.

Original: However, the limited sampling reference framestrategy of MaskTrack R-CNN*+SWA still does not work in this case.
Revised: However, the limited sampling referenceframe strategy of MaskTrack R-CNN*+SWA still does not work in this case.

Original: Mask2Former-VIS* uses afully end-to-end loss instead of an explicit tracking loss to learn temporal association, which makesthe learning even harder in OVIS.
Revised: Mask2Former-VIS*uses a fully end-to-end loss instead of an explicit tracking loss to learn temporal association, whichmakes the learning even harder in OVIS.

Original: On the other hand, MinVIS is image-based and does not need toworry about the temporal sampling strategy to train the model.
Revised: On the other hand, MinVIS is image-based and doesnot need to worry about the temporal sampling strategy to train the model.

Original: This is contrary to common beliefthat per-frame approaches are worse for scenarios with heavy occlusions.
Revised: This is contrary tocommon belief that per-frame approaches are worse for scenarios with heavy occlusions.

Original: Instead, our image-basedapproach leads to easier and better learning on OVIS.
Revised: Instead, ourimage-based approach leads to easier and better learning on OVIS.

Original: We show that an image instance segmentationmodel that can segment occluded instances in each frame is also good at associating such instancesacross frames.
Revised: We show that an image instancesegmentation model that can segment occluded instances in each frame is also good at associatingsuch instances across frames.

Original: Figure 4 shows qualitative results.
Revised: Figure 4 shows qualitative results.

Original: MinVIS stably tracks all the sheep in the video.
Revised: MinVIS stably tracks all the sheepin the video.

Original: Using mask overlap based heuristics instead of query matching leads to multiple identity switchesin tracking.
Revised: Using mask overlap based heuristics instead of query matching leads to multiple identityswitches in tracking.

Original: Mask2Former-VIS* does not have as good segmentation masks because its training isinterfered by heavy occlusions and large appearance deformations between frames in OVIS. 
Revised: Mask2Former-VIS* does not have as good segmentation masks because itstraining is interfered by heavy occlusions and large appearance deformations between frames in OVIS. Appendix F shows additional qualitative results and discuss failure cases of MinVIS.

Original: 4.2 Analyzing Query Matching
Revised: 4.2 Analyzing Query Matching

Original: The success of MinVIS depends on whether query matching is good for tracking instances.
Revised: The success of MinVIS depends on whether query matching is good for tracking instances.

Original: Weconduct ablation studies by comparing it to manually designed heuristics.
Revised: Weconduct ablation studies by comparing it to manually designed heuristics.

Original: We use the bipartitematching heuristics in Section 3.3 for tracking by treating instances in the last frame as groundtruth.
Revised: We use the bipartitematching heuristics in Section 3.3 for tracking by treating instances in the last frame as groundtruth.

Original:  The results are in Table 4.
Revised: [Table] The results are in Table 4.

Original: Using heuristics lead to around 3% AP drop on both YouTube-VIS [-
 [Table] and-] 2021.
Revised: Using heuristics lead to around 3% AP drop on both YouTube-VIS [-
 [Table] and-]2019and 2021.

Original: It leads to more significant drop on OVIS (7.7%) due to heavier occlusions.
Revised: It leads to more significant drop on OVIS (7.7%) due to heavier occlusions.

Original: We alsocombine query matching and heuristics with equal weights, which has mixed results.
Revised: We alsocombine query matching and heuristics with equal weights, which has mixed results.

Original: Our query onlyapproach is simpler and more generalizable without loss of performance.
Revised: Our query onlyapproach is simpler and more generalizable without loss of performance.

Original: We visualize the learned query embeddings by t-SNE [33] in Figure 3.
Revised: We visualize the learned query embeddings by t-SNE [33] in Figure 3.

Original: Each plot is for a video in thetraining set.
Revised: Each plot is for a video in thetraining set.

Original: We visualize the training set to see the effect of an image only objective (to segmentinstances in an image) on query embeddings across different frames.
Revised: We visualize the training set to see the effect of an image only objective (to segmentinstances in an image) on query embeddings across different frames.

Original: Query embeddings of the sameinstance have the same color.
Revised: Query embeddings of the sameinstance have the same color.

Original: We obtain the instance IDs for queries by bipartite matching its outputsto groundtruth instances, which have consistent IDs across frames.
Revised: We obtain the instance IDs for queries by bipartite matching its outputsto groundtruth instances, which have consistent IDs across frames.

Original: Without any video-based trackingobjective, query embeddings of the same instances are already grouped into distinct clusters, even forthe OVIS dataset.
Revised: Without any video-based trackingobjective, query embeddings of the same instances are already grouped into distinct clusters, even forthe OVIS dataset.

Original: This supports our design of only using image-based objectives.
Revised: This supports our design of only using image-based objectives.

Original: 4.3 Effect of Video-based Training
Revised: 4.3 Effect of Video-based Training

Original: While we have shown that MinVIS achieves state-of-the-art VIS performance without video-basedtraining, it is interesting to see how we can leverage video annotation when it is available.
Revised: While we have shown that MinVIS achieves state-of-the-art VIS performance without video-basedtraining, it is interesting to see how we can leverage video annotation when it is available.

Original: We usethe video annotation to supervise our matching as in previous per-frame works [1, 2, 11].
Revised: We usethe video annotation to supervise our matching as in previous per-frame works [1, 2, 11].

Original: Giventwo sampled frames, we use a hinge loss to ensure that the correct associations of queries have thehighest inner products compared to that of other queries between the two frames [11].
Revised: Giventwo sampled frames, we use a hinge loss to ensure that the correct associations of queries have thehighest inner products compared to that of other queries between the two frames [11].

Original: The results arein Table 5.
Revised: The results arein Table 5.

Original: The “Supervised Matching” rows mean directly applying the matching supervision tothe original frame sampling process.
Revised: The “Supervised Matching” rows mean directly applying the matching supervision tothe original frame sampling process.

Original: In our case, this means that the two sampled frames might beseparated up to 20 frames.
Revised: In our case, this means that the two sampled frames might beseparated up to 20 frames.

Original: As pointed out in previous works, frames that are far separated increasethe training difficulty and can hurt model performances especially with occlusions [14].
Revised: As pointed out in previous works, frames that are far separated increasethe training difficulty and can hurt model performances especially with occlusions [14].

Original: We thus alsoconsider the “Limited Range” training to only sample consecutive frames for supervised matching, aswe only need to match consecutive frames.
Revised: We thus alsoconsider the “Limited Range” training to only sample consecutive frames for supervised matching, aswe only need to match consecutive frames.

Original: From the results, directly applying “Supervised Matching”hurt performances on all three datasets.
Revised: From the results, directly applying “Supervised Matching”hurt performances on all three datasets.

Original: Adding “Limited Range” recovers most of the performancesfor YouTube-VIS 2021 and OVIS.
Revised: Adding “Limited Range” recovers most of the performancesfor YouTube-VIS 2021 and OVIS.

Original: On OVIS, it even marginally outperforms the original MinVIS.
Revised: On OVIS, it even marginally outperforms the original MinVIS.

Original: However, this improvement does rely on the dataset dependent sampling range.
Revised: However, this improvement does rely on the dataset dependent sampling range.

Original: While we believe itis possible to use video-based training to further improve MinVIS,  this would take away MinVIS’spractical advantages of only needing sparse annotations and having a simple training pipeline. 
Revised: We believe it ispossible and important to use video-based training to further improve MinVIS, although this wouldtake away MinVIS’s practical advantages of only needing sparse annotations and having a simpletraining pipeline. Appendix A discusses further limitations of not using video information in training.

Original:  Conclusion
Revised:  Conclusion

Original: We show that a purely image-based training procedure can lead to competitive performances forvideoinstance segmentation.
Revised: We show that a purely image-based training procedure can lead to competitive performances for

Original: While being a per-frame approach, MinVIS does not require manually designed heuristics to track instances across frames.
Revised: VIS. Our key finding is that instance tracking naturally emerges in query-based image instancesegmentation models with proper architectural constraint.

Original: In addition to improving state-of-the-art approacheson YouTube-VIS 2019/2021, we show that this is particularly beneficial for OVIS with heavy occlusions.
Revised: In addition to improving state-of-the-artapproaches on YouTube-VIS 2019/2021, we show that this is particularly beneficial for OVIS.

Original: The image-based objective reduces the learning difficulty and leads to better performances.
Revised: Theimage-based objective reduces the learning difficulty and leads to better performances.

Original: The image-based approach also does not require all the frames in a video to be annotated, whichmakes MinVIS  much more applicable to real-world scenarios.
Revised: MinVIS onlyrequires sparse frame annotations, which makes it much more applicable to real-world scenarios.

Original: We believe a promising direction toextend MinVIS is to explore ways to better leverage the video frames that are not annotated to furtherimprove our performances with sub-sampled annotations.
Revised: Webelieve a promising direction to extend MinVIS is to explore ways to better leverage the video framesthat are not annotated to further improve our performances with sub-sampled annotations.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No]  or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer  either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .•
Revised: [Yes] See Section ?? .

Original:  Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.•
Revised: [No] The code and the data areproprietary.•

Original: Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [N/A]Please do not modify the questions and only use the provided macros for your answers
Revised: [N/A] Please do not modify the questions and only use the provided macros for your answers. Note that the

Revised: Checklist section does not count towards the page limit.

Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes](b) Did you describe the limitations of your work?

Original: [Yes] See Section 4.3 and the appendix.
Revised: [Yes] See Section 4.3 and the appendix.(c)

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See theappendix.(d)
Revised: [Yes] See theappendix.

Original:  Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b)
Revised: [N/A](b)

Original: Did you include complete proofs of all theoretical results?
Revised: Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [No] We need further review and approval before code release.
Revised: [No] We need furtherreview and approval before code release.(b)

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Section 4.(c)
Revised: [Yes] See Section 4.(c)

Original: Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] Tables with error bars are in the appendix.(d)
Revised: [Yes] Tables with error bars are in the appendix.(d)

Original: Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [No]4.
Revised: [No]4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: Allthree datasets are public and cited.
Revised: All three datasets are public and cited.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See the appendix.
Revised: [Yes] See the appendix.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]

Original: using/curating?
Revised: using/curating?

Original: [Yes] See the appendix.
Revised: [Yes] See the appendix.

Original: [Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable[Table]
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes] See the appendix.
Revised: [Yes] See the appendix.

Original: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table] (a) Did you include the full text of instructions given to participants and screenshots, if[Table]
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table] (b) Did you describe any potential participant risks, with links to Institutional Review[Table]
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Revised: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Revised: [N/A]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: Limitation. We have discussed in the main paper on the possibility of improving MinVIS withvideo-based training. While we believe there are practical advantages of using our image-based VIStraining pipeline, videos provide lots of extra information that we are not currently leveraging.

Revised: Inparticular, temporal supervision from video should make our query embeddings even more suitable fortracking instances.

Revised: In Appendix F, we visualize failure cases of using our current query embeddingsfor tracking.

Revised: We conduct further analysis of Supervised Matching in Appendix G and believe furtherinvestigation along this direction should improve our approach.

Revised: In addition to improving fullysupervised performance, we believe a promising direction is to explore semi-supervised learning atthe frame level.

Revised: In this case, one can temporally propagate the sub-sampled annotations in training tofurther improve the performance with reduced supervision.

Revised: Potential Negative Social Impacts.

Revised: Video instance segmentation is a challenging video task, andthus provides fine-grained understanding of videos.

Revised: The tracking and segmentation of objects ofinterest might be use for surveillance applications with negative social impact.

Revised: While “person” is acategory in the datasets used in this paper, no further protected attributes are annotated.

Revised: Therefore, ourtrained models’ performance on human subjects might not be fair with respect to protected attributes.

Revised: B Further Details for Datasets

Revised: The YouTube-VIS 2019/2021 datasets are under CC BY 4.0 License, and Occluded VIS is under CC

Revised: BY-NC-SA 4.0 License.

Revised: The videos in YouTube-VIS are from YouTube-VOS [34], whose videos arein turn from YouTube-8M [35].

Revised: YouTube-8M uses public videos on YouTube but does not discussthe process to filter personally identifiable information or offensive content in the paper.

Original: [Table] 
Revised: [Table] [Table]

Revised: C Tables with Standard Deviation

Revised: Tables with standard deviations are shown in Table 6, Table 7, and Table 8.

Revised: D Reducing Supervision for Mask2Former-VIS

Revised: The results for sub-sampling annotated frames for Mask2Former-VIS [3] are shown in Table 9.

Revised: MinVIS consistently outperforms Mask2Former-VIS in all settings.

Original: spent onparticipant compensation?
Revised: The improvement increases forall three datasets when we sub-sample the annotation: +1.2% for full supervision v.s. +1.7% for 1%supervision on YouTube-VIS 2019. +2.7% for full supervision v.s. +5.8% for 1% supervision on

Revised: YouTube-VIS 2021.

Revised: +13.6% for full supervision v.s. +17.2% for 1% supervision on OVIS.

Revised: E Visualizing Query Embeddings in Evaluation

Revised: In the main paper, we visualize the learned query embeddings by t-SNE [33] in Figure 3.

Revised: The videosin Figure 3 are in the training set and the figure is meant to understand how query embeddings intraining cluster by instances without video-based loss function.

Revised: We can similarly apply the samevisualization to videos that are not used in training.

Revised: One complication here is that this visualizationuses groundtruth instance annotations to determine the corresponding instance ID for each query.

Revised: However, the groundtruth annotation is not publicly available for the three datasets considered inthis work.

Revised: Our reported results are obtained by submitting our predictions to the datasets’ evaluationservers.

Revised: We therefore perform this analysis by training a new model that only uses 90% of thetraining videos in YouTube-VIS 2019, and visualize the learned model’s query embeddings duringevaluation on the 10% videos that are not used to train the model.

Revised: While the 10% videos are notused in training the model, we still have their groundtruth instances for visualization purposes.

Revised: Thisprovides a realistic approximation of how our query embeddings would look like for videos not usedin training.

Revised: The visualization is in Figure 5.

Revised: Despite being noisier than training videos, the query

Revised: [Figure]

Revised: F Qualitative Results: Failure Cases

Revised: Figure 6 shows additional qualitative results on failure cases of MinVIS on the challenging OVISdataset.

Revised: As discussed in Section 3.2, MinVIS does not use heuristics to handle the birth and deathof object instances.

Revised: The death of an object instance is correctly handled if its query is matched to aquery in the next frame that produces an empty mask.

Revised: Despite its simplicity and effectiveness, thedrawback of this approach is that there is nothing stopping the model from matching the disappearingquery to a query with a non-empty mask.

Revised: From t 3 to t 4 in the top row of Figure 6, as the fish in thelower left leaves the frame, MinVIS associates it to a mask covering the tail of a nearby fish.

Revised: From t 4to t 5 , when the fish in the upper left leaves the frame, MinVIS again associates it to a mask coveringthe head of a nearby fish.

Revised: Since the associated masks are non-empty, MinVIS fails to correctly handlethe death of these instances.

Revised: On the other hand, when the two dogs in the bottom row of Figure 6are covered in t 2 , MinVIS correctly associates their queries to empty masks.

Revised: In addition, MinVISfurther correctly handles the object births in t 3 .

Revised: However, MinVIS is limited by the segmentation ofthe image segmentation model, which fails to segment the close-up person.

Revised: G Further Analysis of Supervised Matching

Revised: We conduct further analysis on the results in Section 4.3.

Revised: We visualize the query embeddings onthe same training videos with and without using supervised matching.

Revised: In particular, we perform theanalysis on YouTube-VIS 2019 and compare MinVIS v.s. MinVIS + Supervised Matching + Limited

Revised: Range, which hurts performance the most in Table 5.

Revised: The visualizations are in Figure 7.

Revised: While theplots look similar for most videos, one consistent trend we observe is that adding supervised matching

Revised: M i n V

Revised: I

Revised: S + S up e r v

Revised: i s e d M a t

Revised: c h 𝑉 " 𝑉

Original: [N/A]
Revised: [Figure]

Revised: makes the embeddings more evenly distributed and smooths out the outliers in the embedding space.

Revised: This is a reasonable consequence as the objective encourages the embeddings from the same objectinstance to be closer to each other.

Revised: However, it is unclear whether this is overall beneficial to ourtracking by query matching.

Revised: For example, in V 3 , the outliers are removed at the cost of mixingembeddings from different instances.

Revised: We believe it is an important future work to further understandhow to better leverage video information to improve MinVIS.

Revised: H Baseline Training Curves on OVIS

Revised: As discussed in the main paper, it is difficult to optimize our per-clip baseline on the challenging

Revised: OVIS dataset.

Revised: We have included the training curves in Figure 8 for further illustration.

Revised: Blue curvesare MinVIS and orange curves are Mask2Former-VIS.

Revised: While the classification loss still optimizeswell on OVIS, the per-clip baseline has difficulty optimizing mask related losses.