Original: MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training
Revised: MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training

Original: Abstract
Revised: Abstract

Original: We propose MinVIS, a minimal video instance segmentation (VIS) framework that achieves state-of-the-art VIS performance with neither video-based architectures nor training procedures.
Revised: We propose MinVIS, a minimal video instance segmentation (VIS) framework that achieves state-of-the-art VIS performance with neither video-based architectures nor training procedures.

Original: By only training a query-based image instance segmentation model, MinVIS outperforms the previous best result on the challenging
Revised: By only training a query-based image instance segmentation model, MinVIS outperforms the previous best result on the challenging

Original: Occluded VIS dataset by over 10% AP.
Revised: Occluded VIS dataset by over 10% AP.

Original: Since MinVIS treats frames in training videos as independent images, we can drastically sub-sample the annotated frames in training videos without any modifications.
Revised: Since MinVIS treats frames in training videos as independent images, we can drastically sub-sample the annotated frames in training videos without any modifications.

Original: With only 1% of labeled frames, MinVIS outperforms or is comparable to fully-supervised state-of-the-art approaches on YouTube-VIS 2019/2021.
Revised: With only 1% of labeled frames, MinVIS outperforms or is comparable to fully-supervised state-of-the-art approaches on YouTube-VIS 2019/2021.

Original: Our key observation is that queries trained to be discriminative between intra-frame object instances are temporally consistent and can be used to track instances without any manually designed heuristics.
Revised: Our key observation is that queries trained to be discriminative between intra-frame object instances are temporally consistent and can be used to track instances without any manually designed heuristics.

Original: MinVIS thus has the following inference pipeline: we first apply the trained query-based image instance segmentation to video frames independently.
Revised: MinVIS thus has the following inference pipeline: we first apply the trained query-based image instance segmentation to video frames independently.

Original: The segmented instances are then tracked by bipartite matching of the corresponding queries.
Revised: The segmented instances are then tracked by bipartite matching of the corresponding queries.

Original: This inference is done in an online fashion and does not need to process the whole video at once.
Revised: This inference is done in an online fashion and does not need to process the whole video at once.

Original: MinVIS thus has the practical advantages of reducing both the labeling costs and the memory requirements, while not sacrificing the VIS performance.
Revised: MinVIS thus has the practical advantages of reducing both the labeling costs and the memory requirements, while not sacrificing the VIS performance.

Original: Code is available at: https://github.com/NVlabs/MinVIS
Revised: Code is available at: https://github.com/NVlabs/MinVIS

Original:  Introduction
Revised:  Introduction

Original: Video instance segmentation (VIS) aims to simultaneously detect, segment, and track object instances in videos [1].
Revised: Video instance segmentation (VIS) aims to simultaneously detect, segment, and track object instances in videos [1].

Original: The requirement to accurately track object instances through an entire video makes
Revised: The requirement to accurately track object instances through an entire video makes

Original: VIS much more challenging than image instance segmentation.
Revised: VIS much more challenging than image instance segmentation.

Original: Most of the early approaches for VIS build on image instance segmentation models and process videos on a per-frame basis [1, 2].
Revised: Most of the early approaches for VIS build on image instance segmentation models and process videos on a per-frame basis [1, 2].

Original: The segmented object instances for each frame are then matched temporally with a post-processing step.
Revised: The segmented object instances for each frame are then matched temporally with a post-processing step.

Original: This post-processing step often involves manually designed heuristics that do not generalize well to challenging scenarios like occlusions and large appearance deformations.
Revised: This post-processing step often involves manually designed heuristics that do not generalize well to challenging scenarios like occlusions and large appearance deformations.

Original: Recent VIS works address this issue by taking a per-clip approach, where the spatial-temporal volume of a video is processed as a whole to directly predict the spatial-temporal mask for each object instance [3–5].
Revised: Recent VIS works address this issue by taking a per-clip approach, where the spatial-temporal volume of a video is processed as a whole to directly predict the spatial-temporal mask for each object instance [3–5].

Original: Many of these end-to-end VIS approaches are built upon the recent advances of Transformers for end-to-end object detection [6].
Revised: Many of these end-to-end VIS approaches are built upon the recent advances of Transformers for end-to-end object detection [6].

Original: Given learned embeddings called queries ,
Revised: Given learned embeddings called queries ,

Original: Transformers process the queries jointly with the input video using cross-attention, so that each of the processed queries can be used to predict the spatial-temporal mask for an object instance in the video.
Revised: Transformers process the queries jointly with the input video using cross-attention, so that each of the processed queries can be used to predict the spatial-temporal mask for an object instance in the video.

Original: While these per-clip methods have led to considerable improvements for VIS, using attention to process the whole video, especially longer ones, requires large memory and computation.
Revised: While these per-clip methods have led to considerable improvements for VIS, using attention to process the whole video, especially longer ones, requires large memory and computation.

Original: It is also not straightforward to adapt per-clip methods from offline to online processing to reduce the computational requirements.
Revised: It is also not straightforward to adapt per-clip methods from offline to online processing to reduce the computational requirements.

Original: This limits their practical application, and maintaining the effectiveness of these per-clip methods while improving their efficiency remains an active research direction [7, 8].
Revised: This limits their practical application, and maintaining the effectiveness of these per-clip methods while improving their efficiency remains an active research direction [7, 8].

Original: [Figure]
Revised: [Figure]

Original: Another limitation for existing VIS methods is the requirement on annotation.
Revised: Another limitation for existing VIS methods is the requirement on annotation.

Original: Annotating object instance masks for each video frame is prohibitively expensive at scale.
Revised: Annotating object instance masks for each video frame is prohibitively expensive at scale.

Original: While there have been works that alleviate this annotation requirement through weak supervision or image-based annotation, there is still a significant performance gap compared to state-of-the-art fully-supervised methods [9, 10].
Revised: While there have been works that alleviate this annotation requirement through weak supervision or image-based annotation, there is still a significant performance gap compared to state-of-the-art fully-supervised methods [9, 10].

Original: Our Approach.
Revised: Our Approach.

Original: We simultaneously address both of the aforementioned challenges of computational and labeling costs by showing that we can achieve state-of-the-art VIS performance by only training a query-based image instance segmentation model.
Revised: We simultaneously address both of the aforementioned challenges of computational and labeling costs by showing that we can achieve state-of-the-art VIS performance by only training a query-based image instance segmentation model.

Original: During inference, MinVIS first applies the query-based image instance segmentation to video frames independently.
Revised: During inference, MinVIS first applies the query-based image instance segmentation to video frames independently.

Original: The segmented instances are then associated by bipartite matching of the corresponding queries.
Revised: The segmented instances are then associated by bipartite matching of the corresponding queries.

Original: MinVIS processes each frame independently in an online fashion and does not need to process the whole video at once.
Revised: MinVIS processes each frame independently in an online fashion and does not need to process the whole video at once.

Original: MinVIS does not use any video-based training procedure and thus does not need annotations for all the frames in a video.
Revised: MinVIS does not use any video-based training procedure and thus does not need annotations for all the frames in a video.

Original: Our contributions are summarized below:
Revised: Our contributions are summarized below:

Original: We show that video-based architecture and training are not required for competitive VIS performances.
Revised: We show that video-based architecture and training are not required for competitive VIS performances.

Original: MinVIS outperforms previous state-of-the-art on YouTube-VIS 2019 and 2021 datasets by 1% and 3% AP while only training an image instance segmentation model.
Revised: MinVIS outperforms previous state-of-the-art on YouTube-VIS 2019 and 2021 datasets by 1% and 3% AP while only training an image instance segmentation model.

Original: We show that image instance segmentation models capable of segmenting occluded instances are also well suited to track occluded instances in videos in our framework.
Revised: We show that image instance segmentation models capable of segmenting occluded instances are also well suited to track occluded instances in videos in our framework.

Original: MinVIS outperforms its per-clip counterpart by over 13% AP on the challenging Occluded VIS (OVIS) dataset, which is over 10% improvement compared to the previous best performance on the dataset. 3.
Revised: MinVIS outperforms its per-clip counterpart by over 13% AP on the challenging Occluded VIS (OVIS) dataset, which is over 10% improvement compared to the previous best performance on the dataset. 3.

Original: Our image-based approach allows us to significantly sub-sample the required segmentation annotations in training without any change to the model.
Revised: Our image-based approach allows us to significantly sub-sample the required segmentation annotations in training without any change to the model.

Original: With only 1% of labeled frames, MinVIS outperforms or is comparable to fully-supervised state-of-the-art approaches on all three datasets.
Revised: With only 1% of labeled frames, MinVIS outperforms or is comparable to fully-supervised state-of-the-art approaches on all three datasets.

Original: Our key observation is that queries trained to be discriminative between intra-frame object instances are temporally consistent and can be used to track instances without being trained with video-based loss functions.
Revised: Our key observation is that queries trained to be discriminative between intra-frame object instances are temporally consistent and can be used to track instances without being trained with video-based loss functions.

Original: MinVIS achieves this by requiring its image instance segmentation model to generate masks by convolving query embeddings with features of the whole input image, including regions of other object instances.
Revised: MinVIS achieves this by requiring its image instance segmentation model to generate masks by convolving query embeddings with features of the whole input image, including regions of other object instances.

Original: A query is thus trained to only have high responses on features of its corresponding instance.
Revised: A query is thus trained to only have high responses on features of its corresponding instance.

Original: Other query embeddings should instead have low responses on these features because instance masks are non-overlapping.
Revised: Other query embeddings should instead have low responses on these features because instance masks are non-overlapping.

Original: This design encourages the query embeddings for different instances in a frame to be well-separated.
Revised: This design encourages the query embeddings for different instances in a frame to be well-separated.

Original: On the other hand, the query embeddings that segment the same instance from two consecutive frames still need to be similar enough since the instance’s image features to be convoluted do not change drastically between frames.
Revised: On the other hand, the query embeddings that segment the same instance from two consecutive frames still need to be similar enough since the instance’s image features to be convoluted do not change drastically between frames.

Original: This leads to temporally consistent query embeddings for tracking without the need of video-based training.
Revised: This leads to temporally consistent query embeddings for tracking without the need of video-based training.

Original: MinVIS thus has the following design for inference: We first apply a query-based image instance segmentation model on video frames independently.
Revised: MinVIS thus has the following design for inference: We first apply a query-based image instance segmentation model on video frames independently.

Original: The segmented instances are then associated between frames by bipartite matching of the corresponding query embeddings.
Revised: The segmented instances are then associated between frames by bipartite matching of the corresponding query embeddings.

Original: This post-processing step does not need any additional heuristics based on mask overlaps or classification scores as in previous works [1, 11].
Revised: This post-processing step does not need any additional heuristics based on mask overlaps or classification scores as in previous works [1, 11].

Original: This is because query embeddings already contain these information to track the instances.
Revised: This is because query embeddings already contain these information to track the instances.

Original: An overview of MinVIS’s training and inference is shown in Figure 1.
Revised: An overview of MinVIS’s training and inference is shown in Figure 1.

Original: Since video frames are treated as independent images to train MinVIS, there is no requirement to annotate all the frames in a video for training.
Revised: Since video frames are treated as independent images to train MinVIS, there is no requirement to annotate all the frames in a video for training.

Original: This allows us to significantly sub-sample and reduce the annotation without any change to our model.
Revised: This allows us to significantly sub-sample and reduce the annotation without any change to our model.

Original: We find that on YouTube-VIS 2019/ datasets [1], where there are less variations between video frames, using only 1% of labeled frames leads to less than 3% drop in AP for MinVIS.
Revised: We find that on YouTube-VIS 2019/ datasets [1], where there are less variations between video frames, using only 1% of labeled frames leads to less than 3% drop in AP for MinVIS.

Original: We further evaluate MinVIS on the Occluded VIS (OVIS) dataset [12].
Revised: We further evaluate MinVIS on the Occluded VIS (OVIS) dataset [12].

Original: One common critique of per-frame approaches is that their tracking heuristics based on mask overlaps would not work when there are heavy occlusions.
Revised: One common critique of per-frame approaches is that their tracking heuristics based on mask overlaps would not work when there are heavy occlusions.

Original: This is not the case for MinVIS, as we do not use any manually designed heuristics.
Revised: This is not the case for MinVIS, as we do not use any manually designed heuristics.

Original: We show that our query-matching approach generalizes to occluded scenarios.
Revised: We show that our query-matching approach generalizes to occluded scenarios.

Original: MinVIS with Swin Transformers backbone [13] achieves 39.4% AP on OVIS, which is over 10% improvement from the previous best result on the dataset [14].
Revised: MinVIS with Swin Transformers backbone [13] achieves 39.4% AP on OVIS, which is over 10% improvement from the previous best result on the dataset [14].

Original: We further show that our image-based strategy leads to easier and better learning on OVIS.
Revised: We further show that our image-based strategy leads to easier and better learning on OVIS.

Original: MinVIS outperforms its per-clip counterpart by over 13% AP.
Revised: MinVIS outperforms its per-clip counterpart by over 13% AP.

Original:  Related Work
Revised:  Related Work

Original: Video Instance Segmentation.
Revised: Video Instance Segmentation.

Original: Per-frame approaches for VIS process each frame independently and later track instances by post-processing.
Revised: Per-frame approaches for VIS process each frame independently and later track instances by post-processing.

Original: MaskTrack R-CNN [1] adds a tracking head to Mask
Revised: MaskTrack R-CNN [1] adds a tracking head to Mask

Original: R-CNN [15] for VIS.
Revised: R-CNN [15] for VIS.

Original: MaskProp [16] instead adds a mask propagation head to propagate object instance masks.
Revised: MaskProp [16] instead adds a mask propagation head to propagate object instance masks.

Original: CrossVIS [2] uses crossover learning to improve instance representation across video frames.
Revised: CrossVIS [2] uses crossover learning to improve instance representation across video frames.

Original: QueryTrack [11] adds a contrastive tracking head to QueryInst [17] for instance association.
Revised: QueryTrack [11] adds a contrastive tracking head to QueryInst [17] for instance association.

Original: Concurrent work IDOL [18] shows that per-frame models can still outperform per-clip models.
Revised: Concurrent work IDOL [18] shows that per-frame models can still outperform per-clip models.

Original: Our approach also builds on image instance segmentation models, but unlike previous approaches, we need neither additional parameters nor additional losses to apply to VIS.
Revised: Our approach also builds on image instance segmentation models, but unlike previous approaches, we need neither additional parameters nor additional losses to apply to VIS.

Original: Our query embeddings from image instance segmentation can directly be used for tracking without video-based training.
Revised: Our query embeddings from image instance segmentation can directly be used for tracking without video-based training.

Original: Recent per-clip approaches build on the success of Detection Transformer (DETR) [6].
Revised: Recent per-clip approaches build on the success of Detection Transformer (DETR) [6].

Original: VisTR [4] adopts the query-based approach of DETR to VIS, and there has been several follow-up works, such as Mask2Former-VIS [3] and SeqFormer [5].
Revised: VisTR [4] adopts the query-based approach of DETR to VIS, and there has been several follow-up works, such as Mask2Former-VIS [3] and SeqFormer [5].

Original: One limitation of these approaches is the need to process the whole video at once.
Revised: One limitation of these approaches is the need to process the whole video at once.

Original: IFC [7] reduces the overhead of temporal message passing by using memory tokens.
Revised: IFC [7] reduces the overhead of temporal message passing by using memory tokens.

Original: TeViT [8] uses a parameter-shared self attention to efficiently model temporal contexts.
Revised: TeViT [8] uses a parameter-shared self attention to efficiently model temporal contexts.

Original: We also use a query-based approach, but instead of using cross-attention to process the whole video, we process each frame independently while not losing VIS performance.
Revised: We also use a query-based approach, but instead of using cross-attention to process the whole video, we process each frame independently while not losing VIS performance.

Original: Our use of queries to associate instances is also related to other works that build on DETR for tracking in related fields.
Revised: Our use of queries to associate instances is also related to other works that build on DETR for tracking in related fields.

Original: For example, MOTR [19] and TrackFormer [20] use identity preserving track queries for multi-object tracking (MOT).
Revised: For example, MOTR [19] and TrackFormer [20] use identity preserving track queries for multi-object tracking (MOT).

Original: Reducing Supervision for Video Instance Segmentation.
Revised: Reducing Supervision for Video Instance Segmentation.

Original: Annotating instance masks for each video frame can be prohibitively expensive.
Revised: Annotating instance masks for each video frame can be prohibitively expensive.

Original: Compared to video object segmentation [21–23] and image instance segmentation [24–27], there have been less works on reducing supervision for VIS [28].
Revised: Compared to video object segmentation [21–23] and image instance segmentation [24–27], there have been less works on reducing supervision for VIS [28].

Original: FlowIRN [10] extends IRN [24] with motion and temporal consistency cues to have a weaklysupervised VIS framework that only requires classification labels.
Revised: FlowIRN [10] extends IRN [24] with motion and temporal consistency cues to have a weaklysupervised VIS framework that only requires classification labels.

Original: SOLO-Track [9] learns to track instances without video annotations.
Revised: SOLO-Track [9] learns to track instances without video annotations.

Original: It uses instance contrastive learning on SOLO [29] to learn grid cell embeddings for instance tracking.
Revised: It uses instance contrastive learning on SOLO [29] to learn grid cell embeddings for instance tracking.

Original: We make the same observation that disciminating between instances within frames is beneficial or even sufficient for instance tracking.
Revised: We make the same observation that disciminating between instances within frames is beneficial or even sufficient for instance tracking.

Original: However, unlike our query-based association, the grid cell embeddings still need threshold-based post-processing and additional loss functions to better handle birth and death of objects.
Revised: However, unlike our query-based association, the grid cell embeddings still need threshold-based post-processing and additional loss functions to better handle birth and death of objects.

Original:  Method
Revised:  Method

Original: MinVIS is a minimal VIS framework that does not require video-based training and thus can be easily applied to real-world applications that only have sparse image instance segmentation annotations.
Revised: MinVIS is a minimal VIS framework that does not require video-based training and thus can be easily applied to real-world applications that only have sparse image instance segmentation annotations.

Original: MinVIS is a two stage approach: (1) image instance segmentation on each frame independently, (2) associating instances between frames by matching queries.
Revised: MinVIS is a two stage approach: (1) image instance segmentation on each frame independently, (2) associating instances between frames by matching queries.

Original: We will first discuss the image instance segmentation architecture in MinVIS.
Revised: We will first discuss the image instance segmentation architecture in MinVIS.

Original: We will then discuss the temporal association of object instances.
Revised: We will then discuss the temporal association of object instances.

Original: Finally, we will discuss training and reducing supervision for MinVIS.
Revised: Finally, we will discuss training and reducing supervision for MinVIS.

Original: [Figure]
Revised: [Figure]

Original: Cls
Revised: Cls

Original: [Figure]
Revised: [Figure]

Original: 𝑂 ’ :
Revised: 𝑂 ’ :

Original: Skateboard 𝑂 ( : ∅ 𝑂 )
Revised: Skateboard 𝑂 ( : ∅ 𝑂 )

Original: [Figure]
Revised: [Figure]

Original: TrainingVideo
Revised: TrainingVideo

Original: SparseMask
Revised: SparseMask

Original: Annotation 𝑀 ) 𝑀
Revised: Annotation 𝑀 ) 𝑀

Original: [Figure]
Revised: [Figure]

Original: 3.1 Image Instance Segmentation Architecture for VIS
Revised: 3.1 Image Instance Segmentation Architecture for VIS

Original: MinVIS builds on the query-based transformer architectures for detection and segmentation [6, 17, 30, 31], which has the following main components: (1) Image Encoder that learn to extract features from input images.
Revised: MinVIS builds on the query-based transformer architectures for detection and segmentation [6, 17, 30, 31], which has the following main components: (1) Image Encoder that learn to extract features from input images.

Original: (2) Transformer Decoder that processes the outputs of Image Encoder to iteratively update the query embeddings.
Revised: (2) Transformer Decoder that processes the outputs of Image Encoder to iteratively update the query embeddings.

Original: (3) Prediction Heads that use the final query embeddings to predict desired outputs ( e.g., segmentation masks, class labels).
Revised: (3) Prediction Heads that use the final query embeddings to predict desired outputs ( e.g., segmentation masks, class labels).

Original: The queries play an important role for the success of such end-to-end pipeline for set prediction with unknown number of outputs.
Revised: The queries play an important role for the success of such end-to-end pipeline for set prediction with unknown number of outputs.

Original: The number of queries are selected as the maximum number of output instanes of the model.
Revised: The number of queries are selected as the maximum number of output instanes of the model.

Original: During inference, a subset of queries predict ∅ outputs to dynamically adjust the number of valid outputs.
Revised: During inference, a subset of queries predict ∅ outputs to dynamically adjust the number of valid outputs.

Original: An overview of MinVIS’s image instance segmentation is shown in Figure 2(a).
Revised: An overview of MinVIS’s image instance segmentation is shown in Figure 2(a).

Original: Given an image
Revised: Given an image

Original: X ∈ R H,W , the Image Encoder E extracts a set of features F = E ( X ) from the image.
Revised: X ∈ R H,W , the Image Encoder E extracts a set of features F = E ( X ) from the image.

Original: F = { F 0 . . .
Revised: F = { F 0 . . .

Original: F − 1 } is a sequence of multi-scale feature maps
Revised: F − 1 } is a sequence of multi-scale feature maps

Original: F i ∈ R H
Revised: F i ∈ R H

Original: i ,W i ,C i .
Revised: i ,W i ,C i .

Original: F − 1 denotes the final output of E .
Revised: F − 1 denotes the final output of E .

Original: The N initial query embeddings ˆ Q ∈ R N,C are learnable parameters, where N is a large enough number of outputs.
Revised: The N initial query embeddings ˆ Q ∈ R N,C are learnable parameters, where N is a large enough number of outputs.

Original: The Transformer Decoder T then take both F and ˆ Q to iteratively obtain
Revised: The Transformer Decoder T then take both F and ˆ Q to iteratively obtain

Original: Q = T ( F , ˆ Q ) , Q ∈ R N,C .
Revised: Q = T ( F , ˆ Q ) , Q ∈ R N,C .

Original: While most recent works focus on the design of T to better process F for Q , MinVIS’s architectural constraints are on the Prediction Heads.
Revised: While most recent works focus on the design of T to better process F for Q , MinVIS’s architectural constraints are on the Prediction Heads.

Original: There are two outputs for each instance: classification and segmentation mask.
Revised: There are two outputs for each instance: classification and segmentation mask.

Original: The classification scores O = C ( Q ) , O ∈ R N,K for K classes are the output of Classification Head C , and Q should contain the class information for each instance.
Revised: The classification scores O = C ( Q ) , O ∈ R N,K for K classes are the output of Classification Head C , and Q should contain the class information for each instance.

Original: For segmentation masks M ∈ R N,H,W , MinVIS requires that M be generated by convolving the query embeddings Q with the final feature map F − 1 .
Revised: For segmentation masks M ∈ R N,H,W , MinVIS requires that M be generated by convolving the query embeddings Q with the final feature map F − 1 .

Original: The shape for F − 1 is thus
Revised: The shape for F − 1 is thus

Original: H, W, C .
Revised: H, W, C .

Original: We have M = σ ( Q ∗ F − 1 ), where σ ( · ) is the sigmoid function.
Revised: We have M = σ ( Q ∗ F − 1 ), where σ ( · ) is the sigmoid function.

Original: The constraint to have Q convolve with the whole feature map F − 1 is important for MinVIS.
Revised: The constraint to have Q convolve with the whole feature map F − 1 is important for MinVIS.

Original: Consider two queries Q i and Q j that corresponds to two distinct object instances and thus non-overlapping masks.
Revised: Consider two queries Q i and Q j that corresponds to two distinct object instances and thus non-overlapping masks.

Original: This formulation ensures that Q i should only have high inner products with features in
Revised: This formulation ensures that Q i should only have high inner products with features in

Original: F − 1 that are covered by the mask of instance i .
Revised: F − 1 that are covered by the mask of instance i .

Original: Since the instance masks are non-overlapping, Q j should instead have low inner products with these features.
Revised: Since the instance masks are non-overlapping, Q j should instead have low inner products with these features.

Original: This implicitly constrains the query embeddings to be discriminative between each other.
Revised: This implicitly constrains the query embeddings to be discriminative between each other.

Original: On the other hand, if we apply this pipeline to two consecutive frames X t and X t +1 .
Revised: On the other hand, if we apply this pipeline to two consecutive frames X t and X t +1 .

Original: Then Q ti should still have higher inner product with Q t
Revised: Then Q ti should still have higher inner product with Q t

Original: +1 i compared to Q t +1 j .
Revised: +1 i compared to Q t +1 j .

Original: This is because Q t +1 i and Q t +1 j are also discriminative between each other, while
Revised: This is because Q t +1 i and Q t +1 j are also discriminative between each other, while

Original: Q ti and Q t +1 i both need to have high inner products with features of the same instance, which do not change drastically between consecutive frames.
Revised: Q ti and Q t +1 i both need to have high inner products with features of the same instance, which do not change drastically between consecutive frames.

Original: We visualize our learned query embeddings in Figure 3.
Revised: We visualize our learned query embeddings in Figure 3.

Original: Each plot is for a video.
Revised: Each plot is for a video.

Original: Query embeddings belonging to the same instance (from different frames) have the same color.
Revised: Query embeddings belonging to the same instance (from different frames) have the same color.

Original: These embeddings are already grouped by instances without any video-based training.
Revised: These embeddings are already grouped by instances without any video-based training.

Original: Further details are in Section 4.2.
Revised: Further details are in Section 4.2.

Original: While not all image instance segmentation models satisfy our architectural constraints ( e.g., ROIbased architectures), we believe these are rather flexible designs that are compatible with various query-based instance segmentation models.
Revised: While not all image instance segmentation models satisfy our architectural constraints ( e.g., ROIbased architectures), we believe these are rather flexible designs that are compatible with various query-based instance segmentation models.

Original: We use Mask2Former [30] in this work.
Revised: We use Mask2Former [30] in this work.

Original: The Image
Revised: The Image

Original: Encoder E includes both the backbone and the pixel decoder of Mask2Former.
Revised: Encoder E includes both the backbone and the pixel decoder of Mask2Former.

Original: We also find that having fully-connected layers to further process Q before convolution is beneficial to the performance.
Revised: We also find that having fully-connected layers to further process Q before convolution is beneficial to the performance.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Query embeddings are already grouped into clusters by instance without any video-based training.
Revised: Query embeddings are already grouped into clusters by instance without any video-based training.

Original: 3.2 Tracking by Query Matching
Revised: 3.2 Tracking by Query Matching

Original: MinVIS is a per-frame two-stage approach, which requires a post-processing step to temporally associate instances.
Revised: MinVIS is a per-frame two-stage approach, which requires a post-processing step to temporally associate instances.

Original: This post-processing often involves heuristics like mask overlaps, which does not generalize well to scenarios with heavy occlusions.
Revised: This post-processing often involves heuristics like mask overlaps, which does not generalize well to scenarios with heavy occlusions.

Original: Unlike previous two-stage approaches, MinVIS associate instances solely based on the query embeddings Q .
Revised: Unlike previous two-stage approaches, MinVIS associate instances solely based on the query embeddings Q .

Original: Given two consecutive frames X t and X t +1 .
Revised: Given two consecutive frames X t and X t +1 .

Original: We have Q t =
Revised: We have Q t =

Original: T ( F t , ˆ Q ) , where F t = E ( X t ) , and similarly for Q t +1 .
Revised: T ( F t , ˆ Q ) , where F t = E ( X t ) , and similarly for Q t +1 .

Original: Q ti is the query embedding for instance i .
Revised: Q ti is the query embedding for instance i .

Original: Tracking in MinVIS is done by the assignment of applying the Hungarian algorithm on a score matrix S , where S ij = cos ( Q ti , Q t +1 j ) .
Revised: Tracking in MinVIS is done by the assignment of applying the Hungarian algorithm on a score matrix S , where S ij = cos ( Q ti , Q t +1 j ) .

Original: cos ( · , · ) is the cosine similarity.
Revised: cos ( · , · ) is the cosine similarity.

Original: This approach is less affected by occlusions because each instance is represented by a query that does not have a spatial extent.
Revised: This approach is less affected by occlusions because each instance is represented by a query that does not have a spatial extent.

Original: In addition, we do not need heuristics to handle the birth and death of object instances in this framework.
Revised: In addition, we do not need heuristics to handle the birth and death of object instances in this framework.

Original: Since the number of queries is larger than the number of instances, there are queries that produce empty masks.
Revised: Since the number of queries is larger than the number of instances, there are queries that produce empty masks.

Original: The death of an object instance happens when its embedding is matched to such a null query.
Revised: The death of an object instance happens when its embedding is matched to such a null query.

Original: On the other hand, the birth of an instance is correctly handled if the matched query embeddings have been null before the actual birth of the object instance.
Revised: On the other hand, the birth of an instance is correctly handled if the matched query embeddings have been null before the actual birth of the object instance.

Original: 3.3 Training with Less Supervision for VIS
Revised: 3.3 Training with Less Supervision for VIS

Original: Since the matching process does not need training, only the image instance segmentation model needs to be trained.
Revised: Since the matching process does not need training, only the image instance segmentation model needs to be trained.

Original: There are two outputs of the model: classification scores O ∈ R N,K and segmentation masks M ∈ R N,H,W for N queries, K object classes, and image size H , W .
Revised: There are two outputs of the model: classification scores O ∈ R N,K and segmentation masks M ∈ R N,H,W for N queries, K object classes, and image size H , W .

Original: We can process the groundtruth video instances to groundtruth image instances O ∗ ∈ R L,K and M ∗ ∈ R L,H,W , where
Revised: We can process the groundtruth video instances to groundtruth image instances O ∗ ∈ R L,K and M ∗ ∈ R L,H,W , where

Original: L is the number of groundtruth instances ( N >> L ) and O ∗ j is a one-hot vector of groundtruth class.
Revised: L is the number of groundtruth instances ( N >> L ) and O ∗ j is a one-hot vector of groundtruth class.

Original: Given a loss function L ( O i , M i , O ∗ j , M ∗ j ) between predicted instance i and groundtruth instance j , we first use bipartite matching to find the assignments between predicted and groundtruth instances that minimize the overall loss function, and train on those matched predictions with the loss function.
Revised: Given a loss function L ( O i , M i , O ∗ j , M ∗ j ) between predicted instance i and groundtruth instance j , we first use bipartite matching to find the assignments between predicted and groundtruth instances that minimize the overall loss function, and train on those matched predictions with the loss function.

Original: More specifically, there are two terms in the loss function: L cls and L mask .
Revised: More specifically, there are two terms in the loss function: L cls and L mask .

Original: We use cross entropy loss for L cls and binary cross entropy plus dice loss [32] for L mask as in previous works [30].
Revised: We use cross entropy loss for L cls and binary cross entropy plus dice loss [32] for L mask as in previous works [30].

Original: Both terms are purely image-based.
Revised: Both terms are purely image-based.

Original: The groudtruth video instances are first processed to instances for each frame independently.
Revised: The groudtruth video instances are first processed to instances for each frame independently.

Original: Therefore, even if there are only sparse frames labeled with instance, we can still train our model with the annotated frames.
Revised: Therefore, even if there are only sparse frames labeled with instance, we can still train our model with the annotated frames.

Original: This provide a straightforward way to reduce the supervision for VIS.
Revised: This provide a straightforward way to reduce the supervision for VIS.

Original: Figure 2(b) shows the annotation sub-sampling to reduce supervision.
Revised: Figure 2(b) shows the annotation sub-sampling to reduce supervision.

Original:  Experiments
Revised:  Experiments

Original: Datasets.
Revised: Datasets.

Original: We evaluate MinVIS on three datasets: YouTube-VIS 2019/2021 [1] and Occluded VIS (OVIS) [12].
Revised: We evaluate MinVIS on three datasets: YouTube-VIS 2019/2021 [1] and Occluded VIS (OVIS) [12].

Original: The YouTube-VIS datasets contain 40 object classes.
Revised: The YouTube-VIS datasets contain 40 object classes.

Original: YouTube-VIS 2019 contains 2238/302/343 videos for training/validation/testing, while YouTube-VIS 2021 expands the dataset to 2985/421/453 videos for training/validation/testing, and includes higher quality annotations.
Revised: YouTube-VIS 2019 contains 2238/302/343 videos for training/validation/testing, while YouTube-VIS 2021 expands the dataset to 2985/421/453 videos for training/validation/testing, and includes higher quality annotations.

Original: OVIS has 25 object classes and contains 607/140/154 for training/validation/testing.
Revised: OVIS has 25 object classes and contains 607/140/154 for training/validation/testing.

Original: While the number of videos is smaller, OVIS has more objects per frame, and the videos are also longer.
Revised: While the number of videos is smaller, OVIS has more objects per frame, and the videos are also longer.

Original: This leads to more annotated instance masks compared to the YouTube-VIS datasets.
Revised: This leads to more annotated instance masks compared to the YouTube-VIS datasets.

Original: In addition, OVIS also has much higher Bounding-box Occlusion Rate (0.22 v.s. 0.06/0.07) compared to the YouTube-VIS datasets, which indicates heavier occlusions between object instances.
Revised: In addition, OVIS also has much higher Bounding-box Occlusion Rate (0.22 v.s. 0.06/0.07) compared to the YouTube-VIS datasets, which indicates heavier occlusions between object instances.

Original: Metrics.
Revised: Metrics.

Original: We follow previous works and use Average Precision (AP) and Average Recall (AR) as evaluation metrics [1].
Revised: We follow previous works and use Average Precision (AP) and Average Recall (AR) as evaluation metrics [1].

Original: AP is computed based on 10 intersection-over-union (IoU) thresholds from 50% to 95% with 5% increment.
Revised: AP is computed based on 10 intersection-over-union (IoU) thresholds from 50% to 95% with 5% increment.

Original: The reported AP and AR are first computed for each object class and then averaged over all classes.
Revised: The reported AP and AR are first computed for each object class and then averaged over all classes.

Original: All three datasets have public evaluation servers.
Revised: All three datasets have public evaluation servers.

Original: Baselines.
Revised: Baselines.

Original: We focus on results using ResNet50 and Swin-L backbones.
Revised: We focus on results using ResNet50 and Swin-L backbones.

Original: ResNet50 is still the most widely used backbone for VIS, while Swin-L gives the best performances.
Revised: ResNet50 is still the most widely used backbone for VIS, while Swin-L gives the best performances.

Original: Not all methods report both backbones on all three datasets.
Revised: Not all methods report both backbones on all three datasets.

Original: We include results that are available.
Revised: We include results that are available.

Original: For YouTube-VIS datasets, we include recent state-of-the-art results from SeqFormer [5], TeViT [8], and Mask2Former-VIS [3].
Revised: For YouTube-VIS datasets, we include recent state-of-the-art results from SeqFormer [5], TeViT [8], and Mask2Former-VIS [3].

Original: These are all Transformer-based per-clip approaches as this paradigm has been recently dominating the field.
Revised: These are all Transformer-based per-clip approaches as this paradigm has been recently dominating the field.

Original: On the other, out of of these methods, only TeViT is applied to OVIS.
Revised: On the other, out of of these methods, only TeViT is applied to OVIS.

Original: Therefore, we further compare to CMaskTrack R-CNN [12], CrossVIS [2], and STC [33].
Revised: Therefore, we further compare to CMaskTrack R-CNN [12], CrossVIS [2], and STC [33].

Original: These are all methods that allow online processing.
Revised: These are all methods that allow online processing.

Original: Even TeViT uses a near online inference for OVIS [34].
Revised: Even TeViT uses a near online inference for OVIS [34].

Original: This is because OVIS has longer videos that would lead to out-of-memory for most of the per-clip approaches.
Revised: This is because OVIS has longer videos that would lead to out-of-memory for most of the per-clip approaches.

Original: Out of all the baselines, Mask2Former-VIS [3] is the most related to MinVIS, as MinVIS is built on
Revised: Out of all the baselines, Mask2Former-VIS [3] is the most related to MinVIS, as MinVIS is built on

Original: Mask2Former in this work.
Revised: Mask2Former in this work.

Original: Mask2Former-VIS thus can be seen as the per-clip version of ours and is an important baseline for comparison.
Revised: Mask2Former-VIS thus can be seen as the per-clip version of ours and is an important baseline for comparison.

Original: Therefore, we further apply Mask2Former-VIS on the OVIS dataset.
Revised: Therefore, we further apply Mask2Former-VIS on the OVIS dataset.

Original: Due to memory constraints, the videos in OVIS are first split into clips of length 30.
Revised: Due to memory constraints, the videos in OVIS are first split into clips of length 30.

Original: We use the same post-processing as MinVIS to merge the outputs from these clips.
Revised: We use the same post-processing as MinVIS to merge the outputs from these clips.

Original: Implementation Details.
Revised: Implementation Details.

Original: Unless otherwise noted, our hyper-parameters follow Mask2FormerVIS [3].
Revised: Unless otherwise noted, our hyper-parameters follow Mask2FormerVIS [3].

Original: All models are pre-trained with COCO instance segmentation [35].
Revised: All models are pre-trained with COCO instance segmentation [35].

Original: For OVIS, we use the same hyper-parameters as YouTube-VIS 2019 except training for 10 k iterations instead of 6 k .
Revised: For OVIS, we use the same hyper-parameters as YouTube-VIS 2019 except training for 10 k iterations instead of 6 k .

Original: For training losses, the weights are 5 . 

Revised: For training losses, the weights are 5 .

Original: for L mask and 2 . 

Revised: for L mask and 2 .

Original: for L cls .
Revised: for L cls .

Original: All results of MinVIS are averaged over 3 random seeds.
Revised: All results of MinVIS are averaged over 3 random seeds.

Original: We sub-sample training to X% by uniformly sampling frames in the video.
Revised: We sub-sample training to X% by uniformly sampling frames in the video.

Original: We set a minimum of 1 frame per video.
Revised: We set a minimum of 1 frame per video.

Original: Since YouTube-VIS datasets often have videos less than a hundred frames.
Revised: Since YouTube-VIS datasets often have videos less than a hundred frames.

Original: Our 1% results are better seen as one frame per video results for YouTube-VIS.
Revised: Our 1% results are better seen as one frame per video results for YouTube-VIS.

Original: 4.1 Main Results
Revised: 4.1 Main Results

Original: YouTube-VIS 2019.
Revised: YouTube-VIS 2019.

Original: The results for YouTube-VIS 2019 are shown in Table 1.
Revised: The results for YouTube-VIS 2019 are shown in Table 1.

Original: MinVIS achieves highest AP and most other metrics for both ResNet-50 and Swin-L backbones.
Revised: MinVIS achieves highest AP and most other metrics for both ResNet-50 and Swin-L backbones.

Original: SeqFormer shows that it is beneficial to jointly train with images from COCO [35] that contain YouTubeVIS categories (+C80k in table).
Revised: SeqFormer shows that it is beneficial to jointly train with images from COCO [35] that contain YouTubeVIS categories (+C80k in table).

Original: TeViT proposes messenger shift transformer (MsgShifT) that are as efficient as ResNet backbones, while improving the VIS performances.
Revised: TeViT proposes messenger shift transformer (MsgShifT) that are as efficient as ResNet backbones, while improving the VIS performances.

Original: Our ResNet-50 results match or outperform their results without further modifications.
Revised: Our ResNet-50 results match or outperform their results without further modifications.

Original: Compared to the state-of-the-art Mask2FormerVIS, which can be seen as the per-clip approach to apply Mask2Former to VIS, MinVIS consistently outperforms by around 1% for both backbones.
Revised: Compared to the state-of-the-art Mask2FormerVIS, which can be seen as the per-clip approach to apply Mask2Former to VIS, MinVIS consistently outperforms by around 1% for both backbones.

Original: MinVIS with X% means sub-sampling the annotated frames for each video in training.
Revised: MinVIS with X% means sub-sampling the annotated frames for each video in training.

Original: Since there are less temporal variations for videos in YouTube-VIS 2019, MinVIS with 1% of training frames only reduces AP by 2.6%.
Revised: Since there are less temporal variations for videos in YouTube-VIS 2019, MinVIS with 1% of training frames only reduces AP by 2.6%.

Original: This significantly reduces the annotation effort while not sacrificing much performance.
Revised: This significantly reduces the annotation effort while not sacrificing much performance.

Original: YouTube-VIS 2021.
Revised: YouTube-VIS 2021.

Original: The results for YouTube-VIS 2021 are shown in Table 2.
Revised: The results for YouTube-VIS 2021 are shown in Table 2.

Original: On this more challenging dataset, the performance improvements for MinVIS increase compared to YouTube-VIS 2019.
Revised: On this more challenging dataset, the performance improvements for MinVIS increase compared to YouTube-VIS 2019.

Original: Without better backbone like TeViT and additional training data like SeqFormer, our ResNet50 results outperform by a large margin for all metrics.
Revised: Without better backbone like TeViT and additional training data like SeqFormer, our ResNet50 results outperform by a large margin for all metrics.

Original: This is the also case for Swin-L. MinVIS outperforms previous state-of-the-art Mask2Former-VIS by 2.7%.
Revised: This is the also case for Swin-L. MinVIS outperforms previous state-of-the-art Mask2Former-VIS by 2.7%.

Original: By using only 1% of training frames, MinVIS’s AP decrease by only 2.4%, which means that our 1% result still outperforms
Revised: By using only 1% of training frames, MinVIS’s AP decrease by only 2.4%, which means that our 1% result still outperforms

Original: [Table] [Table]
Revised: [Table] [Table]

Original: -VIS 2021.
Revised: -VIS 2021.

Original: Our 1% results already outperform previous state-of-the-art.
Revised: Our 1% results already outperform previous state-of-the-art.

Original: [Table]
Revised: [Table]

Original: previous state-of-the art.
Revised: previous state-of-the art.

Original: We also see that on YouTube-VIS datasets, reducing the annotations by 10x does not significantly affect the performances (-0.6% AP for 2019 and -0.4% AP for 2021).
Revised: We also see that on YouTube-VIS datasets, reducing the annotations by 10x does not significantly affect the performances (-0.6% AP for 2019 and -0.4% AP for 2021).

Original: Occluded VIS (OVIS).
Revised: Occluded VIS (OVIS).

Original: The results for OVIS are shown in Table 3.
Revised: The results for OVIS are shown in Table 3.

Original: Mask2Former-VIS* denotes our application of Mask2Former-VIS to OVIS.
Revised: Mask2Former-VIS* denotes our application of Mask2Former-VIS to OVIS.

Original: Since videos in OVIS can have up to hundreds of frames, we apply Mask2Former-VIS to non-overlapping sliding windows of length 30.
Revised: Since videos in OVIS can have up to hundreds of frames, we apply Mask2Former-VIS to non-overlapping sliding windows of length 30.

Original: The outputs from these clips are then merged by our post-processing.
Revised: The outputs from these clips are then merged by our post-processing.

Original: MinVIS is an online method and does not need modification to apply to OVIS.
Revised: MinVIS is an online method and does not need modification to apply to OVIS.

Original: MinVIS shows significant improvement compared to existing works on OVIS.
Revised: MinVIS shows significant improvement compared to existing works on OVIS.

Original: With ResNet-50 backbone, MinVIS outperforms previous state-of-the-art TeViT with
Revised: With ResNet-50 backbone, MinVIS outperforms previous state-of-the-art TeViT with

Original: MsgShifT backbone by 7.6% AP.
Revised: MsgShifT backbone by 7.6% AP.

Original: With Swin-L backbone, MinVIS outperforms previous best result
Revised: With Swin-L backbone, MinVIS outperforms previous best result

Original: MaskTrack R-CNN*+SWA by 10.5% AP, which is the winner of the 1st OVIS Challenge.
Revised: MaskTrack R-CNN*+SWA by 10.5% AP, which is the winner of the 1st OVIS Challenge.

Original: Their key observation is that sampling frames that are far apart in OVIS leads to drastically different features and makes it hard to train MaskTrack R-CNN.
Revised: Their key observation is that sampling frames that are far apart in OVIS leads to drastically different features and makes it hard to train MaskTrack R-CNN.

Original: This is in contrast to YouTube-VIS datasets, in which the videos are shorter and there are less temporal variations within the video.
Revised: This is in contrast to YouTube-VIS datasets, in which the videos are shorter and there are less temporal variations within the video.

Original: We observe the same phenomenon when training Mask2Former-VIS*.
Revised: We observe the same phenomenon when training Mask2Former-VIS*.

Original: However, the limited sampling reference frame strategy of MaskTrack R-CNN*+SWA still does not work in this case.
Revised: However, the limited sampling reference frame strategy of MaskTrack R-CNN*+SWA still does not work in this case.

Original: Mask2Former-VIS* uses a fully end-to-end loss instead of an explicit tracking loss to learn temporal association, which makes the learning even harder in OVIS.
Revised: Mask2Former-VIS* uses a fully end-to-end loss instead of an explicit tracking loss to learn temporal association, which makes the learning even harder in OVIS.

Original: On the other hand, MinVIS is image-based and does not need to worry about the temporal sampling strategy to train the model.
Revised: On the other hand, MinVIS is image-based and does not need to worry about the temporal sampling strategy to train the model.

Original: This is contrary to common belief that per-frame approaches are worse for scenarios with heavy occlusions.
Revised: This is contrary to common belief that per-frame approaches are worse for scenarios with heavy occlusions.

Original: Instead, our image-based approach leads to easier and better learning on OVIS.
Revised: Instead, our image-based approach leads to easier and better learning on OVIS.

Original: We show that an image instance segmentation model that can segment occluded instances in each frame is also good at associating such instances
Revised: We show that an image instance segmentation model that can segment occluded instances in each frame is also good at associating such instances

Original: [Table]
Revised: [Table]

Original: I S
Revised: I S

Original: [Table]
Revised: [Table]

Original: H
Revised: H

Original: [Table]
Revised: [Table]

Original: u r
Revised: u r

Original: i s t
Revised: i s t

Original: i c s
Revised: i c s

Original: M i n V
Revised: M i n V

Original: I S
Revised: I S

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: on
Revised: on

Original: MinVIS tracks in
Revised: MinVIS tracks in

Original: Using mask overlap based instead leads to multiple in
Revised: Using mask overlap based instead leads to multiple in

Original: Mask2Former
Revised: Mask2Former

Original: * uses per training that is difficult to optimize on the challenging OVIS across frames.
Revised: * uses per training that is difficult to optimize on the challenging OVIS across frames.

Original: Figure 4 shows qualitative results.
Revised: Figure 4 shows qualitative results.

Original: MinVIS stably tracks all the sheep in the video.
Revised: MinVIS stably tracks all the sheep in the video.

Original: Using mask overlap based heuristics instead of query matching leads to multiple identity switches in tracking.
Revised: Using mask overlap based heuristics instead of query matching leads to multiple identity switches in tracking.

Original: Mask2Former-VIS* does not have as good segmentation masks because its training is interfered by heavy occlusions and large appearance deformations between frames in OVIS.
Revised: Mask2Former-VIS* does not have as good segmentation masks because its training is interfered by heavy occlusions and large appearance deformations between frames in OVIS.

Original: Figure 5 shows additional qualitative results on failure cases of MinVIS on the OVIS dataset.
Revised: Figure 5 shows additional qualitative results on failure cases of MinVIS on the OVIS dataset.

Original: As discussed in Section 3.2, MinVIS does not use heuristics to handle the birth and death of object instances.
Revised: As discussed in Section 3.2, MinVIS does not use heuristics to handle the birth and death of object instances.

Original: The death of an object instance is correctly handled if its query is matched to a query in the next frame that produces an empty mask.
Revised: The death of an object instance is correctly handled if its query is matched to a query in the next frame that produces an empty mask.

Original: Despite its simplicity and effectiveness, the drawback of this approach is that there is nothing stopping the model from matching the disappearing query to a query with a non-empty mask.
Revised: Despite its simplicity and effectiveness, the drawback of this approach is that there is nothing stopping the model from matching the disappearing query to a query with a non-empty mask.

Original: From t 3 to t 4 in the top row of Figure 5, as the fish in the lower left leaves the frame, MinVIS associates it to a mask covering the tail of a nearby fish.
Revised: From t 3 to t 4 in the top row of Figure 5, as the fish in the lower left leaves the frame, MinVIS associates it to a mask covering the tail of a nearby fish.

Original: From t 4 to t 5 , when the fish in the upper left leaves the frame, MinVIS again associates it to a mask covering the head of a nearby fish.
Revised: From t 4 to t 5 , when the fish in the upper left leaves the frame, MinVIS again associates it to a mask covering the head of a nearby fish.

Original: Since the associated masks are non-empty, MinVIS fails to correctly handle the death of these instances.
Revised: Since the associated masks are non-empty, MinVIS fails to correctly handle the death of these instances.

Original: On the other hand, when the two dogs in the bottom row of
Revised: On the other hand, when the two dogs in the bottom row of

Original: Figure 5 are covered in t 2 , MinVIS correctly associates their queries to empty masks.
Revised: Figure 5 are covered in t 2 , MinVIS correctly associates their queries to empty masks.

Original: MinVIS further correctly handles the object births in t 3 .
Revised: MinVIS further correctly handles the object births in t 3 .

Original: However, MinVIS is limited by the segmentation of the image segmentation model, which fails to segment the close-up person.
Revised: However, MinVIS is limited by the segmentation of the image segmentation model, which fails to segment the close-up person.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: 4.2 Analyzing Query Matching
Revised: 4.2 Analyzing Query Matching

Original: The success of MinVIS depends on whether query matching is good for tracking instances.
Revised: The success of MinVIS depends on whether query matching is good for tracking instances.

Original: We conduct ablation studies by comparing it to manually designed heuristics.
Revised: We conduct ablation studies by comparing it to manually designed heuristics.

Original: We use the bipartite matching heuristics in Section 3.3 for tracking by treating instances in the last frame as groundtruth.
Revised: We use the bipartite matching heuristics in Section 3.3 for tracking by treating instances in the last frame as groundtruth.

Original: The results are in Table 4.
Revised: The results are in Table 4.

Original: Using heuristics lead to around 3% AP drop on both YouTube-VISand 2021.
Revised: Using heuristics lead to around 3% AP drop on both YouTube-VISand 2021.

Original: It leads to more significant drop on OVIS (7.7%) due to heavier occlusions.
Revised: It leads to more significant drop on OVIS (7.7%) due to heavier occlusions.

Original: We also combine query matching and heuristics with equal weights, which has mixed results.
Revised: We also combine query matching and heuristics with equal weights, which has mixed results.

Original: Our query only approach is simpler and more generalizable without loss of performance.
Revised: Our query only approach is simpler and more generalizable without loss of performance.

Original: We visualize the learned query embeddings by t-SNE [36] in Figure 3.
Revised: We visualize the learned query embeddings by t-SNE [36] in Figure 3.

Original: Each plot is for a video in the training set.
Revised: Each plot is for a video in the training set.

Original: We visualize the training set to see the effect of an image only objective (to segment instances in an image) on query embeddings across different frames.
Revised: We visualize the training set to see the effect of an image only objective (to segment instances in an image) on query embeddings across different frames.

Original: Query embeddings of the same instance have the same color.
Revised: Query embeddings of the same instance have the same color.

Original: We obtain the instance IDs for queries by bipartite matching its outputs to groundtruth instances, which have consistent IDs across frames.
Revised: We obtain the instance IDs for queries by bipartite matching its outputs to groundtruth instances, which have consistent IDs across frames.

Original: Without any video-based tracking objective, query embeddings of the same instances are already grouped into distinct clusters, even for the OVIS dataset.
Revised: Without any video-based tracking objective, query embeddings of the same instances are already grouped into distinct clusters, even for the OVIS dataset.

Original: This supports our design of only using image-based objectives.
Revised: This supports our design of only using image-based objectives.

Original: In Appendix E, we further visualize query embeddings on videos not used in training.
Revised: In Appendix ?? , we further visualize query embeddings on videos not used in training.

Original: 4.3 Effect of Video-based Training
Revised: 4.3 Effect of Video-based Training

Original: While we have shown that MinVIS achieves state-of-the-art VIS performance without video-based training, it is interesting to see how we can leverage video annotation when it is available.
Revised: While we have shown that MinVIS achieves state-of-the-art VIS performance without video-based training, it is interesting to see how we can leverage video annotation when it is available.

Original: We use the video annotation to supervise our matching as in previous per-frame works [1, 2, 11].
Revised: We use the video annotation to supervise our matching as in previous per-frame works [1, 2, 11].

Original: Given two sampled frames, we use a hinge loss to ensure that the correct associations of queries have the highest inner products compared to that of other queries between the two frames [11].
Revised: Given two sampled frames, we use a hinge loss to ensure that the correct associations of queries have the highest inner products compared to that of other queries between the two frames [11].

Original: The results are
Revised: The results are

Original: [Table]
Revised: [Table]

Original: in Table 5.
Revised: in Table 5.

Original: The “Supervised Matching” rows mean directly applying the matching supervision to the original frame sampling process.
Revised: The “Supervised Matching” rows mean directly applying the matching supervision to the original frame sampling process.

Original: In our case, this means that the two sampled frames might be separated up to 20 frames.
Revised: In our case, this means that the two sampled frames might be separated up to 20 frames.

Original: As pointed out in previous works, frames that are far separated increase the training difficulty and can hurt model performances especially with occlusions [14].
Revised: As pointed out in previous works, frames that are far separated increase the training difficulty and can hurt model performances especially with occlusions [14].

Original: We thus also consider the “Limited Range” training to only sample consecutive frames for supervised matching, as we only need to match consecutive frames.
Revised: We thus also consider the “Limited Range” training to only sample consecutive frames for supervised matching, as we only need to match consecutive frames.

Original: From the results, directly applying “Supervised Matching” hurt performances on all three datasets.
Revised: From the results, directly applying “Supervised Matching” hurt performances on all three datasets.

Original: Adding “Limited Range” recovers most of the performances for YouTube-VIS 2021 and OVIS.
Revised: Adding “Limited Range” recovers most of the performances for YouTube-VIS 2021 and OVIS.

Original: On OVIS, it even marginally outperforms the original MinVIS.
Revised: On OVIS, it even marginally outperforms the original MinVIS.

Original: However, this improvement does rely on the dataset dependent sampling range.
Revised: However, this improvement does rely on the dataset dependent sampling range.

Original: We believe it is possible and important to use video-based training to further improve MinVIS, although this would take away MinVIS’s practical advantages of only needing sparse annotations and having a simple training pipeline.
Revised: We believe it is possible and important to use video-based training to further improve MinVIS, although this would take away MinVIS’s practical advantages of only needing sparse annotations and having a simple training pipeline.

Original: Appendix A discusses further limitations of not using video information in training.
Revised: Appendix ?? discusses further limitations of not using video information in training.

Original:  Conclusion
Revised:  Conclusion

Original: We show that a purely image-based training procedure can lead to competitive performances for
Revised: We show that a purely image-based training procedure can lead to competitive performances for

Original: VIS.
Revised: VIS.

Original: Our key finding is that instance tracking naturally emerges in query-based image instance segmentation models with proper architectural constraints.
Revised: Our key finding is that instance tracking naturally emerges in query-based image instance segmentation models with proper architectural constraints.

Original: In addition to improving state-of-the-art approaches on YouTube-VIS 2019/2021, we show that this is particularly beneficial for OVIS.
Revised: In addition to improving state-of-the-art approaches on YouTube-VIS 2019/2021, we show that this is particularly beneficial for OVIS.

Original: The image-based objective reduces the learning difficulty and leads to better performances.
Revised: The image-based objective reduces the learning difficulty and leads to better performances.

Original: MinVIS only requires sparse frame annotations, which makes it much more applicable to real-world scenarios.
Revised: MinVIS only requires sparse frame annotations, which makes it much more applicable to real-world scenarios.

Original: We believe a promising direction to extend MinVIS is to explore ways to better leverage the video frames that are not annotated to further improve our performances with sub-sampled annotations.
Revised: We believe a promising direction to extend MinVIS is to explore ways to better leverage the video frames that are not annotated to further improve our performances with sub-sampled annotations.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Section 4.3 and the appendix.
Revised: [Yes] See Section 4.3 and the appendix.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See the appendix.
Revised: [Yes] See the appendix.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See link in abstract.
Revised: [Yes] See link in abstract.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] Tables with error bars are in the appendix.
Revised: [Yes] Tables with error bars are in the appendix.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?

Original: [No]4.
Revised: [No]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: All three datasets are public and cited.
Revised: All three datasets are public and cited.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See the appendix.
Revised: [Yes] See the appendix.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] See the appendix.
Revised: [Yes] See the appendix.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [Yes] See the appendix.
Revised: [Yes] See the appendix.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?[N/A] A Limitations and Potential Negative Social Impacts Limitation.
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: We have discussed in the main paper on the possibility of improving MinVIS with video-based training.

Original: While we believe there are practical advantages of using our image-based VIS training pipeline, videos provide lots of extra information that we are not currently leveraging.

Original: In particular, temporal supervision from video should make our query embeddings even more suitable for tracking instances.

Original: In Figure 5, we visualize failure cases of using our current query embeddings for tracking.

Original: We conduct further analysis of Supervised Matching in Appendix F and believe further investigation along this direction should improve our approach.

Original: In addition to improving fullysupervised performance, we believe a promising direction is to explore semi-supervised learning at the frame level.

Original: In this case, one can temporally propagate the sub-sampled annotations in training to further improve the performance with reduced supervision.

Original: Potential Negative Social Impacts.

Original: Video instance segmentation is a challenging video task, and thus provides fine-grained understanding of videos.

Original: The tracking and segmentation of objects of interest might be use for surveillance applications with negative social impact.

Original: While “person” is a category in the datasets used in this paper, no further protected attributes are annotated.

Original: Therefore, our trained models’ performance on human subjects might not be fair with respect to protected attributes.

Original: B Further Details for Datasets

Original: The YouTube-VIS 2019/2021 datasets are under CC BY 4.0 License, and Occluded VIS is under CC

Original: BY-NC-SA 4.0 License.

Original: The videos in YouTube-VIS are from YouTube-VOS [37], whose videos are in turn from YouTube-8M [38].

Original: YouTube-8M uses public videos on YouTube but does not discuss the process to filter personally identifiable information or offensive content in the paper.

Original: C Tables with Standard Deviation

Original: Tables with standard deviations are shown in Table 6, Table 7, and Table 8.

Original: [Table]

Original: [Table]

Original: [Table]

Original: D Reducing Supervision for Mask2Former-VIS

Original: The results for sub-sampling annotated frames for Mask2Former-VIS [3] are shown in Table 9.

Original: MinVIS consistently outperforms Mask2Former-VIS in all settings.

Original: The improvement increases for all three datasets when we sub-sample the annotation:

Original: +1.2% for full supervision v.s. +1.7% for 1% supervision on YouTube-VIS 2019.

Original: +2.7% for full supervision v.s.

Original: +5.8% for 1% supervision on

Original: YouTube-VIS 2021.

Original: +13.6% for full supervision v.s. +17.2% for 1% supervision on OVIS.

Original: E Visualizing Query Embeddings in Evaluation

Original: In the main paper, we visualize the learned query embeddings by t-SNE [36] in Figure 3.

Original: The videos in Figure 3 are in the training set and the figure is meant to understand how query embeddings in training cluster by instances without video-based loss function.

Original: We can similarly apply the same

Original: [Table]
Revised: [N/A]

Original: visualization to videos that are not used in training.

Original: One complication here is that this visualization uses groundtruth instance annotations to determine the corresponding instance ID for each query.

Original: However, the groundtruth annotation is not publicly available for the three datasets considered in this work.

Original: Our reported results are obtained by submitting our predictions to the datasets’ evaluation servers.

Original: We therefore perform this analysis by training a new model that only uses 90% of the training videos in YouTube-VIS 2019, and visualize the learned model’s query embeddings during evaluation on the 10% videos that are not used to train the model.

Original: While the 10% videos are not used in training the model, we still have their groundtruth instances for visualization purposes.

Original: This provides a realistic approximation of how our query embeddings would look like for videos not used in training.

Original: The visualization is in Figure 6.

Original: Despite being noisier than training videos, the query embeddings are still grouped into clusters by object instances without any video-based training.

Original: This is also quantitatively supported by our state-of-the-art VIS performance on the three datasets.

Original: F Further Analysis of Supervised Matching

Original: We conduct further analysis on the results in Section 4.3.

Original: We visualize the query embeddings on the same training videos with and without using supervised matching.

Original: In particular, we perform the analysis on YouTube-VIS 2019 and compare MinVIS v.s. MinVIS + Supervised Matching + Limited

Original: Range, which hurts performance the most in Table 5.

Original: The visualizations are in Figure 7.

Original: While the plots look similar for most videos, one consistent trend we observe is that adding supervised matching makes the embeddings more evenly distributed and smooths out the outliers in the embedding space.

Original: This is a reasonable consequence as the objective encourages the embeddings from the same object instance to be closer to each other.

Original: However, it is unclear whether this is overall beneficial to our tracking by query matching.

Original: For example, in V 3 , the outliers are removed at the cost of mixing embeddings from different instances.

Original: We believe it is an important future work to further understand how to better leverage video information to improve MinVIS.

Original: M i n V

Original: I

Original: S + S up e r v

Original: i s e d M a t c h 𝑉 "

Original: [Figure]

Original: G Baseline Training Curves on OVIS

Original: As discussed in the main paper, it is difficult to optimize our per-clip baseline on the challenging

Original: OVIS dataset.

Original: We have included the training curves in Figure 8 for further illustration.

Original: Blue curves are MinVIS and orange curves are Mask2Former-VIS.

Original: While the classification loss still optimizes well on OVIS, the per-clip baseline has difficulty optimizing mask related losses.