Original: U NSUPERVISED S EMANTIC S EGMENTATION BY D ISTILLING F EATURE C ORRESPONDENCES
Revised: U NSUPERVISED S EMANTIC S EGMENTATION BY D ISTILLING F EATURE C ORRESPONDENCES

Original: A BSTRACT
Revised: A BSTRACT

Original: Unsupervised semantic segmentation aims to discover and localize semantically meaningful categories within image corpora without any form of annotation.
Revised: Unsupervised semantic segmentation aims to discover and localize semantically meaningful categories within image corpora without any form of annotation.

Original: To solve this task, algorithms must produce features for every pixel that are both semantically meaningful and compact enough to form distinct clusters.
Revised: To solve this task, algorithms must produce features for every pixel that are both semantically meaningful and compact enough to form distinct clusters.

Original: Unlike previous works which achieve this with a single end-to-end framework, we propose to separate feature learning from cluster compactiﬁcation.
Revised: Unlike previous works which achieve this with a single end-to-end framework, we propose to separate feature learning from cluster compactiﬁcation.

Original: Empirically, we show that current unsupervised feature learning frameworks already generate dense features whose correlations are semantically consistent.
Revised: Empirically, we show that current unsupervised feature learning frameworks already generate dense features whose correlations are semantically consistent.

Original: This observation motivates us to design STEGO ( S elf-supervised T ransformer with E nergy-based G raph
Revised: This observation motivates us to design STEGO ( S elf-supervised T ransformer with E nergy-based G raph

Original: O ptimization), a novel framework that distills unsupervised features into highquality discrete semantic labels.
Revised: O ptimization), a novel framework that distills unsupervised features into highquality discrete semantic labels.

Original: At the core of STEGO is a novel contrastive loss function that encourages features to form compact clusters while preserving their relationships across the corpora.
Revised: At the core of STEGO is a novel contrastive loss function that encourages features to form compact clusters while preserving their relationships across the corpora.

Original: STEGO yields a signiﬁcant improvement over the prior state of the art, on both the CocoStuff ( +14 mIoU ) and Cityscapes ( + mIoU ) semantic segmentation challenges
Revised: STEGO yields a signiﬁcant improvement over the prior state of the art, on both the CocoStuff ( +14 mIoU ) and Cityscapes ( + mIoU ) semantic segmentation challenges

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Semantic segmentation is the process of classifying each individual pixel of an image into a known ontology.
Revised: Semantic segmentation is the process of classifying each individual pixel of an image into a known ontology.

Original: Though semantic segmentation models can detect and delineate objects at a much ﬁner granularity than classiﬁcation or object detection systems, these systems are hindered by the difﬁculties of creating labelled training data.
Revised: Though semantic segmentation models can detect and delineate objects at a much ﬁner granularity than classiﬁcation or object detection systems, these systems are hindered by the difﬁculties of creating labelled training data.

Original: In particular, segmenting an image can take over 100 × more effort for a human annotator than classifying or drawing bounding boxes (Zlateski et al., 2018).
Revised: In particular, segmenting an image can take over 100 × more effort for a human annotator than classifying or drawing bounding boxes (Zlateski et al., 2018).

Original: Furthermore, in complex domains such as medicine, biology, or astrophysics, ground-truth segmentation labels may be unknown, ill-deﬁned, or require considerable domain-expertise to provide (Yu et al., 2018).
Revised: Furthermore, in complex domains such as medicine, biology, or astrophysics, ground-truth segmentation labels may be unknown, ill-deﬁned, or require considerable domain-expertise to provide (Yu et al., 2018).

Original: Recently, several works introduced semantic segmentation systems that could learn from weaker forms of labels such as classes, tags, bounding boxes, scribbles, or point annotations (Ren et al., 2020; Pan et al., 2021; Liu et al., 2020; Bilen et al.).
Revised: Recently, several works introduced semantic segmentation systems that could learn from weaker forms of labels such as classes, tags, bounding boxes, scribbles, or point annotations (Ren et al., 2020; Pan et al., 2021; Liu et al., 2020; Bilen et al.).

Original: However, comparatively few works take up the challenge of semantic segmentation without any form of human supervision or motion cues.
Revised: However, comparatively few works take up the challenge of semantic segmentation without any form of human supervision or motion cues.

Original: Attempts such as Independent Information Clustering (IIC)
Revised: Attempts such as Independent Information Clustering (IIC)

Original: (Ji et al., 2019) and PiCIE (Cho et al., 2021) aim to learn semantically meaningful features through transformation equivariance, while imposing a clustering step to improve the compactness of the learned features.
Revised: (Ji et al., 2019) and PiCIE (Cho et al., 2021) aim to learn semantically meaningful features through transformation equivariance, while imposing a clustering step to improve the compactness of the learned features.

Original: In contrast to these previous methods, we utilize pre-trained features from unsupervised feature learning frameworks and focus on distilling them into a compact and discrete structure while preserving their relationships across the image corpora.
Revised: In contrast to these previous methods, we utilize pre-trained features from unsupervised feature learning frameworks and focus on distilling them into a compact and discrete structure while preserving their relationships across the image corpora.

Original: This is motivated by the observation that correlations between unsupervised features, such as ones learned by DINO (Caron et al., 2021), are already semantically consistent, both within the same image and across image collections.
Revised: This is motivated by the observation that correlations between unsupervised features, such as ones learned by DINO (Caron et al., 2021), are already semantically consistent, both within the same image and across image collections.

Original: As a result, we introduce STEGO ( S elf-supervised T ransformer with E nergy-based G raph
Revised: As a result, we introduce STEGO ( S elf-supervised T ransformer with E nergy-based G raph

Original: O ptimization), which is capable of jointly discovering and segmenting objects without human supervision.
Revised: O ptimization), which is capable of jointly discovering and segmenting objects without human supervision.

Original: STEGO distills pretrained unsupervised visual features into semantic clusters using a novel contrastive loss.
Revised: STEGO distills pretrained unsupervised visual features into semantic clusters using a novel contrastive loss.

Original: STEGO dramatically improves over prior art and is a considerable step towards closing the gap with supervised segmentation systems.
Revised: STEGO dramatically improves over prior art and is a considerable step towards closing the gap with supervised segmentation systems.

Original: We include a short video detailing the work at https://aka.ms/stego-video .
Revised: We include a short video detailing the work at https://aka.ms/stego-video .

Original: Speciﬁcally, we make the following contributions:
Revised: Speciﬁcally, we make the following contributions:

Original: • Show that unsupervised deep network features have correlation patterns that are largely consistent with true semantic labels.
Revised: • Show that unsupervised deep network features have correlation patterns that are largely consistent with true semantic labels.

Original: • Introduce STEGO, a novel transformer-based architecture for unsupervised semantic segmentation.
Revised: • Introduce STEGO, a novel transformer-based architecture for unsupervised semantic segmentation.

Original: • Demonstrate that STEGO achieves state of the art performance on both the CocoStuff ( + mIoU ) and Cityscapes ( +9 mIoU ) segmentation challenges.
Revised: • Demonstrate that STEGO achieves state of the art performance on both the CocoStuff ( + mIoU ) and Cityscapes ( +9 mIoU ) segmentation challenges.

Original: • Justify STEGO’s design with an ablation study on the CocoStuff dataset.
Revised: • Justify STEGO’s design with an ablation study on the CocoStuff dataset.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Self-supervised Visual Feature Learning Learning meaningful visual features without human annotations is a longstanding goal of computer vision.
Revised: Self-supervised Visual Feature Learning Learning meaningful visual features without human annotations is a longstanding goal of computer vision.

Original: Approaches to this problem often optimize a surrogate task, such as denoising (Vincent et al., 2008), inpainting (Pathak et al., 2016), jigsaw puzzles, colorization (Zhang et al., 2017), rotation prediction (Gidaris et al., 2018), and most recently, contrastive learning over multiple augmentations (Hjelm et al., 2018; Chen et al., 2020a;a;c; Oord et al., 2018).
Revised: Approaches to this problem often optimize a surrogate task, such as denoising (Vincent et al., 2008), inpainting (Pathak et al., 2016), jigsaw puzzles, colorization (Zhang et al., 2017), rotation prediction (Gidaris et al., 2018), and most recently, contrastive learning over multiple augmentations (Hjelm et al., 2018; Chen et al., 2020a;a;c; Oord et al., 2018).

Original: Contrastive learning approaches, whose performance surpass all other surrogate tasks, assume visual features are invariant under a certain set of image augmentation operations.
Revised: Contrastive learning approaches, whose performance surpass all other surrogate tasks, assume visual features are invariant under a certain set of image augmentation operations.

Original: These approaches maximize feature similarities between an image and its augmentations, while minimizing similarity between negative samples, which are usually randomly sampled images.
Revised: These approaches maximize feature similarities between an image and its augmentations, while minimizing similarity between negative samples, which are usually randomly sampled images.

Original: Some notable examples of positive pairs include temporally adjacent images in videos (Oord et al., 2018), image augmentations (Chen et al., 2020a;c), and local crops of a single image (Hjelm et al., 2018).
Revised: Some notable examples of positive pairs include temporally adjacent images in videos (Oord et al., 2018), image augmentations (Chen et al., 2020a;c), and local crops of a single image (Hjelm et al., 2018).

Original: Many works highlight the importance of large numbers of negative samples during training.
Revised: Many works highlight the importance of large numbers of negative samples during training.

Original: To this end Wu et al.
Revised: To this end Wu et al.

Original: (2018) propose keeping a memory bank of negative samples and Chen et al.
Revised: (2018) propose keeping a memory bank of negative samples and Chen et al.

Original: (2020c)
Revised: (2020c)

Original: propose momentum updates that can efﬁciently simulate large negative batch sizes.
Revised: propose momentum updates that can efﬁciently simulate large negative batch sizes.

Original: Recently some works have aimed to produce spatially dense feature maps as opposed to a single global vector per image.
Revised: Recently some works have aimed to produce spatially dense feature maps as opposed to a single global vector per image.

Original: In this vein, VADeR (Pinheiro et al., 2020) contrasts local per-pixel features based on random compositions of image transformations that induce known correspondences among pixels which act as positive pairs for contrastive training.
Revised: In this vein, VADeR (Pinheiro et al., 2020) contrasts local per-pixel features based on random compositions of image transformations that induce known correspondences among pixels which act as positive pairs for contrastive training.

Original: Instead of trying to learn visual features and clustering from scratch, STEGO treats pretrained self-supervised features as input and is agnostic to the underlying feature extractor.
Revised: Instead of trying to learn visual features and clustering from scratch, STEGO treats pretrained self-supervised features as input and is agnostic to the underlying feature extractor.

Original: This makes it easy to integrate future advances in self-supervised feature learning into STEGO.
Revised: This makes it easy to integrate future advances in self-supervised feature learning into STEGO.

Original: Unsupervised Semantic Segmentation
Revised: Unsupervised Semantic Segmentation

Original: Many unsupervised semantic segmentation approaches use techniques from self-supervised feature learning.
Revised: Many unsupervised semantic segmentation approaches use techniques from self-supervised feature learning.

Original: IIC (Ji et al., 2019) maximizes mutual information of patch-level cluster assignments between an image and its augmentations.
Revised: IIC (Ji et al., 2019) maximizes mutual information of patch-level cluster assignments between an image and its augmentations.

Original: Contrastive
Revised: Contrastive

Original: Clustering (Li et al., 2020), and SCAN (Van Gansbeke et al., 2020) improve on IIC’s image clustering results with supervision from negative samples and nearest neighbors but do not attempt semantic segmentation.
Revised: Clustering (Li et al., 2020), and SCAN (Van Gansbeke et al., 2020) improve on IIC’s image clustering results with supervision from negative samples and nearest neighbors but do not attempt semantic segmentation.

Original: PiCIE (Cho et al., 2021) improves on IIC’s semantic segmentation results by using invariance to photometric effects and equivariance to geometric transformations as an inductive bias.
Revised: PiCIE (Cho et al., 2021) improves on IIC’s semantic segmentation results by using invariance to photometric effects and equivariance to geometric transformations as an inductive bias.

Original: In PiCIE, a network minimizes the distance between features under different transformations, where the distance is deﬁned by an in-the-loop k-means clustering process.
Revised: In PiCIE, a network minimizes the distance between features under different transformations, where the distance is deﬁned by an in-the-loop k-means clustering process.

Original: SegSort (Hwang et al., 2019) adopts a different approach.
Revised: SegSort (Hwang et al., 2019) adopts a different approach.

Original: First, SegSort learns good features using superpixels as proxy segmentation maps, then uses Expectation-Maximization to iteratively reﬁne segments over a spherical embedding space.
Revised: First, SegSort learns good features using superpixels as proxy segmentation maps, then uses Expectation-Maximization to iteratively reﬁne segments over a spherical embedding space.

Original: In a similar vein, MaskContrast (Van Gansbeke et al., 2021) achieves promising results on PascalVOC by ﬁrst using an off-the-shelf saliency model to generate a binary mask for each image.
Revised: In a similar vein, MaskContrast (Van Gansbeke et al., 2021) achieves promising results on PascalVOC by ﬁrst using an off-the-shelf saliency model to generate a binary mask for each image.

Original: MaskContrast then contrasts learned features within and across the saliency masks.
Revised: MaskContrast then contrasts learned features within and across the saliency masks.

Original: In contrast, our method focuses reﬁning existing pretrained self-supervised visual features to distill their correspondence information and encourage cluster formation.
Revised: In contrast, our method focuses reﬁning existing pretrained self-supervised visual features to distill their correspondence information and encourage cluster formation.

Original: This is similar to the work of
Revised: This is similar to the work of

Original: Collins et al.
Revised: Collins et al.

Original: (2018) who show that low rank factorization of deep network features can be useful for unsupervised co-segmentation.
Revised: (2018) who show that low rank factorization of deep network features can be useful for unsupervised co-segmentation.

Original: We are not aware of any previous work that achieves the goal of high-quality, pixel-level unsupervised semantic segmentation on large scale datasets with diverse images.
Revised: We are not aware of any previous work that achieves the goal of high-quality, pixel-level unsupervised semantic segmentation on large scale datasets with diverse images.

Original: Visual Transformers Convolutional neural networks (CNNs) have long been state of the art for many computer vision tasks, but the nature of the convolution operator makes it hard to model longrange interactions.
Revised: Visual Transformers Convolutional neural networks (CNNs) have long been state of the art for many computer vision tasks, but the nature of the convolution operator makes it hard to model longrange interactions.

Original: To circumvent such shortcomings, Wang et al.
Revised: To circumvent such shortcomings, Wang et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2019) use self-attention operations within a CNN to model long range interactions.
Revised: (2019) use self-attention operations within a CNN to model long range interactions.

Original: Transformers (Vaswani et al., 2017), or purely self-attentive networks, have made signiﬁcant progress in NLP and have recently been used for many computer vision tasks (Dosovitskiy et al., 2020; Touvron et al., 2021;
Revised: Transformers (Vaswani et al., 2017), or purely self-attentive networks, have made signiﬁcant progress in NLP and have recently been used for many computer vision tasks (Dosovitskiy et al., 2020; Touvron et al., 2021;

Original: Ranftl et al., 2021; Caron et al., 2021).
Revised: Ranftl et al., 2021; Caron et al., 2021).

Original: Visual Transformers (ViT) (Vaswani et al., 2017) apply self-attention mechanisms to image patches and positional embeddings in order to generate features and predictions.
Revised: Visual Transformers (ViT) (Vaswani et al., 2017) apply self-attention mechanisms to image patches and positional embeddings in order to generate features and predictions.

Original: Several modiﬁcations of ViT have been proposed to improve supervised learning, unsupervised learning, multi-scale processing, and dense predictions.
Revised: Several modiﬁcations of ViT have been proposed to improve supervised learning, unsupervised learning, multi-scale processing, and dense predictions.

Original: In particular, DINO (Caron et al., 2021) uses a ViT within a self-supervised learning framework that performs self-distillation with exponential moving average updates.
Revised: In particular, DINO (Caron et al., 2021) uses a ViT within a self-supervised learning framework that performs self-distillation with exponential moving average updates.

Original: Caron et al.
Revised: Caron et al.

Original: (2021) show that DINO’s class-attention can produce localized and semantically meaningful salient object segmentations.
Revised: (2021) show that DINO’s class-attention can produce localized and semantically meaningful salient object segmentations.

Original: Our work shows that
Revised: Our work shows that

Original: DINO’s features not only detect salient objects but can be used to extract dense and semantically meaningful correspondences between images.
Revised: DINO’s features not only detect salient objects but can be used to extract dense and semantically meaningful correspondences between images.

Original: In STEGO, we reﬁne the features of this pre-trained backbone to yield semantic segmentation predictions when clustered.
Revised: In STEGO, we reﬁne the features of this pre-trained backbone to yield semantic segmentation predictions when clustered.

Original: We focus on DINO’s embeddings because of their quality but note that STEGO can work with any deep network features.
Revised: We focus on DINO’s embeddings because of their quality but note that STEGO can work with any deep network features.

Original:  M ETHODS 3. F EATURE C ORRESPONDENCES P REDICT C LASS C O -O CCURRENCE
Revised:  M ETHODS 3. F EATURE C ORRESPONDENCES P REDICT C LASS C O -O CCURRENCE

Original: Recent progress in self-supervised visual feature learning has yielded methods with powerful and semantically relevant features that improve a variety of downstream tasks.
Revised: Recent progress in self-supervised visual feature learning has yielded methods with powerful and semantically relevant features that improve a variety of downstream tasks.

Original: Though most works aim to generate a single vector for an image, many works show that intermediate dense features are semantically relevant (Hamilton et al., 2021;
Revised: Though most works aim to generate a single vector for an image, many works show that intermediate dense features are semantically relevant (Hamilton et al., 2021;

Original: Collins et al., 2018;
Revised: Collins et al., 2018;

Original: Zhou et al., 2016).
Revised: Zhou et al., 2016).

Original: To use this information, we focus on the “correlation volume” (Teed & Deng, 2020) between the dense feature maps.
Revised: To use this information, we focus on the “correlation volume” (Teed & Deng, 2020) between the dense feature maps.

Original: For convolutional or transformer architectures, these dense feature maps can be the activation map of a speciﬁc layer.
Revised: For convolutional or transformer architectures, these dense feature maps can be the activation map of a speciﬁc layer.

Original: Additionally, the Q, K or V matrices in transformers can also serve as candidate features, though we ﬁnd these attention tensors do not perform as well in practice.
Revised: Additionally, the Q, K or V matrices in transformers can also serve as candidate features, though we ﬁnd these attention tensors do not perform as well in practice.

Original: More formally, let f ∈ R CHW , g ∈ R CIJ be the feature tensors for two different images where C represents the channel dimension and ( H, W ) , ( I, J ) represent spatial dimensions.
Revised: More formally, let f ∈ R CHW , g ∈ R CIJ be the feature tensors for two different images where C represents the channel dimension and ( H, W ) , ( I, J ) represent spatial dimensions.

Original: We form the feature correspondence tensor:
Revised: We form the feature correspondence tensor:

Original: [Equation]
Revised: [Equation]

Original: whose entries represent the cosine similarity between the feature at spatial position ( h, w ) of feature tensor f and position ( i, j ) of feature tensor g .
Revised: whose entries represent the cosine similarity between the feature at spatial position ( h, w ) of feature tensor f and position ( i, j ) of feature tensor g .

Original: In the special case where f = g these correspon- feature self-correspondences strongly predict true label cooccurrence.
Revised: In the special case where f = g these correspon- feature self-correspondences strongly predict true label cooccurrence.

Original: DINO outperforms
Revised: DINO outperforms

Original: MoCoV2 and a CRF kernel, which shows its power as an unsupervised learning signal.
Revised: MoCoV2 and a CRF kernel, which shows its power as an unsupervised learning signal.

Original: dences measure the similarity between two regions of the same image.
Revised: dences measure the similarity between two regions of the same image.

Original: We note that this quantity appears often as the “cost-volume” within the optical ﬂow literature, and Hamilton et al.
Revised: We note that this quantity appears often as the “cost-volume” within the optical ﬂow literature, and Hamilton et al.

Original: (2021) show this acts a higher-order generalization of Class Activation Maps (Zhou et al., 2016) for contrastive architectures and visual search engines.
Revised: (2021) show this acts a higher-order generalization of Class Activation Maps (Zhou et al., 2016) for contrastive architectures and visual search engines.

Original: By examining slices of the correspondence tensor, F , at a given ( h, w ) we are able to visualize how two images relate according the featurizer.
Revised: By examining slices of the correspondence tensor, F , at a given ( h, w ) we are able to visualize how two images relate according the featurizer.

Original: For example,
Revised: For example,

Original: Figure 2 shows how three different points from the source image (shown in blue, red, and green) are in correspondence with relevant semantic areas within the image and its K-nearest neighbors with respect to the DINO (Caron et al., 2021) as the feature extractor.
Revised: Figure 2 shows how three different points from the source image (shown in blue, red, and green) are in correspondence with relevant semantic areas within the image and its K-nearest neighbors with respect to the DINO (Caron et al., 2021) as the feature extractor.

Original: This feature correspondence tensor not only allows us to visualize image correspondences but is strongly correlated with the true label co-occurrence tensor.
Revised: This feature correspondence tensor not only allows us to visualize image correspondences but is strongly correlated with the true label co-occurrence tensor.

Original: In particular, we can form the ground truth label co-occurrence tensor given a pair of ground-truth semantic segmentation labels k ∈
Revised: In particular, we can form the ground truth label co-occurrence tensor given a pair of ground-truth semantic segmentation labels k ∈

Original: C HW , l ∈ C IJ where C represents the set of possible classes:
Revised: C HW , l ∈ C IJ where C represents the set of possible classes:

Original: [Equation]
Revised: [Equation]

Original: By examining how well the feature correspondences, F , predict the ground-truth label cooccurrences, L , we can measure how compatible the features are with the semantic segmentation labels.
Revised: By examining how well the feature correspondences, F , predict the ground-truth label cooccurrences, L , we can measure how compatible the features are with the semantic segmentation labels.

Original: More speciﬁcally we treat the feature correspondences as a probability logit and compute the average precision when used as a classiﬁer for L .
Revised: More speciﬁcally we treat the feature correspondences as a probability logit and compute the average precision when used as a classiﬁer for L .

Original: This approach not only acts as a quick diagnostic tool to determine the efﬁcacy of features, but also allows us to compare with other forms of supervision such as the fully connected Conditional Random Field (CRF)
Revised: This approach not only acts as a quick diagnostic tool to determine the efﬁcacy of features, but also allows us to compare with other forms of supervision such as the fully connected Conditional Random Field (CRF)

Original: (Kr ¨ ahenb ¨ uhl & Koltun, 2011), which uses correspondences between pixels to reﬁne low-resolution label predictions.
Revised: (Kr ¨ ahenb ¨ uhl & Koltun, 2011), which uses correspondences between pixels to reﬁne low-resolution label predictions.

Original: In
Revised: In

Original: Figure 3 we plot precision-recall curves for the DINO backbone, the MoCoV2 backbone, the CRF
Revised: Figure 3 we plot precision-recall curves for the DINO backbone, the MoCoV2 backbone, the CRF

Original: Kernel, and our trained STEGO architecture.
Revised: Kernel, and our trained STEGO architecture.

Original: Interestingly, we ﬁnd that DINO is already a spectacular predictor of label co-occurrence within the Coco stuff dataset despite never seeing the labels .
Revised: Interestingly, we ﬁnd that DINO is already a spectacular predictor of label co-occurrence within the Coco stuff dataset despite never seeing the labels .

Original: In particular, DINO recalls 50% of true label co-occurrences with a precision of 90% and signiﬁcantly outperforms both MoCoV2 feature correspondences and the CRF kernel.
Revised: In particular, DINO recalls 50% of true label co-occurrences with a precision of 90% and signiﬁcantly outperforms both MoCoV2 feature correspondences and the CRF kernel.

Original: One curious note is that our ﬁnal trained model is a better label predictor than the supervisory signal it learns from.
Revised: One curious note is that our ﬁnal trained model is a better label predictor than the supervisory signal it learns from.

Original: We attribute this to the distillation process discussed in Section 3.2 which ampliﬁes this supervisory signal and drives consistency across the entire dataset.
Revised: We attribute this to the distillation process discussed in Section 3.2 which ampliﬁes this supervisory signal and drives consistency across the entire dataset.

Original: Finally, we stress that our comparison to ground truth labels within this section is solely to provide intuition about the quality of feature correspondences as a supervisory signal.
Revised: Finally, we stress that our comparison to ground truth labels within this section is solely to provide intuition about the quality of feature correspondences as a supervisory signal.

Original: We do not use the ground truth labels to tune any parameters of
Revised: We do not use the ground truth labels to tune any parameters of

Original: STEGO.
Revised: STEGO.

Original: 3. D ISTILLING F EATURE C ORRESPONDENCES
Revised: 3. D ISTILLING F EATURE C ORRESPONDENCES

Original: In Section 3.1 we have shown that feature correspondences have the potential to be a quality learning signal for unsupervised segmentation.
Revised: In Section 3.1 we have shown that feature correspondences have the potential to be a quality learning signal for unsupervised segmentation.

Original: In this section we explore how to harness this signal to create pixel-wise embeddings that, when clustered, yield a quality semantic segmentation.
Revised: In this section we explore how to harness this signal to create pixel-wise embeddings that, when clustered, yield a quality semantic segmentation.

Original: In particular, we seek to learn a low-dimensional embedding that “distills” the feature correspondences.
Revised: In particular, we seek to learn a low-dimensional embedding that “distills” the feature correspondences.

Original: To achieve this aim, we draw inspiration from the CRF which uses an undirected graphical model to reﬁne noisy or low-resolution class predictions by aligning them with edges and color-correlated regions in the original image.
Revised: To achieve this aim, we draw inspiration from the CRF which uses an undirected graphical model to reﬁne noisy or low-resolution class predictions by aligning them with edges and color-correlated regions in the original image.

Original: More formally, let N : R C (cid:48)
Revised: More formally, let N : R C (cid:48)

Original: H (cid:48) W (cid:48)
Revised: H (cid:48) W (cid:48)

Original: → R CHW represent a deep network backbone, which maps an image x with C (cid:48) channels and spatial dimensions ( H (cid:48) , W (cid:48) ) to a feature tensor f with C channels and spatial dimensions ( H, W ) .
Revised: → R CHW represent a deep network backbone, which maps an image x with C (cid:48) channels and spatial dimensions ( H (cid:48) , W (cid:48) ) to a feature tensor f with C channels and spatial dimensions ( H, W ) .

Original: In this work, we keep this backbone network frozen and focus on training a light-weight segmentation head S : R CHW → R KHW , that maps our feature space to a code space of dimension K , where K < C .
Revised: In this work, we keep this backbone network frozen and focus on training a light-weight segmentation head S : R CHW → R KHW , that maps our feature space to a code space of dimension K , where K < C .

Original: The goal of S is to learn a nonlinear projection,
Revised: The goal of S is to learn a nonlinear projection,

Original: S ( f ) =: s ∈ R KHW , that forms compact clusters and ampliﬁes the correlation patterns of f .
Revised: S ( f ) =: s ∈ R KHW , that forms compact clusters and ampliﬁes the correlation patterns of f .

Original: To build our loss function let f and g be two feature tensors from a pair of images x , and y and let s :
Revised: To build our loss function let f and g be two feature tensors from a pair of images x , and y and let s :

Original: = S ( f ) ∈ R CHW and t :=
Revised: = S ( f ) ∈ R CHW and t :=

Original: S ( g ) ∈ R CIJ be their respective segmentation features.
Revised: S ( g ) ∈ R CIJ be their respective segmentation features.

Original: Next, using
Revised: Next, using

Original: Equation 1 we compute a feature correlation tensor F ∈ R HWIJ from f and g and a segmentation correlation tensor S ∈ R HWIJ from s and t .
Revised: Equation 1 we compute a feature correlation tensor F ∈ R HWIJ from f and g and a segmentation correlation tensor S ∈ R HWIJ from s and t .

Original: Our loss function aims to push the entries of s and t together if there is a signiﬁcant coupling between two corresponding entries of f and g .
Revised: Our loss function aims to push the entries of s and t together if there is a signiﬁcant coupling between two corresponding entries of f and g .

Original: As shown in Figure 4, we can achieve this with a simple element-wise multiplication of the tensors F and S :
Revised: As shown in Figure 4, we can achieve this with a simple element-wise multiplication of the tensors F and S :

Original: [Equation]
Revised: [Equation]

Original: Where b is a hyper-parameter which adds uniform “negative pressure” to the equation to prevent collapse.
Revised: Where b is a hyper-parameter which adds uniform “negative pressure” to the equation to prevent collapse.

Original: Minimizing L with respect to S encourages elements of S to be large when elements of F − b are positive and small when elements of F − b are negative.
Revised: Minimizing L with respect to S encourages elements of S to be large when elements of F − b are positive and small when elements of F − b are negative.

Original: More explicitly, because the elements of F and S are cosine similarities, this exerts an attractive or repulsive force on pairs of segmentation features with strength proportional to their feature correspondences.
Revised: More explicitly, because the elements of F and S are cosine similarities, this exerts an attractive or repulsive force on pairs of segmentation features with strength proportional to their feature correspondences.

Original: We note that the elements of S are not just encouraged to equal the elements of F but rather to push to total anti-alignment ( − 1) or alignment (1) depending on the sign of F − b .
Revised: We note that the elements of S are not just encouraged to equal the elements of F but rather to push to total anti-alignment ( − 1) or alignment (1) depending on the sign of F − b .

Original: In practice, we found that L simple − corr is sometimes unstable and does not provide enough learning signal to drive the optimization.
Revised: In practice, we found that L simple − corr is sometimes unstable and does not provide enough learning signal to drive the optimization.

Original: Empirically, we found that optimizing the segmentation features towards total anti-alignment when the corresponding features do not correlate leads to instability, likely because this increases co-linearity.
Revised: Empirically, we found that optimizing the segmentation features towards total anti-alignment when the corresponding features do not correlate leads to instability, likely because this increases co-linearity.

Original: Therefore, we optimize weakly-correlated segmentation features to be orthogonal instead.
Revised: Therefore, we optimize weakly-correlated segmentation features to be orthogonal instead.

Original: This can be efﬁciently achieved by clamping the segmentation correspondence, S , at 0 , which dramatically improved the optimization stability.
Revised: This can be efﬁciently achieved by clamping the segmentation correspondence, S , at 0 , which dramatically improved the optimization stability.

Original: Additionally, we encountered challenges when balancing the learning signal for small objects which have concentrated correlation patterns.
Revised: Additionally, we encountered challenges when balancing the learning signal for small objects which have concentrated correlation patterns.

Original: In these cases, F hwij − b is negative in most locations, and the loss drives the features to diverge instead of aggregate.
Revised: In these cases, F hwij − b is negative in most locations, and the loss drives the features to diverge instead of aggregate.

Original: To make the optimization more balanced, we introduce a S patial C entering operation on the feature correspondences:
Revised: To make the optimization more balanced, we introduce a S patial C entering operation on the feature correspondences:

Original: [Equation]
Revised: [Equation]

Original: Together with the zero clamping, our ﬁnal correlation loss is deﬁned as:
Revised: Together with the zero clamping, our ﬁnal correlation loss is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: We demonstrate the positive effect of both the aforementioned “0-Clamp” and “SC” modiﬁcations in the ablation study of Table 2.
Revised: We demonstrate the positive effect of both the aforementioned “0-Clamp” and “SC” modiﬁcations in the ablation study of Table 2.

Original: 3. STEGO A RCHITECTURE
Revised: 3. STEGO A RCHITECTURE

Original: STEGO uses three instantiations of the correspondence loss of Equation 4 to train a segmentation head to distill feature relationships between an image and itself, its K-Nearest Neighbors (KNNs), and random other images.
Revised: STEGO uses three instantiations of the correspondence loss of Equation 4 to train a segmentation head to distill feature relationships between an image and itself, its K-Nearest Neighbors (KNNs), and random other images.

Original: The self and KNN correspondence losses primarily provide positive, attractive, signal and random image pairs tend to provide negative, repulsive, signal.
Revised: The self and KNN correspondence losses primarily provide positive, attractive, signal and random image pairs tend to provide negative, repulsive, signal.

Original: We illustrate this and other major architecture components of STEGO in Figure 4.
Revised: We illustrate this and other major architecture components of STEGO in Figure 4.

Original: STEGO is made up of a frozen backbone that serves as a source of learning feedback, and as an input to the segmentation head for predicting distilled features.
Revised: STEGO is made up of a frozen backbone that serves as a source of learning feedback, and as an input to the segmentation head for predicting distilled features.

Original: This segmentation head is a simple
Revised: This segmentation head is a simple

Original: [Figure]
Revised: [Figure]

Original: Training
Revised: Training

Original: Prediction
Revised: Prediction

Original: V i s .
Revised: V i s .

Original: T r a n s .
Revised: T r a n s .

Original: Self Corr.
Revised: Self Corr.

Original: S e g .
Revised: S e g .

Original: H e a d
Revised: H e a d

Original: C R F C l u s t e r
Revised: C R F C l u s t e r

Original: KNN Corr.
Revised: KNN Corr.

Original: feed forward network with ReLU activations (Glorot et al., 2011).
Revised: feed forward network with ReLU activations (Glorot et al., 2011).

Original: In contrast to other works, our method does not re-train or ﬁne-tune the backbone.
Revised: In contrast to other works, our method does not re-train or ﬁne-tune the backbone.

Original: This makes our method very efﬁcient to train: it only takes less than 2 hours on a single NVIDIA V100 GPU card.
Revised: This makes our method very efﬁcient to train: it only takes less than 2 hours on a single NVIDIA V100 GPU card.

Original: We ﬁrst use our backbone to extract global image features by global average pooling (GAP) our spatial features: GAP ( f ) .
Revised: We ﬁrst use our backbone to extract global image features by global average pooling (GAP) our spatial features: GAP ( f ) .

Original: We then construct a lookup table of each image’s K-Nearest Neighbors according to cosine similarity in the backbone’s feature space.
Revised: We then construct a lookup table of each image’s K-Nearest Neighbors according to cosine similarity in the backbone’s feature space.

Original: Each training minibatch consists of a collection of random images x and random nearest neighbors x knn .
Revised: Each training minibatch consists of a collection of random images x and random nearest neighbors x knn .

Original: In our experiments we sample x knn randomly from each image’s top 7 KNNs.
Revised: In our experiments we sample x knn randomly from each image’s top 7 KNNs.

Original: We also sample random images, x rand , by shufﬂing x and ensuring that no image matched with itself.
Revised: We also sample random images, x rand , by shufﬂing x and ensuring that no image matched with itself.

Original: STEGO’s full loss is:
Revised: STEGO’s full loss is:

Original: [Equation]
Revised: [Equation]

Original: Where the λ ’s and the b ’s control the balance of the learning signals and the ratio of positive to negative pressure respectively.
Revised: Where the λ ’s and the b ’s control the balance of the learning signals and the ratio of positive to negative pressure respectively.

Original: In practice, we found that a ratio of λ self ≈ λ rand ≈ 2 λ knn worked well.
Revised: In practice, we found that a ratio of λ self ≈ λ rand ≈ 2 λ knn worked well.

Original: The b parameters tended to be dataset and network speciﬁc, but we aimed to keep the system in a rough balance between positive and negative forces.
Revised: The b parameters tended to be dataset and network speciﬁc, but we aimed to keep the system in a rough balance between positive and negative forces.

Original: More speciﬁcally we tuned the b s to keep mean KNN feature similarity at ≈ 0 .
Revised: More speciﬁcally we tuned the b s to keep mean KNN feature similarity at ≈ 0 .

Original: 3 and mean random similarity at ≈ 0 . 0 .
Revised: 3 and mean random similarity at ≈ 0 . 0 .

Original: Many images within the CocoStuff and Cityscapes datasets are cluttered with small objects that are hard to resolve at a feature resolution of (40 , 40) .
Revised: Many images within the CocoStuff and Cityscapes datasets are cluttered with small objects that are hard to resolve at a feature resolution of (40 , 40) .

Original: To better handle small objects and maintain fast training times we ﬁve-crop training images prior to learning KNNs.
Revised: To better handle small objects and maintain fast training times we ﬁve-crop training images prior to learning KNNs.

Original: This not only allows the network to look at closer details of the images, but also improves the quality of the KNNs.
Revised: This not only allows the network to look at closer details of the images, but also improves the quality of the KNNs.

Original: More speciﬁcally, global image embeddings are computed for each crop.
Revised: More speciﬁcally, global image embeddings are computed for each crop.

Original: This allows the network to resolve ﬁner details and yields ﬁve times as many images to ﬁnd close matching KNNs from.
Revised: This allows the network to resolve ﬁner details and yields ﬁve times as many images to ﬁnd close matching KNNs from.

Original: Five-cropping improved both our Cityscapes results and CocoStuff segmentations, and we detail this in Table 2.
Revised: Five-cropping improved both our Cityscapes results and CocoStuff segmentations, and we detail this in Table 2.

Original: The ﬁnal components of our architecture are the clustering and CRF reﬁnement step.
Revised: The ﬁnal components of our architecture are the clustering and CRF reﬁnement step.

Original: Due to the feature distillation process, STEGO’s segmentation features tend to form clear clusters.
Revised: Due to the feature distillation process, STEGO’s segmentation features tend to form clear clusters.

Original: We apply a cosine distance based minibatch K-Means algorithm (MacQueen et al., 1967) to extract these clusters and compute concrete class assignments from STEGO’s continuous features.
Revised: We apply a cosine distance based minibatch K-Means algorithm (MacQueen et al., 1967) to extract these clusters and compute concrete class assignments from STEGO’s continuous features.

Original: After clustering, we reﬁne these labels with a CRF to improve their spatial resolution further.
Revised: After clustering, we reﬁne these labels with a CRF to improve their spatial resolution further.

Original: 3. R ELATION TO P OTTS M ODELS AND E NERGY - BASED G RAPH O PTIMIZATION
Revised: 3. R ELATION TO P OTTS M ODELS AND E NERGY - BASED G RAPH O PTIMIZATION

Original: Equation 4 can be viewed in the context of Potts models or continuous Ising models from statistical physics (Potts, 1952; Baker Jr & Kincaid, 1979).
Revised: Equation 4 can be viewed in the context of Potts models or continuous Ising models from statistical physics (Potts, 1952; Baker Jr & Kincaid, 1979).

Original: We brieﬂy overview this connection, and point interested readers to Section A.8 for a more detailed discussion.
Revised: We brieﬂy overview this connection, and point interested readers to Section A.8 for a more detailed discussion.

Original: To build the general Ising model, let
Revised: To build the general Ising model, let

Original: G =
Revised: G =

Original: ( V , w ) be a fully connected, weighted, and undirected graph on |V| vertices.
Revised: ( V , w ) be a fully connected, weighted, and undirected graph on |V| vertices.

Original: In our applications we take V to be the set of pixels in the training dataset.
Revised: In our applications we take V to be the set of pixels in the training dataset.

Original: Let w : V × V → R represent an edge
Revised: Let w : V × V → R represent an edge

Original: STEGO signiﬁcantly outperforms
Revised: STEGO signiﬁcantly outperforms

Original: [Table]
Revised: [Table]

Original: weighting function.
Revised: weighting function.

Original: Let φ : V → C be a vertex valued function mapping into a generic code space C such as the probability simplex over cluster labels P
Revised: Let φ : V → C be a vertex valued function mapping into a generic code space C such as the probability simplex over cluster labels P

Original: ( L )
Revised: ( L )

Original: [Table]
Revised: [Table]

Original: or the K -dimensional continuous feature space R K .
Revised: or the K -dimensional continuous feature space R K .

Original: The function φ can be a parameterized neural network
Revised: The function φ can be a parameterized neural network

Original: [Table]
Revised: [Table]

Original: or a simple lookup table that assigns a code to each graph node.
Revised: or a simple lookup table that assigns a code to each graph node.

Original: Finally, we deﬁne a compatibility function µ : C × C
Revised: Finally, we deﬁne a compatibility function µ : C × C

Original: → R that measures the cost of comparing two codes.
Revised: → R that measures the cost of comparing two codes.

Original: We can now deﬁne the following graph energy functional:
Revised: We can now deﬁne the following graph energy functional:

Original: [Equation]
Revised: [Equation]

Original: Constructing the Boltzmann Distribution (Hinton, 2002) yields a normalized distribution over the function space Φ :
Revised: Constructing the Boltzmann Distribution (Hinton, 2002) yields a normalized distribution over the function space Φ :

Original: [Equation]
Revised: [Equation]

Original: In general, sampling from this probability distribution is difﬁcult because of the often-intractable normalization factor.
Revised: In general, sampling from this probability distribution is difﬁcult because of the often-intractable normalization factor.

Original: However, it is easier to compute the maximum likelihood estimate (MLE), arg max φ ∈
Revised: However, it is easier to compute the maximum likelihood estimate (MLE), arg max φ ∈

Original: Φ p ( φ | w, µ ) .
Revised: Φ p ( φ | w, µ ) .

Original: In particular, if Φ is a smoothly parameterized space of functions and φ and µ are differentiable functions, one can compute the MLE using stochastic gradient descent (SGD) with highly-optimized automatic differentiation frameworks (Paszke et al., 2019; Abadi et al., 2015).
Revised: In particular, if Φ is a smoothly parameterized space of functions and φ and µ are differentiable functions, one can compute the MLE using stochastic gradient descent (SGD) with highly-optimized automatic differentiation frameworks (Paszke et al., 2019; Abadi et al., 2015).

Original: In Section A.8 of the supplement we prove that the ﬁnding the MLE of Equation 7 is equivalent to minimizing the loss of Equation 4 when | V | is the set of pixels in our image training set, φ = S ◦N , w is the cosine distance between features, and µ is cosine distance.
Revised: In Section A.8 of the supplement we prove that the ﬁnding the MLE of Equation 7 is equivalent to minimizing the loss of Equation 4 when | V | is the set of pixels in our image training set, φ = S ◦N , w is the cosine distance between features, and µ is cosine distance.

Original: Like STEGO, the CRF is also a
Revised: Like STEGO, the CRF is also a

Original: Potts model, and we use this connection to re-purpose the STEGO loss function to create continuous, minibatch, and unsupervised variants of the CRF.
Revised: Potts model, and we use this connection to re-purpose the STEGO loss function to create continuous, minibatch, and unsupervised variants of the CRF.

Original: We detail this exploration in Section A.9 of the
Revised: We detail this exploration in Section A.9 of the

Original: Supplement.
Revised: Supplement.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate STEGO on standard semantic segmentation datasets and compare with current state-ofthe-art.
Revised: We evaluate STEGO on standard semantic segmentation datasets and compare with current state-ofthe-art.

Original: We then justify different design choices of STEGO through ablation studies.
Revised: We then justify different design choices of STEGO through ablation studies.

Original: Additional details on datasets, model hyperparameters, hardware, and other implementation details can be found in Section A.10 of the Supplement.
Revised: Additional details on datasets, model hyperparameters, hardware, and other implementation details can be found in Section A.10 of the Supplement.

Original: 4. E VALUATION D ETAILS
Revised: 4. E VALUATION D ETAILS

Original: Datasets Following Cho et al.
Revised: Datasets Following Cho et al.

Original: (2021), we evaluate STEGO on the 27 mid-level classes of the
Revised: (2021), we evaluate STEGO on the 27 mid-level classes of the

Original: CocoStuff class hierarchy and on the 27 classes of Cityscapes.
Revised: CocoStuff class hierarchy and on the 27 classes of Cityscapes.

Original: Like prior art, we ﬁrst resize images to 320 pixels along the minor axis followed by a (320 × 320) center crops of each validation image.
Revised: Like prior art, we ﬁrst resize images to 320 pixels along the minor axis followed by a (320 × 320) center crops of each validation image.

Original: We use mean intersection over union (mIoU) and Accuracy for evaluation metrics.
Revised: We use mean intersection over union (mIoU) and Accuracy for evaluation metrics.

Original: Our CocoStuff evaluation setting originated in Ji et al.
Revised: Our CocoStuff evaluation setting originated in Ji et al.

Original: and is common in the literature.
Revised: and is common in the literature.

Original: Our Cityscapes evaluation setting is adopted from Cho et al.
Revised: Our Cityscapes evaluation setting is adopted from Cho et al.

Original: The latter is newer and more challenging, and thus fewer baselines are available.
Revised: The latter is newer and more challenging, and thus fewer baselines are available.

Original: Finally we also compare on the Potsdam-3 setting fro Ji et al.
Revised: Finally we also compare on the Potsdam-3 setting fro Ji et al.

Original: (2019) in Section A.2 of the Appendix.
Revised: (2019) in Section A.2 of the Appendix.

Original: Linear Probe The ﬁrst way we evaluate the quality of the distilled segmentation features is through transfer learning effectiveness.
Revised: Linear Probe The ﬁrst way we evaluate the quality of the distilled segmentation features is through transfer learning effectiveness.

Original: As in Van Gansbeke et al.
Revised: As in Van Gansbeke et al.

Original: Cho et al.
Revised: Cho et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2020b), we train a linear projection from segmentation features to class labels using the cross entropy loss.
Revised: (2020b), we train a linear projection from segmentation features to class labels using the cross entropy loss.

Original: This loss solely evaluates feature quality and is not part of the STEGO training process.
Revised: This loss solely evaluates feature quality and is not part of the STEGO training process.

Original: Clustering Unlike the linear probe, the clustering step does not have access to ground truth supervised labels.
Revised: Clustering Unlike the linear probe, the clustering step does not have access to ground truth supervised labels.

Original: As in prior art, we use a Hungarian matching algorithm to align our unlabeled clusters and the ground truth labels for evaluation and visualization purposes.
Revised: As in prior art, we use a Hungarian matching algorithm to align our unlabeled clusters and the ground truth labels for evaluation and visualization purposes.

Original: This measures how consistent the predicted semantic segments are with the ground truth labels and is invariant to permutations of the predicted class labels.
Revised: This measures how consistent the predicted semantic segments are with the ground truth labels and is invariant to permutations of the predicted class labels.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: We summarize our main results on the 27 classes of CocoStuff in Table 1.
Revised: We summarize our main results on the 27 classes of CocoStuff in Table 1.

Original: STEGO signiﬁcantly outperforms the prior state of the art, PiCIE, on both linear probe and clustering (Unsupervised) metrics.
Revised: STEGO signiﬁcantly outperforms the prior state of the art, PiCIE, on both linear probe and clustering (Unsupervised) metrics.

Original: In particular, STEGO improves by +14 unsupervised mIoU, +6.9 unsupervised accuracy, +26 linear probe mIoU, and +21 linear probe accuracy compared to the next best baseline.
Revised: In particular, STEGO improves by +14 unsupervised mIoU, +6.9 unsupervised accuracy, +26 linear probe mIoU, and +21 linear probe accuracy compared to the next best baseline.

Original: In Table 3, we ﬁnd a similarly large improvement of +8.7 unsupervised mIoU and +7.7 unsupervised accuracy on the Cityscapes validation set.
Revised: In Table 3, we ﬁnd a similarly large improvement of +8.7 unsupervised mIoU and +7.7 unsupervised accuracy on the Cityscapes validation set.

Original: These two experiments demonstrate that even though we do not ﬁne-tune the backbone for these datasets, DINO’s self-supervised weights on ImageNet (Deng et al., 2009) are enough to simultaneously solve both settings.
Revised: These two experiments demonstrate that even though we do not ﬁne-tune the backbone for these datasets, DINO’s self-supervised weights on ImageNet (Deng et al., 2009) are enough to simultaneously solve both settings.

Original: STEGO also outperforms simply clustering the features from unmodiﬁed DINO, MoCoV2, and ImageNet supervised ResNet50 backbones.
Revised: STEGO also outperforms simply clustering the features from unmodiﬁed DINO, MoCoV2, and ImageNet supervised ResNet50 backbones.

Original: This demonstrates the beneﬁts of training a segmentation head to distill feature correspondences.
Revised: This demonstrates the beneﬁts of training a segmentation head to distill feature correspondences.

Original: We show some example segmentations from STEGO and our baseline PiCIE on the CocoStuff dataset in Figure 1.
Revised: We show some example segmentations from STEGO and our baseline PiCIE on the CocoStuff dataset in Figure 1.

Original: We include additional examples and failure cases in Sections A.4 and A.5.
Revised: We include additional examples and failure cases in Sections A.4 and A.5.

Original: We note that STEGO is signiﬁcantly better at resolving ﬁne-grained details within the images such as the legs of horses in the third image from the left column of Figure 1, and the individual birds in the right-most column.
Revised: We note that STEGO is signiﬁcantly better at resolving ﬁne-grained details within the images such as the legs of horses in the third image from the left column of Figure 1, and the individual birds in the right-most column.

Original: Though the PiCIE baseline uses a feature pyramid network to output high resolution predictions, the network does not attune to ﬁne grained details, potentially demonstrating the limitations of the sparse training signal induced by data augmentations alone.
Revised: Though the PiCIE baseline uses a feature pyramid network to output high resolution predictions, the network does not attune to ﬁne grained details, potentially demonstrating the limitations of the sparse training signal induced by data augmentations alone.

Original: In contrast, STEGO’s predictions capture small objects and ﬁne details.
Revised: In contrast, STEGO’s predictions capture small objects and ﬁne details.

Original: In part, this can be attributed to DINO backbone’s higher resolution features, the 5-crop training described in 3.3, and the CRF post-processing which helps to align the predictions to image edges.
Revised: In part, this can be attributed to DINO backbone’s higher resolution features, the 5-crop training described in 3.3, and the CRF post-processing which helps to align the predictions to image edges.

Original: We show qualitative results on the Cityscapes dataset in Figure 5.
Revised: We show qualitative results on the Cityscapes dataset in Figure 5.

Original: STEGO successfully identiﬁes people, street, sidewalk, cars, and street signs with high
Revised: STEGO successfully identiﬁes people, street, sidewalk, cars, and street signs with high

Original: [Table] [Table]
Revised: [Table] [Table]

Original: detail and ﬁdelity.
Revised: detail and ﬁdelity.

Original: We note that prior works did not publish pretrained models or linear probe results on Cityscapes so we exclude this information from Table 3 and Figure 5.
Revised: We note that prior works did not publish pretrained models or linear probe results on Cityscapes so we exclude this information from Table 3 and Figure 5.

Original: To better understand the predictions and failures of STEGO
Revised: To better understand the predictions and failures of STEGO

Original: [Table]
Revised: [Table]

Original: we include confusion matrices for
Revised: we include confusion matrices for

Original: CocoStuff (Figure 6) and Cityscapes (Figure 11 of the Supplement).
Revised: CocoStuff (Figure 6) and Cityscapes (Figure 11 of the Supplement).

Original: Some salient STEGO errors include confusing the “food” category from the CocoStuff “things”, and the “food” category from
Revised: Some salient STEGO errors include confusing the “food” category from the CocoStuff “things”, and the “food” category from

Original: CocoStuff “stuff”.
Revised: CocoStuff “stuff”.

Original: STEGO also does not properly separate “ceilings” from “walls”, and lacks consistent segmentations for classes such as “indoor”, “accessory”, “rawmaterial” and “textile”.
Revised: STEGO also does not properly separate “ceilings” from “walls”, and lacks consistent segmentations for classes such as “indoor”, “accessory”, “rawmaterial” and “textile”.

Original: These errors also draw our attention to the challenges of evaluating unsupervised segmentation methods: label ontologies can be arbitrary .
Revised: These errors also draw our attention to the challenges of evaluating unsupervised segmentation methods: label ontologies can be arbitrary .

Original: In these circumstances the divisions between classes are not well deﬁned and it is hard to imagine a system that can segment the results consistently without additional information.
Revised: In these circumstances the divisions between classes are not well deﬁned and it is hard to imagine a system that can segment the results consistently without additional information.

Original: In these regimes, the linear probe provides a more important barometer for quality because the limited supervision can help disambiguate these cases.
Revised: In these regimes, the linear probe provides a more important barometer for quality because the limited supervision can help disambiguate these cases.

Original: Nevertheless, we feel that there is still considerable progress to be made on the purely unsupervised benchmark, and that even with the improvements of STEGO there is still a measurable performance gap with supervised systems.
Revised: Nevertheless, we feel that there is still considerable progress to be made on the purely unsupervised benchmark, and that even with the improvements of STEGO there is still a measurable performance gap with supervised systems.

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: To understand the impact of STEGO’s architectural components we perform an ablation analysis on the CocoStuff dataset, and report the results in Table 2.
Revised: To understand the impact of STEGO’s architectural components we perform an ablation analysis on the CocoStuff dataset, and report the results in Table 2.

Original: We examine the effect of using several different backbones in STEGO including MoCoV2, the ViT-Small, and ViT-Base architectures of
Revised: We examine the effect of using several different backbones in STEGO including MoCoV2, the ViT-Small, and ViT-Base architectures of

Original: DINO.
Revised: DINO.

Original: We ﬁnd that ViT-Base is the best feature extractor of the group and leads by a signiﬁcant margin both in terms of accuracy and mIoU. We also evaluate the several loss function and architecture decisions described in Section 3.3.
Revised: We ﬁnd that ViT-Base is the best feature extractor of the group and leads by a signiﬁcant margin both in terms of accuracy and mIoU. We also evaluate the several loss function and architecture decisions described in Section 3.3.

Original: In particular, we explore clamping the segmentation feature correspondence tensor at 0 to prevent the negative pressure from introducing co-linearity (0-Clamp), ﬁve-cropping the dataset prior to mining KNNs to improve the resolution of the learning signal (5Crop), spatially centering the feature correspondence tensor to improve resolution of small objects (SC), and Conditional Random Field post-processing to reﬁne predictions (CRF).
Revised: In particular, we explore clamping the segmentation feature correspondence tensor at 0 to prevent the negative pressure from introducing co-linearity (0-Clamp), ﬁve-cropping the dataset prior to mining KNNs to improve the resolution of the learning signal (5Crop), spatially centering the feature correspondence tensor to improve resolution of small objects (SC), and Conditional Random Field post-processing to reﬁne predictions (CRF).

Original: We ﬁnd that these modiﬁcations improve both the cluster and linear probe evaluation metrics.
Revised: We ﬁnd that these modiﬁcations improve both the cluster and linear probe evaluation metrics.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have found that modern self-supervised visual backbones can be reﬁned to yield state of the art unsupervised semantic segmentation methods.
Revised: We have found that modern self-supervised visual backbones can be reﬁned to yield state of the art unsupervised semantic segmentation methods.

Original: We have motivated this architecture by showing that correspondences between deep features are directly correlated with ground truth label cooccurrence.
Revised: We have motivated this architecture by showing that correspondences between deep features are directly correlated with ground truth label cooccurrence.

Original: We take advantage of this strong, yet entirely unsupervised, learning signal by introducing a novel contrastive loss that “distills” the correspondences between features.
Revised: We take advantage of this strong, yet entirely unsupervised, learning signal by introducing a novel contrastive loss that “distills” the correspondences between features.

Original: Our system,
Revised: Our system,

Original: STEGO, produces low rank representations that cluster into accurate semantic segmentation predictions.
Revised: STEGO, produces low rank representations that cluster into accurate semantic segmentation predictions.

Original: We connect STEGO’s loss to CRF inference by showing it is equivalent to MLE in Potts models over the entire collection of pixels in our dataset.
Revised: We connect STEGO’s loss to CRF inference by showing it is equivalent to MLE in Potts models over the entire collection of pixels in our dataset.

Original: We show STEGO yields a signiﬁcant improvement over the prior state of the art, on both the CocoStuff ( +14 mIoU ) and Cityscapes ( + mIoU ) semantic segmentation challenges.
Revised: We show STEGO yields a signiﬁcant improvement over the prior state of the art, on both the CocoStuff ( +14 mIoU ) and Cityscapes ( + mIoU ) semantic segmentation challenges.

Original: Finally, we justify the architectural decisions of STEGO with an ablation study on the CocoStuff dataset.
Revised: Finally, we justify the architectural decisions of STEGO with an ablation study on the CocoStuff dataset.

Revised: A CKNOWLEDGMENTS

Revised: We would like to thank Karen Hamilton for proofreading the work and Siddhartha Sen for sponsoring access to the Microsoft Research compute infrastructure.

Revised: We also thank Jang Hyun Cho for helping us run and evaluate the PiCIE baseline.

Revised: We thank Kavital Bala, Vincent Sitzmann, Marc

Original:  A A PPENDIX A. V IDEO AND C ODE
Revised: Bosch, Desalegn Delelegn, Cody Champion and Markus Weimer for their helpful commentary on the work. This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. 2021323067. Any opinion, ﬁndings, and conclusions or recommendations expressed in this material are those of the authors(s) and do not necessarily reﬂect the views of the National Science Foundation A A PPENDIX A. V IDEO AND C ODE

Original: We include a short video description of our work at https://aka.ms/stego-video .
Revised: We include a short video description of our work at https://aka.ms/stego-video .

Original: We also provide training and evaluation code at https://aka.ms/stego-code
Revised: We also provide training and evaluation code at https://aka.ms/stego-code

Original: A.
Revised: A.

Original: A DDITIONAL R ESULTS ON THE P OTSDAM -3 D ATASET
Revised: A DDITIONAL R ESULTS ON THE P OTSDAM -3 D ATASET

Original: In addition to our evaluations in Section 4.1 we compare STEGO to prior art on the Potsdam 3-class aerial image segmentation task presented in Ji et al.
Revised: In addition to our evaluations in Section 4.1 we compare STEGO to prior art on the Potsdam 3-class aerial image segmentation task presented in Ji et al.

Original: In Table ??
Revised: In Table ??

Original: We ﬁnd that STEGO is able to achieve +12% accuracy compared to the previous state of the art, IIC.
Revised: We ﬁnd that STEGO is able to achieve +12% accuracy compared to the previous state of the art, IIC.

Original: We show example qualitative results in Figure
Revised: We show example qualitative results in Figure

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. A DDITIONAL A BLATION S TUDY
Revised: A. A DDITIONAL A BLATION S TUDY

Original: In addition to the ablation study of Table
Revised: In addition to the ablation study of Table

Original: [Table]
Revised: [Table]

Original: we investigate the effect of each major architectural decision in isolation.
Revised: we investigate the effect of each major architectural decision in isolation.

Original: We ﬁnd that in most metrics, removing each architectural component hurts performance.
Revised: We ﬁnd that in most metrics, removing each architectural component hurts performance.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: A. F AILURE C ASES
Revised: A. F AILURE C ASES

Original: Unsupervised Segmentation is prone to a variety of issues.
Revised: Unsupervised Segmentation is prone to a variety of issues.

Original: We include some of the following to segmentations to demonstrate cases where STEGO breaks down.
Revised: We include some of the following to segmentations to demonstrate cases where STEGO breaks down.

Original: In the ﬁrst column of Figure 9 we can see that STEGO improperly segments ground from trees and backgrounds.
Revised: In the ﬁrst column of Figure 9 we can see that STEGO improperly segments ground from trees and backgrounds.

Original: In the second column we see that STEGO makes an understandable error and assigns the barn ﬂoor to the “outdoor” class and the barn wall to the “building” class.
Revised: In the second column we see that STEGO makes an understandable error and assigns the barn ﬂoor to the “outdoor” class and the barn wall to the “building” class.

Original: In the third column STEGO misses the boundary between wall and ceiling.
Revised: In the third column STEGO misses the boundary between wall and ceiling.

Original: The fourth column demonstrates the challenge between food (thing) and food (stuff) characterization.
Revised: The fourth column demonstrates the challenge between food (thing) and food (stuff) characterization.

Original: Interestingly PiCIE makes the same type of error both here, and in the barn case.
Revised: Interestingly PiCIE makes the same type of error both here, and in the barn case.

Original: The last column shows an example of STEGO missing a human in the lower left.
Revised: The last column shows an example of STEGO missing a human in the lower left.

Original: In this image it is challenging to spot the person, probably because it is grayscale.
Revised: In this image it is challenging to spot the person, probably because it is grayscale.

Original: A. F EATURE C ORRESPONDENCES P REDICT STEGO’ S E RRORS
Revised: A. F EATURE C ORRESPONDENCES P REDICT STEGO’ S E RRORS

Original: Section 3.1 demonstrates how unsupervised feature correspondences serve as an excellent proxy for the true label co-occurrence information.
Revised: Section 3.1 demonstrates how unsupervised feature correspondences serve as an excellent proxy for the true label co-occurrence information.

Original: In this section we explore how and where DINO’s feature correspondences systematically differ from the ground truth labels, and show that these insights allow us to directly predict STEGO’s ﬁnal confusion matrix.
Revised: In this section we explore how and where DINO’s feature correspondences systematically differ from the ground truth labels, and show that these insights allow us to directly predict STEGO’s ﬁnal confusion matrix.

Original: More speciﬁcally we consider the setting of Section 3.1.
Revised: More speciﬁcally we consider the setting of Section 3.1.

Original: Instead of computing precision-recall curves from our feature correspondence scores we can instead threshold these scores, select the strongest couplings between the images, and evaluate whether these couplings are between objects of the same class or objects of different classes.
Revised: Instead of computing precision-recall curves from our feature correspondence scores we can instead threshold these scores, select the strongest couplings between the images, and evaluate whether these couplings are between objects of the same class or objects of different classes.

Original: In particular, Figure 10 shows a confusion matrix capturing how well DINO feature correspondences between images and their K-Nearest Neighbors align with the ground truth label ontology in the CocoStuff27 dataset.
Revised: In particular, Figure 10 shows a confusion matrix capturing how well DINO feature correspondences between images and their K-Nearest Neighbors align with the ground truth label ontology in the CocoStuff27 dataset.

Original: We ﬁnd that that this analysis predicts many of the areas where the ﬁnal STEGO architecture fails.
Revised: We ﬁnd that that this analysis predicts many of the areas where the ﬁnal STEGO architecture fails.

Original: In particular, we can see that
Revised: In particular, we can see that

Original: DINO conﬂates the “Food (things)” and “Food (stuff)” and this error also appears in STEGO’s confusion matrix in Figure 12.
Revised: DINO conﬂates the “Food (things)” and “Food (stuff)” and this error also appears in STEGO’s confusion matrix in Figure 12.

Original: Likewise both visualizations show confusion between “appliance” and “furniture”, “window” and “wall”, and several other common errors.
Revised: Likewise both visualizations show confusion between “appliance” and “furniture”, “window” and “wall”, and several other common errors.

Original: This analysis demonstrates that many of STEGO’s errors originate from the structure of the DINO features used to train STEGO as opposed to other aspects of the architecture.
Revised: This analysis demonstrates that many of STEGO’s errors originate from the structure of the DINO features used to train STEGO as opposed to other aspects of the architecture.

Original: However we note that the question of whether whether this is an issue with the DINO features, or due to ambiguities in the
Revised: However we note that the question of whether whether this is an issue with the DINO features, or due to ambiguities in the

Original: CocoStuff label ontology is still outstanding.
Revised: CocoStuff label ontology is still outstanding.

Original: Finally we note that this analysis is able to predict the results of a fully-trained STEGO architecture, and could be used as a way to select better backbones without having to training STEGO.
Revised: Finally we note that this analysis is able to predict the results of a fully-trained STEGO architecture, and could be used as a way to select better backbones without having to training STEGO.

Original: This analysis shows where our unsupervised supervisory signal, the DINO feature correspondences, fails to align with the CocoStuff27 label ontology.
Revised: This analysis shows where our unsupervised supervisory signal, the DINO feature correspondences, fails to align with the CocoStuff27 label ontology.

Original: [Figure]
Revised: [Figure]

Original: IGHER
Revised: IGHER

Original: [Figure]
Revised: [Figure]

Original: ESOLUTION 
Revised: ESOLUTION C ONFUSION M ATRICES

Original: [Figure]

Original: A. R ELATIONSHIP WITH G RAPH E NERGY M INIMIZATION
Revised: A. R ELATIONSHIP WITH G RAPH E NERGY M INIMIZATION

Original: In section 3.4 we brieﬂy mention that STEGO’s feature correlation distillation loss deﬁned in Equation 4 can be seen as a particular case of Maximum Likelihood (ML) estimation on a undirected graphical model or Ising model.
Revised: In section 3.4 we brieﬂy mention that STEGO’s feature correlation distillation loss deﬁned in Equation 4 can be seen as a particular case of Maximum Likelihood (ML) estimation on a undirected graphical model or Ising model.

Original: In this section we demonstrate this connection in greater detail using the formalism deﬁned in 3.4.
Revised: In this section we demonstrate this connection in greater detail using the formalism deﬁned in 3.4.

Original: In particular, we recall the energy for a Potts model:
Revised: In particular, we recall the energy for a Potts model:

Original: [Equation]
Revised: [Equation]

Original: We then construct the Boltzmann Distribution (Hinton, 2002) yields a normalized distribution over the function space Φ :
Revised: We then construct the Boltzmann Distribution (Hinton, 2002) yields a normalized distribution over the function space Φ :

Original: [Equation]
Revised: [Equation]

Original: In general, sampling from this probability distribution is difﬁcult because of the often-intractable normalization factor.
Revised: In general, sampling from this probability distribution is difﬁcult because of the often-intractable normalization factor.

Original: However, it is easier to compute the maximum likelihood estimate (MLE):
Revised: However, it is easier to compute the maximum likelihood estimate (MLE):

Original: [Equation]
Revised: [Equation]

Original: Where Z is the unknown constant normalization factor.
Revised: Where Z is the unknown constant normalization factor.

Original: Simplifying the right-hand side yields:
Revised: Simplifying the right-hand side yields:

Original: [Equation]
Revised: [Equation]

Original: We are now in the position to connect this to the STEGO loss function.
Revised: We are now in the position to connect this to the STEGO loss function.

Original: First, we take our nodes
Revised: First, we take our nodes

Original: V to be the set of all spatial locations across our entire dataset of images.
Revised: V to be the set of all spatial locations across our entire dataset of images.

Original: For concreteness we can represent v ∈ V by the tuple ( n, h, w ) where h, w represent height and width n represents the image number.
Revised: For concreteness we can represent v ∈ V by the tuple ( n, h, w ) where h, w represent height and width n represents the image number.

Original: We now let φ ( v i ) be the output of the segmentation head, s v
Revised: We now let φ ( v i ) be the output of the segmentation head, s v

Original: i , at the image and spatial location v i .
Revised: i , at the image and spatial location v i .

Original: Using cosine distance, d cos ( x, y )
Revised: Using cosine distance, d cos ( x, y )

Original: =
Revised: =

Original: 1 − x | x | y | y | as the compatibility function, µ , yields the following:
Revised: 1 − x | x | y | y | as the compatibility function, µ , yields the following:

Original: [Equation]
Revised: [Equation]

Original: Wherte the argmin now ranges over the parameters of the segmentation head S .
Revised: Wherte the argmin now ranges over the parameters of the segmentation head S .

Original: We can now observe that the sum over all pairs v i , v j ∈ V can be written as a sum over pairs of images x, y ∈ X and pairs of spatial locations ( h, w ) , ( i, j ) where we note that ( i, j ) in this context refers to the spatial coordinates of image y
Revised: We can now observe that the sum over all pairs v i , v j ∈ V can be written as a sum over pairs of images x, y ∈ X and pairs of spatial locations ( h, w ) , ( i, j ) where we note that ( i, j ) in this context refers to the spatial coordinates of image y

Original: as in 3.1 and not the indices of the vertices.
Revised: as in 3.1 and not the indices of the vertices.

Original: [Equation]
Revised: [Equation]

Original: Where we deﬁne S ( x, y ) to be the segmentation feature correlation tensor for images x, y as deﬁned in Section 3.2.
Revised: Where we deﬁne S ( x, y ) to be the segmentation feature correlation tensor for images x, y as deﬁned in Section 3.2.

Original: Finally letting W ( x, y ) hwij =
Revised: Finally letting W ( x, y ) hwij =

Original: F hwij − b we recover our loss:
Revised: F hwij − b we recover our loss:

Original: [Equation]
Revised: [Equation]

Original: Finally we note that in practice we approximate the minimization using minibatch SGD, and our inclusion of KNN and Self-correspondence distillation changes the weight function w , but does not change its functional form.
Revised: Finally we note that in practice we approximate the minimization using minibatch SGD, and our inclusion of KNN and Self-correspondence distillation changes the weight function w , but does not change its functional form.

Original: Switching to the ML formulation of this problem allows us to solve this optimization for φ by gradient descent on the parameters of the segmentation head, S , and makes this computationally tractable.
Revised: Switching to the ML formulation of this problem allows us to solve this optimization for φ by gradient descent on the parameters of the segmentation head, S , and makes this computationally tractable.

Original: For large image datasets that can contain millions of high-resolution images, the induced graph can contain billions of image locations.
Revised: For large image datasets that can contain millions of high-resolution images, the induced graph can contain billions of image locations.

Original: Other graph embedding and clustering approaches such as Spectral methods require solving for eigenvalues of the graph Laplacian, which can take
Revised: Other graph embedding and clustering approaches such as Spectral methods require solving for eigenvalues of the graph Laplacian, which can take

Original: O ( |V| 3 ) time (Yan et al., 2009).
Revised: O ( |V| 3 ) time (Yan et al., 2009).

Original: More recent attempts to accelerate Spectral clustering such as (Yan et al., 2009) and (Han & Filippone, 2017) further assume a “Nonparametric” structure on the function φ , where a separate cluster assignment is learned for each vertex.
Revised: More recent attempts to accelerate Spectral clustering such as (Yan et al., 2009) and (Han & Filippone, 2017) further assume a “Nonparametric” structure on the function φ , where a separate cluster assignment is learned for each vertex.

Original: This assumption of a “nonparametric” function φ can be undesirable as one cannot cluster or embed new data without recomputing the entire clustering.
Revised: This assumption of a “nonparametric” function φ can be undesirable as one cannot cluster or embed new data without recomputing the entire clustering.

Original: In contrast, STEGO’s backbone and segmentation head act as a parametric form for the function φ allowing the approach to output predictions for novel images.
Revised: In contrast, STEGO’s backbone and segmentation head act as a parametric form for the function φ allowing the approach to output predictions for novel images.

Original: A. C ONTINUOUS , U NSUPERVISED , AND M INI - BATCH CRF
Revised: A. C ONTINUOUS , U NSUPERVISED , AND M INI - BATCH CRF

Original: Fully connected Gaussian Conditional Random Fields (CRFs) (Lafferty et al., 2001) are an extremely popular addition to semantic segmentation architectures.
Revised: Fully connected Gaussian Conditional Random Fields (CRFs) (Lafferty et al., 2001) are an extremely popular addition to semantic segmentation architectures.

Original: The CRF has the ability to improve initial predictions of locations, and can “sharpen” predictions to make them consistent with edges and areas with consistent color in the original image.
Revised: The CRF has the ability to improve initial predictions of locations, and can “sharpen” predictions to make them consistent with edges and areas with consistent color in the original image.

Original: CRF post-processing for reﬁning supervised and weakly supervised semantic segmentation predictions is ubiquitous in the literature (Lafferty et al., 2001;
Revised: CRF post-processing for reﬁning supervised and weakly supervised semantic segmentation predictions is ubiquitous in the literature (Lafferty et al., 2001;

Original: Chen et al., 2014; Long et al., 2015; Liu et al., 2020; Ahn et al., 2019).
Revised: Chen et al., 2014; Long et al., 2015; Liu et al., 2020; Ahn et al., 2019).

Original: Recently, new connections between CRF message passing and convolutional networks have allowed CRFs to be embedded into existing models (Chen et al., 2017; Teichmann & Cipolla, 2018) and trained jointly for better performance.
Revised: Recently, new connections between CRF message passing and convolutional networks have allowed CRFs to be embedded into existing models (Chen et al., 2017; Teichmann & Cipolla, 2018) and trained jointly for better performance.

Original: By connecting the STEGO correspondence distillation loss to the energy of an undirected model on image pixels we can use the same minibatch MLE strategy to estimate other similar graphical models.
Revised: By connecting the STEGO correspondence distillation loss to the energy of an undirected model on image pixels we can use the same minibatch MLE strategy to estimate other similar graphical models.

Original: For example, in the fully connected Gaussian edge potential CRF, one forms a pairwise potential function potential function for the pixels of a single image:
Revised: For example, in the fully connected Gaussian edge potential CRF, one forms a pairwise potential function potential function for the pixels of a single image:

Original: [Equation]
Revised: [Equation]

Original: Where p i represent the pixel coordinates associated with node v i and I i represents pixel colors associated with node v i .
Revised: Where p i represent the pixel coordinates associated with node v i and I i represents pixel colors associated with node v i .

Original: The parameters a, b, θ α , θ β , θ γ are hyperparameters and control the behavior of the model.
Revised: The parameters a, b, θ α , θ β , θ γ are hyperparameters and control the behavior of the model.

Original: These parameters balance the effect of long- and short-range color similarities against smoothness.
Revised: These parameters balance the effect of long- and short-range color similarities against smoothness.

Original: The CRF directly learns a pixel-wise array of probabilistic class assignments over k labels corresponding to the probability simplex code space C
Revised: The CRF directly learns a pixel-wise array of probabilistic class assignments over k labels corresponding to the probability simplex code space C

Original: = P ( l ) and a non-parametric clustering function f .
Revised: = P ( l ) and a non-parametric clustering function f .

Original: For a compatibility function µ the CRF chooses the Potts Model (Potts, 1952): µ potts ( φ ( v i ) , φ ( v j )) :
Revised: For a compatibility function µ the CRF chooses the Potts Model (Potts, 1952): µ potts ( φ ( v i ) , φ ( v j )) :

Original: = P ( φ ( v i ) (cid:54) = φ ( v j )) .
Revised: = P ( φ ( v i ) (cid:54) = φ ( v j )) .

Original: With this setting of the weights and compatibility function, we directly recover the binary potentials of the fully connected Gaussian edge potential CRF (Kr¨ahenb¨uhl & Koltun, 2011).
Revised: With this setting of the weights and compatibility function, we directly recover the binary potentials of the fully connected Gaussian edge potential CRF (Kr¨ahenb¨uhl & Koltun, 2011).

Original: We can also add the unary potentials which are often the outputs of another model.
Revised: We can also add the unary potentials which are often the outputs of another model.

Original: However, for our analysis we explore the case without unary potentials which yields an “unsupervised” variant of the CRF.
Revised: However, for our analysis we explore the case without unary potentials which yields an “unsupervised” variant of the CRF.

Original: However, without external unary potential terms, the strictly positive similarity kernel encourages the maximum likelihood estimator (MLE) of the graph to be the constant function.
Revised: However, without external unary potential terms, the strictly positive similarity kernel encourages the maximum likelihood estimator (MLE) of the graph to be the constant function.

Original: To rectify this, we can add small negative constant, − b , to the weight tensor to push unrelated pixels apart.
Revised: To rectify this, we can add small negative constant, − b , to the weight tensor to push unrelated pixels apart.

Original: This negative force is the direct analogue of the negative pressure hyper-parameter in STEGO and can be interpreted through the lens of negative sampling (Mikolov et al., 2013).
Revised: This negative force is the direct analogue of the negative pressure hyper-parameter in STEGO and can be interpreted through the lens of negative sampling (Mikolov et al., 2013).

Original: This negative shift also appears in the word2vec and graph2vec embedding techniques (Narayanan et al., 2017; Levy & Goldberg, 2014).
Revised: This negative shift also appears in the word2vec and graph2vec embedding techniques (Narayanan et al., 2017; Levy & Goldberg, 2014).

Original: Our shifted CRF potential encourages natural clusters to form that respect the structure of the potentials that capture similarities in pixel colors and locations.
Revised: Our shifted CRF potential encourages natural clusters to form that respect the structure of the potentials that capture similarities in pixel colors and locations.

Original: In the discrete case, solutions to this equation resemble superpixel algorithms such as SLIC (Zhang et al., 2015).
Revised: In the discrete case, solutions to this equation resemble superpixel algorithms such as SLIC (Zhang et al., 2015).

Original: Additionally lifting this to the continuous code space and provide a natural continuous generalization of superpixels and seems to avoid challenging local minima.
Revised: Additionally lifting this to the continuous code space and provide a natural continuous generalization of superpixels and seems to avoid challenging local minima.

Original: We illustrate these solutions to just the unsupervised CRF potential in Figure 13.
Revised: We illustrate these solutions to just the unsupervised CRF potential in Figure 13.

Original: Finally, we note that the second term of Equation 15, referred to as the smoothness kernel, matches IIC’s notion of local class consistency.
Revised: Finally, we note that the second term of Equation 15, referred to as the smoothness kernel, matches IIC’s notion of local class consistency.

Original: However, we found that adding these CRF terms to the self-correspondence loss of STEGO did not improve performance.
Revised: However, we found that adding these CRF terms to the self-correspondence loss of STEGO did not improve performance.

Original: A. I MPLEMENTATION D ETAILS
Revised: A. I MPLEMENTATION D ETAILS

Original: Model STEGO uses the “ViT-Base” architecture of DINO pre-trained on ImageNet.
Revised: Model STEGO uses the “ViT-Base” architecture of DINO pre-trained on ImageNet.

Original: This backbone was trained using self-supervision without access to ground-truth labels.
Revised: This backbone was trained using self-supervision without access to ground-truth labels.

Original: We use the “teacher” weights when creating our backbone.
Revised: We use the “teacher” weights when creating our backbone.

Original: We take the ﬁnal layer of spatially varying features and apply a small amount ( p = 0 . 1 ) of channel-wise dropout (Srivastava et al., 2014) before using them throughout the architecture during training.
Revised: We take the ﬁnal layer of spatially varying features and apply a small amount ( p = 0 . 1 ) of channel-wise dropout (Srivastava et al., 2014) before using them throughout the architecture during training.

Original: Our segmentation head consists of a linear network and a two-layer ReLU MLP added together and outputs a 70 dimensional vector.
Revised: Our segmentation head consists of a linear network and a two-layer ReLU MLP added together and outputs a 70 dimensional vector.

Original: We use the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0 . 0005 and a batch size of 32 .
Revised: We use the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0 . 0005 and a batch size of 32 .

Original: To make our losses resolution independent we sample 121 random spatial locations in the source and target implementations and use grid sampling (Jaderberg et al., 2015) to sample features from the backbone and segmentation heads.
Revised: To make our losses resolution independent we sample 121 random spatial locations in the source and target implementations and use grid sampling (Jaderberg et al., 2015) to sample features from the backbone and segmentation heads.

Original: Our cluster probe is trained alongside the STEGO architecture using a minibatch k-means loss where closeness is measured by cosine distance.
Revised: Our cluster probe is trained alongside the STEGO architecture using a minibatch k-means loss where closeness is measured by cosine distance.

Original: Cluster and linear probes are trained with separate Adam optimizers using a learning rate of .
Revised: Cluster and linear probes are trained with separate Adam optimizers using a learning rate of .

Original: Datasets We use the training and validation sets of Cocostuff described ﬁrst in Ji et al.
Revised: Datasets We use the training and validation sets of Cocostuff described ﬁrst in Ji et al.

Original: and used throughout the literature including in Cho et al.
Revised: and used throughout the literature including in Cho et al.

Original: We note that the validation set used in
Revised: We note that the validation set used in

Original: Ji et al.
Revised: Ji et al.

Original: (2019) is a subset of the full CocoStuff validation set
Revised: (2019) is a subset of the full CocoStuff validation set

Original: and we use this validation subset to be consistent with prior benchmarks.
Revised: and we use this validation subset to be consistent with prior benchmarks.

Original: We note that using the full validation set does not change results signiﬁcantly.
Revised: We note that using the full validation set does not change results signiﬁcantly.

Original: When ﬁve-cropping images we use a target size of ( . 5 h, .
Revised: When ﬁve-cropping images we use a target size of ( . 5 h, .

Original: 5 w ) for each crop where h, w are the original image height and width.
Revised: 5 w ) for each crop where h, w are the original image height and width.

Original: Training images are then scaled to have minor axis equal to 224 and are then center cropped to (224 , 224) , validation images are ﬁrst scaled to 320 then are center cropped to (320 , 320) .
Revised: Training images are then scaled to have minor axis equal to 224 and are then center cropped to (224 , 224) , validation images are ﬁrst scaled to 320 then are center cropped to (320 , 320) .

Original: All image resizing uses bilinear interpolation and resizing of target tensors for evaluation uses nearest neighbor interpolation.
Revised: All image resizing uses bilinear interpolation and resizing of target tensors for evaluation uses nearest neighbor interpolation.

Original: CRF
Revised: CRF

Original: We use PyDenseCRF
Revised: We use PyDenseCRF

Original: (Kr ¨ ahenb ¨ uhl & Koltun, 2011) with 10 iterations with parameters a = 4 , b = 3 , θ α
Revised: (Kr ¨ ahenb ¨ uhl & Koltun, 2011) with 10 iterations with parameters a = 4 , b = 3 , θ α

Original: = 67 , θ β = 3 , θ γ = 1 as written in Section A.9.
Revised: = 67 , θ β = 3 , θ γ = 1 as written in Section A.9.

Original: Compute All experiments use PyTorch (Paszke et al., 2019) v1.7 pre-trained models, on an
Revised: Compute All experiments use PyTorch (Paszke et al., 2019) v1.7 pre-trained models, on an

Original: Ubuntu 16.04 Azure NV24 Virtual Machine with Python 3.6.
Revised: Ubuntu 16.04 Azure NV24 Virtual Machine with Python 3.6.

Original: Experiments use PyTorch Lightning for distributed and multi-gpu training when necessary (Falcon et al., 2019).
Revised: Experiments use PyTorch Lightning for distributed and multi-gpu training when necessary (Falcon et al., 2019).

Original: Hyperparameters We use the following hyperparameters for our results in Tables 1 and 3:
Revised: Hyperparameters We use the following hyperparameters for our results in Tables 1 and 3:

Original: [Table]
Revised: [Table]

Original: A. A H EURISTIC FOR S ETTING H YPER - PARAMETERS
Revised: A. A H EURISTIC FOR S ETTING H YPER - PARAMETERS

Original: Setting hyperparameters without cross-validation on ground truth data can be difﬁcult and this is an outstanding challenges with the STEGO architecture that we hope can be solved in future work.
Revised: Setting hyperparameters without cross-validation on ground truth data can be difﬁcult and this is an outstanding challenges with the STEGO architecture that we hope can be solved in future work.

Original: Nevertheless we have identiﬁed some key intuition to guide manual hyperparameter tuning.
Revised: Nevertheless we have identiﬁed some key intuition to guide manual hyperparameter tuning.

Original: More speciﬁcally, we ﬁnd that the most important factor affecting performance is the balance of positive and negative forces.
Revised: More speciﬁcally, we ﬁnd that the most important factor affecting performance is the balance of positive and negative forces.

Original: Too much negative feedback and vectors will all push apart and clusters will not form well, too much positive feedback and the system will tend towards a small number of clusters.
Revised: Too much negative feedback and vectors will all push apart and clusters will not form well, too much positive feedback and the system will tend towards a small number of clusters.

Original: To debug this balance, we found it useful to visualize the distribution of feature correspondence similarities as a function of training step as shown in Figure 14.
Revised: To debug this balance, we found it useful to visualize the distribution of feature correspondence similarities as a function of training step as shown in Figure 14.

Original: A balanced system (Orange distribution) will tend towards a bi-modal distribution with peaks at alignment 1 or orthogonality at 0.
Revised: A balanced system (Orange distribution) will tend towards a bi-modal distribution with peaks at alignment 1 or orthogonality at 0.

Original: This bi-modal structure is indicative that there is some clustering within images, but that not everything is assigned to the same cluster.
Revised: This bi-modal structure is indicative that there is some clustering within images, but that not everything is assigned to the same cluster.

Original: Pink and blue distributions show too much positive and negative signal respectively.
Revised: Pink and blue distributions show too much positive and negative signal respectively.

Original: We ﬁnd that given a reasonable balance of the λ ’s, this balance can be achieved by tuning the b s to achieve the desired balance.
Revised: We ﬁnd that given a reasonable balance of the λ ’s, this balance can be achieved by tuning the b s to achieve the desired balance.

Original: A. A NOTE ON 5-C ROP N EAREST N EIGHBORS
Revised: A. A NOTE ON 5-C ROP N EAREST N EIGHBORS

Original: We found that pre-processing the dataset by 5-cropping images was a simple and effective way to improve the spatial resolution of STEGO and the quality of K-Nearest Neighbors.
Revised: We found that pre-processing the dataset by 5-cropping images was a simple and effective way to improve the spatial resolution of STEGO and the quality of K-Nearest Neighbors.

Original: We consider each resulting 5-crop as a separate image when computing KNNs and patches from the same image are valid KNNs.
Revised: We consider each resulting 5-crop as a separate image when computing KNNs and patches from the same image are valid KNNs.

Original: Figure 15 shows the distribution of these self-matches for the CocoStuff dataset.
Revised: Figure 15 shows the distribution of these self-matches for the CocoStuff dataset.

Original: We note that the majority of patches do not have any nearest neighbors from the same image.
Revised: We note that the majority of patches do not have any nearest neighbors from the same image.