Original: CAST: C ONCURRENT R ECOGNITION AND S EGMENTATION WITH A DAPTIVE S EGMENT T OKENS
Revised: CAST: C ONCURRENT R ECOGNITION AND S EGMENTATION WITH A DAPTIVE S EGMENT T OKENS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recognizing an image and segmenting it into coherent regions are often treated as separate tasks.
Revised: Recognizing an image and segmenting it into coherent regions are often treated as separate tasks.

Original: Human vision, however, has a general sense of segmentation hierarchy before recognition occurs.
Revised: Human vision, however, has a general sense of segmentation hierarchy before recognition occurs.

Original: We are thus inspired to learn image recognition with hierarchical image segmentation based entirely on unlabeled images.
Revised: We are thus inspired to learn image recognition with hierarchical image segmentation based entirely on unlabeled images.

Original: Our insight is to learn fine-to-coarse features concurrently at superpixels, segments, and full image levels, enforcing consistencyand goodness of feature induced segmentations while maximizing discrimination among image instances.
Revised: Our insight is to learn ﬁne-to-coarse features concurrently at superpixels, segments, and full image levels, enforcing consistency induced segmentations while maximizing discrimination among image instances.

Original: Our model innovates vision transformers on three aspects.
Revised: Our model innovates vision transformers in three aspects.

Original: We use adaptive segment tokens instead of fixed-shape patch tokens.
Revised: We use adaptive segment tokens instead of ﬁxed-shape patch tokens.

Original: We create a token hierarchy by inserting graph pooling between transformer blocks, naturally producing consistent multi-scale segmentations while increasing the segment size and reducing the number of tokens.
Revised: We create a token hierarchy by inserting graph pooling between transformer blocks, naturally producing consistent multi-scale segmentations while increasing the segment size and reducing the number of tokens.

Original: We produce hierarchical image segmentation for free while training for recognition by maximizing image-wise discrimination.
Revised: We produce hierarchical image segmentation for free while training for recognition by maximizing image-wise discrimination.

Original: Our work delivers the first concurrent recognition and hierarchical segmentation model without any supervision .
Revised: Our work delivers the ﬁrst concurrent recognition and hierarchical segmentation model without any supervision .

Original: Validated on ImageNet and PASCAL VOC, it achieves better recognition and segmentation with higher computational efficiency.
Revised: Validated on ImageNet and PASCAL VOC, it achieves better recognition and segmentation with higher computational efﬁciency.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Convolutional neural networks (CNN) (LeCun et al., 1989; Krizhevsky et al., 2012; He et al., 2016) and Vision Transformers (ViT)
Revised: Convolutional neural networks (CNN) (LeCun et al., 1989; Krizhevsky et al., 2012; He et al., 2016) and Vision Transformers (ViT)

Original: (Dosovitskiy et al., 2020) have been very successful in computer vision.
Revised: (Dosovitskiy et al., 2020) have been very successful in computer vision.

Original: However, recognizing an image and segmenting it into coherent regions are treated as separate tasks or learned sequentially (Martin et al., 2001).
Revised: However, recognizing an image and segmenting it into coherent regions are treated as separate tasks or learned sequentially (Martin et al., 2001).

Original: Fig.
Revised: Fig.

Original: 1 illustrates a common practice: CNN (ViT) predicts the semantic class of an image based on the image-level feature from the output of the final convolutional layer (transformer block), and additional clustering based on earlier pixel-wise features is required to generate image segmentation (Hwang et al., 2019; Ke et al., 2022).
Revised: 1 illustrates a common practice: CNN (ViT) predicts the semantic class of an image based on the image-level feature from the output of the ﬁnal convolutional layer (transformer block), and additional clustering based on earlier pixel-wise features is required to generate image segmentation (Hwang et al., 2019; Ke et al., 2022).

Original: However, human vision has a general sense of segmentation hierarchy, in terms of groups of pixels or segments , before recognition even occurs.
Revised: However, human vision has a general sense of segmentation hierarchy, in terms of groups of pixels or segments , before recognition even occurs.

Original: This perceptual organization perspective (Witkin &
Revised: This perceptual organization perspective (Witkin &

Original: Tenenbaum, 1983; Biederman, 1987) has been overlooked in CNN and ViT architectures. Models optimized for image classification have no concepts of parts and wholes; recognition is achieved without understanding how different parts such aseyes and face  are organized for the whole animal .
Revised: Tenenbaum, 1983; Biederman, 1987) has been overlooked in CNN and ViT architectures: models optimized for image classiﬁcation tend to latch onto discriminative parts (Selvaraju et al., 2017) such as faces, often missing inconspicuous body parts that go with the face. Previous methods seldom model how different parts such as face and body are organized for the whole animal explicitly.

Original: To understand the connections between parts and wholes, visual information must be extracted locally and globally.
Revised: To understand the connections between parts and wholes, visual information must be extracted locally and globally.

Original: There are three major approaches (Fig. 2):
Revised: There are three major approaches (Fig. 2):

Original: Spatial downsampling: With pixels laid on a regular grid, features are extracted from patches. 
Revised: Spatial downsampling: With pixels laid on a regular grid, features are extracted from patches. The

Original: The granularity of visual information is determined by the patch size.
Revised: granularity of visual information is determined by the patch size.

Original: Max or mean pooling (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014), uniform sub-sampling (striding) (He et al., 2016) and patch merging (Liu et al., 2021) are performed multiple times to increase the effective receptive field size (Luo et al., 2016).
Revised: Max or mean pool- ing (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014), uniform sub-sampling (striding) (He et al., 2016) and patch merging (Liu et al., 2021) are performed multiple times to increase the effective receptive ﬁeld size (Luo et al., 2016).

Revised: To generate a segmentation, the output features need to be upsampled and

Revised: clustered

Original: To generate a segmentation, the output features need to be upsampled and clustered (e.g. K-Means (Hwang et al., 2019)), often resulting in over-smoothed boundaries.
Revised: (e.g. K -Means (Hwang et al., 2019)), often resulting in over-smoothed boundaries.

Original: Attention: Inspired by Natural Language Processing (NLP), image patches are treated as visual word tokens of the entire image document .
Revised: Attention: Inspired by Natural Language Processing (NLP), image patches are treated as visual word tokens of the entire image document .

Original: To extract more global information, ViT contextually updates feature representations based on pair-wise correlation among all the tokens of an image, using attention modules (Vaswani et al.,2017).
Revised: To extract more global information, ViT contextually updates feature representations based on pair-wise correlation among all the tokens of an image, using attention modules (Vaswani et al.,

Original: However, ViT is computationally inefficient as all image tokens are kept in every transformer block. 
Revised: However, ViT is computationally inefﬁcient as all image tokens are kept in every transformer block. 3.

Original: Significance-based subsampling: To increase ViT’s computational efficiency, tokens are subsampled at higher levels based on their significance scores. 
Revised: Signiﬁcance-based subsampling: To increase ViT’s computational efﬁciency, tokens are subsampled at higher levels based on their signiﬁcance scores. PoWER-BERT

Original: PoWER-BERT (Goyal et al., 2020) and Token Pooling (Marin et al., 2021) define the significance score as the total attention given to each token from all other tokens.
Revised: Goyal et al., 2020) and Token Pooling (Marin et al. 2021) deﬁne the signiﬁcance score as the total attention given to each token from all other tokens.

Original: Downsampling then retains only  most dominant visual features in the image.
Revised: Downsampling then retains only the most dominant visual features in the image.

Original: Such methods only keep the most informative tokens in final output representations.
Revised: Such methods only keep the most informative tokens in ﬁnal output representations.

Original: These existing methods have two major issues.
Revised: These existing methods have two major issues.

Original: 1) Both CNN and ViT models take regularly shaped patch features as inputs, regardless of what is in the image.
Revised: 1) Both CNN and ViT models take regularly shaped patch features as inputs, regardless of what is in the image.

Original: Image segmentation derived from such representations often fails to align with contours.
Revised: Image segmentation derived from such representations often fails to align with contours.

Original: 2) Image segmentation does not involve local-toglobal feature extraction, which is treated as a separate visual task from image-wise recognition.
Revised: 2) Image segmentation does not involve local-toglobal feature extraction, which is treated as a separate visual task from image-wise recognition.

Original: Our first insight is that pixel groupings are not a computational inconvenience (as opposed to regular patches), but a natural structure to be exploited for better visual computing.
Revised: Our ﬁrst insight is that pixel groupings are not a computational inconvenience (as opposed to regular patches), but a natural structure to be exploited for better visual computing.

Original: Unlike existing CNN and
Revised: Unlike existing CNN and

Original: ViT which extract features on a regular grid throughout the entire model, we directly get to low-level pixel groupings at an early stage and develop feature representations subsequently.
Revised: ViT which extract features on a regular grid throughout the entire model, we directly get to low-level pixel groupings at an early stage and develop feature representations subsequently.

Original: Our model takes segment features as input tokens and carries this adaptive segment representation through deeper layers.
Revised: Our model takes segment features as input tokens and carries this adaptive segment representation through deeper layers.

Original: Post-processing with pixel-wise clustering methods is no longer needed.
Revised: Post-processing with pixel-wise clustering methods is no longer needed.

Original: Our second insight is to derive fine-to-coarse pixel groupings jointly with local-to-global feature extraction.
Revised: Our second insight is to derive ﬁne-to-coarse pixel groupings jointly with local-to-global feature extraction.

Original: Given a set of token features, we cluster them into fewer components.
Revised: Given a set of token features, we cluster them into fewer components.

Original: The next-level feature is the result of pooling current features within each cluster.
Revised: The next-level feature is the result of pooling current features within each cluster.

Original: Since our input tokens come from segments of an image, feature clustering turns fine-grained segments into coarse-grained regions.
Revised: Since our input tokens come from segments of an image, feature clustering turns ﬁne-grained segments into coarse-grained regions.

Original: By repeating the procedure, we obtain a consistent fine-to-coarse (hierarchical) image segmentation and corresponding feature representations at each level of granularity.
Revised: By repeating the procedure, we obtain a consistent ﬁne-to-coarse (hierarchical) image segmentation and corresponding feature representations at each level of granularity.

Original: We propose to integrate such data-driven perceptual organization into Vision Transformers
Revised: We propose to integrate such data-driven perceptual organization into Vision Transformers

Original: (Dosovitskiy et al., 2020).
Revised: (Dosovitskiy et al., 2020).

Original: We develop Concurrent recognition and segmentation with Adaptive Segment
Revised: We develop Concurrent recognition and segmentation with Adaptive Segment

Original: Tokens (CAST).
Revised: Tokens (CAST).

Original: It has three novel aspects (Fig. 2).
Revised: It has three novel aspects (Fig. 2).

Original: We use adaptive segment tokens instead of fixed-shape patch tokens.
Revised: We use adaptive segment tokens instead of ﬁxed-shape patch tokens.

Original: They no longer live on a regular grid, and their shapes and numbers vary with the image.
Revised: They no longer live on a regular grid, and their shapes and numbers vary with the image.

Original: We create a token hierarchy by inserting graph pooling between transformer blocks, naturally producing consistent multi-scale segmentations while increasing the segment sizeand reducing the number of tokens. We learn segmentation for free while training the model for
Revised: We create a token hierarchy by inserting graph pooling between transformer blocks, naturally producing consistent multi-scale segmentations while increasing the segment size

Original: [Figure]
Revised: [Figure]

Original: models
Revised: models

Original: CNN
Revised: CNN

Original: ViT
Revised: [Figure]

Original: Token Pooling
Revised: Token Pooling

Original: Our CAST nodes patches patches patches segments edges local / fixed global / dynamic global / dynamic global / dynamic coarsening ✓ ✗ ✓ ✓ segmentation ✗
Revised: Our CAST nodes patches patches patches segments edges local / ﬁxed global / dynamic global / dynamic global / dynamic coarsening (cid:51) (cid:55) (cid:51) (cid:51) segmentation (cid:55) (cid:55) (cid:55) (cid:51) and reducing the number of tokens.

Original: ✗ ✗ ✓ unsupervised recognition by maximizing image-wise discrimination (Chen et al., 
Revised: We learn segmentation for free while training the model for unsupervised recognition by maximizing image-wise discrimination (Chen et al., 2021).

Original: Neither recognition nor segmentation requires any labeled supervision.
Revised: Neither recognition nor segmentation requires any labeled supervision.

Original: Our experimental results demonstrate our superior computational efficiency and segmentation accuracy on ImageNet and PASCAL VOC.
Revised: Our experimental results demonstrate our superior computational efﬁciency and segmentation accuracy on ImageNet and PASCAL VOC.

Original: More importantly, our model delivers far more precise foreground masks which can be very useful in a wide range of dense pixel applications.
Revised: More importantly, our model delivers far more precise foreground masks which can be very useful in a wide range of dense pixel applications.

Original: In short our work makes three major contributions.
Revised: In short, our work makes three major contributions.

Original: We develop the first vision transformer model that can concurrently achieve image-wise recognition and hierarchical image segmentation without any additional processing.
Revised: We develop the ﬁrst vision transformer model that can concurrently achieve image-wise recognition and hierarchical image segmentation without any additional processing.

Original: We outperform existing token coarsening methods for both image classification and segmentation tasks.
Revised: We outperform existing token coarsening methods for both image classiﬁcation and segmentation tasks.

Original: We achieve a better trade-off between model efficiency and task performance.
Revised: We achieve a better trade-off between model efﬁciency and task performance.

Original: We deliver better attention maps that capture foreground semantics without supervision, with many potential applications beyond segmentation.
Revised: We deliver better attention maps that capture foreground semantics without supervision, with many potential applications beyond segmentation.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Vision Transformers.
Revised: Vision Transformers.

Original: Vision transformers (ViT)
Revised: Vision transformers (ViT)

Original: (Dosovitskiy et al., 2020) and its followups (Touvron et al., 2021; Yuan et al., 2021; Wang et al., 2021a) adopt the transformer architecture originally for NLP proposed in (Vaswani et al., 2017).
Revised: (Dosovitskiy et al., 2020) and its followups (Touvron et al., 2021; Yuan et al., 2021; Wang et al., 2021a) adopt the transformer architecture originally for NLP proposed in Vaswani et al.

Original: Remarkably, ViT achieves state-of-the-art performance on ImageNet1k (Deng et al., 2009) with pretraining on JFT-300M. However, its high computation costs limit its applications.
Revised: ViT achieves remarking performance on image classiﬁcation (Deng et al., 2009), however, its high computation costs limit its applications.

Original: Its computational complexity is derived from two factors: the latent feature dimensions and  number of tokens.
Revised: Its computational complexity is derived from two factors: the latent feature dimensions and the number of tokens.

Original: Many research works then tackle one of these aspects.

Original: To reduce the latent feature computation, one direction is to restrict the attention connections by leveraging spatial relationship in the data (Parmar et al., 2018;
Revised: To reduce the latent feature computation, one direction is to restrict the attention connections by leveraging spatial relationships in the data (Parmar et al., 2018;

Original: Ramachandran et al., 2019; Beltagy et al., 2020; Child et al., 2019; Zaheer et al., 2020) or utilizing hashing, sorting, or compression (Kitaev et al., 2020; Vyas et al., 2020; Tay et al., 2020;
Revised: Ramachandran et al., 2019; Beltagy et al., 2020; Child et al., 2019; Zaheer et al., 2020) or utilizing hashing, sorting, or compression (Kitaev et al., 2020; Vyas et al., 2020; Tay et al., 2020;

Original: Liu et al., 2018; Wang et al., 2020).
Revised: Liu et al., 2018; Wang et al., 2020).

Original: To reduce the number of tokens, two camps of approaches are proposed.
Revised: To reduce the number of tokens, two camps of approaches are proposed.

Original: The first is to apply the concepts of hierarchical convolutional neural nets to downsample tokens using various pooling methods (Liu et al., 2021; Heo et al., 2021; Dong et al., 2022).
Revised: The ﬁrst is to apply the concepts of hierarchical convolutional neural nets to downsample tokens using various pooling methods (Liu et al., 2021; Heo et al., 2021; Dong et al., 2022).

Original: The other attempts to measure the significance scores among the tokens and drop or prune tokens accordingly (Goyal et al., 2020; Rao et al., 2021; Marin et al., 2021).
Revised: The other attempts to measure the signiﬁcance scores among the tokens and drop or prune tokens accordingly (Goyal et al., 2020; Rao et al., 2021; discarded coarser

Original: This camp is the most related to our work.

Original: Our work differs that tokens are not discarded but merged into coarser ones.

Original: TCFormer (Zeng et al., 2022) and GroupViT (Xu et al., 2022) are two recently proposed hierarchical vision transformers.
Revised: TCFormer (Zeng et al., 2022) and GroupViT (Xu et al., 2022) are two recently proposed hierarchical vision transformers.

Original: Both models still usefixed-shape patch tokens, resulting in noise among object boundaries.
Revised: Both models still use patch tokens.

Original: In addition, TCFormer is only demonstrated in supervised settings, and GroupViT requires text supervision.
Revised: TCFormer is only applied to supervised tasks, and GroupViT requires text supervision.

Original: In contrast, our work gets to fine-grained pixel groupings early and develops adaptive segment tokens throughout the model, which naturally induces hierarchical image segmentations.
Revised: Our work operates on adaptive segment tokens, which naturally induce hierarchical image segmentations.

Original: Our model does not require any human labels.
Revised: Our model does not require any human label.

Original: Image Segmentation and Clustering.
Revised: Superpixels. Superpixels are referred to as sets of locally connected pixels that contain coherent structures (e.g. colors) (Ren & Malik, 2003). Superpixels have been applied to densely labeling tasks, such as body-part parsing (Mori et al., 2004), saliency detection (Ren et al., 2013), image segmentation (Gould et al., 2008; Fulkerson et al., 2009; Sharma et al., 2014; Gadde et al., 2016), and hierarchical segmentation (Wei et al., 2018).

Revised: Recently, Zhang et al.

Revised: (2022) tackle semantic segmentation by replacing patch with superpixel tokens in ViT architectures.

Revised: In contrast, our model further creates a segment hierarchy and performs both classiﬁcation and segmentation concurrently.

Revised: Image Segmentation and Clustering.

Original: Image segmentation is referred to as partitioning an image by coherent regions.
Revised: Image segmentation is referred to as partitioning an image into coherent regions.

Original: Classic methods have two steps: extracting local features and clustering them based on different criteria, e.g., mode-finding (Comaniciu & Meer, 2002; Banerjee et al., 2005), or graph partitioning (Felzenszwalb & Huttenlocher, 2004; Shi & Malik, 2000; Malik et al., 2001; Yu & Shi, 2003; 2004).
Revised: Classic methods have two steps: extracting local features and clustering them based on different criteria, e.g., mode-ﬁnding (Comaniciu & Meer, 2002; Banerjee et al., 2005), or graph partitioning (Felzenszwalb & Huttenlocher, 2004; Shi & Malik, 2000; Malik et al., 2001; Yu & Shi, 2003a; 2004).

Original: Conventionally, a hierarchical segmentation is predicted as output for comparing against human perception (Arbelaez et al., 2010).
Revised: A hierarchical segmentation is predicted as output for comparing against human perception (Arbelaez et al., 2010).

Original: The common approaches, to avoid ambiguities along object boundaries, typically resort to contour detection (Hwang & Liu, 2015; Xie & Tu, 2015) and eliminate contours one by one to form different levels of segmentation (OWT-UCM) (Arbelaez et al., 2010).
Revised: The common approaches, to avoid ambiguities along object boundaries, typically resort to contour detection (Hwang & Liu, 2015; Xie & Tu, 2015) and eliminate contours iteratively to form multi-scale segmentations (Arbelaez et al., 2010).

Original: Such approaches train on the finest level of groundtruth segmentation and hope to produce coarser levels of segmentation automatically at inference time.
Revised: Such approaches train on the ﬁnest level of ground-truth segmentation and hope to produce coarser levels of segmentation automatically for inference.

Original: On the other hand, our work operates directly on segments as opposed to contour proxies.
Revised: Our work operates directly on segments as opposed to contour proxies.

Revised: Concurrent Recognition and Segmentation.

Revised: This idea was explored before the deep learning era:

Revised: Recognition by grouping compatible patches and segmentation by grouping visually similar pixels are solved together through detected pixel-patch relations, resulting in object-speciﬁc segmentation (Yu et al., 2002; Yu & Shi, 2003b) and ﬁgure-ground segmentation (Maire, 2010; Maire et al., 2011).

Revised: These methods rely on not only handcrafted features and grouping cues, but also pre-trained object part detectors, whereas our work does not use any such priors or supervised training.

Original: Self-supervised Segmentation and Representation Learning.
Revised: Self-supervised Segmentation and Representation Learning.

Original: Recent works can be categorized into three camps.
Revised: Recent works can be categorized into three camps.

Original: A straightforward approach is to leverage self-supervised image recognition and transfer the model to segmentation by increasing the location sensitivity (Wu et al., 2018;
Revised: A straightforward approach is to leverage self-supervised image recognition and transfer the model to segmentation by increasing the location sensitivity (Wu et al., 2018;

Original: He et al., 2020; Chen et al., 2020; Wang et al., 2021c), adding an contrastive loss across views (Wang et al., 2021b), or by stronger augmentation and constrained cropping (Van Gansbeke et al., 2021;
Revised: He et al., 2020; Chen et al., 2020; Wang et al., 2021c), adding a contrastive loss across views (Wang et al., 2021b), or by stronger augmentation and constrained cropping (Van Gansbeke et al., 2021;

Original: Selvaraju et al., 2021).
Revised: Selvaraju et al., 2021).

Original: A pixel-wise cluster predictor can be learned by maximizing the mutual information between cluster predictions on augmented views of the same instance at corresponding pixels (Ji et al., 2019; Ouali et al., 2020).
Revised: A pixel-wise cluster predictor can be learned by maximizing the mutual information between cluster predictions on augmented views of the same instance at corresponding pixels (Ji et al., 2019; Ouali et al., 2020).

Original: A pixel-level feature encoder can be learned directly by maximizing discrimination between pixels based on either contour-induced segments (Hwang et al., 2019), pre-computed region hierarchies (Zhang & Maire, 2020), or hierarchical clustering transformers (Ke et al., 2022).
Revised: A pixel-level feature encoder can be learned directly by maximizing discrimination between pixels based on either contour-induced segments (Hwang et al., 2019), pre-computed region hierarchies (Zhang & Maire, 2020), or hierarchical clustering transformers (Ke et al., 2022).

Original: Segmentation is thus derived from pixel feature similarities.
Revised: Segmentation is thus derived from pixel feature similarities.

Original: Our work unifies the first camp and the third camp, as we train ViT with a self-supervised image recognition framework while naturally producing unsupervised hierarchical segmentation.
Revised: Our work uniﬁes the ﬁrst and the third camp, as we train ViT with a self-supervised image recognition framework while naturally producing unsupervised hierarchical segmentation.

Original:  C ONCURRENT R ECOGNITION AND H IERARCHICAL S EGMENTATION
Revised:  C ONCURRENT R ECOGNITION AND H IERARCHICAL S EGMENTATION

Original: At the core of our method is to consider image recognition and segmentation as concurrent, not separate, tasks.
Revised: At the core of our method is to consider image recognition and segmentation as concurrent, not separate, tasks.

Original: The basic idea is to extract local-to-global visual information by producing fine-tocoarse image segmentations and corresponding feature representations.
Revised: The basic idea is to extract local-to-global visual information by producing ﬁne-tocoarse image segmentations and corresponding feature representations.

Original: Meanwhile, such multi-scale feature representations should support final image-level recognition.
Revised: Meanwhile, such multi-scale feature representations should support ﬁnal image-level recognition.

Original: We ground our approach on a general perspective: images as graphs where pixels are nodes.
Revised: We ground our approach on a general perspective: images as graphs where pixels are nodes.

Original: We first generate low-level superpixels given an input image and extract corresponding segment tokens.
Revised: We ﬁrst generate low-level superpixels given an input image and extract corresponding segment tokens.

Original: Then we integrate image segmentation into the ViT architecture, where transformer blocks and our proposed graph pooling modules take arbitrarily shaped segment, not fixed-shape patch, tokens as inputs and outputs coarsened segment tokens for the next transformer block.
Revised: Then we integrate image segmentation into the ViT architecture, where transformer blocks and our proposed graph pooling modules take arbitrarily shaped segment, not ﬁxed-shape patch, tokens as inputs and outputs coarsened segment tokens for the next transformer block.

Original: In other words, we augment ViT to cluster fine-grained segments into coarse-grained regions w.r.t group-wise correlationin the token feature space, resulting in hierarchical segmentation.
Revised: In other words, we augment ViT to cluster ﬁne-grained segments into coarse-grained regions w.r.t group-wise correlation

Original: By doing so, our model performs classification and hierarchical segmentation concurrently and more efficiently.

Original: We describe the three components in our model in each subsection. 1) Oversegmentations of input images based on low-level visual cues.
Revised: We describe the three components in our model in each subsection of input images based on low-level visual cues.

Original: 2) Transformer encoder blocks to update token features.
Revised: 2) Transformer encoder blocks to update token features.

Original: Graph pooling modules to cluster segment tokens and generate next-level representations.
Revised: Graph pooling modules to cluster segment tokens and generate next-level representations.

Original: See Fig.
Revised: See Fig.

Original: . 3. T HE F INEST - LEVEL P IXEL G ROUPINGS AND T OKEN F EATURES
Revised: T HE F INEST - LEVEL P IXEL G ROUPINGS AND T OKEN F EATURES

Original: Existing methods tackle image segmentation separately from feature extraction.
Revised: Existing methods tackle image segmentation separately from feature extraction.

Original: Such models extract patch features from the input image and then generate after-the-fact image segmentation by clustering the fixed patch features (similar to SegSort (Hwang et al., 2019) and HSG (Ke et al., 2022)).
Revised: Such models extract patch features from the input image and then generate after-the-fact image segmentation by clustering the ﬁxed patch features (similar to SegSort (Hwang et al., 2019) and HSG (Ke et al., 2022)).

Original: In stark contrast, our core idea is to involve image segmentation in feature extraction in the model architecture.
Revised: In stark contrast, our core idea is to involve image segmentation in feature extraction in the model architecture.

Original: We derive pixel groupings at an early stage, where we extract corresponding features for every segment.
Revised: We derive pixel groupings at an early stage, where we extract corresponding features for every segment.

Original: Our model carries such segment features to the subsequent layers and obtains image segmentations directly from the segment index of each pixel.
Revised: Our model carries such segment features to the subsequent layers and obtains image segmentations directly from the segment index of each pixel.

Original: Post-processing is thus not needed.
Revised: Post-processing is thus not needed.

Original: Given an input image, we start with the finest-level pixel groupings.
Revised: Given an input image, we start with the ﬁnest-level pixel groupings.

Original: We perform pixel groupings based on low-level visual cues to align segmentations with image contours.
Revised: We perform pixel groupings based on low-level visual cues to align segmentations with image contours.

Original: Specifically, we applyimage oversegmentation methods, such as Seeds (Bergh et al., 2012), to partition an image into locally connected and color-wise coherent regions, a.k.a superpixels.
Revised: Speciﬁcally, we apply oversegmentation methods, e.g. Seeds (Bergh et al., 2012), to partition an image into locally connected and color-wise coherent regions–superpixels.

Original: Detailed in section 3.3, we can produce a hierarchical image segmentation with precisely localized contours by progressively grouping these superpixels.
Revised: Detailed in section 3.3, we can produce a hierarchical image segmentation with precisely localized contours by progressively grouping these superpixels.

Original: To extract features of the superpixels, we first convolve the image with multiple convolutional layers, resulting in overlapping patch features.
Revised: To extract features of the superpixels, we ﬁrst convolve the image with multiple convolutional layers, resulting in overlapping patch features.

Original: We then average pool those patch features within each superpixel to derive initial segment tokens x s (with dimensions of the number of superpixels and number of feature channels).
Revised: We then average pool those patch features within each superpixel to derive initial segment tokens x s (with dimensions of the number of superpixels and number of feature channels).

Original: We then input the initial segment tokens x s , along with additional priors to our model.
Revised: We then input the initial segment tokens x s , along with additional priors to our model.

Original: Following
Revised: Following

Original: ViT (Dosovitskiy et al., 2020), we append x s with a learnable embedding, a.k.a [CLASS] token( x class ), to encode the most prominent features of an image.
Revised: ViT (Dosovitskiy et al., 2020), we append x s with a learnable embedding ( [CLASS] token x class ) to encode the most prominent features of an image.

Original: x class is randomly initialized and shared among different input images.
Revised: x class is randomly initialized and shared among different input images.

Original: We also enforce a spatial prior by adding x s with relative position encodings E pos .
Revised: We also enforce a spatial prior by adding x s with relative position encodings E pos .

Original: We initiate E pos at the same resolution as the convolutional patch features and then average pool within each superpixel.
Revised: We initiate E pos at the same resolution as the convolutional patch features and then average pool within each superpixel.

Original: To sum up, the input segment token is z 0 =
Revised: To sum up, the input segment token is z 0 =

Original: [ x class ; x s ] + E pos .
Revised: [ x class ; x s ] + E pos .

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: G ENERAL AND G LOBALIZED B ACKBONE A RCHITECTURES
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: 3. G ENERAL AND G LOBALIZED B ACKBONE A RCHITECTURES

Original: Most existing vision models digest pixels on a regular grid and update features within a limited range of neighboring grids.
Revised: Most existing vision models digest pixels on a regular grid and update features within a limited range of neighboring grids.

Original: Such methods have two limitations: 1) all features correspond to the same-shape patches in images, and 2) pixels/patches are locally connected andthus have little knowledge of global correlation.
Revised: Such methods have two limitations: 1) all features correspond to the same-shape patches in images, and 2) pixels/patches are locally connected and have little knowledge of global correlation.

Original: However, segmentations are adaptive to images: every segment has different shapes.
Revised: Yet, segmentations are adaptive to images: every segment has different shapes.

Original: Optimal segmentation also requires group-wise correlation amongimage pixels (Shi & Malik, 2000).
Revised: Optimal segmentation also requires group-wise correlation among pixels (Shi & Malik, 2000).

Original: Therefore, we prefer globalized architectures, such as ViT(Dosovitskiy et al., 2020) and GNN (Kipf & Welling, 2016), over the grid-based models.[Table] [Table]
Revised: We thus prefer globalized architectures, such as ViT and GNN (Kipf & Welling, 2016), over the grid-based models.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table] [Table] [Table] Specifically, we select ViT as the backbone, which updates features in the context of all inputs without the need for a regular grid.
Revised: Speciﬁcally, we select ViT as the backbone, which updates features in the context of all inputs without the need for a regular grid.

Original: That is, ViT computes features from a globally-connected graph.
Revised: That is, ViT computes features from a globally-connected graph.

Original: It consists of multi-headed self-attention modules (MSA) (Vaswani et al., 2017), where features are updated according to pair-wise correlation among all the input features.
Revised: It consists of multi-headed self-attention modules (MSA) (Vaswani et al., 2017), where features are updated according to pair-wise correlation among all the input features.

Original: As a result, ViT encodes global correlation and allows inputs to have different shapes, which is ideal to enable optimal segmentation.
Revised: As a result, ViT encodes global correlation and allows inputs to have different shapes, which is ideal to enable optimal segmentation.

Original: Building upon ViT, our model contains multiple encoder blocks that take segment tokens as inputs: at level l , the encoder block updates its inputs z l − 1 to ˆ z l .
Revised: Building upon ViT, our model contains multiple encoder blocks that take segment tokens as inputs: at level l , the encoder block updates its inputs z l − 1 to ˆ z l .

Original: Notably, to extract information at different scales, we vary the shapes and sizes of segment tokens throughout the model.
Revised: Notably, to extract information at different scales, we vary the shapes and sizes of segment tokens throughout the model.

Original: Coarser segmentations have fewer segments (tokens), which improves our model efficiency.
Revised: Coarser segmentations have fewer segments (tokens), which improves our model efﬁciency.

Original: For the first block, zcorresponds to superpixels.
Revised: For the ﬁrst block, zcorresponds to superpixels.

Original: For all the other blocks, z l − 1 corresponds to segments at different scales.
Revised: For all the other blocks, z l − 1 corresponds to segments at different scales.

Original: 3. H IERARCHICAL G ROUPINGS OF S EGMENT T OKENS
Revised: 3. H IERARCHICAL G ROUPINGS OF S EGMENT T OKENS

Original: Starting with superpixels, we group fine-grained segment tokens into coarse-grained region tokens to obtain more global visual information.
Revised: Starting with superpixels, we group ﬁne-grained segment tokens into coarse-grained region tokens to obtain more global visual information.

Original: With fine-to-coarse segment groupings, we can directly induce hierarchical image segmentations of an input image.
Revised: With ﬁne-to-coarse segment groupings, we can directly induce hierarchical image segmentations of an input image.

Original: As we map superpixels into token features, hierarchical segment groupings become a multi-scale feature clustering and pooling problem.
Revised: As we map superpixels into token features, hierarchical segment groupings become a multi-scale feature clustering and pooling problem.

Original: We have two considerations: 1)
Revised: We have two considerations: 1)

Original: The number of coarser segments (tokens) should be freely adjustable during training and inference.
Revised: The number of coarser segments (tokens) should be freely adjustable during training and inference.

Original: The model should achieve optimal partitioning of inputs.
Revised: The model should achieve optimal partitioning of inputs.

Original: We summarize our feature clustering algorithm, dubbed Graph Pooling Module, in Alg. 1.
Revised: We summarize our feature clustering algorithm, dubbed Graph Pooling Module, in Alg. 1.

Original: Existing methods (Xu et al., 2022; Ke et al., 2022) perform clustering with a set of learnable embeddings, resulting in fixed segmentation granularity.
Revised: Existing methods (Xu et al., 2022; Ke et al., 2022) perform clustering with a set of learnable embeddings, resulting in ﬁxed segmentation granularity.

Original: Yet, the optimal number of segments vary with image scales: larger (smaller) images require more (less) segments.
Revised: Yet, the optimal number of segments varies with image scales: larger (smaller) images require more (fewer) segments.

Original: Instead, we conduct feature clustering with an arbitrary number of centroids that are initialized by sampled inputs.
Revised: Instead, we conduct feature clustering with an arbitrary number of centroids that are initialized by sampled inputs.

Original: The number of centroids corresponds to the segmentation granularity.
Revised: The number of centroids corresponds to the segmentation granularity.

Original: We apply the Farthest Point Sampling (FPS) algorithm
Revised: We apply the Farthest Point Sampling (FPS) algorithm

Original: (Qi et al., 2017) to select a subset of token features as initial centroids.
Revised: (Qi et al., 2017) to select a subset of token features as initial centroids.

Original: The FPS algorithm enables the sampled centroids to cover the input feature distributions, unbiased w.r.t.
Revised: The FPS algorithm enables the sampled centroids to cover the input feature distributions, unbiased w.r.t.

Original: dominant features.
Revised: dominant features.

Original: We can set the number of clusters flexibly during training and inference.
Revised: We can set the number of clusters ﬂexibly during training and inference.

Original: In particular, we predict the soft clustering assignments P l , which indicates how input features are assigned to sampled centroids.
Revised: In particular, we predict the soft clustering assignments P l , which indicates how input features are assigned to sampled centroids.

Original: We calculate P l as softmax-normalized pair-wise similarity among inputs and sampled centroids.
Revised: We calculate P l as softmax-normalized pair-wise similarity among inputs and sampled centroids.

Original: We then generate coarsened tokens z l by average aggregating within a cluster.
Revised: We then generate coarsened tokens z l by weighted-average aggregating within a cluster.

Original: For detailed computation, please refer to Alg. 1.
Revised: For detailed computation, please refer to Alg. 1.

Original: We apply graph clustering losses (Tsitsulin et al., 2020) to cluster input tokens and optimize our graph pooling modules.

Original: At each level, we construct an affinity matrix A and the degree vector D from the pair-wise similarity among inputs.

Original: Let m and n l be the total number of connection edges and output tokens at level l .

Original: We follow Ke et al.

Original: and apply three losses: 1) Modularity maximization : L mod l = − 1 2 m trace( P ⊤ l ( A − 1 2 m DD ⊤ ) P l ) .

Original: Based on group-wise correlations among inputs, the loss seeks an optimal partitioning of inputs, such that in-cluster (out-cluster) similarity is higher (lower) than total expectation.

Original: 2) Collapse regularization : L col l =

Original: [Equation]

Original: The loss encourages an even (similar-sized) partitioning of inputs.

Original: 3) Coarse-region contrastive loss :

Original: L crc l =

Original: [Equation]

Original: i

Original: [Equation]

Original: log

Original: [Equation]

Original: j exp

Original: [Equation]

Original: i ˜

Original: z l,j ) , where ˜ z l,i =

Original: z l,i ∥

Original: z l,i ∥ are the normalized coarsened tokens.

Original: The loss contrasts each coarsened region against others to increase the distinctiveness.

Original: 3. T RAINING AND I NFERENCE
Revised: 3. T RAINING AND I NFERENCE

Original: We summarize our overall framework in Alg. 2.
Revised: We summarize our overall framework in Alg. 2.

Original: We conduct segment grouping at certain levels in the model.
Revised: We conduct segment grouping at certain levels in the model.

Original: We train our CAST using an image-wise contrastive learning framework–Moco-v3 (Chen et al., 2021).
Revised: We train our CAST using an image-wise contrastive learning framework–Moco-v3 (Chen et al., 2021).

Original: The objective is to contrast each image against others in the latent feature space.
Revised: The objective is to contrast each image against others in the latent feature space.

Original: Let L moco denote the corresponding loss.

Original: Our overall loss is: L

Original: [Equation]

Original: λ [Equation] We only impose the coarse-region contrastive loss in the final graph pooling module. To predict classification, we follow MoCo-v3 to apply a 3-layer MultiLayer Perceptron (MLP) head on output [CLASS] token.
Revised: To predict classiﬁcation, we follow MoCo-v3 to apply a 3-layer MultiLayer Perceptron (MLP) head on output [CLASS] token.

Original: To predict segmentation, we fuse multi-level features as outputs (Hariharan et al., 2015) and unpool higher-level features based on the grouping index w.r.t superpixels.
Revised: To predict segmentation, we fuse multi-level features as outputs (Hariharan et al., 2015) and unpool higher-level features based on the grouping index w.r.t superpixels.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Datasets .
Revised: Datasets .

Original: 1) ImageNet (Deng et al., 2009) is an object-centric image classification dataset, annotated with 1 , 000 object categories (a.k.a IN-1k).
Revised: 1) ImageNet (Deng et al., 2009) is an object-centric image classiﬁcation dataset, annotated with 1 , 000 object categories–IN-1k.

Original: Each image is labeled with one object category, and objects mostly locate at the image center.
Revised: Each image is labeled with one object category, and objects mostly locate at the image center.

Original: The training and validation set includes 1 .
Revised: The training and validation set includes 1 .

Original: M and 50 K images, respectively.
Revised: M and 50 K images, respectively.

Original: We follow Tian et al.
Revised: We follow Tian et al.

Original: (2020) to subsample 100 object categories to create IN-100.
Revised: (2020) to subsample 100 object categories to create IN-100.

Original: The subset consists of 127 K and 5 K images for training and testing.
Revised: The subset consists of 127 K and 5 K images for training and testing.

Original: 2) Pascal VOC 2012 (Everingham et al., 2010) is an object-centric semantic segmentation dataset that containsobject categories and a background class.
Revised: 2) Pascal VOC 2012 (Everingham et al., 2010) is an object-centric semantic segmentation dataset that contains 20 object categories and a background class.

Original: We use the augmented training set (Hariharan et al., 2011) with 10 , 582 images and  validation set with 1 , 449 images.
Revised: We use the augmented training set (Hariharan et al., 2011) with 10 , 582 images and the validation set with 1 , 449 images.

Original: 3) MSCOCO (Lin et al., 2014) is a generic scene parsing dataset.
Revised: 3) MSCOCO (Lin et al., 2014) is a generic scene parsing dataset.

Original: The scene contexts are more complex and include more objects in each image ( 7 . 3 vs. 2 . 3 ) than VOC.
Revised: The scene contexts are more complex and include more objects in each image ( 7 . 3 vs. 2 . 3 ) than VOC.

Original: Following Van Gansbeke et al.
Revised: Following Van Gansbeke et al.

Original: (2021), we train on 118 , 287 images of train2017 split and test on the VOC validation set.
Revised: (2021), we train on 118 , 287 images of train2017 split and test on the VOC validation set.

Original: Model architecture.
Revised: Model architecture.

Original: For most experiments, we base our architecture on ViT-S (Dosovitskiy et al., 2020), which has 384 channel dimensions of all encoder blocks.
Revised: For most experiments, we base our architecture on ViT-S (Dosovitskiy et al., 2020), which has 384 channel dimensions of all encoder blocks.

Original: We follow Xiao et al.
Revised: We follow Xiao et al.

Original: (2021) to 1) replace the patch-wise linear projection layer with a stack of four 3 × 3 and one 1 × 1 convolutional layers; 2) use 11 , not 12 , encoder blocks to maintain similar model capacity.
Revised: (2021) to 1) replace the patch-wise linear projection layer with a stack of four 3 × 3 and one 1 × 1 convolutional layers; 2) use 11 , not 12 , encoder blocks to maintain similar model capacity.

Original: We adopt the same design choice for our vanilla ViT baselines.
Revised: We adopt the same design choice for our vanilla ViT baselines.

Original: For vanilla ViT (our CAST), we set stride to 2 among the first four (three) convolutional layers.
Revised: For vanilla ViT (our CAST), we set stride to 2 among the ﬁrst four (three) convolutional layers.

Original: Our models aggregate pixel features within superpixels, resulting in a similar number of input tokens as ViT baselines.
Revised: Our models aggregate pixel features within superpixels, resulting in a similar number of input tokens as ViT baselines.

Original: We set graph pooling step ∆ = { 3 , 6 , 9 } and reduce the number of tokens to 13 , 16 , 112 of initial inputs.
Revised: We set graph pooling step ∆ = { 3 , 6 , 9 } and reduce the number of tokens to 13 , 16 , 112 of initial inputs.

Original: K-Medoids clustering baselines.
Revised: K-Medoids clustering baselines.

Original: There is no released code for Token Pooling (Marin et al., 2021).
Revised: There is no released code for Token Pooling (Marin et al., 2021).

Original: We thus re-implement the method by combining PoWER-BERT (Goyal et al., 2020) with K-Medoids clustering algorithm.
Revised: We thus re-implement the method by combining PoWER-BERT (Goyal et al., 2020) with K-Medoids clustering algorithm.

Original: We follow the same settings as our  CAST.
Revised: We follow the same settings as our CAST: we adopt segment tokens and use the same number of tokens at intermediate layers as our CAST.

Original: Image resolution and token numbers.
Revised: Image resolution and token numbers.

Original: For training on all datasets and testing on ImageNet, we set crop _ size to 224 and partition 196 superpixels from an image, resulting in around 196 input tokens.
Revised: For training on all datasets and testing on ImageNet, we set crop _ size to 224 and partition 196 superpixels from an image, resulting in around 196 input tokens.

Original: For testing on VOC, we generate 1024 ( 384 ) superpixels from a 512 × 512 input image, resulted in 1024 ( 384 ) input tokens for semantic (figure-ground) segmentation.
Revised: For testing on VOC, we generate 1024 ( 384 ) superpixels from a 512 × 512 input image, resulting in 1024 ( 384 ) input tokens for semantic (ﬁgure-ground) segmentation.

Original: For vanilla ViT, we adopt the same image resolution and use 196 and 1024 input tokens on ImageNet and VOC.
Revised: For vanilla ViT, we adopt the same image resolution and use 196 and 1024 input tokens on ImageNet and VOC.

Original: Model training.
Revised: Model training.

Original: Based on MoCo, we train all models from scratch without any human-labeled supervision.
Revised: Based on MoCo, we train all models from scratch without any human-labeled supervision.

Original: Mostly follow MoCo, we set batch _ size to 256 , learning _ rate to 1 .

Original: 5 e − 4 , weight _ decay to 0 .

Original: 1 , and momentum to 0 .

Original: We use AdamW (Kingma & Ba, 2014) optimizer. For hyper-parameters of MoCoframework, we set temperature to 0 . 2 , output dimension to 256 , momentum _ coefficients to 0 . The 3-layer MLP head has a hidden dimension of 4096 .
Revised: We mostly follow MoCo to set up the hyper-parameters.

Original: For IN-100, IN-1k and COCO, we set training epochs to 200 , 100 and 400 , along with warmup _ epochs to 20 , 10 and 40 , respectively. Cosine decay schedule is applied to adjust the learning rate. We set λ m , λ g , λ c to 1 . 0 , 0 . 4 , and 0 . 1 . The gradients calculated from the modularity maximization and collapse regularization loss are used to update only our graph pooling modules.
Revised: See Appendix for more details.

Original: We do not propagate the gradients to the rest of the model.

Original: Model testing.
Revised: Model testing.

Original: On ImageNet , we follow MoCo to apply linear probing to evaluate model performance.
Revised: On ImageNet , we follow MoCo to apply linear probing to evaluate model performance.

Original: We freeze the trained model weights and replace the 3-layer MLP head with a randomly initialized linear projection layer as the classifier.
Revised: We freeze the trained model weights and replace the 3-layer MLP head with a randomly initialized linear projection layer as the classiﬁer.

Original: We train the linear classifier with ground-truth labels and report Top-1 accuracy.
Revised: We train the classiﬁer with ground-truth labels.

Original:  On VOC , we follow Van Gansbeke et al.
Revised: 2) On VOC , we follow Van Gansbeke et al.

Original: (2021) to predict semantic segmentation via nearest neighbor search from the labeled VOC training set.
Revised: (2021) to predict semantic segmentation via nearest neighbor search from the labeled VOC training set.

Original: We also evaluatetransfer learning performance by fine-tuning models on the training set and testing on the validation set.
Revised: We also evaluate performance by ﬁne-tuning models on the training set and testing on the validation set.

Original: On figure-ground segmentation , we binarize multi-head attention maps: Following DINO (Caron et al., 2021) to generate binary segmentation, we threshold attention mapsby keeping only 60% of the mass and select the best binary segmentation among all attention maps for each image.
Revised: On ﬁgure-ground segmentation , following DINO Caron et al. (2021), we binarize multi-head attention maps and select the best binary segmentation among all attention maps for each image.

Original: See Appendix for more details.
Revised: See Appendix for more details.

Original: [Table]
Revised: [Table]

Original: model sizes.

Original: Our CAST outperforms vanilla ViT using the same computational budget.

Original: Result 1: Better accuracy on ImageNet classification.
Revised: Result 1: Better accuracy on ImageNet classiﬁcation.

Original: Our model achieves a better trade-off between model efficiency and task performance for unsupervised image classification.
Revised: Our model achieves a better trade-off between model efﬁciency and task performance for unsupervised image classiﬁcation.

Original: We report top-1 accuracy on both IN-1k and IN-100 (see the left of Table 1).
Revised: We report top-1 accuracy on both IN-1k and IN-100 (see the left of Table 1).

Original: On IN-100, using ViT-S and
Revised: On IN-100, using ViT-S and

Original: ViT-B backbone, our CAST achieves comparable performance as vanilla ViT, yet our model is 63 .
Revised: ViT-B backbone, our CAST achieves comparable performance as vanilla ViT, yet our model is 66 .

Original: 4% and 59 .
Revised: 8% and 65 .

Original: 1% computationally more efficient.
Revised: 7% computationally more efﬁcient.

Original: On IN-1k, our CAST maintains 97 .
Revised: On IN-1k, our CAST maintains 97 .

Original: 3% performance of ViT baseline ( 66 .
Revised: 3% performance of ViT baseline ( 66 .

Original: 1% vs. 67 . 9% accuracy).
Revised: 1% vs. 67 . 9% accuracy).

Original: Our graph pooling module consistently outperforms
Revised: Our graph pooling module consistently outperforms

Original: K-Medoids clustering.
Revised: K-Medoids clustering.

Original: Due to the computation limitations, we set smaller batch _ size and training epochs on IN-1K.
Revised: Due to the computation limitations, we set smaller batch _ size and training epochs on IN-1K.

Original: We also do not apply the pre-training (Marin et al., 2021) or model distillation (Rao et al., 2021) strategy.
Revised: We also do not apply the pre-training (Marin et al., 2021) or model distillation (Rao et al., 2021) strategy.

Original: We can further improve the performance based on such settings.
Revised: We can further improve the performance based on such settings.

Original: On IN-100, our model outperforms baselines under different model sizes (see the right plot of Table 1).
Revised: On IN-100, our model outperforms baselines under different model sizes (see the right plot of Table 1).

Original: Our CAST outperforms vanilla ViT using the same computational budget.
Revised: Our CAST outperforms vanilla ViT using the same computational budget.

Original: [Table]
Revised: [Table]

Original: Result 2: Better semantic segmentation on VOC.
Revised: Result 2: Better semantic segmentation on VOC.

Original: We evaluate unsupervised semantic segmentation on VOC in Table 2.
Revised: We evaluate unsupervised semantic segmentation on VOC in Table 2.

Original: We report the regional mean Intersection of Union (mIoU) and boundary Fmeasure metric.
Revised: We report the regional mean Intersection of Union (mIoU) and boundary Fmeasure metric.

Original: For both segment retrieval and transfer learning, our segmentations are more precise ( +1 .
Revised: For both segment retrieval and transfer learning, our segmentations are more precise ( +5 .

Original:  and +1 .
Revised: 0% and +1 .

Original: 0% mIoU) and respect object boundaries better ( +3 .
Revised: 0% mIoU) and respect object boundaries better ( +8 . 6% and +7 .

Original: 9% and +5 .
Revised: 4% F-measure) than vanilla ViT, yet model efﬁciency is greatly improved. Notably, without applying CRF for post-processing, our model is still able to preserve thin structures (e [Table]

Original: 1% F-measure) than baselines, yet model efficiency is greatly improved.
Revised: g. horse legs) well, whereas ViT results are over-smoothed.

Original: Result 3: Better figure-ground segmentation on VOC.
Revised: Result 3: Better ﬁgure-ground segmentation on VOC.

Original: We show that our latent attentions capture semantics more precisely.
Revised: We show that our latent attentions capture semantics more precisely.

Original: We report the Jaccard similarity between ground truths and predicted foreground masks.
Revised: We report the Jaccard similarity between ground truths and predicted foreground masks.

Original: Summarized in Table 3, our CAST outperforms DINO and ViT by +2 .
Revised: Summarized in Table 3, our CAST outperforms DINO and ViT by +2 .

Original: Our foreground masks can cover dominant foreground objects and better align with object boundaries.
Revised: Our foreground masks can cover dominant foreground objects and better align with object boundaries.

Original: method
Revised: [Table]

Original: IoU Our CAST 48. Vanilla ViT 45. DINO 45.
Revised: [Table]

Original: Result 4: Better hierarchical segmentation on IN-1K. Fig.
Revised: Result 4: Better hierarchical segmentation on IN-1K. Fig.

Original: 4 shows that our hierarchical segmentations respect semantics at different levels of granularity better than ViT and K-Medoids baselines.
Revised: 4 shows that our hierarchical segmentations respect semantics at different levels of granularity better than ViT and K-Medoids baselines.

Original: Our model directly produces hierarchical segmentations without needing additional clustering.
Revised: Our model directly produces hierarchical segmentations without needing additional clustering.

Original: [Figure]
Revised: Bootstrapping from superpixels, our segmentations (e.g. dog regions) align with image contours more precisely than vanilla ViT.

Original: requires additional pixel-wise clustering (e.g. K-Means) on fixed features, yet the results are over-smoothed. Our CAST naturally generates hierarchical segmentations which capture semantics more precisely.
Revised: Compared to K-Medoids clustering, our model maps similar-semantic ﬁne segments into the same coarse regions (e.g. street-sign pole regions) more consistently..

Original: Summary.
Revised: Summary.

Original: We develop a novel vision transformer that performs image-wise recognition atop of consistent hierarchical image segmentation, by learning fine-to-coarse features over adaptive segment tokens instead of regular patch tokens.
Revised: We develop a novel vision transformer that performs image-wise recognition atop of consistent hierarchical image segmentation, by learning ﬁne-to-coarse features over adaptive segment tokens instead of regular patch tokens.

Original: We deliver the first concurrent recognition and hierarchical segmentation model without any supervision, achieving better accuracy and efficiency.
Revised: We deliver the ﬁrst concurrent recognition and hierarchical segmentation model without any supervision, achieving better accuracy and efﬁciency.

Original: The idea can be extended to supervised image classification, with hierarchical semantic segmentation for free.
Revised: The idea can be extended to supervised image classiﬁcation, with hierarchical semantic segmentation for free.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: We develop the first vision transformer that concurrently achieves image-wise recognition and hierarchical image segmentation without additional processing.
Revised: We develop the ﬁrst vision transformer that concurrently achieves image-wise recognition and hierarchical image segmentation without additional processing.

Original: Our CAST outperforms existing token coarsening methods for image classification, semantic segmentation, and attention-induced figure-ground segmentation tasks.
Revised: Our CAST outperforms existing token coarsening methods for image classiﬁcation, semantic segmentation, and attention-induced ﬁgure-ground segmentation tasks.

Original: In this supplementary, we include more details as followings:
Revised: In this supplementary, we include more details as followings:

Original: • We present more visual results of hierarchical segmentation on ImageNet in A.1.
Revised: • We present more visual results of hierarchical segmentation on ImageNet in A.1.

Original: • We present more visual results of attention-induced figure-ground segmentations on VOC in A.2.
Revised: • We present more visual results of attention-induced ﬁgure-ground segmentations on VOC in A.2.

Original: • We present visual results of attention maps from our CAST on ImageNet in A.3.
Revised: • We present visual results of attention maps from our CAST on ImageNet in A.3.

Original: • We present the ablation study regarding the proposedregularizations and graph pooling module in A.4.
Revised: • We present the ablation study regarding the proposed graph pooling module in A.4.

Original: • We present the ablation study on inference latency of our CAST in A.5.
Revised: • We present the ablation study on inference latency of our CAST in A.5.

Original: • We present the ablation study on effectiveness of superpixels in A. •
Revised: • We present the ablation study on effectiveness of superpixels in A. •

Original: We present quantitative results of hierarchical segmentation on DensePose in A. • We illustrate more details of inference in A.8. 
Revised: We present quantitative results of hierarchical segmentation on DensePose in A. • We illustrate more details of hyper-parameters for training in A.8. • We illustrate more details of inference in A.9.

Original: [Figure]
Revised: [Figure]

Revised: requires

Original: [Figure]
Revised: [Figure]

Original: A. V ISUAL R ESULTS ON H IERARCHICAL S EGMENTATION .
Revised: A. V ISUAL R ESULTS ON H IERARCHICAL S EGMENTATION .

Original: We present more visualization results of hierarchical segmentations induced by vanilla ViT, KMedoids, and our CAST (see Fig. 5).
Revised: We present more visualization results of hierarchical segmentations induced by vanilla ViT, KMedoids, and our CAST (see Fig. 5).

Original: We hierarchically segment the input image into 32 , 16 , and 8 regions.
Revised: We hierarchically segment the input image into 32 , 16 , and 8 regions.

Original: Notably, Vit baseline requires additional pixel-wise K-Means clustering on fixed feature representations, yet the results are over-smoothed.
Revised: Notably, Vit baseline requires additional pixel-wise K-Means clustering on ﬁxed feature representations, yet the results are over-smoothed.

Original: Our CAST naturally generates hierarchical segmentations without any post-processing.
Revised: Our CAST naturally generates hierarchical segmentations without any post-processing.

Original: Our multi-scale segmentations align with image boundaries and capture semantics more precisely.
Revised: Our multi-scale segmentations align with image boundaries and capture semantics more precisely.

Original: A. V ISUAL R ESULTS ON A TTENTION - INDUCED F IGURE - GROUND S EGMENTATION .
Revised: A. V ISUAL R ESULTS ON A TTENTION - INDUCED F IGURE - GROUND S EGMENTATION .

Original: We present more visual results of figure-ground segmentations generated from multi-head attention maps on VOC.
Revised: We present more visual results of ﬁgure-ground segmentations generated from multi-head attention maps on VOC.

Original: Our CAST attends to foreground semantics more precisely than vanilla ViT, and the segmentations preserve object boundaries more accurately.
Revised: Our CAST attends to foreground semantics more precisely than vanilla ViT, and the segmentations preserve object boundaries more accurately.

Original: See Fig.
Revised: See Fig.

Original: . A. V ISUAL R ESULTS ON M ULTI - HEAD A TTENTION M APS .
Revised: . A. V ISUAL R ESULTS ON M ULTI - HEAD A TTENTION M APS .

Original: We visualize the multi-head attention maps of the [CLASS] token to all the other segment tokens in our vision transformer.
Revised: We visualize the multi-head attention maps of the [CLASS] token to all the other segment tokens in our vision transformer.

Original: As the [CLASS] token is optimized for image-wise discrimination, such attention maps indicate the most informative groupings of segments that will induce the most discriminative image-wise representations.
Revised: As the [CLASS] token is optimized for image-wise discrimination, such attention maps indicate the most informative groupings of segments that will induce the most discriminative image-wise representations.

Original: We visualize the same attention maps used to generate the figure-ground segmentation, which are the ones in the 9 th transformer encoder block.
Revised: We visualize the same attention maps used to generate the ﬁgure-ground segmentation, which are the ones in the 9 th transformer encoder block.

Original: The layer takes 32 coarsened segment tokens as inputs, resulting in 12 heads of 32 × 32 attention maps.
Revised: The layer takes 32 coarsened segment tokens as inputs, resulting in 12 heads of 32 × 32 attention maps.

Original: We follow the same procedure as DINO (Caron et al., 2021) to display the binarized attention maps.
Revised: We follow the same procedure as DINO (Caron et al., 2021) to display the binarized attention maps.

Original: The threshold is adjusted to keep 60% of the mass.
Revised: The threshold is adjusted to keep 60% of the mass.

Original: See Caron et al.
Revised: See Caron et al.

Original: (2021) for more details.
Revised: (2021) for more details.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 7, our attention maps reveal parts-of-the-whole information of the image.
Revised: 7, our attention maps reveal parts-of-the-whole information of the image.

Original: We observe that the same object parts are together attended in the same attention head, e.g. face vs. ears vs. nose of the dog.
Revised: We observe that the same object parts are together attended in the same attention head, e.g. face vs. ears vs. nose of the dog.

Original: It indicates that image-wise recognition requires parts-of-the-whole information.
Revised: It indicates that image-wise recognition requires parts-of-the-whole information.

Original: Additionally, our model carries segment, not patch, tokens through the layers, resulting in attention maps better aligned with object boundaries. 
Revised: Additionally, our model carries segment, not patch, tokens through the layers, resulting in attention maps better aligned with object boundaries. Token Pooling Acc.

Revised: Patch -

Original: [Table]
Revised: [Table]

Revised: Segment -

Revised: [Table]

Revised: Segment (cid:88)

Revised: [Table]

Revised: Pooling

Revised: Acc.

Revised: Our Graph Pooling 78.

Revised: [Table]

Revised: BERT 75.

Revised: [Table]

Revised: BERT 73.

Revised: K

Original:  A. A BLATION S TUDY ON P ROPOSEDR EGULARIZATIONS AND G RAPH P OOLING M ODULE .
Revised: [Table] [Table] Sampling 72. A. A BLATION S TUDY ON P ROPOSED G RAPH P OOLING M ODULE .

Original: Summarized in Table 4, we demonstrate the efficacy of the proposedloss regularizations and graph pooling module.
Revised: Summarized in Table 4, we demonstrate the efﬁcacy of the proposed graph pooling module.

Original: We report top-1 accuracy results on IN-100 dataset.
Revised: We report top-1 accuracy results on IN-100 dataset.

Original: We show that we improve our model’s performance by adding each proposed loss.

Original: In addition, our graph pooling module outperforms K-Medoids and K-Means clustering by a large margin.
Revised: We show that our graph pooling module outperforms K-Medoids and K-Means clustering by a large margin.

Original: [Table]
Revised: [Table]

Original: A. A BLATION S TUDY ON I NFERENCE L ATENCY
Revised: A. A BLATION S TUDY ON I NFERENCE L ATENCY

Original: We present the comparison of inference latency among our CAST and vanilla ViT architectures.
Revised: We present the comparison of inference latency among our CAST and vanilla ViT architectures.

Original: We report the inference time (ms) of each module with 384 channel dimensions and 256 batch sizes on ImageNet-100.
Revised: We report the inference time (ms) of each module with 384 channel dimensions and 256 batch sizes on ImageNet-100.

Original: As summarized in Table 5, our graph pool module with FPS indeed requires additional computation.
Revised: As summarized in Table 5, our graph pool module with FPS indeed requires additional computation.

Original: However, our method can also reduce inference time by decreasing the number of tokens in deeper layers.
Revised: However, our method can also reduce inference time by decreasing the number of tokens in deeper layers.

Original: Disregarding the Conv Stem, vanilla ViT and our CAST take 316.91 and 220.57 ms for inference, respectively.
Revised: Disregarding the Conv Stem, vanilla ViT and our CAST take 316.91 and 220.57 ms for inference, respectively.

Original: Our model is 30.4% faster than ViT. Optimizing the token sampling technique to increase model efficiency is our future work.
Revised: Our model is 30.4% faster than ViT. Optimizing the token sampling technique to increase model efﬁciency is our future work.

Original: A. A BLATION S TUDY ON S UPERPIXEL
Revised: A. A BLATION S TUDY ON S UPERPIXEL

Original: We next verify the superior efficacy of superpixel tokens on dense pixel applications.
Revised: We next verify the superior efﬁcacy of superpixel tokens on dense pixel applications.

Original: As shown in Table 6, we compare patch tokens to superpixel tokens on image classification and semantic segmentation tasks.
Revised: As shown in Table 6, we compare patch tokens to superpixel tokens on image classiﬁcation and semantic segmentation tasks.

Original: For both ViT baseline and our CAST, we observe significant performance gain for semantic segmentation, yet the performance gap for classification is negligible.
Revised: For both ViT baseline and our CAST, we observe signiﬁcant performance gain for semantic segmentation, yet the performance gap for classiﬁcation is negligible.

Original: We conclude that using superpixels can be very useful in a wide range of dense pixel labeling tasks.
Revised: We conclude that using superpixels can be very useful in a wide range of dense pixel labeling tasks.

Original: A. Q UANTITATIVE R ESULTS ON H IERARCHICAL S EGMENTATION
Revised: A. Q UANTITATIVE R ESULTS ON H IERARCHICAL S EGMENTATION

Original: To demonstrate the efficacy of our hierarchical segmentation results, we compare CAST with KMedoids on the DensePose dataset (Alp Güler et al., 2018).
Revised: To demonstrate the efﬁcacy of our hierarchical segmentation results, we compare CAST with KMedoids on the DensePose dataset (Alp Güler et al., 2018).

Original: We process body-part-level labels into the head, torso, upper limb, and lower limb regions.
Revised: We process body-part-level labels into the head, torso, upper limb, and lower limb regions.

Original: We segment an image into 64 , 32 , and 16 regions, and evaluate the region-wise coverings (Arbelaez et al., 2010) with ground truths by F1-score.
Revised: We segment an image into 64 , 32 , and 16 regions, and evaluate the region-wise coverings (Arbelaez et al., 2010) with ground truths by F1-score.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 8, our CAST outperforms K-Medoids and is better at capturing semantics at every level of granularity.
Revised: 8, our CAST outperforms K-Medoids and is better at capturing semantics at every level of granularity.

Revised: A. H YPER - PARAMETERS FOR T RAINING

Revised: We list the hyper-parameters for training in Table 7.

Original: A. I NFERENCE AND T ESTING . 
Revised: Mostly follow MoCo, we set batch _ size to 256 , learning _ rate to 1 . 5 e − 4 , weight _ decay to 0 . 1 , and momentum to 0 . We use AdamW (Kingma & Ba, 2014) optimizer.

Revised: For hyper-parameters of MoCo framework, we set temperature to 0 .

Revised: 2 , output dimension to 256 , momentum _ coeﬃcients to 0 .

Revised: The 3-layer MLP head has a hidden dimension of 4096 .

Revised: For IN-100, IN-1k and COCO, we set training epochs to 200 , 100 and 400 , along with warmup _ epochs to 20 , 10 and 40 , respectively.

Revised: Cosine decay schedule is applied to adjust the learning rate.

Revised: [Table]

Revised: A. I NFERENCE AND T ESTING

Original: For image classification, we apply the linear probing procedure for evaluation.
Revised: For image classiﬁcation, we apply the linear probing procedure for evaluation.

Original: For semantic segmentation, we use the segment retrieval and transfer learning procedure to evaluate model performance.
Revised: For semantic segmentation, we use the segment retrieval and transfer learning procedure to evaluate model performance.

Original: Image classification: linear probing.
Revised: Image classiﬁcation: linear probing.

Original: We follow MoCo-v3 (Chen et al., 2021) to evaluate imagewise discrimination model performance using a linear probing protocol.
Revised: We follow MoCo-v3 (Chen et al., 2021) to evaluate imagewise discrimination model performance using a linear probing protocol.

Original: We freeze the trained model weights and replace the 3-layer MLP head with a randomly initialized linear projection layer as classifier.
Revised: We freeze the trained model weights and replace the 3-layer MLP head with a randomly initialized linear projection layer as classiﬁer.

Original: We train the linear classifier with ground-truth labels and report the top-1 accuracy.
Revised: We train the linear classiﬁer with ground-truth labels and report the top-1 accuracy.

Original: Following Chen et al.
Revised: Following Chen et al.

Original: (2021), we train the linear classifier with 90 epochs on ImageNet dataset.
Revised: (2021), we train the linear classiﬁer with 90 epochs on ImageNet dataset.

Original: We set momentum to 0 .
Revised: We On DensePose (Alp Güler et al., 2018), we parse body-part-level labels into head, torso, upper and lower limb regions.

Original: 9 and weight _ decay to 0 for all experiments. On IN-1k, we set batch _ size to
Revised: Using CAST or K-Medoids, we partition an image into 64 , 32 and 16 regions, and evaluate the region-wise converings with ground-truths by F1-score. Our CAST captures semantics better at every level of granularity.

Original: [Table]
Revised: [Table]

Original: 1024 , learning _ rate to 30 ; on IN-100, we set batch _ size to 256 , learning _ rate to 0 . 8 . SGD is used as the
Revised: set momentum to 0 . 9 and weight _ decay to experiments.

Original: Semantic segmentation: segment retrieval.
Revised: Semantic segmentation: segment retrieval.

Original: We follow Hwang et al.
Revised: We follow Hwang et al.

Original: Van Gansbeke et al.
Revised: Van Gansbeke et al.

Original: (2021); Ke et al.
Revised: (2021); Ke et al.

Original: (2022) to evaluate semantic segmentation using segment retrieval.
Revised: (2022) to evaluate semantic segmentation using segment retrieval.

Original: We partition an image into several segments and conduct nearest neighbor search to predict the label for each segment.
Revised: We partition an image into several segments and conduct nearest neighbor search to predict the label for each segment.

Original: We assign the majority labels from the 20 retrieved segments.
Revised: We assign the majority labels from the 20 retrieved segments.

Original: For ViT baselines, we apply the MLP head on each token to generate unit-length output features and upsample the feature maps to the original resolution of the input image.
Revised: For ViT baselines, we apply the MLP head on each token to generate unit-length output features and upsample the feature maps to the original resolution of the input image.

Original: Followed by spherical
Revised: Followed by spherical

Original: K-Means clustering algorithm, we partition the image into 36 segments using the output features.
Revised: K-Means clustering algorithm, we partition the image into 36 segments using the output features.

Original: On DensePose (Alp Güler et al., 2018), we parse body-part-level labels into head, torso, upper and lower limb regions.

Original: Using CAST or K-Medoids, we partition an image into 64 , 32 and 16 regions, and evaluate the region-wise converings with ground-truths by F1-score.

Original: Our CAST captures semantics better at every level of granularity.

Original: Our CAST does not require additional upsampling and K-Means clustering.
Revised: Our CAST does not require additional upsampling and K-Means clustering.

Original: For segmentation, our model follows Hypercolumn design (Hariharan et al., 2015) to unpool and fuse multi-level segment tokens.
Revised: For segmentation, our model follows Hypercolumn design (Hariharan et al., 2015) to unpool and fuse multi-level segment tokens.

Original: Our model generates the same number of output tokens as the superpixels.
Revised: Our model generates the same number of output tokens as the superpixels.

Original: We gather pixel features from output tokens based on the superpixel index.
Revised: We gather pixel features from output tokens based on the superpixel index.

Original: Without the need for spherical K-Means clustering, our CAST predicts an image segmentation using the graph pooling modules.
Revised: Without the need for spherical K-Means clustering, our CAST predicts an image segmentation using the graph pooling modules.

Original: We compute normalized segment features according to such image segmentation.
Revised: We compute normalized segment features according to such image segmentation.

Original: Semantic segmentation: transfer learning.
Revised: Semantic segmentation: transfer learning.

Original: We follow Van Gansbeke et al.
Revised: We follow Van Gansbeke et al.

Original: (2021) to evaluate model performance using transfer learning protocol.
Revised: (2021) to evaluate model performance using transfer learning protocol.

Original: All models are unsupervisedly trained on MSCOCO, and supervisedly fine-tuned on Pascal VOC.
Revised: All models are unsupervisedly trained on MSCOCO, and supervisedly ﬁne-tuned on Pascal VOC.

Original: We replace the 3-layer MLP head with 2-layer 1 ×convolutional layers.
Revised: We replace the 3-layer MLP head with 2-layer 1 ×convolutional layers.

Original: We set the training steps to 30 K, learning _ rate to 0 .
Revised: We set the training steps to 30 K, learning _ rate to 0 .

Original: 003 , weight _ decay to 0 .
Revised: 003 , weight _ decay to 0 .

Original: 0001 , batch _ size to 16 , crop _ size to 512 .
Revised: 0001 , batch _ size to 16 , crop _ size to 512 .

Original: Following Chen et al.
Revised: Following Chen et al.

Original: (2016), we adopt poly learning rate policy by multiplying base learning rate by 1 − iter max _ iter 0 .
Revised: (2016), we adopt poly learning rate policy by multiplying base learning rate by 1 − iter max _ iter 0 .

Original: We adopt the SGD optimizer.
Revised: We adopt the SGD optimizer.

Original: We use only a single-scale image for inference.
Revised: We use only a single-scale image for inference.