Original: I mage as S et of P oints
Revised: I mage as S et of P oints

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A bstract
Revised: A bstract

Original: What is an image and how to extract latent features?
Revised: What is an image and how to extract latent features?

Original: Convolutional Networks (ConvNets) consider an image as organized pixels in a rectangular shape and extract features via convolutional operation in local region;
Revised: Convolutional Networks (ConvNets) consider an image as organized pixels in a rectangular shape and extract features via convolutional operation in local region;

Original: Vision Transformers (ViTs) treat an image as a sequence of patches and extract features via attention mechanism in a global range.
Revised: Vision Transformers (ViTs) treat an image as a sequence of patches and extract features via attention mechanism in a global range.

Original: In this work, we introduce a straightforward and promising paradigm for visual representation, which is called
Revised: In this work, we introduce a straightforward and promising paradigm for visual representation, which is called

Original: Context Clusters .
Revised: Context Clusters .

Original: Context clusters (CoCs) view an image as a set of unorganized points and extract features via simplified clustering algorithm.
Revised: Context clusters (CoCs) view an image as a set of unorganized points and extract features via simplified clustering algorithm.

Original: In detail, each point includes the raw feature ( e.g. , color) and positional information ( e.g. coordinates), and a simplified clustering algorithm is employed to group and extract deep features hierarchically.
Revised: In detail, each point includes the raw feature ( e.g. , color) and positional information ( e.g. coordinates), and a simplified clustering algorithm is employed to group and extract deep features hierarchically.

Original: Our CoCs are convolution- and attention-free, and only rely on clustering algorithm for spatial interaction.
Revised: Our CoCs are convolution- and attention-free, and only rely on clustering algorithm for spatial interaction.

Original: Owing to the simple design, we show CoCs endow gratifying interpretability via the visualization of clustering process.
Revised: Owing to the simple design, we show CoCs endow gratifying interpretability via the visualization of clustering process.

Original: Our
Revised: Our

Original: CoCs aim at providing a new perspective on image and visual representation, which may enjoy broad applications in di ff erent domains and exhibit profound insights.
Revised: CoCs aim at providing a new perspective on image and visual representation, which may enjoy broad applications in di ff erent domains and exhibit profound insights.

Original: Even though we are not targeting SOTA performance, COCs still achieve comparable or even better performance than ConvNets or ViTs on several benchmarks.
Revised: Even though we are not targeting SOTA performance, COCs still achieve comparable or even better performance than ConvNets or ViTs on several benchmarks.

Original: Codes are made available at: https: // anonymous.4open.science / r / ContextCluster.
Revised: Codes are made available at: https: // anonymous.4open.science / r / ContextCluster.

Original:  I ntroduction
Revised:  I ntroduction

Original: The way we extract features depends a lot on how we interpret an image.
Revised: The way we extract features depends a lot on how we interpret an image.

Original: As a fundamental paradigm, Convolutional Neural Networks (ConvNets) have dominated the field of computer vision and considerably improved the performance of various vision tasks in recent years He et al.
Revised: As a fundamental paradigm, Convolutional Neural Networks (ConvNets) have dominated the field of computer vision and considerably improved the performance of various vision tasks in recent years (He et al., 2016;

Original: Xie et al. ; Ge et al.
Revised: Xie et al., 2021; Ge et al., 2021).

Original: Methodologically, ConvNets conceptualize a picture as a collection of arranged pixels in a rectangle form, and extract local features using convolution in a sliding window fashion.
Revised: Methodologically, ConvNets conceptualize a picture as a collection of arranged pixels in a rectangle form, and extract local features using convolution in a sliding window fashion.

Original: Benefiting from some important inductive biases like Locality and translation equivariance, ConvNets are made to be e ffi cient and e ff ective.
Revised: Benefiting from some important inductive biases like Locality and translation equivariance, ConvNets are made to be e ffi cient and e ff ective.

Original: Recently, Vision Transformers (ViTs) have significantly challenged ConvNets’ hegemony in the vision domain.
Revised: Recently, Vision Transformers (ViTs) have significantly challenged ConvNets’ hegemony in the vision domain.

Original: Derived from language processing, Transformers Vaswani et al. (2017) treat an image as a sequence of patches, and a globalrange self-attention operation is employed to adaptively fuse information from patches.
Revised: Derived from language processing, Transformers (Vaswani et al., 2017) treat an image as a sequence of patches, and a global-range self-attention operation is employed to adaptively fuse information from patches.

Original: With the resulting models ( i.e. , ViTs), the inherent inductive biases in ConvNets are abandoned, and gratifying results are obtained Touvron et al. (2021b).
Revised: With the resulting models ( i.e. , ViTs), the inherent inductive biases in ConvNets are abandoned, and gratifying results are obtained (Touvron et al., 2021b).

Original: Recent work has shown tremendous improvements in vision community, which are mainly built on top of convolution or attention ( e.g. , ConvNeXt Liu et al. MAE He et al. (2022), and
Revised: Recent work has shown tremendous improvements in vision community, which are mainly built on top of convolution or attention ( e.g. , ConvNeXt (Liu et al., 2022), MAE (He et al., 2022), and

Original: CLIP Radford et al. (2021), etc .)
Revised: CLIP (Radford et al., 2021), etc .)

Original: Meanwhile, some attempts combine convolution and attention together, like CMT Guo et al. (2022a) and CoAtNet Dai et al.
Revised: Meanwhile, some attempts combine convolution and attention together, like CMT (Guo et al., 2022a) and CoAtNet (Dai et al., 2021).

Original: These methods scan images in grid (convolution) yet explore mutual relationships of a sequence (attention), enjoying locality prior (convolution) without sacrificing global reception (attention).
Revised: These methods scan images in grid (convolution) yet explore mutual relationships of a sequence (attention), enjoying locality prior (convolution) without sacrificing global reception (attention).

Original: While they inherit the advantages from both and achieve better empirical performance, the insights and knowledge are still restricted to
Revised: While they inherit the advantages from both and achieve better empirical performance, the insights and knowledge are still restricted to

Original: ConvNets and ViTs.
Revised: ConvNets and ViTs.

Original: Instead of being lured into the trap of chasing incremental improvements, we underline that some feature extractors are also worth investigating beyond convolution and attention.
Revised: Instead of being lured into the trap of chasing incremental improvements, we underline that some feature extractors are also worth investigating beyond convolution and attention.

Original: While convolution and attention are acknowledged to have significant benefits and an enormous influence on the field of vision, they are not the only choices.
Revised: While convolution and attention are acknowledged to have significant benefits and an enormous influence on the field of vision, they are not the only choices.

Original: MLP-based architectures Touvron et al. ; Tolstikhin et al. (2021) have demonstrated that a pure MLP-based design can also achieve similar performance.
Revised: MLP-based architectures (Touvron et al., 2022; Tolstikhin et al., 2021) have demonstrated that a pure MLP-based design can also achieve similar performance.

Original: Besides, considering graph network as the feature extractor is proven to be feasible Han et al. (2022).
Revised: Besides, considering graph network as the feature extractor is proven to be feasible (Han et al., 2022).

Original: Hence, we expect a new paradigm of feature extraction that can provide some novel insights instead of incremental performance improvements.
Revised: Hence, we expect a new paradigm of feature extraction that can provide some novel insights instead of incremental performance improvements.

Original: [Figure]
Revised: [Figure]

Original: In this

Original: , we back the - cal algorithm for the fundamental visual representation, clustering Bishop & Nasrabadi (2006).
Revised: into the classical algorithm for the - mental visual representation clustering method (Bishop & Nasrabadi, 2006).

Original: Holistically, we view an image as a set of data points and group all points into clusters.
Revised: Holistically, we view an image as a set of data points and group all points into clusters.

Original: In each cluster, we aggregate the points into a center and then dispatch the center point to all the points adaptively.
Revised: In each cluster, we aggregate the points into a center and then dispatch the center point to all the points adaptively.

Original: We call this design as context cluster .
Revised: We call this design as context cluster.

Original: Fig.
Revised: Fig.

Original: 1 illustrates the process.
Revised: 1 illustrates the process.

Original: Specifically, we consider each pixel as a 5-dimensional data point with the information of color and position.
Revised: Specifically, we consider each pixel as a 5-dimensional data point with the information of color and position.

Original: In a sense, we convert an image as a set of point clouds and utilize methodologies from point cloud analysis Qi et al. (2017b)
Revised: In a sense, we convert an image as a set of point clouds and utilize methodologies from point cloud analysis (Qi et al., 2017b;

Original: ; Ma et al. for image visual representation.
Revised: Ma et al., 2022) for image visual representation.

Original: This bridges the representations of image and point cloud, showing a strong generalization and opening the possibility for an easy fusion of multi-modalities.
Revised: This bridges the representations of image and point cloud, showing a strong generalization and opening the possibility for an easy fusion of multimodalities.

Original: With a set of points, we introduce a simplified clustering method to group the points into clusters.
Revised: With a set of points, we introduce a simplified clustering method to group the points into clusters.

Original: The clustering processing shares a similar idea as SuperPixel Ren & Malik (2003), where similar pixels are grouped, but they are fundamentally di ff erent.
Revised: The clustering processing shares a similar idea as SuperPixel (Ren & Malik, 2003), where similar pixels are grouped, but they are fundamentally di ff erent.

Original: To our best knowledge, we are the first ones to introduce the clustering method for the general visual representation and make it work.
Revised: To our best knowledge, we are the first ones to introduce the clustering method for the general visual representation and make it work.

Original: On the contrary, SuperPixel and later incarnations are mainly for image pre-processing Jampani et al. (2018) or particular tasks like semantic segmentationYang et al.
Revised: On the contrary, SuperPixel and later incarnations are mainly for image pre-processing (Jampani et al., 2018) or particular tasks like semantic segmentation

Original:  Yu et al. (2022b).
Revised: (Yang et al., 2020; Yu et al., 2022b).

Original: We instantiate our deep network based on the context cluster and name the resulting models as
Revised: We instantiate our deep network based on the context cluster and name the resulting models as

Original: Context Clusters (CoCs).
Revised: Context Clusters (CoCs).

Original: Our new design is inherently di ff erent from ConvNets or ViTs, but we also inherit some positive philosophy from them, including the hierarchical representation Liu et al. (2022) from ConvNets and Metaformer Yu et al. (2022c) framework from ViTs.
Revised: Our new design is inherently di ff erent from ConvNets or ViTs, but we also inherit some positive philosophy from them, including the hierarchical representation (Liu et al., 2022) from ConvNets and Metaformer (Yu et al., 2022c) framework from ViTs.

Original: CoCs reveal distinct advantages.
Revised: CoCs reveal distinct advantages.

Original: First, by considering image as a set of points, CoCs show great generalization ability to di ff erent data domains, like point clouds, RGBD images, etc.
Revised: First, by considering image as a set of points, CoCs show great generalization ability to di ff erent data domains, like point clouds, RGBD images, etc.

Original: Second, the context clustering processing provides CoCs gratifying interpretability.
Revised: Second, the context clustering processing provides CoCs gratifying interpretability.

Original: By visualizing the clustering in each layer, we can explicitly understand the learning in each layer.
Revised: By visualizing the clustering in each layer, we can explicitly understand the learning in each layer.

Original: Even though our method does not target SOTA performance, it still achieves on par or even better performance than ConvNets or ViTs on several benchmarks.
Revised: Even though our method does not target SOTA performance, it still achieves on par or even better performance than ConvNets or ViTs on several benchmarks.

Original: We hope our context cluster will bring some new breaks to the vision community.
Revised: We hope our context cluster will bring some new breaks to the vision community.

Original:  R elated W ork
Revised:  R elated W ork

Original: Clustering in Image Processing While clustering approaches in image processing Castleman (1996) have gone out of favor in the deep learning era, they never disappear from computer vision.
Revised: Clustering in Image Processing While clustering approaches in image processing (Castleman, 1996) have gone out of favor in the deep learning era, they never disappear from computer vision.

Original: A time-honored work is SuperPixel Ren & Malik (2003), which segments an image into regions by grouping a set of pixels that share common characteristics.
Revised: A time-honored work is SuperPixel (Ren & Malik, 2003), which segments an image into regions by grouping a set of pixels that share common characteristics.

Original: Given the desired sparsity and simple representation, SuperPixel has become a common practice for image preprocessing.
Revised: Given the desired sparsity and simple representation, SuperPixel has become a common practice for image preprocessing.

Original: Naive application of SuperPixel exhaustively clusters ( e.g. , via K-means algorithm)
Revised: Naive application of SuperPixel exhaustively clusters ( e.g. , via K-means algorithm)

Original: pixels over the entire image, making the computational cost heavy.
Revised: pixels over the entire image, making the computational cost heavy.

Original: To this end, SLIC Achanta et al. (2012) limits the clustering operation in a local region and evenly initializes the K-means centers for better and faster convergence.
Revised: To this end, SLIC (Achanta et al., 2012) limits the clustering operation in a local region and evenly initializes the K-means centers for better and faster convergence.

Original: In recent years, clustering methods have been experiencing a surge of interest and are closely bound with deep networks Li & Chen (2015); Jampani et al.
Revised: In recent years, clustering methods have been experiencing a surge of interest and are closely bound with deep networks (Li & Chen, 2015; Jampani et al., 2018; Yang et al., 2020).

Original: ; Yang et al.
Revised: To create the superpixels for deep networks, SSN (Jampani et al., 2018) proposes a di ff erentiable SLIC method, which is end-to-end trainable and enjoys favorable runtime.

Original: Most recently, tentative e ff orts have been made towards applying clustering methods into networks for specific vision tasks, like segmentation Yu et al. (2022a;b) and fine-grained recognition Huang & Li (2020).
Revised: Most recently, tentative e ff orts have been made towards applying clustering methods into networks for specific vision tasks, like segmentation (Yu et al., 2022a;b) and fine-grained recognition (Huang & Li, 2020).

Revised: For example, CMT-DeepLab (Yu et al., 2022a) interprets the object queries in segmentation tasks as cluster centers, and the grouped pixels are assigned to the segmentation result for each cluster.

Original: Nevertheless,  no workis conducted for a general visual representation via clustering method.
Revised: Nevertheless, to our best knowledge, there is no work conducted for a general visual representation via clustering method.

Original: We aim to make up for the vacancy, along with proving the feasibility numerically and visually.
Revised: We aim to make up for the vacancy, along with proving the feasibility numerically and visually.

Original: ConvNets & ViTs Convolutional Networks (ConvNets) have dominated the vision community since the deep learning era Simonyan & Zisserman (2015); He et al. ; Liu et al.
Revised: ConvNets & ViTs ConvNets have dominated the vision community since the deep learning era (Simonyan & Zisserman, 2015;

Original: Benefiting from the local relation and weight sharing, ConvNets can e ff ectively and e ffi ciently model visual representations He et al. ; Tan & Le (2019) ; Liu et al. Meanwhile, an emerging research topic, Vision Transformer, applies purely attention-based transformers Vaswani et al.
Revised: He et al., 2016).

Original: (2017) to the vision community and hasattracted much attention. Recent e ff orts on Vision Transformers have set new SOTA performances on various vision tasks Dosovitskiy et al. ; Carion et al.
Revised: Recently, ViTs (Dosovitskiy et al., 2020) introduce purely attention-based transformers (Vaswani et al., 2017) to the vision community and has set new SOTA performances on various vision tasks.

Original: Xie et al.

Original: A common and plausible conjecture is that these gratifying achievements are credited to the self-attention mechanism.
Revised: A common and plausible conjecture is that these gratifying achievements are credited to the self-attention mechanism.

Original: However, this intuitive conjecture has soon been challenged, from language Dong et al.
Revised: However, this intuitive conjecture has soon been challenged.

Original: (2021); Jain & Wallace (2019) to vision Yu et al.

Original: (2022c)

Original: ; Wightman et al. (2021) community. Extensive experiments also showcase that aclassical ResNet He et al. (2016) can achieve on par or even better performance thanadvanced ViTs, with proper training recipe and minimalarchitecture modifications Wightman et al.
Revised: Extensive experiments also showcase that a ResNet (He et al., 2016) can achieve on par or even better performance than ViTs, with proper training recipe and minimal modifications (Wightman et al., 2021;

Original: Liu et al. (2022).
Revised: Liu et al., 2022).

Original: We emphasize that while convolution and attention may have unique virtues ( i.e. , ConvNets enjoy inductive biases Liu et al.
Revised: We emphasize that while convolution and attention may have unique virtues ( i.e. , ConvNets enjoy inductive biases (Liu et al., 2022) while

Original: (2022) while ViTs excel at generalization and scalability Yuan et al. (2021b)) , they did not show significant gap in visual representational ability.
Revised: ViTs excel at generalization (Yuan et al., 2021b)), they did not show significant performance gap.

Original: Di ff erent from convolution and attention, in this work, we radically present a new paradigm for visual representation using clustering algorithm.
Revised: Di ff erent from convolution and attention, in this work, we radically present a new paradigm for visual representation using clustering algorithm.

Original: With both quantitative and qualitative analysis, we show that our method can serve as a new  backbonefor visual representation and enjoys better interpretability.
Revised: With both quantitative and qualitative analysis, we show that our method can serve as a new general backbone and enjoys gratifying interpretability.

Original: Recent AdvancesExtensive e
Revised: Recent Advances

Original:  ff orts have been made to push up the vision tasks’ performances within the framework of ConvNets and ViTs Liu et al. (2021b)
Revised: Extensive e ff orts have been made to push up the vision tasks’ performances within the framework of ConvNets and ViTs (Liu et al., 2021b;

Original: ; Ding et al. (2022b).
Revised: Ding et al., 2022b).

Original: To take advantage of both convolution and attention, some work learns to mix the two designs in a hybrid mode, like
Revised: To take advantage of both convolution and attention, some work learns to mix the two designs in a hybrid mode, like

Original: CoAtNet Dai et al. (2021), Mobile-Former Chen et al. (2022), and CVT Wu et al.
Revised: CoAtNet (Dai et al., 2021), Mobile-Former (Chen et al., 2022), and CVT (Wu et al., 2021).

Original: We also note that some recent advances explored more methods for visual representation, beyond convolution and attention.
Revised: We also note that some recent advances explored more methods for visual representation, beyond convolution and attention.

Original: MLP-like models Tolstikhin et al. (2021); Touvron et al.
Revised: MLP-like models (Tolstikhin et al., 2021; Touvron et al., 2022;

Original: ; Hou et al. (2022) directly consider a MLP layer for spatial interaction.
Revised: Hou et al., 2022) directly consider a MLP layer for spatial interaction.

Original: Besides, some work employs shifting Lian et al. ; Huang et al. or pooling Yu et al. (2022c) for local communication.
Revised: Besides, some work employs shifting (Lian et al., 2021; Huang et al., 2021) or pooling (Yu et al., 2022c) for local communication.

Original: Similar to our work that treats the image as unordered data set, Vision GNN (ViG) Han et al. (2022) extracts graph-level features for visual tasks.
Revised: Similar to our work that treats the image as unordered data set, Vision GNN (ViG) (Han et al., 2022) extracts graph-level features for visual tasks.

Original: Di ff erently, we directly apply the clustering method from conventional image processing and exhibit promising generalization ability and interpretability.
Revised: Di ff erently, we directly apply the clustering method from conventional image processing and exhibit promising generalization ability and interpretability.

Original:  M ethod
Revised:  M ethod

Original: [Figure]
Revised: [Figure]

Original: Context Clusters forgo the fashionable convolution or attention in favor of novelly considering the classical algorithm, clustering, for the representation of visual learning.
Revised: Context Clusters forgo the fashionable convolution or attention in favor of novelly considering the classical algorithm, clustering, for the representation of visual learning.

Original: In this section, we first describe the Context Clusters pipeline.
Revised: In this section, we first describe the Context Clusters pipeline.

Original: The proposed context cluster operation (as shown in Fig. 2) for feature extraction is then thoroughly explained.
Revised: The proposed context cluster operation (as shown in Fig. 2) for feature extraction is then thoroughly explained.

Original: After that, we set up the Context Cluster architectures.
Revised: After that, we set up the Context Cluster architectures.

Original: Finally, some open discussions might aid individuals in comprehending our work and exploring more directions following our Context Cluster.
Revised: Finally, some open discussions might aid individuals in comprehending our work and exploring more directions following our Context Cluster.

Original: 3.1 C ontext C lusters P ipeline
Revised: 3.1 C ontext C lusters P ipeline

Original: From Image to Set of Points.
Revised: From Image to Set of Points.

Original: given an input image I ∈ R 3 × w × h , we begin by enhancing the image with the 2D coordinates of each pixel
Revised: given an input image I ∈ R 3 × w × h , we begin by enhancing the image with the 2D coordinates of each pixel

Original: I i , j , where each pixel’s coordinate is presented as (cid:104) iw − 0 .
Revised: I i , j , where each pixel’s coordinate is presented as (cid:104) iw − 0 .

Original: 5 , jh − 0 .
Revised: 5 , jh − 0 .

Original: 5 (cid:105) .
Revised: 5 (cid:105) .

Original: It is feasible to investigate further positional augmentation techniques to potentially improve performance.
Revised: It is feasible to investigate further positional augmentation techniques to potentially improve performance.

Original: This design is taken into consideration for its simplicity and practicality.
Revised: This design is taken into consideration for its simplicity and practicality.

Original: The augmented image is then converted to a collection of points ( i.e. , pixels) P ∈ R 5 × n , where n = w × h is the number of points, and each point contains both feature (color) and position (coordinates) information; hence, the points set could be unordered and disorganized.
Revised: The augmented image is then converted to a collection of points ( i.e. , pixels) P ∈ R 5 × n , where n = w × h is the number of points, and each point contains both feature (color) and position (coordinates) information; hence, the points set could be unordered and disorganized.

Original: C o n t e x t C l u s t
Revised: C o n t e x t C l u s t

Original: er
Revised: er

Original: B l o c k s
Revised: B l o c k s

Original: C o n t e x t C l u s t
Revised: C o n t e x t C l u s t

Original: er
Revised: er

Original: B l o c k s
Revised: B l o c k s

Original: C o n t e x t C l u s t
Revised: C o n t e x t C l u s t

Original: er
Revised: er

Original: B l o c k s
Revised: B l o c k s

Original: C o n t e x t C l u s t
Revised: C o n t e x t C l u s t

Original: er
Revised: er

Original: B l o c k s
Revised: B l o c k s

Original: Stage
Revised: Stage

Original: Stage Stage
Revised: Stage Stage

Original: Stage
Revised: Stage

Original: Image Points Set
Revised: Image Points Set

Original: [𝟓,𝒏] [𝑪 𝟏 , 𝒏 𝟏𝟔] [𝑪 𝟐 , 𝒏 𝟔𝟒] [𝑪 𝟑 , 𝒏 𝟐𝟓𝟔] [𝑪 𝟒 , 𝒏 𝟏𝟎𝟐𝟒] ×𝑵×𝑵×𝑵×𝑵
Revised: [𝟓,𝒏] [𝑪 𝟏 , 𝒏 𝟏𝟔] [𝑪 𝟐 , 𝒏 𝟔𝟒] [𝑪 𝟑 , 𝒏 𝟐𝟓𝟔] [𝑪 𝟒 , 𝒏 𝟏𝟎𝟐𝟒] ×𝑵×𝑵×𝑵×𝑵

Original: P o i n t s R e du cer
Revised: P o i n t s R e du cer

Original: P o i n t s R e du cer P o i n t s R e du cer
Revised: P o i n t s R e du cer P o i n t s R e du cer

Original: P o i n t s R e du cer
Revised: P o i n t s R e du cer

Original: We are rewarded with excellent generalization ability for o ff ering a fresh perspective of image, a set of points.
Revised: We are rewarded with excellent generalization ability for o ff ering a fresh perspective of image, a set of points.

Original: A set of data points can be considered as a universal data representation because data in most domains can be given as a combination of feature and position information (or either of the two).
Revised: A set of data points can be considered as a universal data representation because data in most domains can be given as a combination of feature and position information (or either of the two).

Original: This inspires us to conceptualize an image as a set of points.
Revised: This inspires us to conceptualize an image as a set of points.

Original: Feature Extraction with Image Set Points.
Revised: Feature Extraction with Image Set Points.

Original: Following the ConvNets methodology He et al.
Revised: Following the ConvNets methodology (He et al., 2016;

Original: Liu et al. (2022), we extract deep features using context cluster blocks (see Fig.
Revised: Liu et al., 2022), we extract deep features using context cluster blocks (see Fig.

Original: 2 for reference and § 3.2 for explanation) hierarchically.
Revised: 2 for reference and § 3.2 for explanation) hierarchically.

Original: Fig.
Revised: Fig.

Original: 3 shows our Context Cluster architecture.
Revised: 3 shows our Context Cluster architecture.

Original: Given a set of points P ∈ R 5 × n , we first reduce the points number for computational e ffi ciency, then a succession of context cluster blocks are applied to extract features.
Revised: Given a set of points P ∈ R 5 × n , we first reduce the points number for computational e ffi ciency, then a succession of context cluster blocks are applied to extract features.

Original: To reduce the points number, we evenly select some anchors in space, and the nearest k points are concatenated and fused by a linear projection.
Revised: To reduce the points number, we evenly select some anchors in space, and the nearest k points are concatenated and fused by a linear projection.

Original: Note that this reduction can be achieved by a convolutional operation if all points are arranged in order and k is properly set ( i.e. , 4 and 9), like in ViT Dosovitskiy et al.
Revised: Note that this reduction can be achieved by a convolutional operation if all points are arranged in order and k is properly set ( i.e. , 4 and 9), like in ViT (Dosovitskiy et al., 2020).

Original: For clarity on the centers and anchors stated previously, we strongly suggest the readers check appendix § 5 .
Revised: For clarity on the centers and anchors stated previously, we strongly suggest the readers check appendix § 5.

Original: Task-Specific Applications.
Revised: Task-Specific Applications.

Original: For classification, we average all points of the last block’s output and use a FC layer for classification.
Revised: For classification, we average all points of the last block’s output and use a FC layer for classification.

Original: For downstream dense prediction tasks like detection and segmentation, we need to rearrange the output points by position after each stage to satisfy the needs of most detection and segmentation heads ( e.g. , Mask-RCNN He et al.
Revised: For downstream dense prediction tasks like detection and segmentation, we need to rearrange the output points by position after each stage to satisfy the needs of most detection and segmentation heads ( e.g. , Mask-RCNN (He et al., 2017)).

Original: In other words, Context
Revised: In other words, Context

Original: Cluster o ff ers remarkable flexibility in classification task, but is limited to a compromise between dense prediction tasks’ requirements and our model configurations.
Revised: Cluster o ff ers remarkable flexibility in classification task, but is limited to a compromise between dense prediction tasks’ requirements and our model configurations.

Original: We expect innovative detection & segmentation heads (like DETR Carion et al. (2020)) can seamlessly integrate with our method.
Revised: We expect innovative detection & segmentation heads (like DETR (Carion et al., 2020)) can seamlessly integrate with our method.

Original: 3.2 C ontext C luster O peration
Revised: 3.2 C ontext C luster O peration

Original: In this subsection, we introduce the key contribution in our work, the context cluster operation.
Revised: In this subsection, we introduce the key contribution in our work, the context cluster operation.

Original: Holistically, we first group the feature points into clusters; then, feature points in each cluster will be aggregated and then dispatched back, as illustrated in Fig.
Revised: Holistically, we first group the feature points into clusters; then, feature points in each cluster will be aggregated and then dispatched back, as illustrated in Fig.

Original: Context Clustering.
Revised: Context Clustering.

Original: Given a set of feature points P ∈ R n × d , we group all the points into several groups based on the similarities, with each point being solely assigned to one cluster.
Revised: Given a set of feature points P ∈ R n × d , we group all the points into several groups based on the similarities, with each point being solely assigned to one cluster.

Original: We first linearly project P to P s for similarity computation.
Revised: We first linearly project P to P s for similarity computation.

Original: Following the conventional SuperPixel method
Revised: Following the conventional SuperPixel method

Original: SLIC Achanta et al. (2012), we evenly propose c centers in space, and the center feature is computed by averaging its k nearest points.
Revised: SLIC (Achanta et al., 2012), we evenly propose c centers in space, and the center feature is computed by averaging its k nearest points.

Original: We then calculate the pair-wise cosine similarity matrix S ∈ R c × n between P s and the resulting set of center points.
Revised: We then calculate the pair-wise cosine similarity matrix S ∈ R c × n between P s and the resulting set of center points.

Original: Since each point contains both feature and position information, while computing similarity, we implicitly highlight the points’ distances (locality) as well as the feature similarity.
Revised: Since each point contains both feature and position information, while computing similarity, we implicitly highlight the points’ distances (locality) as well as the feature similarity.

Original: After that, we allocate each point to the most similar center, resulting in c clusters.
Revised: After that, we allocate each point to the most similar center, resulting in c clusters.

Original: Of note is that each cluster may have a di ff erent number of points.
Revised: Of note is that each cluster may have a di ff erent number of points.

Original: Extremely, some clusters might have zero points, in which case they would be otiose.
Revised: Extremely, some clusters might have zero points, in which case they would be otiose.

Original: Feature Aggregating.
Revised: Feature Aggregating.

Original: We dynamically aggregate all points in a cluster based on the similarities to the center point.
Revised: We dynamically aggregate all points in a cluster based on the similarities to the center point.

Original: Assuming a cluster contains m points (a subset in P ) and the similarity between the m points and the center is s ∈ R m (a subset in S ), we map the points to a value space to get
Revised: Assuming a cluster contains m points (a subset in P ) and the similarity between the m points and the center is s ∈ R m (a subset in S ), we map the points to a value space to get

Original: P v ∈ R m × d ′ , where d ′ is the value dimension.
Revised: P v ∈ R m × d ′ , where d ′ is the value dimension.

Original: We also propose a center v c in the value space like the clustering center proposal.
Revised: We also propose a center v c in the value space like the clustering center proposal.

Original: The aggregated feature g ∈ R d ′ is given by:
Revised: The aggregated feature g ∈ R d ′ is given by:

Original: [Equation]
Revised: [Equation]

Original: Here α and β are learnable scalars to scale and shift the similarity and sig ( · ) is a sigmoid function to re-scale the similarity to (0 , 1).
Revised: Here α and β are learnable scalars to scale and shift the similarity and sig ( · ) is a sigmoid function to re-scale the similarity to (0 , 1).

Original: v i indicates i -th point in P v .
Revised: v i indicates i -th point in P v .

Original: Empirically, this strategy would achieve much better results than directly applying the original similarity because no negative value is involved.
Revised: Empirically, this strategy would achieve much better results than directly applying the original similarity because no negative value is involved.

Original: Softmax is not considered since the points do not contradict with one another.
Revised: Softmax is not considered since the points do not contradict with one another.

Original: We incorporate the value center v c in Eq.
Revised: We incorporate the value center v c in Eq.

Original: 1 for numerical stability 1 as well as further emphasize the locality.
Revised: 1 for numerical stability 1 as well as further emphasize the locality.

Original: To control the magnitude, the aggregated feature is normalized by a factor of C .
Revised: To control the magnitude, the aggregated feature is normalized by a factor of C .

Original: Feature Dispatching.
Revised: Feature Dispatching.

Original: The aggregated feature g is then adaptively dispatched to each point in a cluster based on the similarity.
Revised: The aggregated feature g is then adaptively dispatched to each point in a cluster based on the similarity.

Original: By doing so, the points can communicate with one another and shares features from all points in the cluster, as shown in Fig.
Revised: By doing so, the points can communicate with one another and shares features from all points in the cluster, as shown in Fig.

Original: For each point p i , we update it by
Revised: For each point p i , we update it by

Original: [Equation]
Revised: [Equation]

Original: Here we follow the same procedures to handle the similarity and apply a fully-connected (FC) layer to match the feature dimension (from value space dimension d ′ to original dimension d ).
Revised: Here we follow the same procedures to handle the similarity and apply a fully-connected (FC) layer to match the feature dimension (from value space dimension d ′ to original dimension d ).

Original: Multi-Head Computing.
Revised: Multi-Head Computing.

Original: We acknowledge the multi-head design in the self-attention mechanism Vaswani et al. and use it to enhance our context cluster.
Revised: We acknowledge the multi-head design in the self-attention mechanism (Vaswani et al., 2017) and use it to enhance our context cluster.

Original: We consider h heads and set the dimension number of both value space P v and similarity space P s to d ′ for simplicity.
Revised: We consider h heads and set the dimension number of both value space P v and similarity space P s to d ′ for simplicity.

Original: The outputs of multi-head operations are concatenated and fused by a FC layer.
Revised: The outputs of multi-head operations are concatenated and fused by a FC layer.

Original: The multi-head architecture also contributes to a satisfying improvement in our context cluster, as we empirically demonstrate.
Revised: The multi-head architecture also contributes to a satisfying improvement in our context cluster, as we empirically demonstrate.

Original: 3.3 A rchitecture I nitialization
Revised: 3.3 A rchitecture I nitialization

Original: While Context Cluster is fundamentally distinct from convolution and attention, the design philosophies from ConvNets and ViTs, such as hierarchical representations and meta Transformer architecture Yu et al. (2022c), are still applicable to Context Cluster.
Revised: While Context Cluster is fundamentally distinct from convolution and attention, the design philosophies from ConvNets and ViTs, such as hierarchical representations and meta Transformer architecture (Yu et al., 2022c), are still applicable to Context Cluster.

Original: To align with other networks and make our method compatible with most detection and segmentation algorithms, we progressively reduce the number of points by a factor of 16, 4, 4, and 4 in each stage.
Revised: To align with other networks and make our method compatible with most detection and segmentation algorithms, we progressively reduce the number of points by a factor of 16, 4, 4, and 4 in each stage.

Original: We consider 16 nearest neighbors for selected anchors in the first stage, and we choose their 9 nearest neighbors in the rest stages.
Revised: We consider 16 nearest neighbors for selected anchors in the first stage, and we choose their 9 nearest neighbors in the rest stages.

Original: An underlying issue is computational e ffi ciency.
Revised: An underlying issue is computational e ffi ciency.

Original: Assume we have n d-dimensional points and c clusters, the time complexity to calculate the feature similarity would be O ( ncd ), which is unacceptable when the input image resolution is high ( e.g. , 224 × 224).
Revised: Assume we have n d-dimensional points and c clusters, the time complexity to calculate the feature similarity would be O ( ncd ), which is unacceptable when the input image resolution is high ( e.g. , 224 × 224).

Original: To circumvent this problem, we introduce region partition by splitting the points into several local regions like Swin Transformer Liu et al. (2021b), and compute similarity locally.
Revised: To circumvent this problem, we introduce region partition by splitting the points into several local regions like Swin Transformer (Liu et al., 2021b), and compute similarity locally.

Original: As a result, when the number of local regions is set to r , we noticeably lower the time complexity by a factor of r , from O ( ncd ) to O (cid:16) r nr c r d (cid:17) .
Revised: As a result, when the number of local regions is set to r , we noticeably lower the time complexity by a factor of r , from O ( ncd ) to O (cid:16) r nr c r d (cid:17) .

Original: See appendix for detailed configurations.
Revised: See appendix for detailed configurations.

Original: Note that if we split the set of points to several local regions, we limit the receptive field for context cluster, and no communications among local regions are available.
Revised: Note that if we split the set of points to several local regions, we limit the receptive field for context cluster, and no communications among local regions are available.

Original: 3.4 D iscussion
Revised: 3.4 D iscussion

Original: Fixed or Dynamic centers for clusters?
Revised: Fixed or Dynamic centers for clusters?

Original: Both conventional clustering algorithms and SuperPixel techniques iteratively update the centers until converge.
Revised: Both conventional clustering algorithms and SuperPixel techniques iteratively update the centers until converge.

Original: However, this will result in exorbitant computing costs when clustering is used as a key component in each building block.
Revised: However, this will result in exorbitant computing costs when clustering is used as a key component in each building block.

Original: The inference time will increase exponentially.
Revised: The inference time will increase exponentially.

Original: In Context Cluster, we view fixed centers as an alternative for inference e ffi ciency, which can be considered as a compromise between accuracy and speed.
Revised: In Context Cluster, we view fixed centers as an alternative for inference e ffi ciency, which can be considered as a compromise between accuracy and speed.

Original: Overlap or non-overlap clustering?
Revised: Overlap or non-overlap clustering?

Original: We allocate the points solely to a specific center, which di ff ers from previous point cloud analysis design philosophies.
Revised: We allocate the points solely to a specific center, which di ff ers from previous point cloud analysis design philosophies.

Original: We intentionally adhere to the conventional clustering approach (non-overlap clustering) since we want to demonstrate that the simple and
Revised: We intentionally adhere to the conventional clustering approach (non-overlap clustering) since we want to demonstrate that the simple and

Original: [Table]
Revised: [Table]

Original: ViT-7 Yuan et al.
Revised: C o n vo l u t i o n

Original: [Table]
Revised: [Table]

Original: 2021a)

Original: [-[Table] (cid:113) DeiT-Tiny / 
 Touvron et al.-]
Revised: [-[Table] (cid:113) DeiT-Tiny / 
 Touvron et al.-]C l u s t e r

Original: [Table]
Revised: [Table]

Original: 2021b) [Table] (cid:113) DeiT [Table]
Revised: C l u s t er (cid:114) Context-Cluster

Original: Small / 16 Touvron et al.

Original: [Table]
Revised: [Table]

Original: 2021b)

Original: [Table]

Original: [Table]

Original: [Table]

Original: 16 Trockman et [Table] [Table] 1024 / 12 Trockman et al
Revised: Medium

Original: [Table]
Revised: [Table]

Original: [Table]

Original: ConvMixer [Table] [Table] Trockman et
Revised: ours)

Original: [Table]

Original: [Table]
Revised: [Table]

Original: traditional algorithm can serve as a generic backbone.
Revised: traditional algorithm can serve as a generic backbone.

Original: Although it might produce higher performance, overlapped clustering is not essential to our approach and could result in extra computing burdens.
Revised: Although it might produce higher performance, overlapped clustering is not essential to our approach and could result in extra computing burdens.

Original:  E xperiments
Revised:  E xperiments

Original: We validate Context Cluster on ImageNet-1K Deng et al. ScanObjectNN Uy et al.
Revised: We validate Context Cluster on ImageNet-1K (Deng et al., 2009), ScanObjectNN (Uy et al., 2019),

Original: MS COCO Lin et al. (2014), and ADE20k Zhou et al. (2017) datasets for image classification, point cloud classification, object detection, instance segmentation, and semantic segmentation tasks.
Revised: MS COCO (Lin et al., 2014), and ADE20k (Zhou et al., 2017) datasets for image classification, point cloud classification, object detection, instance segmentation, and semantic segmentation tasks.

Original: Even we are not in pursuit of state-of-the-art performance like ConvNeXt Liu et al. and
Revised: Even we are not in pursuit of state-of-the-art performance like ConvNeXt (Liu et al., 2022) and

Original: DaViT Ding et al. (2022a), Context Cluster still presents promising results on all tasks.
Revised: DaViT (Ding et al., 2022a), Context Cluster still presents promising results on all tasks.

Original: Detailed studies demonstrate the interpretability and the generalization ability of our Context Cluster.
Revised: Detailed studies demonstrate the interpretability and the generalization ability of our Context Cluster.

Original: 4.1 I mage C lassification on I mage N et -1K
Revised: 4.1 I mage C lassification on I mage N et -1K

Original: We train Context Clusters on the ImageNet-1K training set (about 1.3M images) and evaluate upon the validation set.
Revised: We train Context Clusters on the ImageNet-1K training set (about 1.3M images) and evaluate upon the validation set.

Original: In this work, we adhere to the conventional training recipe in Dai et al. (2021); Wightman (2019); Touvron et al. (2021b) ; Yu et al. (2022c).
Revised: In this work, we adhere to the conventional training recipe in (Dai et al., 2021; Wightman, 2019; Touvron et al., 2021b; Yu et al., 2022c).

Original: For data augmentation, we mainly adopt random horizontal flipping, random pixel erase, mixup, cutmix, and label smoothing.
Revised: For data augmentation, we mainly adopt random horizontal flipping, random pixel erase, mixup, cutmix, and label smoothing.

Original: AdamW Loshchilov & Hutter (2019) is used to train all of our models across 310 epochs with a momentum of 0.9 and a weight decay of 0.05.
Revised: AdamW (Loshchilov & Hutter, 2019) is used to train all of our models across 310 epochs with a momentum of 0.9 and a weight decay of 0.05.

Original: The learning rate is set to 0.001 by default and adjusted using a cosine schedular Loshchilov & Hutter (2017).
Revised: The learning rate is set to 0.001 by default and adjusted using a cosine schedular (Loshchilov & Hutter, 2017).

Original: By default, the models are trained on 8 A GPUs with a 128 mini-batch size (that is 1024 in total).
Revised: By default, the models are trained on 8 A100 GPUs with a 128 mini-batch size (that is 1024 in total).

Original: We use Exponential Moving Average (EMA) to enhance the training, similar to earlier studies Guo et al. (2022b); Touvron et al. (2021b).
Revised: We use Exponential Moving Average (EMA) to enhance the training, similar to earlier studies (Guo et al., 2022b; Touvron et al., 2021b).

Original: Tablereports the parameters used, FLOPs, classification accuracy, and throughputs.
Revised: Table 1 reports the parameters used, FLOPs, classification accuracy, and throughputs.

Original: ‡ denotes a di ff erent region partition approach that we used to divide the points into [49 , 49 , 1 , 1] in the four stages.
Revised: ‡ denotes a di ff erent region partition approach that we used to divide the points into [49 , 49 , 1 , 1] in the four stages.

Original: Empirically, results in Table 1 indicate the e ff ectiveness of our proposed Context Cluster.
Revised: Empirically, results in Table 1 indicate the e ff ectiveness of our proposed Context Cluster.

Original: Our Context
Revised: Our Context

Original: Cluster is capable of attaining comparable or even better performance than the widely-used baselines
Revised: Cluster is capable of attaining comparable or even better performance than the widely-used baselines

Original: V i T - B /
Revised: V i T - B /

Original: R e s N e t
Revised: R e s N e t

Original: C o CM ( a tt e n ti on m a p ) ( c l a ss ac t .
Revised: C o CM ( a tt e n ti on m a p ) ( c l a ss ac t .

Original: m a p ) ( c l u s t e r i ng m a p ) using a similar number of parameters and FLOPs.
Revised: m a p ) ( c l u s t e r i ng m a p ) using a similar number of parameters and FLOPs.

Original: With about 25M parameters, our Context Cluster surpasses the enhanced ResNet50 Wightman et al. (2021) and PVT-small by 1.1% and achieves 80.9% top-1 accuracy.
Revised: With about 25M parameters, our Context Cluster surpasses the enhanced ResNet50 (Wightman et al., 2021) and PVT-small by 1.1% and achieves 80.9% top-1 accuracy.

Original: Additionally, our Context Cluster obviously outperforms MLP-based methods.
Revised: Additionally, our Context Cluster obviously outperforms MLP-based methods.

Original: This phenomenon indicates that the performance of our method is not credited to MLP blocks, and context cluster blocks substantially contribute to the visual representation.
Revised: This phenomenon indicates that the performance of our method is not credited to MLP blocks, and context cluster blocks substantially contribute to the visual representation.

Original: The performance di ff erences between Context-Cluster-Ti and Context-Cluster-Ti ‡ are negligible, demonstrating the robustness of our Context Cluster to the local region partitioning strategy.
Revised: The performance di ff erences between Context-Cluster-Ti and Context-Cluster-Ti ‡ are negligible, demonstrating the robustness of our Context Cluster to the local region partitioning strategy.

Original: We concede that our results cannot match the SOTA performance ( e.g. , CoAtNet-0 arrives 81.6% accuracy with a comparable number of parameters as CoC-Tiny), but we emphasize that we are pursuing and proving the viability of a new feature extraction paradigm.
Revised: We concede that our results cannot match the SOTA performance ( e.g. , CoAtNet-0 arrives 81.6% accuracy with a comparable number of parameters as CoC-Tiny), but we emphasize that we are pursuing and proving the viability of a new feature extraction paradigm.

Original: We successfully forsake convolution and attention in our networks by conceptualizing image as a set of points and naturally applying clustering algorithm for feature extraction.
Revised: We successfully forsake convolution and attention in our networks by conceptualizing image as a set of points and naturally applying clustering algorithm for feature extraction.

Original: In contrast to convolution and attention, our context cluster has excellent generalizability to other domain data and enjoys promising interpretability.
Revised: In contrast to convolution and attention, our context cluster has excellent generalizability to other domain data and enjoys promising interpretability.

Original: [Table]
Revised: [Table]

Original: Component Ablation .
Revised: Component Ablation .

Original: Table 2 reports the results on
Revised: Table 2 reports the results on

Original: ImageNet-1K of eliminating each individual component in Context-Cluster-Small variant.
Revised: ImageNet-1K of eliminating each individual component in Context-Cluster-Small variant.

Original: To remove the multi-head design, we utilize one head for each block and set the head dimension number to [16 , 32 , 96 , 128] in the four stages, respectively.
Revised: To remove the multi-head design, we utilize one head for each block and set the head dimension number to [16 , 32 , 96 , 128] in the four stages, respectively.

Original: When the positional information is removed, the model becomes untrainable since points are disorganized.
Revised: When the positional information is removed, the model becomes untrainable since points are disorganized.

Original: A similar phenomenon can also be seen from cifar Krizhevsky et al. (2009) datasets.
Revised: A similar phenomenon can also be seen from cifar (Krizhevsky et al., 2009) datasets.

Original: Performance dropped 3.3% without the context cluster operation.
Revised: Performance dropped 3.3% without the context cluster operation.

Original: Besides, multi-head design is able to boost the result by 0.9%.
Revised: Besides, multi-head design is able to boost the result by 0.9%.

Original: Combining all the components, we arrive at a 77.5% top-1 accuracy.
Revised: Combining all the components, we arrive at a 77.5% top-1 accuracy.

Original: 4.2 V isualization of C lustering
Revised: 4.2 V isualization of C lustering

Original: To better understand Context Cluster, we draw the clustering map in Fig. 4, and we also show the attention map of ViTs and the class activation map 
Revised: To better understand Context Cluster, we draw the clustering map in Fig. 4, and we also show the attention map of ViTs and the class activation map ( i.e. , CAM)

Original: ( i.e. , CAM) Zhou et al. (2016) of ConvNets.
Revised: (Zhou et al., 2016) of ConvNets.

Original: Notice that the three kinds of maps are conceptually di ff erent and cannot be compared directly.
Revised: Notice that the three kinds of maps are conceptually di ff erent and cannot be compared directly.

Original: We list the other two (attention and class activation) maps for reference and demonstrate the inner operations in ViTs, ConvNets, and our Context Cluster.
Revised: We list the other two (attention and class activation) maps for reference and demonstrate the inner operations in ViTs, ConvNets, and our Context Cluster.

Original: Detail settings can be found in the caption of Fig.
Revised: Detail settings can be found in the caption of Fig.

Original: As the number of points is reduced, the details are merged to form a context cluster.
Revised: As the number of points is reduced, the details are merged to form a context cluster.

Original: Three observations justify the correctness and e ff ectiveness of our Context Cluster.
Revised: Three observations justify the correctness and e ff ectiveness of our Context Cluster.

Original: First, our method clearly clusters the goose as one object context in the last stage and groups the background grass together.
Revised: First, our method clearly clusters the goose as one object context in the last stage and groups the background grass together.

Original: A similar phenomenon can also be observed from the previous stages but in more detailed and local regions.
Revised: A similar phenomenon can also be observed from the previous stages but in more detailed and local regions.

Original: Second, our context cluster can even cluster similar contexts in the very early stages ( e [Table] g. , the first and second stages).
Revised: Second, our context cluster can even cluster similar contexts in the very early stages ( e.g. , the first and second stages).

Original: Zoom in the details in the red boxes, we can see that the points belonging to the goose’s neck are clearly clustered together, suggesting the strong clustering ability of our method.
Revised: Zoom in the details in the red boxes, we can see that the points belonging to the goose’s neck are clearly clustered together, suggesting the strong clustering ability of our method.

Original: Last, we notice that most clusters emphasize the locality, while some (colored in bright green) show the globality a lot, as shown in the clustering map of the last stage.
Revised: Last, we notice that most clusters emphasize the locality, while some (colored in bright green) show the globality a lot, as shown in the clustering map of the last stage.

Original: This further demonstrates the design philosophy; we encourage similar points (considering both color feature and spatial position) to be grouped but make no restriction to the receptive field.
Revised: This further demonstrates the design philosophy; we encourage similar points (considering both color feature and spatial position) to be grouped but make no restriction to the receptive field.

Original: Visual clustering map and detailed analysis indicate that our Context Cluster is e ff ective and exhibit promising interpretability.
Revised: Visual clustering map and detailed analysis indicate that our Context Cluster is e ff ective and exhibit promising interpretability.

Original: 4.3 3D P oint C loud C lassification on S can O bject NN
Revised: 4.3 3D P oint C loud C lassification on S can O bject NN

Original: [Table]
Revised: [Table]

Original: Context Clusters are a natural fit for point clouds.
Revised: Context Clusters are a natural fit for point clouds.

Original: Therefore we also examine our method for the task of point cloud classification.
Revised: Therefore we also examine our method for the task of point cloud classification.

Original: We choose
Revised: We choose

Original: PointMLP Ma et al. as the foundation for our model because of its performance and ease of use.
Revised: PointMLP (Ma et al., 2022) as the foundation for our model because of its performance and ease of use.

Original: In detail, we only consider one head and set the head dimension number to min (cid:16) c 4 , 32 (cid:17) where c indicates the channel number in each layer.
Revised: In detail, we only consider one head and set the head dimension number to min (cid:16) c 4 , 32 (cid:17) where c indicates the channel number in each layer.

Original: We place our
Revised: We place our

Original: Context Cluster block before each Residual Point Block in PointMLP.
Revised: Context Cluster block before each Residual Point Block in PointMLP.

Original: The resulting model is termed PointMLPCoC. Note that better settings would result in improved performance, but that is not the focus of our study.
Revised: The resulting model is termed PointMLPCoC. Note that better settings would result in improved performance, but that is not the focus of our study.

Original: We report the mean accuracy over all classes (mAcc) and overall accuracy over all examples (OA) in Table 3.
Revised: We report the mean accuracy over all classes (mAcc) and overall accuracy over all examples (OA) in Table 3.

Original: In Table 3, we present the mean accuracy across all classes (mAcc) and the overall accuracy across all samples (OA).
Revised: In Table 3, we present the mean accuracy across all classes (mAcc) and the overall accuracy across all samples (OA).

Original: Experimental results show that our method can substantially increase PointMLP’s performance, with improvements in mean accuracy of 0.5% (84.4% vs. 83.9%) and overall accuracy of 0.8% (86.2% vs. 85.4%).
Revised: Experimental results show that our method can substantially increase PointMLP’s performance, with improvements in mean accuracy of 0.5% (84.4% vs. 83.9%) and overall accuracy of 0.8% (86.2% vs. 85.4%).

Original: Note that the promising gain has only been made by the introduction of one head in the context cluster; with more heads and elaborate settings, performance would be improved.
Revised: Note that the promising gain has only been made by the introduction of one head in the context cluster; with more heads and elaborate settings, performance would be improved.

Original: Most importantly, the outcomes show that our approach is highly generalizable to di ff erent domains, such as point clouds.
Revised: Most importantly, the outcomes show that our approach is highly generalizable to di ff erent domains, such as point clouds.

Original: We anticipate that our Context Cluster will operate satisfactorily when applied to more domains with little to no modifications.
Revised: We anticipate that our Context Cluster will operate satisfactorily when applied to more domains with little to no modifications.

Original: 4.4 O bject D etection and I nstance S egmentation on MS-COCO
Revised: 4.4 O bject D etection and I nstance S egmentation on MS-COCO

Original: Next, we investigate Context Cluster’s generalisability to downstream tasks, including object detection and instance segmentation.
Revised: Next, we investigate Context Cluster’s generalisability to downstream tasks, including object detection and instance segmentation.

Original: We conduct our experiments on the MS COCO 2017 benchmark Lin et al. (2014), which has 118k images for training and 5k images for validation.
Revised: We conduct our experiments on the MS COCO 2017 benchmark (Lin et al., 2014), which has 118k images for training and 5k images for validation.

Original: Following previous work, we train and test our model integrating with Mask RCNN He et al. (2017) for both object detection and instance segmentation tasks.
Revised: Following previous work, we train and test our model integrating with Mask RCNN (He et al., 2017) for both object detection and instance segmentation tasks.

Original: All models are trained with 1 × scheduler (12 epochs) and initialized with ImageNet pre-trained weights.
Revised: All models are trained with 1 × scheduler (12 epochs) and initialized with ImageNet pre-trained weights.

Original: For comparison, we consider ResNet as a representative for
Revised: For comparison, we consider ResNet as a representative for

Original: ConvNets and PVT for ViTs.
Revised: ConvNets and PVT for ViTs.

Original: We report evaluation metric mean Average Precision (mAP) in Table 4.
Revised: We report evaluation metric mean Average Precision (mAP) in Table 4.

Original: We notice that owing to the di ff erences in image resolution, directly adopting the Context Cluster configuration for ImageNet may not be appropriate for the downstream tasks.
Revised: We notice that owing to the di ff erences in image resolution, directly adopting the Context Cluster configuration for ImageNet may not be appropriate for the downstream tasks.

Original: For classification task,
Revised: For classification task,

Original: [Table]
Revised: [Table]

Original: we would have 49 points and 4 centers in a local region.
Revised: we would have 49 points and 4 centers in a local region.

Original: The detection and segmentation tasks would have 1000 points with the same configuration for image size (1280, 800).
Revised: The detection and segmentation tasks would have 1000 points with the same configuration for image size (1280, 800).

Original: It is obvious that grouping 1000 points into 4 clusters would produce an inferior result.
Revised: It is obvious that grouping 1000 points into 4 clusters would produce an inferior result.

Original: To this end, we investigate 4,
Revised: To this end, we investigate 4,

Original: [Table]
Revised: [Table]

Original: and 49 centers for a local region
Revised: and 49 centers for a local region

Original: [Table]
Revised: [Table]

Original: and we refer to the resulting models as Small / 4, Small /
Revised: and we refer to the resulting models as Small / 4, Small /

Original: [Table]
Revised: [Table]

Original: and
Revised: and

Original: Small / 49, respectively.
Revised: Small / 49, respectively.

Original: Results in Table 4 indicate that our Context Cluster demonstrates promising generalisability to downstream tasks.
Revised: Results in Table 4 indicate that our Context Cluster demonstrates promising generalisability to downstream tasks.

Original: Our CoC-Small / 25 outperforms the ConvNet and ViT baselines on both detection and instance segmentation tasks when properly configured (25 centers in a local region).
Revised: Our CoC-Small / 25 outperforms the ConvNet and ViT baselines on both detection and instance segmentation tasks when properly configured (25 centers in a local region).

Original: In line with our expectations, only 4 centers cannot accurately model the large local region, and unnecessary centers cannot further enhance the performance.
Revised: In line with our expectations, only 4 centers cannot accurately model the large local region, and unnecessary centers cannot further enhance the performance.

Original: See appendix for more experiments.
Revised: See appendix for more experiments

Revised: [Table]

Original: 4.5 S emantic S egmentation on ADE20K
Revised: 4.5 S emantic S egmentation on ADE20K

Original: We examine our Context Cluster equipped with semantic FPN Kirillov et al. for semantic segmentation task on the ADE20KZhou et al.
Revised: We examine our Context Cluster equipped with semantic FPN (Kirillov et al., 2019) for semantic segmentation task on the ADE20K

Original: (2017) dataset.
Revised: (Zhou et al., 2017) dataset.

Original: For training, validation, and testing,
Revised: For training, validation, and testing,

Original: ADE20K includes 20k, 2k, and 3k images, each of which corresponds to one of 150 semantic categories.
Revised: ADE20K includes 20k, 2k, and 3k images, each of which corresponds to one of 150 semantic categories.

Original: For a fair comparison, we train all of our models for 80k iterations with a batch size of 16 on four V100 GPUs and adopt the standard data augmentation methods used in PVT Wang et al.
Revised: For a fair comparison, we train all of our models for 80k iterations with a batch size ofon four V100 GPUs and adopt the standard data augmentation methods used in PVT (Wang et al., 2021).

Original: With an initial learning rate of 2x10-4, the AdamW optimizer is used to train all of our models.
Revised: With an initial learning rate of 2x10-4, the AdamW optimizer is used to train all of our models.

Original: We use a polynomial decay schedule with a power of 0.9 to decrease the learning rate.
Revised: We use a polynomial decay schedule with a power of 0.9 to decrease the learning rate.

Original: [Table]
Revised: [Table]

Original: Experimental results on ADE20K are reported in Table 5.
Revised: Experimental results on ADE20K are reported in Table 5.

Original: We show that our Context Clusters clearly outperform PVT and ResNet using a similar number of parameters.
Revised: We show that our Context Clusters clearly outperform PVT and ResNet using a similar number of parameters.

Original: The promising improvements can be credited to our novel context cluster operation.
Revised: The promising improvements can be credited to our novel context cluster operation.

Original: Our context cluster is similar to the SuperPixel, which is an oversegmentation technology.
Revised: Our context cluster is similar to the SuperPixel, which is an oversegmentation technology.

Original: When applied for feature extraction, we expect context cluster can over-segment the contexts in intermediate features, and show improvements for semantic segmentation tasks.
Revised: When applied for feature extraction, we expect context cluster can over-segment the contexts in intermediate features, and show improvements for semantic segmentation tasks.

Original: Unlike in object detection and semantic segmentation tasks, the centers number shows little influence on the results.
Revised: Unlike in object detection and semantic segmentation tasks, the centers number shows little influence on the results.

Original: More results can be found in the appendix.
Revised: More results can be found in the appendix.

Original:  C onclusion
Revised:  C onclusion

Original: We introduce Context Cluster, a novel feature extraction paradigm for visual representation.
Revised: We introduce Context Cluster, a novel feature extraction paradigm for visual representation.

Original: Inspired by point cloud analysis and SuperPixel algorithms, we view an image as a set of unorganized points and employ the simplified clustering approach to extract features.
Revised: Inspired by point cloud analysis and SuperPixel algorithms, we view an image as a set of unorganized points and employ the simplified clustering approach to extract features.

Original: In terms of image interpretation and feature extraction operation, Context Cluster is fundamentally distinct from ConvNets and ViTs, and no convolution or attention is involved in our architecture.
Revised: In terms of image interpretation and feature extraction operation, Context Cluster is fundamentally distinct from ConvNets and ViTs, and no convolution or attention is involved in our architecture.

Original: Instead of chasing SOTA performance, we show that our Context Cluster can achieve comparable or even better results than ConvNet and ViT baselines on multiple tasks and domains.
Revised: Instead of chasing SOTA performance, we show that our Context Cluster can achieve comparable or even better results than ConvNet and ViT baselines on multiple tasks and domains.

Original: Most notably, our method shows promising interpretability and generalization properties.
Revised: Most notably, our method shows promising interpretability and generalization properties.

Original: We hope our Context Cluster can be considered as a novel visual representation method in addition to convolution and attention.
Revised: We hope our Context Cluster can be considered as a novel visual representation method in addition to convolution and attention.

Original: As discussed at the end of § 3, our new perspective and design for visual representation also come with new challenges, primarily in the compromise between accuracy and speed.
Revised: As discussed at the end of § 3, our new perspective and design for visual representation also come with new challenges, primarily in the compromise between accuracy and speed.

Original: Better strategies are worth exploring.
Revised: Better strategies are worth exploring.

Original: Departing from the current framework of detection and segmentation to apply our context cluster philosophy to other tasks is also a worthwhile direction to pursue.
Revised: Departing from the current framework of detection and segmentation to apply our context cluster philosophy to other tasks is also a worthwhile direction to pursue.

Original: E thics S tatement
Revised: E thics S tatement

Original: In our paper we strictly follow the ICLR ethical research standards and laws.
Revised: In our paper we strictly follow the ICLR ethical research standards and laws.

Original: All datasets we employed are publicly available, and all related publications and source codes are cited appropriately.
Revised: All datasets we employed are publicly available, and all related publications and source codes are cited appropriately.

Original: R eproducibility S tatement
Revised: R eproducibility S tatement

Original: We adhere to ICLR reproducibility standards and ensure the reproducibility of our work in multiple ways, including:
Revised: We adhere to ICLR reproducibility standards and ensure the reproducibility of our work in multiple ways, including:

Original: • We upload our codes, pre-trained models, and the training log files to an anonymous GitHub repository (https: // anonymous.4open.science / r / ContextCluster), as stated in the abstract.
Revised: • We upload our codes, pre-trained models, and the training log files to an anonymous GitHub repository (https: // anonymous.4open.science / r / ContextCluster), as stated in the abstract.

Original: A clear description is presented to ease the reviewing work.
Revised: A clear description is presented to ease the reviewing work.

Original: • Besides the main results of each task, we also upload the codes and checkpoints for all our ablation studies to ensure each experiment is strictly proved.
Revised: • Besides the main results of each task, we also upload the codes and checkpoints for all our ablation studies to ensure each experiment is strictly proved.

Original: • We clearly present the design of Context Cluster in Section 3.
Revised: • We clearly present the design of Context Cluster in Section 3.

Original: • Detailed framework and more experiments are presented in appendix.
Revised: • Detailed framework and more experiments are presented in appendix.

Original: By doing so, each experiment in our submission is easy to reproduce.
Revised: By doing so, each experiment in our submission is easy to reproduce.

Original: We always open-source our research work for each submission to help the community better understand our work
Revised: We always open-source our research work for each submission to help the community better understand our work

Original: A M odel C onfigurations
Revised: A M odel C onfigurations

Original: We first introduce the detailed configurations of our Context clusters in Table 6.
Revised: We first introduce the detailed configurations of our Context clusters in Table 6.

Original: Point reducer block is consistent with image downsize blocks, like in PVT and ConvNeXt.
Revised: Point reducer block is consistent with image downsize blocks, like in PVT and ConvNeXt.

Original: In the context of points, we select k _ neighbors of the nearest points for a proposed anchor, and fuse all the points using FC layer.
Revised: In the context of points, we select k _ neighbors of the nearest points for a proposed anchor, and fuse all the points using FC layer.

Original: We reduce the number of points by a factor of downsample _ r .
Revised: We reduce the number of points by a factor of downsample _ r .

Original: The key contribution in our work is context cluster blocks.
Revised: The key contribution in our work is context cluster blocks.

Original: We first evenly split the whole set of points into regions local regions in the space.
Revised: We first evenly split the whole set of points into regions local regions in the space.

Original: In each local region, we propose local _ centers for clustering.
Revised: In each local region, we propose local _ centers for clustering.

Original: We set the heads number and dimension of each head to heads and head _ dim , respectively, in our context cluster operation.
Revised: We set the heads number and dimension of each head to heads and head _ dim , respectively, in our context cluster operation.

Original: The channel number is expanded by a factor of mlp _ r ( i.e. , to mlp _ r × dim ) and then reduced to dim in the MLP block.
Revised: The channel number is expanded by a factor of mlp _ r ( i.e. , to mlp _ r × dim ) and then reduced to dim in the MLP block.

Original: The Context Cluster block would be repeated by several times in each stage.
Revised: The Context Cluster block would be repeated by several times in each stage.

Original: We mark all variables in blue for easy understanding.
Revised: We mark all variables in blue for easy understanding.

Original: For the Context-Cluster-Ti ‡ variation, it shares the same network structure as Context-Cluster-Ti, with the exception that we configure the region partition and local center numbers di ff erently.
Revised: For the Context-Cluster-Ti ‡ variation, it shares the same network structure as Context-Cluster-Ti, with the exception that we configure the region partition and local center numbers di ff erently.

Original: In particular, the number of region partitions is set to [49 , 49 , 1 , 1] and the number of centers in each local region is set to [16 , 4 , 49 , 16] for the four stages.
Revised: In particular, the number of region partitions is set to [49 , 49 , 1 , 1] and the number of centers in each local region is set to [16 , 4 , 49 , 16] for the four stages.

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: 

Original: [Table]

Original: 

Original: [Table]

Original:  
Revised: 

Original: [Table]
Revised: [Table]

Revised: 

Original:  B D etail E xplanations
Revised: [Table]   [Table] B D etail E xplanations

Original: One may be confused about how to specify the anchors in our point reducer block and the centers in our context cluster block.
Revised: One may be confused about how to specify the anchors in our point reducer block and the centers in our context cluster block.

Original: We provide illustrative and thorough explanations of them in this section.
Revised: We provide illustrative and thorough explanations of them in this section.

Original: For both anchor and center, they are generated evenly in the space .
Revised: For both anchor and center, they are generated evenly in the space .

Original: In order to better illustrate this, we plot organized image points in Fig.
Revised: In order to better illustrate this, we plot organized image points in Fig.

Original: [Figure]
Revised: [Figure]

Original: In a sense, the anchors are for reducing point numbers, and centers are used for clustering.
Revised: In a sense, the anchors are for reducing point numbers, and centers are used for clustering.

Original: Both of them are evenly distributed left we dot) with 4 neighbors for each
Revised: Both of them are evenly distributed left we dot) with 4 neighbors for each

Original: On right , we evenly sample
Revised: On right , we evenly sample

Original: (marked in red block), hence leading to 9 irregular clusters.
Revised: (marked in red block), hence leading to 9 irregular clusters.

Original: The center feature value is achieved by averaging its k neighbors.
Revised: The center feature value is achieved by averaging its k neighbors.

Original: In this figure, we show the neighbors in the big blue circle for the second center.
Revised: In this figure, we show the neighbors in the big blue circle for the second center.

Original: On the left, we display 16 points with 4 proposed anchors for point reduction, each of which takes its closest 4 neighbors into account.
Revised: On the left, we display 16 points with 4 proposed anchors for point reduction, each of which takes its closest 4 neighbors into account.

Original: All neighbors are concatenated along the channel dimension, and a FC layer is used to lower the dimensional number and fuse the information.
Revised: All neighbors are concatenated along the channel dimension, and a FC layer is used to lower the dimensional number and fuse the information.

Original: After reducing the number of points, we arrive at a new set of points with the same number of proposed anchors.
Revised: After reducing the number of points, we arrive at a new set of points with the same number of proposed anchors.

Original: On the right, we show 9 centers (red blocks) generated from the set of image points and corresponding 9 clusters.
Revised: On the right, we show 9 centers (red blocks) generated from the set of image points and corresponding 9 clusters.

Original: The feature of generated centers will be given by averaging the k neighbors second average
Revised: The feature of generated centers will be given by averaging the k neighbors second average

Original: The number of neighbors can be of any value.
Revised: The number of neighbors can be of any value.

Original: We set it to 4 or 9 due to three reasons.
Revised: We set it to 4 or 9 due to three reasons.

Original: First, we follow the design of ConvNets and pyramid ViTs to ensure the set of points can be reorganized to a rectangular feature map.
Revised: First, we follow the design of ConvNets and pyramid ViTs to ensure the set of points can be reorganized to a rectangular feature map.

Original: Second, this strategy eases our coding by employing convolution or pooling operations (which are equivalent to our description) and avoids heavy indices searching work.
Revised: Second, this strategy eases our coding by employing convolution or pooling operations (which are equivalent to our description) and avoids heavy indices searching work.

Original: Last, a rectangular feature map is required by most detection and segmentation methods.
Revised: Last, a rectangular feature map is required by most detection and segmentation methods.

Original: C M ore E xperiments
Revised: C M ore E xperiments

Original: We conduct more experiments to validate the e ff ectiveness of our Context Cluster.
Revised: We conduct more experiments to validate the e ff ectiveness of our Context Cluster.

Original: [Table]
Revised: [Table]

Original: More results for Segmentation.
Revised: More results for Segmentation.

Original: We report the CoC-Small with Semantic FPN’s performance on the ADE20K validation set in Table 7 under the same conditions as previous.
Revised: We report the CoC-Small with Semantic FPN’s performance on the ADE20K validation set in Table 7 under the same conditions as previous.

Original: We notice that performance increases modestly as more centers are added in a local region, but the computational cost rises a lot.
Revised: We notice that performance increases modestly as more centers are added in a local region, but the computational cost rises a lot.

Original: With 4 V
Revised: With 4 V

Original: GPUs, training CoC-Small / 4 would take 9 hours.
Revised: GPUs, training CoC-Small / 4 would take 9 hours.

Original: As a comparison, it would take 11 hours for CoC-Small / 25 and 14 hours for CoC-Small / 49.
Revised: As a comparison, it would take 11 hours for CoC-Small / 25 and 14 hours for CoC-Small / 49.

Original: The cost of computing increases linearly with the complexity of the computations in our context cluster blocks.
Revised: The cost of computing increases linearly with the complexity of the computations in our context cluster blocks.

Original: More results for Detection.
Revised: More results for Detection.

Original: We also conduct more results for object detection and instance segmentation task.
Revised: We also conduct more results for object detection and instance segmentation task.

Original: Besides the experiments reported in Table 4, we also conduct experiments based on the pre-trained CoC-Medium.
Revised: Besides the experiments reported in Table 4, we also conduct experiments based on the pre-trained CoC-Medium.

Original: Results are reported in Table 8.
Revised: Results are reported in Table 8.

Original: In line with CoC-Small, CoC-Medium shows comparable performance as in ResNet50 and PVT-Small.
Revised: In line with CoC-Small, CoC-Medium shows comparable performance as in ResNet50 and PVT-Small.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: or

Original: Clustering map of all heads.
Revised: Clustering map of all heads.

Original: To have a better understanding of our context clustering operation, we show the clustering maps of all heads in the last block of stage3 and stage4 in our Context ClusterMedium.
Revised: To have a better understanding of our context clustering operation, we show the clustering maps of all heads in the last block of stage3 and stage4 in our Context Cluster-Medium.

Original: As we expected, results in Fig.
Revised: As we expected, results in Fig.

Original: 6 indicate that our method is able to cluster semantically similar contexts together and exhibits decent locality.
Revised: 6 indicate that our method is able to cluster semantically similar contexts together and exhibits decent locality.

Original: An interesting observation is that the context cluster operation also tends to cluster contexts along vertical or horizontal directions .
Revised: An interesting observation is that the context cluster operation also tends to cluster contexts along vertical or horizontal directions.

Original: Similar phenomena can also be observed in other images and model variants.
Revised: Similar phenomena can also be observed in other images and model variants.