Original: SAPA: Similarity-Aware Point Affiliation for Feature Upsampling
Revised: SAPA: Similarity-Aware Point Affiliation for Feature Upsampling

Original: Abstract
Revised: Abstract

Original: We introduce point affiliation into feature upsampling, a notion that describes the affiliation of each upsampled point to a semantic cluster formed by local decoder feature points with semantic similarity.
Revised: We introduce point affiliation into feature upsampling, a notion that describes the affiliation of each upsampled point to a semantic cluster formed by local decoder feature points with semantic similarity.

Original: By rethinking point affiliation, we present a generic formulation for generating upsampling kernels.
Revised: By rethinking point affiliation, we present a generic formulation for generating upsampling kernels.

Original: The kernels encourage not only semantic smoothness but also boundary sharpness in the upsampled feature maps.
Revised: The kernels encourage not only semantic smoothness but also boundary sharpness in the upsampled feature maps.

Original: Such properties are particularly useful for some dense prediction tasks such as semantic segmentation.
Revised: Such properties are particularly useful for some dense prediction tasks such as semantic segmentation.

Original: The key idea of our formulation is to generate similarity-aware kernels by comparing the similarity between each encoder feature point and the spatially associated local region of decoder features.
Revised: The key idea of our formulation is to generate similarity-aware kernels by comparing the similarity between each encoder feature point and the spatially associated local region of decoder features.

Original: In this way, the encoder feature point can function as a cue to inform the semantic cluster of upsampled feature points.
Revised: In this way, the encoder feature point can function as a cue to inform the semantic cluster of upsampled feature points.

Original: To embody the formulation, we further instantiate a lightweight upsampling operator, termed Similarity-Aware
Revised: To embody the formulation, we further instantiate a lightweight upsampling operator, termed Similarity-Aware

Original: Point Affiliation (SAPA), and investigate its variants.
Revised: Point Affiliation (SAPA), and investigate its variants.

Original: SAPA invites consistent performance improvements on a number of dense prediction tasks, including semantic segmentation, object detection, depth estimation, and image matting.
Revised: SAPA invites consistent performance improvements on a number of dense prediction tasks, including semantic segmentation, object detection, depth estimation, and image matting.

Original: Code is available at: https://github.com/poppinace/sapa
Revised: Code is available at: https://github.com/poppinace/sapa

Original:  Introduction
Revised:  Introduction

Original: We introduce the notion of point affiliation into feature upsampling.
Revised: We introduce the notion of point affiliation into feature upsampling.

Original: Point affiliation defines a relation between each upsampled point and a semantic cluster 2 to which the point should belong.
Revised: Point affiliation defines a relation between each upsampled point and a semantic cluster 2 to which the point should belong.

Original: It highlights the spatial arrangement of upsampled points at the semantic level.
Revised: It highlights the spatial arrangement of upsampled points at the semantic level.

Original: Considering an example shown in Fig.
Revised: Considering an example shown in Fig.

Original: 1 w.r t. × 2 upsampling in semantic segmentation, the orange point of low resolution will correspond to four upsampled points of high resolution, in which the red and yellow ones should be assigned the ‘picture’ cluster and the ‘wall’ cluster, respectively.
Revised: 1 w.r t. × 2 upsampling in semantic segmentation, the orange point of low resolution will correspond to four upsampled points of high resolution, in which the red and yellow ones should be assigned the ‘picture’ cluster and the ‘wall’ cluster, respectively.

Original: Designating point affiliation is difficult and sometimes can be erroneous, however.
Revised: Designating point affiliation is difficult and sometimes can be erroneous, however.

Original: In × 2 upsampling, nearest neighbor (NN) interpolation directly copies four identical points from the low-res one, which assigns the same semantic cluster to the four points.
Revised: In × 2 upsampling, nearest neighbor (NN) interpolation directly copies four identical points from the low-res one, which assigns the same semantic cluster to the four points.

Original: On regions in need of details, the four points probably do not share the same cluster but are forced to share.
Revised: On regions in need of details, the four points probably do not share the same cluster but are forced to share.

Original: Bilinear interpolation assigns point affiliation with distance priors.
Revised: Bilinear interpolation assigns point affiliation with distance priors.

Original: Yet, when tackling points of different semantic clusters, it not only cannot inform clear point affiliation, but also blurs the boundary between different semantic clusters.
Revised: Yet, when tackling points of different semantic clusters, it not only cannot inform clear point affiliation, but also blurs the boundary between different semantic clusters.

Original: Recent dynamic upsampling operators have similar issues.
Revised: Recent dynamic upsampling operators have similar issues.

Original: CARAFE [1, 2] judges the affiliation of an upsampled point with content-aware kernels.
Revised: CARAFE [1, 2] judges the affiliation of an upsampled point with content-aware kernels.

Original: Certain semantic clusters will receive larger weights than the rest and therefore dominate the affiliation of upsampled points.
Revised: Certain semantic clusters will receive larger weights than the rest and therefore dominate the affiliation of upsampled points.

Original: However, the affiliation near boundaries or on regions full of details can still be ambiguous.
Revised: However, the affiliation near boundaries or on regions full of details can still be ambiguous.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, the
Revised: 2, the

Original: [Figure]
Revised: [Figure]

Original: boundaries are unclear in the feature maps upsampled by CARAFE.
Revised: boundaries are unclear in the feature maps upsampled by CARAFE.

Original: The reason is that the kernels are conditioned on decoder features alone; the decoder features carry little useful information about high-res structure.
Revised: The reason is that the kernels are conditioned on decoder features alone; the decoder features carry little useful information about high-res structure.

Original: Inferring structure requires high-res encoder features.
Revised: Inferring structure requires high-res encoder features.

Original: For instance, if the orange point in Fig.
Revised: For instance, if the orange point in Fig.

Original: 1 lies on the low-res boundary, it is difficult to judge to which cluster the four upsampled points should belong.
Revised: 1 lies on the low-res boundary, it is difficult to judge to which cluster the four upsampled points should belong.

Original: However, the encoder feature in Fig.
Revised: However, the encoder feature in Fig.

Original: 1 actually tells that, the yellow point is on the wall, and the red one is on the picture, which suggests one may extract useful information from the encoder feature to assist the designation of point affiliation.
Revised: 1 actually tells that, the yellow point is on the wall, and the red one is on the picture, which suggests one may extract useful information from the encoder feature to assist the designation of point affiliation.

Original: Indeed IndexNet [3, 4] and A2U [5] have attempted to encode such information to improve detail delineation in encoder-dependent upsampling kernels; however, the encoder feature can easily introduce noise into kernels, engendering discontinuous feature maps shown in Fig.
Revised: Indeed IndexNet [3, 4] and A2U [5] have attempted to encode such information to improve detail delineation in encoder-dependent upsampling kernels; however, the encoder feature can easily introduce noise into kernels, engendering discontinuous feature maps shown in Fig.

Original: Hence, the key problem seems to be how to extract only required information into the upsampling kernels from the encoder feature while filtering out the rest.
Revised: Hence, the key problem seems to be how to extract only required information into the upsampling kernels from the encoder feature while filtering out the rest.

Original: To use encoder features effectively, an important assumption of this paper is that, an encoder feature point is most similar to the semantic cluster into which the point will be classified.
Revised: To use encoder features effectively, an important assumption of this paper is that, an encoder feature point is most similar to the semantic cluster into which the point will be classified.

Original: Per the left of
Revised: Per the left of

Original: Fig. 1, suppose that the encoder point in the red box is assigned into the cluster C 1 by its semantic meaning, then it is similar to C 1 , while not similar to C 2 .
Revised: Fig. 1, suppose that the encoder point in the red box is assigned into the cluster C 1 by its semantic meaning, then it is similar to C 1 , while not similar to C 2 .

Original: As a result, by comparing the similarity between the encoder feature point and different semantic clusters in the decoder feature, the affiliation of the upsampled point can be informed according to the similarity scores.
Revised: As a result, by comparing the similarity between the encoder feature point and different semantic clusters in the decoder feature, the affiliation of the upsampled point can be informed according to the similarity scores.

Original: In particular, we propose to generate upsampling kernels with local mutual similarity between encoder and decoder features.
Revised: In particular, we propose to generate upsampling kernels with local mutual similarity between encoder and decoder features.

Original: For every encoder feature point, we compute the similarity score between this point and each decoder feature point in the spatially associated local window.
Revised: For every encoder feature point, we compute the similarity score between this point and each decoder feature point in the spatially associated local window.

Original: For the green point in Fig.
Revised: For the green point in Fig.

Original: 1, since every point in the window shares the same semantic cluster, the encoder feature point is as similar as every point in the window.
Revised: 1, since every point in the window shares the same semantic cluster, the encoder feature point is as similar as every point in the window.

Original: In this case we expect an ‘average kernel’ which is the key characteristic to filter noise, and the upsampled four points would have the same semantic cluster as before.
Revised: In this case we expect an ‘average kernel’ which is the key characteristic to filter noise, and the upsampled four points would have the same semantic cluster as before.

Original: For the yellow point in the encoder, since it belongs to the ‘wall’ cluster, it is more similar to the points on the wall than those on the picture.
Revised: For the yellow point in the encoder, since it belongs to the ‘wall’ cluster, it is more similar to the points on the wall than those on the picture.

Original: In this case we expect a kernel with larger weights on points related to the ‘wall’ cluster.
Revised: In this case we expect a kernel with larger weights on points related to the ‘wall’ cluster.

Original: This can help to assign the affiliation of the yellow point to be in the ‘wall’ cluster.
Revised: This can help to assign the affiliation of the yellow point to be in the ‘wall’ cluster.

Original: By modeling the local mutual similarity, we derive a generic form of upsampling kernels and show that this form implements our expected upsampling behaviors: encouraging both semantic smoothness and boundary sharpness.
Revised: By modeling the local mutual similarity, we derive a generic form of upsampling kernels and show that this form implements our expected upsampling behaviors: encouraging both semantic smoothness and boundary sharpness.

Original: Following our formulation, we further instantiate a lightweight upsampling operator, termed Similarity-Aware Point Affiliation (SAPA), and investigate its variants.
Revised: Following our formulation, we further instantiate a lightweight upsampling operator, termed Similarity-Aware Point Affiliation (SAPA), and investigate its variants.

Original: We evaluate
Revised: We evaluate

Original: SAPA across a number of mainstream dense prediction tasks, for example: i) semantic segmentation : we test SAPA on several transformer-based segmentation baselines on the ADE20K dataset [6], such as SegFormer [7], MaskFormer [8], and Mask2Former [9], improving the baselines by 1% ∼ 2 .
Revised: SAPA across a number of mainstream dense prediction tasks, for example: i) semantic segmentation : we test SAPA on several transformer-based segmentation baselines on the ADE20K dataset [6], such as SegFormer [7], MaskFormer [8], and Mask2Former [9], improving the baselines by 1% ∼ 2 .

Original: 7% mIoU; ii) object detection : SAPA improves the performance of Faster R-CNN by 0 .
Revised: 7% mIoU; ii) object detection : SAPA improves the performance of Faster R-CNN by 0 .

Original: 4% AP on MS
Revised: 4% AP on MS

Original: COCO [10]; iii) monocular depth estimation : SAPA reduces the rmse metric of BTS [11] from 0 .
Revised: COCO [10]; iii) monocular depth estimation : SAPA reduces the rmse metric of BTS [11] from 0 .

Original: 419 to 0 .
Revised: 419 to 0 .

Original: 408 on NYU Depth V2 [12]; and iv) image matting : SAPA outperforms a strong A2U matting baseline [5] on the Adobe Composition-1k testing set [13] with a further 3 .
Revised: 408 on NYU Depth V2 [12]; and iv) image matting : SAPA outperforms a strong A2U matting baseline [5] on the Adobe Composition-1k testing set [13] with a further 3 .

Original: 8% relative error reduction in the SAD metric.
Revised: 8% relative error reduction in the SAD metric.

Original: SAPA also outperforms or at least is on par with other state-of-the-art
Revised: SAPA also outperforms or at least is on par with other state-of-the-art

Original: [Figure]
Revised: [Figure]

Original: U p s a m p l e d f ea t u r e m a p s
Revised: U p s a m p l e d f ea t u r e m a p s

Original: K e r n e l dynamic upsampling operators.
Revised: K e r n e l dynamic upsampling operators.

Original: Particularly, even without additional parameters, SAPA outperforms the previous best upsampling operator CARAFE on semantic segmentation.
Revised: Particularly, even without additional parameters, SAPA outperforms the previous best upsampling operator CARAFE on semantic segmentation.

Original:  Related Work
Revised:  Related Work

Original: We review work related to feature upsampling.
Revised: We review work related to feature upsampling.

Original: Feature upsampling is a fundamental procedure in encoder-decoder architectures used to recover the spatial resolution of low-res decoder feature maps and has been extensively used in dense prediction tasks such as semantic segmentation [14, 7, 8, 9] and depth estimation [16, 17, 11].
Revised: Feature upsampling is a fundamental procedure in encoder-decoder architectures used to recover the spatial resolution of low-res decoder feature maps and has been extensively used in dense prediction tasks such as semantic segmentation [14, 7, 8, 9] and depth estimation [16, 17, 11].

Original: Standard upsampling operators are hand-crafted.
Revised: Standard upsampling operators are hand-crafted.

Original: NN and bilinear interpolation measure the semantic affiliation in terms of relative distances in upsampling, which follows fixed rules to designate point affiliation, even if the true affiliation may be different.
Revised: NN and bilinear interpolation measure the semantic affiliation in terms of relative distances in upsampling, which follows fixed rules to designate point affiliation, even if the true affiliation may be different.

Original: Max unpooling [14] stores the indices of max-pooled feature points in encoder features and uses the sparse indices to guide upsampling.
Revised: Max unpooling [14] stores the indices of max-pooled feature points in encoder features and uses the sparse indices to guide upsampling.

Original: While it executes location-specific point affiliation which benefits detail recovery, most upsampled points are assigned with null affiliation due to zero filling.
Revised: While it executes location-specific point affiliation which benefits detail recovery, most upsampled points are assigned with null affiliation due to zero filling.

Original: Pixel Shuffle [18] is widely-used in image/video super-resolution.
Revised: Pixel Shuffle [18] is widely-used in image/video super-resolution.

Original: Its upsampling only includes memory operation – reshaping depth channels to space ones.
Revised: Its upsampling only includes memory operation – reshaping depth channels to space ones.

Original: The notion of point affiliation does not apply to this operator, however.
Revised: The notion of point affiliation does not apply to this operator, however.

Original: Another stream of upsampling operators implement learning-based upsampling.
Revised: Another stream of upsampling operators implement learning-based upsampling.

Original: Among them, transposed convolution or deconvolution [19] is known as an inverse convolutional operator.
Revised: Among them, transposed convolution or deconvolution [19] is known as an inverse convolutional operator.

Original: Based on a novel interpretation of deconvolution, PixelTCL [20] is proposed to alleviate the checkerboard artifacts [21] of standard deconvolution.
Revised: Based on a novel interpretation of deconvolution, PixelTCL [20] is proposed to alleviate the checkerboard artifacts [21] of standard deconvolution.

Original: In addition, bilinear additive upsampling [22] attempts to combine learnable convolutional kernels with hand-crafted upsampling operators to achieve composited upsampling.
Revised: In addition, bilinear additive upsampling [22] attempts to combine learnable convolutional kernels with hand-crafted upsampling operators to achieve composited upsampling.

Original: Recently, DUpsample [23] seeks to reconstruct the upsampled feature map with pre-learned projection matrices, expecting to achieve a data-dependent upsampling behavior.
Revised: Recently, DUpsample [23] seeks to reconstruct the upsampled feature map with pre-learned projection matrices, expecting to achieve a data-dependent upsampling behavior.

Original: While these operators are learnable, the upsampling kernels are fixed once learned, still resulting in ambiguous designation of point affiliation.
Revised: While these operators are learnable, the upsampling kernels are fixed once learned, still resulting in ambiguous designation of point affiliation.

Original: In learning-based upsampling, some recent work introduces the idea of generating content-aware dynamic kernels.
Revised: In learning-based upsampling, some recent work introduces the idea of generating content-aware dynamic kernels.

Original: Instead of learning the parameters of the kernels, they learn how to predict the kernels.
Revised: Instead of learning the parameters of the kernels, they learn how to predict the kernels.

Original: In particular, CARAFE [1, 2] predicts dynamic kernels conditioned on the decoder features.
Revised: In particular, CARAFE [1, 2] predicts dynamic kernels conditioned on the decoder features.

Original: IndexNet [3, 4] and A2U [5], by contrast, generate encoder-dependent kernels.
Revised: IndexNet [3, 4] and A2U [5], by contrast, generate encoder-dependent kernels.

Original: While they significantly outperform previous upsampling operators in various tasks, they still can cause uncertain point affiliation, resulting in either unclear predictions near boundaries or fragile predictions in regions.
Revised: While they significantly outperform previous upsampling operators in various tasks, they still can cause uncertain point affiliation, resulting in either unclear predictions near boundaries or fragile predictions in regions.

Original: Our work is closely related to dynamic kernel-based upsampling.
Revised: Our work is closely related to dynamic kernel-based upsampling.

Original: We also seek to predict dynamic kernels; however, we aim to address the uncertain point affiliation in prior arts to achieve simultaneous region smoothness and boundary sharpness.
Revised: We also seek to predict dynamic kernels; however, we aim to address the uncertain point affiliation in prior arts to achieve simultaneous region smoothness and boundary sharpness.

Original:  Dynamic Upsampling Revisited
Revised:  Dynamic Upsampling Revisited

Original: We first revisit two key components shared by existing dynamic upsampling operators: kernel generation and feature assembly.
Revised: We first revisit two key components shared by existing dynamic upsampling operators: kernel generation and feature assembly.

Original: Kernel Generation Given the decoder feature X ∈ R H × W × C , if we upsample it to the target feature X
Revised: Kernel Generation Given the decoder feature X ∈ R H × W × C , if we upsample it to the target feature X

Original: ′ ∈
Revised: ′ ∈

Original: R 2 H × 2 W × C , then for any point at the location l ′ =
Revised: R 2 H × 2 W × C , then for any point at the location l ′ =

Original: ( i ′ , j ′ ) in X ′ , we generate a kernel
Revised: ( i ′ , j ′ ) in X ′ , we generate a kernel

Original: W l ′ based on the neighborhood feature N l ′ that spatially corresponds to l ′ .
Revised: W l ′ based on the neighborhood feature N l ′ that spatially corresponds to l ′ .

Original: In this way, kernel generation can be defined by
Revised: In this way, kernel generation can be defined by

Original: [Equation]
Revised: [Equation]

Original: where ψ refers to a kernel generation module, which is often implemented by a sub-network used to predict the kernel conditioned on N l ′ , and norm is a normalization function.
Revised: where ψ refers to a kernel generation module, which is often implemented by a sub-network used to predict the kernel conditioned on N l ′ , and norm is a normalization function.

Original: The feature N l ′ can originate from two sources, to be specific, from the encoder
Revised: The feature N l ′ can originate from two sources, to be specific, from the encoder

Original: feature Y ∈
Revised: feature Y ∈

Original: R 2 H × 2 W × C or from the decoder feature X .
Revised: R 2 H × 2 W × C or from the decoder feature X .

Original: If the encoder feature Y is chosen as the source, then a local region of Y centered at l ′ is extracted to be N l ′ .
Revised: If the encoder feature Y is chosen as the source, then a local region of Y centered at l ′ is extracted to be N l ′ .

Original: If the decoder feature X is chosen, one first needs to compute the projective location of l ′ , i.e. , l = ( i, j ) =
Revised: If the decoder feature X is chosen, one first needs to compute the projective location of l ′ , i.e. , l = ( i, j ) =

Original: ⌊ l ′ 2 ⌋ =
Revised: ⌊ l ′ 2 ⌋ =

Original: ( ⌊ i ′ 2 ⌋ , ⌊ j ′ 2 ⌋ ) according to the spatial location correspondence, then a local region of X centered at l is regarded as N l ′ .
Revised: ( ⌊ i ′ 2 ⌋ , ⌊ j ′ 2 ⌋ ) according to the spatial location correspondence, then a local region of X centered at l is regarded as N l ′ .

Original: The softmax function is often used as the normalization function such that relevant points can be softly selected to compute the value of the target point using the weight W l ′ .
Revised: The softmax function is often used as the normalization function such that relevant points can be softly selected to compute the value of the target point using the weight W l ′ .

Original: According to Fig. 2, the source of N l ′ can affect the predicted kernel.
Revised: According to Fig. 2, the source of N l ′ can affect the predicted kernel.

Original: The kernels predicted by
Revised: The kernels predicted by

Original: CARAFE, IndexNet, and A2U show significantly distinct characteristics.
Revised: CARAFE, IndexNet, and A2U show significantly distinct characteristics.

Original: With the decoder feature alone, the kernel map is coarse and lacking in details.
Revised: With the decoder feature alone, the kernel map is coarse and lacking in details.

Original: Benefiting from the encoder feature, the kernel maps generated by IndexNet and A2U have rich details; however, they manifest high similarity to the encoder feature, which means noise is introduced into the kernel.
Revised: Benefiting from the encoder feature, the kernel maps generated by IndexNet and A2U have rich details; however, they manifest high similarity to the encoder feature, which means noise is introduced into the kernel.

Original: Feature Assembly For each target feature point at l ′ , we assemble the corresponding sub-region of decoder feature with the predicted K × K kernel W l ′ , whose weight is denoted by W l ′ ,p , p ∈
Revised: Feature Assembly For each target feature point at l ′ , we assemble the corresponding sub-region of decoder feature with the predicted K × K kernel W l ′ , whose weight is denoted by W l ′ ,p , p ∈

Original: I, I = { ( u, v ) : u, v = − r, ..., r, r =
Revised: I, I = { ( u, v ) : u, v = − r, ..., r, r =

Original: ⌊ K 2 ⌋} , to obtain the value of the target point X ′ l ′ by a weighted sum
Revised: ⌊ K 2 ⌋} , to obtain the value of the target point X ′ l ′ by a weighted sum

Original: [Equation]
Revised: [Equation]

Original: By executing feature assembly on every target feature point, we can obtain the target upsampled feature map.
Revised: By executing feature assembly on every target feature point, we can obtain the target upsampled feature map.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, the upsampled feature has a close relation to the kernel.
Revised: 2, the upsampled feature has a close relation to the kernel.

Original: A wellpredicted kernel can encourage both semantic smoothness and boundary sharpness; a kernel without encoding details or with too many details encoded can introduce noise.
Revised: A wellpredicted kernel can encourage both semantic smoothness and boundary sharpness; a kernel without encoding details or with too many details encoded can introduce noise.

Original: We consider an ideal kernel should only response at the position in need of details, while do not response (appearing as an average value over an area) at good semantic regions.
Revised: We consider an ideal kernel should only response at the position in need of details, while do not response (appearing as an average value over an area) at good semantic regions.

Original: More importantly, an ideal kernel should assign weights reasonably so that each point can be designated to a correct semantic cluster.
Revised: More importantly, an ideal kernel should assign weights reasonably so that each point can be designated to a correct semantic cluster.

Original:  Rethinking Point Affiliation with Local Mutual Similarity
Revised:  Rethinking Point Affiliation with Local Mutual Similarity

Original: To obtain an expected upsampling kernel mentioned above, we first derive a generic formulation for generating the upsampling kernel by exploiting local mutual similarity, then explain why the formula- tion encourages semantic smoothness and boundary sharpness, and finally present an upsampling operator, SAPA, that embodies our formulation.
Revised: To obtain an expected upsampling kernel mentioned above, we first derive a generic formulation for generating the upsampling kernel by exploiting local mutual similarity, then explain why the formula- tion encourages semantic smoothness and boundary sharpness, and finally present an upsampling operator, SAPA, that embodies our formulation.

Original: 4.1 Local Mutual Similarity
Revised: 4.1 Local Mutual Similarity

Original: We rethink point affiliation from the view of local mutual similarity between encoder and decoder features.
Revised: We rethink point affiliation from the view of local mutual similarity between encoder and decoder features.

Original: With a detailed analysis, we explain why such similarity can assist point affiliation.
Revised: With a detailed analysis, we explain why such similarity can assist point affiliation.

Original: We first define a generic similarity function sim( x , y ) :
Revised: We first define a generic similarity function sim( x , y ) :

Original: R d × R d → R .
Revised: R d × R d → R .

Original: It scores the similarity between a vector x and a vector y of the same dimension d .
Revised: It scores the similarity between a vector x and a vector y of the same dimension d .

Original: We also define a normalization function involving n real numbers x 1 , x 2 , ..., x n by norm( x i ; x 1 , x 2 , ..., x n )
Revised: We also define a normalization function involving n real numbers x 1 , x 2 , ..., x n by norm( x i ; x 1 , x 2 , ..., x n )

Original: =
Revised: =

Original: h ( x i ) (cid:80) nj =1 h ( x j ) , where h ( x ) :
Revised: h ( x i ) (cid:80) nj =1 h ( x j ) , where h ( x ) :

Original: R → R is an arbitrary function, ignoring zero division.
Revised: R → R is an arbitrary function, ignoring zero division.

Original: Given sim( x , y ) and h ( x ) , we can define a generic formulation for generating the upsampling kernel
Revised: Given sim( x , y ) and h ( x ) , we can define a generic formulation for generating the upsampling kernel

Original: [Equation]
Revised: [Equation]

Original: where w is the kernel value specific to x and y .
Revised: where w is the kernel value specific to x and y .

Original: To analyze the upsampling behavior of the kernel, we further define the following notations.
Revised: To analyze the upsampling behavior of the kernel, we further define the following notations.

Original: Let y l ′ ∈ R C denote the encoder feature vector at position l ′ and x l ∈ R C be the decoder feature vector at position l , where C is the number of channels.
Revised: Let y l ′ ∈ R C denote the encoder feature vector at position l ′ and x l ∈ R C be the decoder feature vector at position l , where C is the number of channels.

Original: Our operation will be done within a local window of size K × K , between each encoder vector y l ′ and all its spatially associated decoder feature vectors, x l + p ’s.
Revised: Our operation will be done within a local window of size K × K , between each encoder vector y l ′ and all its spatially associated decoder feature vectors, x l + p ’s.

Original: To simplify our analysis, we also assume local smoothness.
Revised: To simplify our analysis, we also assume local smoothness.

Original: That is, points with the same semantic cluster will have a similar value, which means a local region will share the same value on every channel of the feature map.
Revised: That is, points with the same semantic cluster will have a similar value, which means a local region will share the same value on every channel of the feature map.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, we choose a and b as the feature vectors in ‘wall’ and ‘picture’ cluster, respectively, then a and b are constant vectors.
Revised: 1, we choose a and b as the feature vectors in ‘wall’ and ‘picture’ cluster, respectively, then a and b are constant vectors.

Original: For ease of analysis, we define two types of windows distinguished by their contents.
Revised: For ease of analysis, we define two types of windows distinguished by their contents.

Original: When all the points inside a window belong to the same semantic cluster, it is called a smooth window; while different semantic clusters appear in a window, it is defined as a detail window.
Revised: When all the points inside a window belong to the same semantic cluster, it is called a smooth window; while different semantic clusters appear in a window, it is defined as a detail window.

Original: Next, we explain why the kernel can filter out noise, why it encourages semantic smoothness in a smooth window, and why it can help to recover details when dealing with boundaries/textures in a detail window.
Revised: Next, we explain why the kernel can filter out noise, why it encourages semantic smoothness in a smooth window, and why it can help to recover details when dealing with boundaries/textures in a detail window.

Original: Upsampling in a Smooth Window Without loss of generality, we consider an encoder point at the position l ′ in Fig.
Revised: Upsampling in a Smooth Window Without loss of generality, we consider an encoder point at the position l ′ in Fig.

Original: Its corresponding window is a smooth window of the semantic cluster ‘picture’, thus x l + p = b , ∀ p ∈
Revised: Its corresponding window is a smooth window of the semantic cluster ‘picture’, thus x l + p = b , ∀ p ∈

Original: I .
Revised: I .

Original: Then the upsampling kernel weight w.r.t.
Revised: Then the upsampling kernel weight w.r.t.

Original: the upsampled point l ′ at the position p takes the form
Revised: the upsampled point l ′ at the position p takes the form

Original: [Equation]
Revised: [Equation]

Original: which has nothing to do with l , and p .
Revised: which has nothing to do with l , and p .

Original: Eq.
Revised: Eq.

Original: (4) reveals a key characteristic of local mutual similarity in a smooth window: the kernel weight is a constant regardless of y .
Revised: (4) reveals a key characteristic of local mutual similarity in a smooth window: the kernel weight is a constant regardless of y .

Original: Therefore, the kernel fundamentally can filter out noise from encoder features with an ‘average’ kernel.
Revised: Therefore, the kernel fundamentally can filter out noise from encoder features with an ‘average’ kernel.

Original: Note that, in the derivation above the necessary conditions include: i) x is from a local window in the decoder feature map; ii) a normalization function in the form of
Revised: Note that, in the derivation above the necessary conditions include: i) x is from a local window in the decoder feature map; ii) a normalization function in the form of

Original: [Equation]
Revised: [Equation]

Original: Upsampling in a Detail Window Again we consider two encoder points at the position m ′ and n ′ in Fig.
Revised: Upsampling in a Detail Window Again we consider two encoder points at the position m ′ and n ′ in Fig.

Original: Ideally y m ′ and y n ′ should be classified into the semantic cluster of ‘wall’ and ’picture’, respectively.
Revised: Ideally y m ′ and y n ′ should be classified into the semantic cluster of ‘wall’ and ’picture’, respectively.

Original: Taking y m ′ as an example, following our assumption, it is more similar to points of the ‘wall’ cluster rather than the ‘picture’ cluster.
Revised: Taking y m ′ as an example, following our assumption, it is more similar to points of the ‘wall’ cluster rather than the ‘picture’ cluster.

Original: From Eq.
Revised: From Eq.

Original: (4), we can tell that sim( x m + s , y m ′ ) is larger than sim( x m + t , y m ′ ) , where s and t are the offsets in the window such that m + s and m + t are within the ‘wall’ and the ‘picture’ cluster, respectively.
Revised: (4), we can tell that sim( x m + s , y m ′ ) is larger than sim( x m + t , y m ′ ) , where s and t are the offsets in the window such that m + s and m + t are within the ‘wall’ and the ‘picture’ cluster, respectively.

Original: Therefore, after computing similarity scores and normalization, one can acquire a kernel with significantly larger weights on points with the semantic cluster of ‘wall’ than that of ‘picture’, i.e. , W m ′ ,s >> W m ′ ,t .
Revised: Therefore, after computing similarity scores and normalization, one can acquire a kernel with significantly larger weights on points with the semantic cluster of ‘wall’ than that of ‘picture’, i.e. , W m ′ ,s >> W m ′ ,t .

Original: After applying the kernel to the corresponding window, the upsampled point at m ′ will be revalued and assigned to the semantic cluster of ‘wall’.
Revised: After applying the kernel to the corresponding window, the upsampled point at m ′ will be revalued and assigned to the semantic cluster of ‘wall’.

Original: Similarly, the upsampled point at n ′ will be assigned into ‘picture’.
Revised: Similarly, the upsampled point at n ′ will be assigned into ‘picture’.

Original: normalization 𝑑 𝐾 !
Revised: normalization 𝑑 𝐾 !

Original: 𝑑 kernels Encoder feature
Revised: 𝑑 kernels Encoder feature

Original: Decoder feature 2𝐻×2𝑊×𝐶 𝐻×𝑊×𝐶
Revised: Decoder feature 2𝐻×2𝑊×𝐶 𝐻×𝑊×𝐶

Original: P r e - p r o ce ss i ng ( 𝑑=𝐶 ) 𝐺 𝐵 𝐼 Pre-processing
Revised: P r e - p r o ce ss i ng ( 𝑑=𝐶 ) 𝐺 𝐵 𝐼 Pre-processing

Original: Computing similarity 2𝐻×2𝑊×𝐶
Revised: Computing similarity 2𝐻×2𝑊×𝐶

Original: Upsampledfeature
Revised: Upsampledfeature

Original: Kernel generation matrix multiplication linear embedding gating mechanism
Revised: Kernel generation matrix multiplication linear embedding gating mechanism

Original: Feature assembly ( 𝑑<𝐶 ) ( 𝑑<𝐶 )
Revised: Feature assembly ( 𝑑<𝐶 ) ( 𝑑<𝐶 )

Original: Note that, in Eq.
Revised: Note that, in Eq.

Original: (4) we have no constraint on y .
Revised: (4) we have no constraint on y .

Original: But here in a detail window, y as an encoder feature vector can play a vital role for designating correct point affiliation.
Revised: But here in a detail window, y as an encoder feature vector can play a vital role for designating correct point affiliation.

Original: Next in our concrete implementation, we discuss how to appropriately use y in the similarity function.
Revised: Next in our concrete implementation, we discuss how to appropriately use y in the similarity function.

Original: 4.2 SAPA: Similarity-Aware Point Affiliation
Revised: 4.2 SAPA: Similarity-Aware Point Affiliation

Original: Here we embody Eq.
Revised: Here we embody Eq.

Original: (3) by investigating different similarity and normalization functions.
Revised: (3) by investigating different similarity and normalization functions.

Original: Normalization Function Though we do not constrain h ( x ) in theory, in reality it must be carefully chosen.
Revised: Normalization Function Though we do not constrain h ( x ) in theory, in reality it must be carefully chosen.

Original: For example, to avoid zero division, the scope for the choice of h ( x ) is narrowed.
Revised: For example, to avoid zero division, the scope for the choice of h ( x ) is narrowed.

Original: Following existing practices [1, 3, 5], we choose h ( x ) =
Revised: Following existing practices [1, 3, 5], we choose h ( x ) =

Original: e x by default, which is equivalent to softmax normalization.
Revised: e x by default, which is equivalent to softmax normalization.

Original: We also test some other h ( x ) ’s, such as h ( x ) =
Revised: We also test some other h ( x ) ’s, such as h ( x ) =

Original: ReLU ( x ) , h ( x ) = sigmoid ( x ) , and h ( x ) = softplus ( x ) .
Revised: ReLU ( x ) , h ( x ) = sigmoid ( x ) , and h ( x ) = softplus ( x ) .

Original: Their performance comparisons will be given in ablation studies.
Revised: Their performance comparisons will be given in ablation studies.

Original: Similarity Function We study three types of similarity functions:
Revised: Similarity Function We study three types of similarity functions:

Original: • Inner-product similarity: sim( x , y ) =
Revised: • Inner-product similarity: sim( x , y ) =

Original: x T y , • (Low-rank) bilinear similarity [24]: sim( x , y )
Revised: x T y , • (Low-rank) bilinear similarity [24]: sim( x , y )

Original: =
Revised: =

Original: x T P T x P y y , • Gated (low-rank) bilinear similarity [25]: sim( x , y ) =
Revised: x T P T x P y y , • Gated (low-rank) bilinear similarity [25]: sim( x , y ) =

Original: g x T P T x P xy y +(1 − g ) x
Revised: g x T P T x P xy y +(1 − g ) x

Original: T P T x P xx x ,
Revised: T P T x P xx x ,

Original: where P x ∈ R d × C , P y ∈ R d × C , P xy ∈ R d × C , and P xx ∈ R d × C are the linear projection matrices, d is the embedding dimension, and g ∈ (0 , 1) is a gate unit learned by linear projection.
Revised: where P x ∈ R d × C , P y ∈ R d × C , P xy ∈ R d × C , and P xx ∈ R d × C are the linear projection matrices, d is the embedding dimension, and g ∈ (0 , 1) is a gate unit learned by linear projection.

Original: We model the projection matrices to be low-rank, i.e. , d < C , to reduce the number of parameters and computational complexity.
Revised: We model the projection matrices to be low-rank, i.e. , d < C , to reduce the number of parameters and computational complexity.

Original: The gate-modulated bilinear similarity is designed to further filter out the encoder noise.
Revised: The gate-modulated bilinear similarity is designed to further filter out the encoder noise.

Original: The gate is generated by learning a linear projection matrix that projects the decoder feature X to a single-channel mask, followed by sigmoid normalization.
Revised: The gate is generated by learning a linear projection matrix that projects the decoder feature X to a single-channel mask, followed by sigmoid normalization.

Original: Finally, we have Y =
Revised: Finally, we have Y =

Original: G Y
Revised: G Y

Original: + (1 − G )
Revised: + (1 − G )

Original: X (nearest neighbor interpolation is used for matching the resolution), where G is the matrix form of the gate unit.
Revised: X (nearest neighbor interpolation is used for matching the resolution), where G is the matrix form of the gate unit.

Original: The use of the gate implies we reduce noise in some spatial regions by replacing encoder features Y with decoder features X .
Revised: The use of the gate implies we reduce noise in some spatial regions by replacing encoder features Y with decoder features X .

Original: Its vector form explains that in the area of noise it tends to switch to the self-similarity mode.
Revised: Its vector form explains that in the area of noise it tends to switch to the self-similarity mode.

Original: We will prove the effectiveness of the gating mechanism by comparing the gate-modulated bilinear similarity with a baseline without the gating mechanism.
Revised: We will prove the effectiveness of the gating mechanism by comparing the gate-modulated bilinear similarity with a baseline without the gating mechanism.

Original: In practice, encoder and decoder features have different data distributions, which is unsuitable to compare similarity directly.
Revised: In practice, encoder and decoder features have different data distributions, which is unsuitable to compare similarity directly.

Original: Therefore LayerNorm is applied to encoder and decoder features before similarity computation.
Revised: Therefore LayerNorm is applied to encoder and decoder features before similarity computation.

Original: Indeed we observe that the loss does not decrease without normalization.
Revised: Indeed we observe that the loss does not decrease without normalization.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, our implementation is similar to the previous dynamic upsampling operators, which first generates the upsampling kernels and then assembles the decoder feature conditioned on the kernels.
Revised: 3, our implementation is similar to the previous dynamic upsampling operators, which first generates the upsampling kernels and then assembles the decoder feature conditioned on the kernels.

Original: We highlight the kernel generation module.
Revised: We highlight the kernel generation module.

Original: By setting a kernel size of K , for each encoder feature point, we compute the similarity scores between this point and each of K × K neighbors in the decoder.
Revised: By setting a kernel size of K , for each encoder feature point, we compute the similarity scores between this point and each of K × K neighbors in the decoder.

Original: Then, the softmax normalization is applied to generate the upsampling kernels.
Revised: Then, the softmax normalization is applied to generate the upsampling kernels.

Original: SAPA is lightweight and can even work without additional parameters (with inner-product similarity).
Revised: SAPA is lightweight and can even work without additional parameters (with inner-product similarity).

Original: [Table]
Revised: [Table]

Original: To intuitively understand its lightweight property
Revised: To intuitively understand its lightweight property

Original: [Table]
Revised: [Table]

Original: we compare the computational complexity and number of parameters of different dynamic upsampling operators in Table 1.
Revised: we compare the computational complexity and number of parameters of different dynamic upsampling operators in Table 1.

Original: For example
Revised: For example

Original: [Table]
Revised: [Table]

Original: when upsampling a 256 -channel feature map, the FLOPs are H ∗ W ∗ 99 K , H ∗ W ∗ 2 M , H ∗ W ∗ 28 K , and H ∗ W ∗ 70 K for CARAFE, IndexNet-HIN, A2U, and SAPA-B, respectively.
Revised: when upsampling a 256 -channel feature map, the FLOPs are H ∗ W ∗ 99 K , H ∗ W ∗ 2 M , H ∗ W ∗ 28 K , and H ∗ W ∗ 70 K for CARAFE, IndexNet-HIN, A2U, and SAPA-B, respectively.

Original: We visualize the feature maps of upsampling kernels and upsampled features in Fig.
Revised: We visualize the feature maps of upsampling kernels and upsampled features in Fig.

Original: Our upsampling kernels show more favorable responses than other upsampling operators, with weights highlighted on boundaries and noise suppressed in regions, which visually supports our proposition and is a concrete embodiment of Eq.
Revised: Our upsampling kernels show more favorable responses than other upsampling operators, with weights highlighted on boundaries and noise suppressed in regions, which visually supports our proposition and is a concrete embodiment of Eq.

Original:  Experiments
Revised:  Experiments

Original: We first focus our experiments on semantic segmentation to justify the effectiveness of SAPA.
Revised: We first focus our experiments on semantic segmentation to justify the effectiveness of SAPA.

Original: We then showcase its universality across three additional dense prediction tasks, including object detection, depth estimation, and image matting.
Revised: We then showcase its universality across three additional dense prediction tasks, including object detection, depth estimation, and image matting.

Original: All our experiments are run on a server with 8 NVIDIA GeForce
Revised: All our experiments are run on a server with 8 NVIDIA GeForce

Original: RTX 3090 GPUs.
Revised: RTX 3090 GPUs.

Original: 5.1 Data Sets, Metrics, Baselines, and Protocols
Revised: 5.1 Data Sets, Metrics, Baselines, and Protocols

Original: For semantic segmentation, we conduct experiments on the ADE20K dataset [6] and report the mIoU metric.
Revised: For semantic segmentation, we conduct experiments on the ADE20K dataset [6] and report the mIoU metric.

Original: Three strong transformer-based models are adopted as the baselines, including SegFormerB1 [7], MaskFormer-Swin-Base [8] and Mask2Former-Swin-Base [9], where the Swin-Base backbone is pretrained on ImageNet-22K.
Revised: Three strong transformer-based models are adopted as the baselines, including SegFormerB1 [7], MaskFormer-Swin-Base [8] and Mask2Former-Swin-Base [9], where the Swin-Base backbone is pretrained on ImageNet-22K.

Original: All training settings and implementation details are kept the same as the original papers.
Revised: All training settings and implementation details are kept the same as the original papers.

Original: We only modify the upsampling stages with specific upsampling operators.
Revised: We only modify the upsampling stages with specific upsampling operators.

Original: For object detection, we use the MS COCO [10] dataset, which involves 80 object categories.
Revised: For object detection, we use the MS COCO [10] dataset, which involves 80 object categories.

Original: We use
Revised: We use

Original: AP as the evaluation metric.
Revised: AP as the evaluation metric.

Original: Faster R-CNN [26] with ResNet-50 [27] is adopted as the baseline.
Revised: Faster R-CNN [26] with ResNet-50 [27] is adopted as the baseline.

Original: We use mmdetection [28] and follow the 1 × ( 12 epochs) training configurations.
Revised: We use mmdetection [28] and follow the 1 × ( 12 epochs) training configurations.

Original: For depth estimation, we use NYU Depth V2 dataset [12] and its default train/test split.
Revised: For depth estimation, we use NYU Depth V2 dataset [12] and its default train/test split.

Original: We choose
Revised: We choose

Original: BTS [11] with ResNet-50 as the baseline and follow its training configurations.
Revised: BTS [11] with ResNet-50 as the baseline and follow its training configurations.

Original: The standard depth metrics used by previous work is employed to evaluate the performance, including root mean squared error (RMS) and its log version (RMS (log)), absolute relative error (Abs Rel), squared relative error (Sq Rel), average log 10 error (log10), and the accuracy with threshold thr ( δ < thr ).
Revised: The standard depth metrics used by previous work is employed to evaluate the performance, including root mean squared error (RMS) and its log version (RMS (log)), absolute relative error (Abs Rel), squared relative error (Sq Rel), average log 10 error (log10), and the accuracy with threshold thr ( δ < thr ).

Original: Readers can
Revised: Readers can

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: SAPA-G +0.
Revised: SAPA-G +

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: refer to [11] for the definitions of the metrics.
Revised: refer to [11] for the definitions of the metrics.

Original: We replace all upsampling stages but the last one for
Revised: We replace all upsampling stages but the last one for

Original: SAPA, due to no available high-res feature map for the last stage.
Revised: SAPA, due to no available high-res feature map for the last stage.

Original: For image matting, we train the model on the Adobe Image Matting dataset [13] and report four metrics on the Composition-1k test set, including Sum of Absolute Difference (SAD), Mean Squared
Revised: For image matting, we train the model on the Adobe Image Matting dataset [13] and report four metrics on the Composition-1k test set, including Sum of Absolute Difference (SAD), Mean Squared

Original: Error (MSE), Gradient (Grad), and Connectivity (Conn) [29].
Revised: Error (MSE), Gradient (Grad), and Connectivity (Conn) [29].

Original: A2U matting [5] is adopted as the baseline.
Revised: A2U matting [5] is adopted as the baseline.

Original: We use the code provided by the authors and follow the same training settings as in the original paper.
Revised: We use the code provided by the authors and follow the same training settings as in the original paper.

Original: 5.2 Main Results
Revised: 5.2 Main Results

Original: We compare SAPA and its variants against different upsampling operators on the three strong segmentation baselines.
Revised: We compare SAPA and its variants against different upsampling operators on the three strong segmentation baselines.

Original: Results are shown in Table 2, from which we can see that SAPA consistently outperforms other upsampling operators.
Revised: Results are shown in Table 2, from which we can see that SAPA consistently outperforms other upsampling operators.

Original: Note that SAPA can work well even without parameters and achieves the best performance with only few additional #Params and #FLOPs.
Revised: Note that SAPA can work well even without parameters and achieves the best performance with only few additional #Params and #FLOPs.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Results on other three dense prediction tasks are shown in Tables 3, 4, and 5.
Revised: Results on other three dense prediction tasks are shown in Tables 3, [Table] and 5.

Original: SAPA outperforms other upsampling operators on depth estimation and image matting, but falls behind CARAFE on object detection.
Revised: SAPA outperforms other upsampling operators on depth estimation and image matting, but falls behind CARAFE on object detection.

Original: One plausible explanation is that the demand of details in object detection is low (Section 6 presents a further in-depth analysis).
Revised: One plausible explanation is that the demand of details in object detection is low (Section 6 presents a further in-depth analysis).

Original: In detail-sensitive tasks like image matting, SAPA significantly outperforms CARAFE.
Revised: In detail-sensitive tasks like image matting, SAPA significantly outperforms CARAFE.

Original: Qualitative comparisons are shown in Fig 4.
Revised: Qualitative comparisons are shown in Fig 4.

Original: 5.3 Ablation Study
Revised: 5.3 Ablation Study

Original: Here we conduct ablation studies to compare the choices of similarity function and normalization function, the effect of different kernel sizes, and the number of embedding dimension.
Revised: Here we conduct ablation studies to compare the choices of similarity function and normalization function, the effect of different kernel sizes, and the number of embedding dimension.

Original: For the default setting, we use the bilinear similarity function, apply the normalization function h ( x ) =
Revised: For the default setting, we use the bilinear similarity function, apply the normalization function h ( x ) =

Original: e x , set the kernel size K = 5 and the embedding dimension d = 32 .
Revised: e x , set the kernel size K = 5 and the embedding dimension d = 32 .

Original: Quantitative results are shown in Table 6.
Revised: Quantitative results are shown in Table 6.

Original: Similarity Function We investigate three types of similarity function aforementioned and an additional ‘plain addition’ baseline.
Revised: Similarity Function We investigate three types of similarity function aforementioned and an additional ‘plain addition’ baseline.

Original: It ablates the gating mechanism from the gated bilinear similarity.
Revised: It ablates the gating mechanism from the gated bilinear similarity.

Original: Among them, gated bilinear similarity generates the best performance, which highlights the complementarity of semantic regions and boundary details in kernel generation.
Revised: Among them, gated bilinear similarity generates the best performance, which highlights the complementarity of semantic regions and boundary details in kernel generation.

Original: Normalization Function We also investigate different normalization functions.
Revised: Normalization Function We also investigate different normalization functions.

Original: ‘None’ indicates no normalization is used.
Revised: ‘None’ indicates no normalization is used.

Original: Among validated functions, h ( x )
Revised: Among validated functions, h ( x )

Original: =
Revised: =

Original: e x works the best, which means normalization matters.
Revised: e x works the best, which means normalization matters.

Original: The other three have to play with the epsilon trick to prevent zero division.
Revised: The other three have to play with the epsilon trick to prevent zero division.

Original: Kernel Size
Revised: Kernel Size

Original: The kernel size controls the region that each point in the upsampled feature can attend to.
Revised: The kernel size controls the region that each point in the upsampled feature can attend to.

Original: Results show that, compared with a large kernel, a small local window is sufficient to distinguish the semantic clusters.
Revised: Results show that, compared with a large kernel, a small local window is sufficient to distinguish the semantic clusters.

Original: Embedding Dimension We further study the influence of embedding dimension in the range of 16 , 32 , 64 , and 128 .
Revised: Embedding Dimension We further study the influence of embedding dimension in the range of 16 , 32 , 64 , and 128 .

Original: Interestingly, results suggest that SAPA is not sensitive to the embedding dimension.
Revised: Interestingly, results suggest that SAPA is not sensitive to the embedding dimension.

Original: This also verifies that SAPA extracts existing knowledge rather than learn unknown knowledge.
Revised: This also verifies that SAPA extracts existing knowledge rather than learn unknown knowledge.

Original:  Discussion
Revised:  Discussion

Original: Understanding SAPA from a Backward Perspective We have explained SAPA in the forward pass, here we further discuss how it may work during training.
Revised: Understanding SAPA from a Backward Perspective We have explained SAPA in the forward pass, here we further discuss how it may work during training.

Original: In fact, originally the model does
Revised: In fact, originally the model does

Original: [Figure]
Revised: [Figure]

Original: not know to which the semantic cluster an encoder point should belong.
Revised: not know to which the semantic cluster an encoder point should belong.

Original: The working mechanism of SAPA assigns each encoder feature point a possibility to choose a cluster.
Revised: The working mechanism of SAPA assigns each encoder feature point a possibility to choose a cluster.

Original: During training, the ground truths produce gradients, thus changes the assignment possibility.
Revised: During training, the ground truths produce gradients, thus changes the assignment possibility.

Original: In SAPA, for every encoder point, the semantic clusters in the corresponding local window in decoder features serve as implicit labels and cooperate with the ground truths.
Revised: In SAPA, for every encoder point, the semantic clusters in the corresponding local window in decoder features serve as implicit labels and cooperate with the ground truths.

Original: The correct cluster is the positive label, while the wrong one is negative.
Revised: The correct cluster is the positive label, while the wrong one is negative.

Original: In the preliminary stage of training, if an encoder feature point is more similar to a wrong cluster, it will be punished by gradients and engender large losses, and vice versa.
Revised: In the preliminary stage of training, if an encoder feature point is more similar to a wrong cluster, it will be punished by gradients and engender large losses, and vice versa.

Original: Therefore, the encoder feature points can gradually learn its affiliation.
Revised: Therefore, the encoder feature points can gradually learn its affiliation.

Original: We find this process is fast by visualizing the epoch-to-epoch feature maps.
Revised: We find this process is fast by visualizing the epoch-to-epoch feature maps.

Original: Limitations compared with CARAFE CARAFE is a purely semantic-driven upsampling operator able to mend semantic information with a single-input flow.
Revised: Limitations compared with CARAFE CARAFE is a purely semantic-driven upsampling operator able to mend semantic information with a single-input flow.

Original: Such a mechanism in CARAFE brings advantages in smoothing semantic regions.
Revised: Such a mechanism in CARAFE brings advantages in smoothing semantic regions.

Original: E.g. , we observe it mends holes in a continuous region.
Revised: E.g. , we observe it mends holes in a continuous region.

Original: However, due to its single-input flow, it cannot compensate the details lost in downsampling.
Revised: However, due to its single-input flow, it cannot compensate the details lost in downsampling.

Original: Our
Revised: Our

Original: SAPA, by contrast, mainly aims to compensate details such as textures and boundaries.
Revised: SAPA, by contrast, mainly aims to compensate details such as textures and boundaries.

Original: SAPA characters in two aspects: semantic preservation and detail delineation.
Revised: SAPA characters in two aspects: semantic preservation and detail delineation.

Original: However, as Eq.
Revised: However, as Eq.

Original: (4) suggests, we do not add any semantic mending mechanism in SAPA.
Revised: (4) suggests, we do not add any semantic mending mechanism in SAPA.

Original: This explains why SAPA is worse than CARAFE on object detection, because detection has less demand for details but more for regional integrity.
Revised: This explains why SAPA is worse than CARAFE on object detection, because detection has less demand for details but more for regional integrity.

Original: In short, CARAFE highlights semantic mending , while SAPA highlights semantic preserving and detail delineation .
Revised: In short, CARAFE highlights semantic mending , while SAPA highlights semantic preserving and detail delineation .

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we introduce similarity-aware point affiliation, i.e. , SAPA.
Revised: In this paper, we introduce similarity-aware point affiliation, i.e. , SAPA.

Original: It not only indicates a lightweight but effective upsampling operator suitable for tasks like semantic segmentation, but also expresses a high-level concept that characterizes feature upsampling.
Revised: It not only indicates a lightweight but effective upsampling operator suitable for tasks like semantic segmentation, but also expresses a high-level concept that characterizes feature upsampling.

Original: SAPA can serve as an universal substitution for conventional upsampling operators.
Revised: SAPA can serve as an universal substitution for conventional upsampling operators.

Original: Experiments show the effectiveness of SAPA and also indicate its limitation: it is more suitable for tasks that favor detail delineation.
Revised: Experiments show the effectiveness of SAPA and also indicate its limitation: it is more suitable for tasks that favor detail delineation.

Original: Acknowledgement This work is supported by the Natural Science Foundation of China under
Revised: Acknowledgement This work is supported by the Natural Science Foundation of China under

Original: Grant No. 62106080.
Revised: Grant No. 62106080.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] Please see Section 6.
Revised: [Yes] Please see Section 6.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] Our method generates stable results.
Revised: [No] Our method generates stable results.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] For each of the datasets used, we cite the creators, as shown in Section 5.1.
Revised: [Yes] For each of the datasets used, we cite the creators, as shown in Section 5.1.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] The data used in our work is open source and can be used for academic research.
Revised: [Yes] The data used in our work is open source and can be used for academic research.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] Our code for this work will appear in the supplementary material.
Revised: [Yes] Our code for this work will appear in the supplementary material.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A]
Revised: [N/A]

Original: The data used in our work is open source and can be used for academic research.
Revised: The data used in our work is open source and can be used for academic research.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: The data do not contain personally identifiable information or offensive content.
Revised: The data do not contain personally identifiable information or offensive content.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]