Original: SAPA: Similarity-Aware Point Affiliation for Feature Upsampling
Revised: SAPA: Similarity-Aware Point Afﬁliation for Feature Upsampling

Original: Anonymous Author s

Original: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: We introduce point affiliation into feature upsampling, a notion that describes theaffiliation of each upsampled point to a semantic cluster formed by local decoderfeature points with semantic similarity.
Revised: We introduce point afﬁliation into feature upsampling, a notion that describes theafﬁliation of each upsampled point to a semantic cluster formed by local decoderfeature points with semantic similarity.

Original: By rethinking point affiliation, we present ageneric formulation for generating upsampling kernels.
Revised: By rethinking point afﬁliation, we present ageneric formulation for generating upsampling kernels.

Original: The kernels encourage notonly semantic smoothness but also boundary sharpness in the upsampled featuremaps.
Revised: The kernels encourage notonly semantic smoothness but also boundary sharpness in the upsampled featuremaps.

Original: Such properties are particularly useful for some dense prediction tasks suchas semantic segmentation.
Revised: Such properties are particularly useful for some dense prediction tasks suchas semantic segmentation.

Original: The key idea of our formulation is to generate similarityaware kernels by comparing the similarity between each encoder feature point andthe spatially associated local region of decoder features.
Revised: The key idea of our formulation is to generate similarityaware kernels by comparing the similarity between each encoder feature point andthe spatially associated local region of decoder features.

Original: In this way, the encoderfeature point can function as a cue to inform the semantic cluster of upsampledfeature points.
Revised: In this way, the encoderfeature point can function as a cue to inform the semantic cluster of upsampledfeature points.

Original: To embody the formulation, we further instantiate a lightweightupsampling operator, termed Similarity-Aware Point Affiliation (SAPA), and investigate its variants.
Revised: To embody the formulation, we further instantiate a lightweightupsampling operator, termed Similarity-Aware Point Afﬁliation (SAPA), and investigate its variants.

Original: SAPA invites consistent performance improvements on a numberof dense prediction tasks, including semantic segmentation, object detection, depthestimation, and image matting.
Revised: SAPA invites consistent performance improvements on a numberof dense prediction tasks, including semantic segmentation, object detection, depthestimation, and image matting.

Original: Code will be available online.
Revised: Code will be available online.

Original:  Introduction
Revised:  Introduction

Original: We introduce the notion of point affiliation into feature upsampling.
Revised: We introduce the notion of point afﬁliation into feature upsampling.

Original: Point affiliation defines a relationbetween each upsampled point and a semantic cluster 1 to which the point should belong.
Revised: Point afﬁliation deﬁnes a relationbetween each upsampled point and a semantic cluster 1 to which the point should belong.

Original: It highlightsthe spatial arrangement of upsampled points at the semantic level.
Revised: It highlightsthe spatial arrangement of upsampled points at the semantic level.

Original: Considering an example shownin Fig.
Revised: Considering an example shownin Fig.

Original: 1 w.r.t.
Revised: 1 w.r.t.

Original: × 2 upsampling in semantic segmentation, the orange point of low resolution willcorrespond to 4 upsampled points of high resolution, in which the red and yellow ones should beassigned the ‘picture’ cluster and the ‘wall’ cluster, respectively.
Revised: × 2 upsampling in semantic segmentation, the orange point of low resolution willcorrespond to 4 upsampled points of high resolution, in which the red and yellow ones should beassigned the ‘picture’ cluster and the ‘wall’ cluster, respectively.

Original: Designating point affiliation isdifficult and sometimes can be erroneous, however.
Revised: Designating point afﬁliation isdifﬁcult and sometimes can be erroneous, however.

Original: In × 2 upsampling, nearest neighbor (NN) interpolation directly copies 4 identical points from thelow-res one, which assigns the same semantic cluster to the 4 points.
Revised: In × 2 upsampling, nearest neighbor (NN) interpolation directly copies 4 identical points from thelow-res one, which assigns the same semantic cluster to the 4 points.

Original: On regions in need of details,the 4 points probably do not share the same cluster but are forced to share.
Revised: On regions in need of details,the 4 points probably do not share the same cluster but are forced to share.

Original: Bilinear interpolationassigns point affiliation with distance priors.
Revised: Bilinear interpolationassigns point afﬁliation with distance priors.

Original: Yet, when tackling points of different semantic clusters,it not only cannot inform clear point affiliation, but also blurs the boundary between differentsemantic clusters.
Revised: Yet, when tackling points of different semantic clusters,it not only cannot inform clear point afﬁliation, but also blurs the boundary between differentsemantic clusters.

Original: Recent dynamic upsampling operators have similar issues.
Revised: Recent dynamic upsampling operators have similar issues.

Original: CARAFE [1] judges theaffiliation of an upsampled point with content-aware kernels.
Revised: CARAFE [1] judges theafﬁliation of an upsampled point with content-aware kernels.

Original: Certain semantic clusters will receivelarger weights than the rest and therefore dominate the affiliation of upsampled points.
Revised: Certain semantic clusters will receivelarger weights than the rest and therefore dominate the afﬁliation of upsampled points.

Original: However, theaffiliation near boundaries or on regions full of details can still be ambiguous.
Revised: However, theafﬁliation near boundaries or on regions full of details can still be ambiguous.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, theboundaries are unclear in the feature maps upsampled by CARAFE.
Revised: 2, theboundaries are unclear in the feature maps upsampled by CARAFE.

Original: The reason is that the kernelsare conditioned on decoder features alone; the decoder features carry little useful information abouthigh-res
Revised: The reason is that the kernelsare conditioned on decoder features alone; the decoder features carry little useful information about Upsample Kernel

Revised: Kernel (𝑖 2𝑝 ,𝑗 2𝑝 ) (𝑖 1 ,𝑗 1 ) (⌊𝑖 2 2⌋,⌊j 2 2⌋) (⌊𝑖 1 2⌋,⌊j 1 2⌋) small large

Revised: Encoder feature

Revised: Decoder feature

Original:  [Figure]
Revised: Upsampled feature (𝑖 2𝑤 ,𝑗 2𝑤 ) Upsample Kernel (𝑖 2𝑤 ,𝑗 2𝑤 ) (𝑖 1 ,𝑗 1 ) (𝑖 2𝑝 ,𝑗 2𝑝 ) 𝐶𝐶𝐶𝐶 [Figure]

Original: Inferring structure requires high-res encoder features.
Revised: Inferring structure requires high-res encoder features.

Original: For instance, if the orange point in Fig.
Revised: For instance, if the orange point in Fig.

Original: 1 lieson the low-res boundary, it is difficult to judge to which cluster the 4 upsampled points should belong.
Revised: 1 lieson the low-res boundary, it is difﬁcult to judge to which cluster the 4 upsampled points should belong.

Original: However, the encoder feature in Fig.
Revised: However, the encoder feature in Fig.

Original: 1 actually tells that, the yellow point is on the wall, and the redone is on the picture, which suggests one may extract useful information from the encoder feature toassist the designation of point affiliation.
Revised: 1 actually tells that, the yellow point is on the wall, and the redone is on the picture, which suggests one may extract useful information from the encoder feature toassist the designation of point afﬁliation.

Original: Indeed IndexNet [2] and A2U [3] have attempted to encodesuch information to improve detail delineation in encoder-dependent upsampling kernels; however,the encoder feature can easily introduce noise into kernels, engendering discontinuous feature mapsshown in Fig.
Revised: Indeed IndexNet [2] and A2U [3] have attempted to encodesuch information to improve detail delineation in encoder-dependent upsampling kernels; however,the encoder feature can easily introduce noise into kernels, engendering discontinuous feature mapsshown in Fig.

Original: Hence, the key problem seems to be how to extract only required information intothe upsampling kernels from the encoder feature while filtering out the rest.
Revised: Hence, the key problem seems to be how to extract only required information intothe upsampling kernels from the encoder feature while ﬁltering out the rest.

Original: To use encoder features effectively, an important assumption of this paper is that, an encoder featurepoint is most similar to the semantic cluster into which the point will be classified.
Revised: To use encoder features effectively, an important assumption of this paper is that, an encoder featurepoint is most similar to the semantic cluster into which the point will be classiﬁed.

Original: Per the left of
Revised: Per the left of

Original: Fig. 1, suppose that the encoder point in the red box is assigned into the cluster C 1 by its semanticmeaning, then it is similar to C 1 , while not similar to C 2 .
Revised: Fig. 1, suppose that the encoder point in the red box is assigned into the cluster C 1 by its semanticmeaning, then it is similar to C 1 , while not similar to C 2 .

Original: As a result, by comparing the similaritybetween the encoder feature point and different semantic clusters in the decoder feature, the affiliationof the upsampled point can be informed according to the similarity scores.
Revised: As a result, by comparing the similaritybetween the encoder feature point and different semantic clusters in the decoder feature, the afﬁliationof the upsampled point can be informed according to the similarity scores.

Original: In particular, we proposeto generate upsampling kernels with local mutual similarity between encoder and decoder features.
Revised: In particular, we proposeto generate upsampling kernels with local mutual similarity between encoder and decoder features.

Original: For every encoder feature point, we compute the similarity score between this point and each decoderfeature point in the spatially associated local window.
Revised: For every encoder feature point, we compute the similarity score between this point and each decoderfeature point in the spatially associated local window.

Original: For the green point in Fig.
Revised: For the green point in Fig.

Original: 1, since every pointin the window shares the same semantic cluster, the encoder feature point is as similar as every pointin the window.
Revised: 1, since every pointin the window shares the same semantic cluster, the encoder feature point is as similar as every pointin the window.

Original: In this case we expect an ‘average kernel’ which is the key characteristic to filternoise, and the upsampled 4 points would have the same semantic cluster as before.
Revised: In this case we expect an ‘average kernel’ which is the key characteristic to ﬁlternoise, and the upsampled 4 points would have the same semantic cluster as before.

Original: For the yellowpoint in the encoder, since it belongs to the ‘wall’ cluster, it is more similar to the points on the wallthan those on the picture.
Revised: For the yellowpoint in the encoder, since it belongs to the ‘wall’ cluster, it is more similar to the points on the wallthan those on the picture.

Original: In this case we expect a kernel with larger weights on points related to the‘wall’ cluster.
Revised: In this case we expect a kernel with larger weights on points related to the‘wall’ cluster.

Original: This can help to assign the affiliation of the yellow point to be in the ’wall’ cluster.
Revised: This can help to assign the afﬁliation of the yellow point to be in the ’wall’ cluster.

Original: By modeling the local mutual similarity, we derive a generic form of upsampling kernels and showthat this form implements our expected upsampling behaviors: encouraging both semantic smoothnessand boundary sharpness.
Revised: By modeling the local mutual similarity, we derive a generic form of upsampling kernels and showthat this form implements our expected upsampling behaviors: encouraging both semantic smoothnessand boundary sharpness.

Original: Following our formulation, we further instantiate a lightweight upsamplingoperator, termed Similarity-Aware Point Affiliation (SAPA), and investigate its variants.
Revised: Following our formulation, we further instantiate a lightweight upsamplingoperator, termed Similarity-Aware Point Afﬁliation (SAPA), and investigate its variants.

Original: We evaluate
Revised: We evaluate

Original: SAPA across a number of mainstream dense prediction tasks, for example: i) semantic segmentation :we test SAPA on several transformer-based segmentation baselines on the ADE20K dataset [4], suchas SegFormer [5], MaskFormer [6], and Mask2Former [7], improving the baselines by 1% ∼ 2 .
Revised: SAPA across a number of mainstream dense prediction tasks, for example: i) semantic segmentation :we test SAPA on several transformer-based segmentation baselines on the ADE20K dataset [4], suchas SegFormer [5], MaskFormer [6], and Mask2Former [7], improving the baselines by 1% ∼ 2 .

Original: 7%mIoU; ii) object detection : SAPA improves the performance of Faster RCNN by 0 .
Revised: 7%mIoU; ii) object detection : SAPA improves the performance of Faster RCNN by 0 .

Original: 4% AP on MS
Revised: 4% AP on MS

Original: COCO [8]; iii) monocular depth estimation : SAPA reduces the rmse metric of BTS [9] from 0 .
Revised: COCO [8]; iii) monocular depth estimation : SAPA reduces the rmse metric of BTS [9] from 0 .

Original: 419to 0 .
Revised: 419to 0 .

Original: 408 on NYU Depth V2 [10]; and iv) image matting
Revised: 408 on NYU Depth V2 [10]; and iv) image matting

Original: : SAPA outperforms a strong A2U mattingbaseline [3] on the Adobe Composition-1k testing set [11] with a further 3 .
Revised: : SAPA outperforms a strong A2U mattingbaseline [3] on the Adobe Composition-1k testing set [11] with a further 3 .

Original: 8% relative error reductionin the SAD metric.
Revised: 8% relative error reductionin the SAD metric.

Original: SAPA also outperforms or at least is on par with other state-of-the-art dynamicupsampling operators.
Revised: SAPA also outperforms or at least is on par with other state-of-the-art dynamicupsampling operators.

Original: Particularly, even without additional parameters, SAPA outperforms theprevious best upsampling operator CARAFE on semantic segmentation.
Revised: Particularly, even without additional parameters, SAPA outperforms theprevious best upsampling operator CARAFE on semantic segmentation.

Original: [Figure]
Revised: [Figure]

Original: U p s a m p l e d f ea t u r e m a p s
Revised: U p s a m p l e d f ea t u r e m a p s

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: K e r n e l
Revised: K e r n e l

Original: The visualization is produced with SegNet [12] as

Original: For eachupsampling operator,we choose thefirst three channelsfrom the feature maps of the lastupsampling stage for visualization.
Revised: For each operator, choose the three channels the feature maps of the last

Original: Only SAPAshows both smooth regions andsharp boundaries.
Revised: Only SAPA both smooth regions and

Original: The kernel map of CARAFE is coarse and lacking in details, while IndexNet and A2U generate kernels with  much details from the encoder.
Revised: The kernel map of CARAFE is coarse and lacking in details, while IndexNet and A2U generate kernels with too much detail from encoder to preserve semantics.

Original: See the supplementary materials for additional visualizations.
Revised: Please refer to supplementary materials for additional visualizations.

Original:  Related Work
Revised:  Related Work

Original: We review work related to feature upsampling.
Revised: We review work related to feature upsampling.

Original: Feature upsampling is a fundamental procedure inencoder-decoder architectures used to recover the spatial resolution of low-res decoder feature mapsand has been extensively used in dense prediction tasks such as semantic segmentation [12, 5, 6, 7]and depth estimation [14, 15, 9].
Revised: Feature upsampling is a fundamental procedure inencoder-decoder architectures used to recover the spatial resolution of low-res decoder feature mapsand has been extensively used in dense prediction tasks such as semantic segmentation [12, 5, 6, 7]and depth estimation [14, 15, 9].

Original: Standard upsampling operators are hand-crafted.
Revised: Standard upsampling operators are hand-crafted.

Original: NN and bilinear interpolation measure the semanticaffiliation in terms of relative distances in upsampling, which follows fixed rules to designate pointaffiliation, even if the true affiliation may be different.
Revised: NN and bilinear interpolation measure the semanticafﬁliation in terms of relative distances in upsampling, which follows ﬁxed rules to designate pointafﬁliation, even if the true afﬁliation may be different.

Original: Max unpooling [12] stores the indices ofmax-pooled feature points in encoder features and uses the sparse indices to guide upsampling.
Revised: Max unpooling [12] stores the indices ofmax-pooled feature points in encoder features and uses the sparse indices to guide upsampling.

Original: Whileit executes location-specific point affiliation which benefits detail recovery, most upsampled pointsare assigned with null affiliation due to zero filling.
Revised: Whileit executes location-speciﬁc point afﬁliation which beneﬁts detail recovery, most upsampled pointsare assigned with null afﬁliation due to zero ﬁlling.

Original: Pixel Shuffle [16] is widely-used in image/videosuper-resolution.
Revised: Pixel Shufﬂe [16] is widely-used in image/videosuper-resolution.

Original: Its upsampling only includes memory operation – reshaping depth channels tospace ones.
Revised: Its upsampling only includes memory operation – reshaping depth channels tospace ones.

Original: The notion of point affiliation does not apply to this operator, however.
Revised: The notion of point afﬁliation does not apply to this operator, however.

Original: Another stream of upsampling operators implement learning-based upsampling.
Revised: Another stream of upsampling operators implement learning-based upsampling.

Original: Among them,transposed convolution or deconvolution [17] is known as an inverse convolutional operator.
Revised: Among them,transposed convolution or deconvolution [17] is known as an inverse convolutional operator.

Original: Basedon a novel interpretation of deconvolution, PixelTCL [18] is proposed to alleviate the checkerboardartifacts [19] of standard deconvolution.
Revised: Basedon a novel interpretation of deconvolution, PixelTCL [18] is proposed to alleviate the checkerboardartifacts [19] of standard deconvolution.

Original: In addition, bilinear additive upsampling [20] attemptsto combine learnable convolutional kernels with hand-crafted upsampling operators to achievecomposited upsampling.
Revised: In addition, bilinear additive upsampling [20] attemptsto combine learnable convolutional kernels with hand-crafted upsampling operators to achievecomposited upsampling.

Original: Recently, DUpsample [21] seeks to reconstruct the upsampled feature mapwith pre-learned projection matrices, expecting to achieve a data-dependent upsampling behavior.
Revised: Recently, DUpsample [21] seeks to reconstruct the upsampled feature mapwith pre-learned projection matrices, expecting to achieve a data-dependent upsampling behavior.

Original: While these operators are learnable, the upsampling kernels are fixed once learned, still resulting infixed designation of point affiliation.
Revised: While these operators are learnable, the upsampling kernels are ﬁxed once learned, still resulting inﬁxed designation of point afﬁliation.

Original: In learning-based upsampling, some recent work introduces the idea of generating content-awaredynamic kernels.
Revised: In learning-based upsampling, some recent work introduces the idea of generating content-awaredynamic kernels.

Original: Instead of learning the parameters of the kernels, they learn how to predict thekernels.
Revised: Instead of learning the parameters of the kernels, they learn how to predict thekernels.

Original: In particular, CARAFE [1] predicts dynamic kernels conditioned on the decoder features.
Revised: In particular, CARAFE [1] predicts dynamic kernels conditioned on the decoder features.

Original: IndexNet [2] and A2U [3], by contrast, generate encoder-dependent kernels.
Revised: IndexNet [2] and A2U [3], by contrast, generate encoder-dependent kernels.

Original: While they significantlyoutperform previous upsampling operators in various tasks, they still can cause uncertain pointaffiliation, resulting in either unclear predictions near boundaries or fragile predictions in regions.
Revised: While they signiﬁcantlyoutperform previous upsampling operators in various tasks, they still can cause uncertain pointafﬁliation, resulting in either unclear predictions near boundaries or fragile predictions in regions.

Original: Our work is closely related to dynamic kernel-based upsampling.
Revised: Our work is closely related to dynamic kernel-based upsampling.

Original: We also seek to predict dynamickernels; however, we aim to address the uncertain point affiliation in prior arts to achieve simultaneousregion smoothness and boundary sharpness.
Revised: We also seek to predict dynamickernels; however, we aim to address the uncertain point afﬁliation in prior arts to achieve simultaneousregion smoothness and boundary sharpness.

Original:  Dynamic Upsampling Revisited
Revised:  Dynamic Upsampling Revisited

Original: We first revisit two key components shared by existing dynamic upsampling operators: kernelgeneration and feature assembly.
Revised: We ﬁrst revisit two key components shared by existing dynamic upsampling operators: kernelgeneration and feature assembly.

Original: Kernel Generation Given the decoder feature X∈ R H × W × C , if we upsample it to the targetfeature X 
Revised: Kernel Generation Given the decoder feature X , if we upsample it to the target feature X (cid:48) , whichis often twice the size of X , then for any point at the location l (cid:48)

Original: ′ ∈
Revised: =

Original: R 2 H × 2 W × C , then for any point at the location l ′ = ( i ′ , j ′ ) in X ′ , we generate a kernel W l ′ based on the neighborhood feature N l ′ that spatially corresponds to l ′ .
Revised: ( i (cid:48) , j (cid:48) ) in X (cid:48) , we generate akernel W l (cid:48) based on the neighborhood feature N l (cid:48) that spatially corresponds to l (cid:48) .

Original: In this way, kernelgeneration can be defined by
Revised: In this way, kernelgeneration can be deﬁned by

Original: [Equation]
Revised: [Equation]

Original:  where ψ refers to a kernel generation module, which is often implemented by a sub-network usedto predict the kernel conditioned on N l ′ , and norm is a normalization function.
Revised: 1) where ψ refers to a kernel generation module, which is often implemented by a sub-network used topredict the kernel conditioned on N l (cid:48) , and norm is a normalization function.

Original: The feature N l ′can originate from two sources, to be specific, from the encoder feature Y∈ R 2 H × 2 W × C or fromthe decoder feature X .
Revised: The feature N l (cid:48) canoriginate from two sources, to be speciﬁc, from the encoder feature Y or from the decoder feature X .

Original: If the encoder feature Y is chosen as the source, then a local region of Ycentered at l ′ is extracted to be N l ′ .
Revised: If the encoder feature Y is chosen as the source, then a local region of Y centered at l (cid:48) is extractedto be N l (cid:48) .

Original: If the decoder feature X is chosen, one first needs to computethe projective location of l ′ , i.e. , l = ( i, j ) = ⌊ l ′ 2 ⌋ = ( ⌊ i ′ 2 ⌋ , ⌊ j ′ 2 ⌋ ) according to the spatial locationcorrespondence, then a local region of Xcentered at l is regarded as N l ′ .
Revised: If the decoder feature X is chosen, one ﬁrst needs to compute the projective locationof l (cid:48) , i.e. , l = ( (cid:98) i (cid:48) 2 (cid:99) , (cid:98) j (cid:48) 2 (cid:99) ) according to the spatial location correspondence, then a neighborhoodregion extracted from X l is regarded as N l (cid:48) .

Original: The softmax function isoften used as the normalization function such that relevant points can be softly selected to computethe value of the target point using the weight W l ′ .
Revised: The softmax function is often used as the normalizationfunction such that relevant points can be softly selected to compute the value of the target point usingthe weight W l (cid:48) .

Original: According to Fig. 2, the source of N l ′ can affect the predicted kernel.
Revised: As illustrated in Fig. 2, the source of N l (cid:48) can affect the predicted kernel.

Original: The kernels predicted by
Revised: The kernels predicted by

Original: CARAFE, IndexNet, and A2U show significantly distinct characteristics.
Revised: CARAFE, IndexNet, and A2U show signiﬁcantly distinct characteristics.

Original: With the decoder featurealone, the kernel map is coarse and lacking in details.
Revised: With the decoder featurealone, the kernel map is coarse and lacking in details.

Original: Benefiting from the encoder feature, the kernelmaps generated by IndexNet and A2U have rich details; however, they manifest high similarity to theencoder feature, which means noise is introduced into the kernel.
Revised: Beneﬁted from the encoder feature, the kernelmaps generated by IndexNet and A2U have rich details; however, they manifest large similarity tothe original encoder feature, which means noise is introduced into the kernel.

Original: Feature Assembly For each target feature point at l ′ , we assemble the corresponding sub-regionof decoder feature with the predictedK × K kernel W l ′ , whose weight is denoted by W l′ ,  ( u,v ),u, v = − r, ..., r, r =
Revised: Feature Assembly In this step, for each target feature point at l (cid:48) , we assemble the correspondingsub-region of decode feature with the predicted kernel W l (cid:48) and assign a value to this target point. Ifthe predicted kernel has a kernel size of K × K , then the sub-region is also of size K × K centeredat l , where l = ( i, j ) =

Original: ⌊ K 2 ⌋ ,  to obtain the value ofthe target point X ′ l ′ bya weighted sum
Revised: ( (cid:98) i (cid:48) 2 (cid:99) , (cid:98) j (cid:48) 2 (cid:99) ) . We compute the weighted sum between the normalized kerneland the sub-region of decoder feature to obtain the value of X (cid:48) l (cid:48) by

Original: [Equation]
Revised: [Equation]

Original: By executing feature assembly on every target feature point, we can obtain the target upsampledfeature map.
Revised: By executing feature assembly on every target feature point, we can obtain the target upsampledfeature map.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, the upsampled feature has a close relation to the kernel.
Revised: 2, the upsampled feature has a close relation to the kernel.

Original: A wellpredicted kernel can encourge both semantic smoothness and boundary sharpness; a kernel withoutencoding details or with too many details encoded can introduce noise.
Revised: A wellpredicted kernel can encourge both semantic smoothness and boundary sharpness; a kernel withoutencoding details or with too many details encoded can introduce noise.

Original: We consider an ideal kernelshould only response at the position in need of details, while do not response (appearing as an averagevalue over an area) at good semantic regions.
Revised: We consider an ideal kernelshould only response at the position in need of details, while do not response (appearing as an averagevalue over an area) at good semantic regions.

Original: More importantly, an ideal kernel should assign weightsreasonably so that each point can be designated to a correct semantic cluster.
Revised: More importantly, an ideal kernel should assign weightsreasonably so that each point can be designated to a correct semantic cluster.

Original:  Rethinking Point Affiliation with Local Mutual Similarity
Revised:  Rethinking Point Afﬁliation with Local Mutual Similarity

Original: To obtain an expected upsampling kernel mentioned above, we first derive a generic formulation forgeneratingthe upsampling kernel by exploiting local mutual similarity, then explain why the formulation encourages semantic smoothness and boundary sharpness, and finally present an upsamplingoperator, SAPA, that embodies our formulation.
Revised: To obtain an expected upsampling kernel mentioned above, we ﬁrst derive a generic formulation forgenerating upsampling kernels by exploiting local mutual similarity, then explain why the formulation encourages semantic smoothness and boundary sharpness, and ﬁnally present an upsampling operator, SAPA, that embodies our formulation.

Original: 4.1 Local Mutual Similarity
Revised: 4.1 Local Mutual Similarity

Original: We rethink point affiliation from the view of local mutual similarity between encoder and decoderfeatures.
Revised: We rethink point afﬁliation from the view of local mutual similarity between encoder and decoderfeatures.

Original: With a detailed analysis, we explain why such similarity can assist point affiliation.
Revised: With a detailed analysis, we explain why such similarity can assist point afﬁliation.

Original: We first define a generic similarity function sim( x , y ) :
Revised: We ﬁrst deﬁne a generic similarity function sim( x , y ) :

Original: R C × R C → R .
Revised: R C × R C → R .

Original: It scores the similaritybetween a vector x and a vector y of the same dimension C .
Revised: It scores the similaritybetween a vector x and a vector y of the same dimension C .

Original: We also define a normalizationfunction involving n real numbers x 1 , x 2 , ..., x n by norm( x i : x 1 , x 2 , ..., x n )
Revised: We also deﬁne a normalizationfunction involving n real numbers x 1 , x 2 , ..., x n by norm( x i : x 1 , x 2 , ..., x n )

Original: =
Revised: =

Original: h( x i ) (cid:80) nj =1 h ( x j ) , whereh( x ) :
Revised: h( x i ) (cid:80) nj =1 h ( x j ) , whereh( x ) :

Original: R → R is an arbitrary function, ignoring zero division.
Revised: R → R is an arbitrary function, ignoring zero division.

Original: Given sim( x , y ) and h( x ) , we candefine a generic formulation for generating the upsampling kernel
Revised: Given sim( x , y ) and h( x ) , we candeﬁne a generic formulation for generating the upsampling kernel

Original: [Equation]
Revised: [Equation]

Original: where w is the kernel value specific to x and y .
Revised: where w is the kernel value speciﬁc to x and y .

Original: To analyze the upsampling behavior of the kernel,we further define the following notations.
Revised: To analyze the upsampling behavior of the kernel,we further deﬁne the following notations.

Revised: Let Y ∈ R 2 H × 2 W × C and X ∈

Revised: R

Original:  Let y l ′ ∈ R C  denote the encoder feature vector at position l ′ and x l ∈ R C  be the decoder featurevector at position l , where C is the number of channels. 
Revised: H × W × C be the encoder and decoder feature maps, respectively. Let y i,j = Y ( i,j ) ∈ R C , ( i, j ) ∈ { 0 , 1 , ..., 2 H − 1 } × { 0 , 1 , ..., 2 W − 1 } denote the encoder featurevector and x k,l = X ( k,l ) ∈ R C , ( k, l ) ∈ { 0 , 1 , ..., H − 1 } × { 0 , 1 , ..., W − 1 } be the decoder featurevector, where C is the number of channels. Let K be the upsampling kernel size.

Original: Our operation will be done within a localwindow of size K × K , between each encoder vector y l ′ and all its spatially associated decoderfeature vectors, x l +( u,v ) ’s, where u, v = − r, ..., r, r = 
Revised: Our operation will be done within a local window of size K × K , between each encoder vector y i,j and all its spatially associated decoder feature vectors,x (cid:98) i 2 (cid:99) + m, (cid:98) j 2 (cid:99) + n ’s, where m, n = −(cid:98)

Original: ⌊ K 2 ⌋ .
Revised: K 2 (cid:99) , ..., (cid:98) K 2 (cid:99) .

Original: To simplify our analysis, we also assume local smoothness.
Revised: To simplify our analysis, we also assume local smoothness.

Original: That is, points with the same semanticcluster will have a similar value, which means a local region will share the same value on everychannel of the feature map.
Revised: That is, points with the same semanticcluster will have a similar value, which means a local region will share the same value on everychannel of the feature map.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, we define x a =
Revised: 1, we deﬁne x w =

Original: ( R a 1 , R a 2 , ..., R aC ) T and x p =( R p 1 , R p 2 , ..., R pC ) T as the semantic clusters related to ‘wall’ and ‘picture’, respectively, where R acand R pc are constants, and c = 1 , 2 , ..., C .
Revised: ( R w 1 , R w 2 , ..., R wC ) T and x p =( R p 1 , R p 2 , ..., R pC ) T as the semantic clusters related to ‘wall’ and ‘picture’, respectively, where R wcand R pc are constants, and c = 1 , 2 , ..., C .

Original: For ease of analysis, we define two types of windowsdistinguished by their contents.
Revised: For ease of analysis, we deﬁne two types of windowsdistinguished by their contents.

Original: When all the points inside a window belong to the same semanticcluster, it is called a smooth window; while different semantic clusters appear in a window, it isdefined as a detail window.
Revised: When all the points inside a window belong to the same semanticcluster, it is called a smooth window; while different semantic clusters appear in a window, it isdeﬁned as a detail window.

Original: Next, we explain why the kernel can filter out noise, why it encourages semantic smoothness in asmooth window, and why it can help to recover details when dealing with boundaries/textures in adetail window.
Revised: Next, we explain why the kernel can ﬁlter noise and can encourage semantic smoothness in a smoothwindow, and why it can help to recover details when dealing with boundaries/textures in a detailwindow.

Original: Upsampling in a Smooth Window Without loss of generality, we consider an encoder point atthe position l ′ in Fig. 
Revised: Upsampling in a Smooth Window Without loss of generality, we consider the point y i 1 ,j 1 in Fig. 1 in the encoder feature.

Original: Its corresponding window is a smooth window of  the semantic cluster‘picture’, thus x l +( u,v ) = 
Revised: Its corresponding window is a smooth window of with the semanticcluster ‘picture’, thus x (cid:98) i 12 (cid:99) + m, (cid:98) j 12 (cid:99) + n = x p , m, n = −(cid:98)

Original: x p ,u, v = − r, ..., r, r = ⌊ K 2 ⌋ .
Revised: K 2 (cid:99) , ..., (cid:98) K 2 (cid:99) .

Original: Then the upsampling kernel weight w.r.t.the upsampled point l ′ at the position ( u, v ) takes the form
Revised: Then the upsampling kernelweight for the upsampled point ( i 1 , j 1 ) at the position ( m, n ) takes the form

Original: [Equation]
Revised: [Equation]

Original: which has nothing to do with l , u , and v .
Revised: which has nothing to do with i 1 , j 1 , m , and n .

Original: Eq.
Revised: Eq.

Original: (4) reveals a key characteristic of local mutualsimilarity in a smooth window: the kernel weight is a constant regardless of y .
Revised: (4) reveals a key characteristic of local mutualsimilarity in a smooth window: the kernel weight is a constant regardless of y .

Original: Therefore, the kernelfundamentally can filter out noise from encoder features with an ‘average’ kernel. 
Revised: Therefore, the kernelfundamentally can ﬁlter out noise from encoder features with an ‘average’ kernel. [Figure]

Original: Note that, in the derivation above the necessary conditions include: i) x is from a local window in thedecoder feature map; ii) a normalization function in the form of
Revised: 𝐻 ×𝑊 ×𝐶 𝐻×𝑊×𝐶 softmaxreshape matrix multiplication 𝑑 𝐾 [Figure] 𝑑 𝑑 𝐶 → 𝑑 𝐶 →𝑑 2𝐻×2𝑊×𝐶 [Figure]

Revised: fusion

Original: [Equation]
Revised: [Figure]

Original: j h ( x j ) .
Revised: The scores are then transformed into upsampling kernel weights after normalization. In this way each of the 4 upsampling kernels can select some decoder points from the 3 × 3 decoder window, conditioned on mutual similarity, and assigns the value to the corresponding upsampled point. Note that, in the derivation above the necessary conditions include: i) x is from a local window in thedecoder feature map; ii) a normalization function in the form of [Figure]

Original: Upsampling in a Detail Window Again we consider twoencoder points at the position m ′ and n ′in Fig. 
Revised: Upsampling in a Detail Window Again we consider two points y i 2 w ,j 2 w and y i 2 p ,j 2 p in Fig. 1 inthe encoder feature.

Original: Ideally y m ′ and y n ′ should be classified into the semantic cluster of ‘wall’ and ’picture’,respectively.
Revised: Ideally y i 2 w ,j 2 w and y i 2 p ,j 2 p should be classiﬁed into the semantic cluster of‘wall’ and ’picture’, respectively.

Original: Taking y m ′ as an example, following our assumption, it is  more similar to points ofthe ‘wall’ cluster rather than the ‘picture’ cluster.
Revised: Taking y i 2 w ,j 2 w as an example, following our assumption, it is 182 more similar to points of the ‘wall’ cluster rather than the ‘picture’ cluster.

Original: From Eq.
Revised: From Eq.

Original: (4), we can tell that sim( x  m  + a , y m ′ ) 
Revised: (4), we cantell that sim( x (cid:98) i 22 (cid:99) + m w , (cid:98) j 22 (cid:99) + n w , y i 2 w ,j 2 w ) is larger than sim( x (cid:98)

Original: islarger than sim( x m  +  p , y m ′ ) , where a and p are the offsets in the window such that m  + a and  m  + pare within the ‘wall’  and the ‘picture’ cluster, respectively.
Revised: i 22 (cid:99) + m p , (cid:98) j 22 (cid:99) + n p , y i 2 w ,j 2 w ) , wherex (cid:98) i 22 (cid:99) + m w , (cid:98) j 22 (cid:99) + n w and x (cid:98) i 22 (cid:99) + m p , (cid:98) j 22 (cid:99) + n p belong to the ‘wall’ cluster and the ‘picture’ cluster,respectively.

Original: Therefore, after computing similarityscores and normalization, one can acquire a kernel with significantly larger weights on points with thesemantic cluster of ‘wall’ than that of ‘picture’, i.e. , W m ′ ,a >> W m ′ ,p .
Revised: Therefore, after computing similarity scores and normalization, one can acquire a kernelwith signiﬁcantly larger weights on points with the semantic cluster of ‘wall’ than that of ‘picture’,i.e. , w i 2 w ,j 2 w ,m w ,n w >> w i 2 w ,j 2 w ,m p ,n p .

Original: After applying the kernel tothe corresponding window, the  upsampled point at m ′ will be revalued and assigned to the semanticcluster of ‘wall’.Similarly, the upsampled point at n ′ will be assigned into ‘picture’. normalization 𝑑 𝐾 ! 𝑑 kernels Encoder feature
Revised: After applying the kernel to the corresponding window,the upsampled point at ( i 2 w , j 2 w ) will be revalued and assigned to the semantic cluster of ‘wall’.

Original: Decoder feature 2𝐻×2𝑊×𝐶 𝐻×𝑊×𝐶 P r e - p r o ce ss i ng ( 𝑑=𝐶 ) 𝐺 𝐵 𝐼 Pre-processing Computing similarity 2𝐻×2𝑊×𝐶 Upsampledfeature
Revised: Similarly, the upsampled point at ( i 2 p , j 2 p ) will be assigned into ‘picture’.

Original: Kernel generation matrix multiplication linear embedding gating mechanism

Original: Feature assembly ( 𝑑<𝐶 ) ( 𝑑<𝐶 )

Original: Note that, in Eq.
Revised: Note that, in Eq.

Original: (4) we have no constraint on y .
Revised: (4) we have no constraint on y .

Original: But here in a detail window, y as an encoderfeature vector can play a vital role for designating correct point affiliation.
Revised: But here in a detail window, y as an encoderfeature vector can play a vital role for designating correct point afﬁliation.

Original: Next in our concreteimplementation, we discuss how to appropriately use y in the similarity function.
Revised: Next in our concreteimplementation, we discuss how to appropriately use y in the similarity function.

Original: 4.2 SAPA: Similarity-Aware Point Affiliation
Revised: 4.2 SAPA: Similarity-Aware Point Afﬁliation

Original: Here we embody Eq.
Revised: Here we embody Eq.

Original: (3) by investigating different similarity and normalization functions.
Revised: (3) by choosing appropriate similarity and normalization functions.

Original: Normalization Function Though we do not constrain h ( x ) in theory, in reality it must be carefullychosen.
Revised: Normalization Function Though we do not constrain h ( x ) in theory, in reality it must be carefullychosen.

Original: For example, to avoid zero division, the scope for the choice of h ( x ) is narrowed.
Revised: For example, to avoid zero division, the scope for the choice of h ( x ) is narrowed.

Original: Followingexisting practices [1, 2, 3], we choose h ( x ) =
Revised: Followingexisting practices [1, 2, 3], we choose h ( x ) =

Original: e x by default, which is equivalent to softmaxnormalization.
Revised: e x by default, which is equivalent to softmaxnormalization.

Original: We also test some other h ( x ) ’s, such as h ( x ) =
Revised: We also test some other h ( x ) ’s, such as h ( x ) =

Original: ReLU ( x ) , h ( x ) = sigmoid ( x ) , andh ( x ) = softplus ( x ) .
Revised: ReLU ( x ) , h ( x ) = sigmoid ( x ) , andh ( x ) = softplus ( x ) .

Original: Their performance comparisons will be given in ablation studies.
Revised: Their performance comparisons will be given in ablation studies.

Original: Similarity Function We  study three types of similarity functions:
Revised: Similarity Function We focus on the choices of similarity functions and study three types offunctions:

Original: • Inner-product similarity: sim( x , y ) =
Revised: • Inner-product similarity: sim( x , y ) =

Original: x T y ,
Revised: x T y ,

Original: • (Low-rank) bilinear similarity [22]: sim( x , y ) =
Revised: • (Low-rank) bilinear similarity [22]: sim( x , y ) =

Original: x T P T x P y y ,
Revised: x T P T x P y y ,

Original: • Gated (low-rank) bilinear similarity: sim( x , y ) =
Revised: • Gated (low-rank) bilinear similarity: sim( x , y ) =

Original: g x T P T x P xy y + (1 − g ) x 
Revised: g x T P T x P xy y + (1 − g ) x T P T x

Original: T P T x P xx x ,
Revised: P xx x ,

Original: where P x ∈ R d × C , P y ∈ R d × C , P xy ∈ R d × C , and P xx ∈ R d × C are the linear projection matrices,d is the embedding dimension, and g ∈ (0 , 1) is a gate unit learned by linear projection.
Revised: where P x ∈ R d × C , P y ∈ R d × C , P xy ∈ R d × C , and P xx ∈ R d × C are the linear projection matrices,d is the embedding dimension, and g ∈ (0 , 1) is a gate unit learned by linear projection.

Original: We model theprojection matrices to be low-rank, i.e. , d < C , to reduce the number of parameters and computationalcomplexity.

Original: The gate-modulated bilinear similarity is designed to further filter out the encoder noise.
Revised: The gatemodulated bilinear similarity is designed to further ﬁlter out the encoder noise.

Original: The gate is generated by learning a linear projection matrix used to project the decoder feature Xto a single-channel mask, and then it is normalized to (0 , 1) by sigmoid function.
Revised: The gate is generatedby learning a linear projection matrix used to project the decoder feature X to a single-channel mask,and then it is normalized to (0 , 1) by sigmoid function.

Original: Finally, we have Y =
Revised: Then, we have Y =

Original: G Y
Revised: G Y

Original: + (1 − G )
Revised: + (1 − G )

Original: X (nearest neighbor interpolation is used for matching the resolution), 
Revised: X(nearest neighbor interpolation is used for matching the resolution), where G is the matrix form of the

Original: where
Revised: [Table]

Original: G is the matrix form of the gate unit.
Revised: gate unit.

Original: The use of the gate implies we reduce noise in some spatialregions by replacing encoder features Y with decoder features X .
Revised: The use of the gate implies we reduce noise in some spatial regions by replacing encoderfeatures Y with decoder features X .

Original: Its vector form explains that in thearea of noise it tends to switch to the self-similarity mode.
Revised: Its vector-form explains that in the area of noise it tends to switchto self-similarity mode.

Original: We will prove the effectiveness of thegating mechanism by comparing the gate-modulated bilinear similarity with a baseline without thegating mechanism.
Revised: We will prove the effectiveness of the gating mechanism by comparing thegate-modulated bilinear similarity with a baseline without the gating mechanism.

Original: In practice, encoder and decoder features may have different data distributions, which is not suitableto compare similarity directly.

Original: Therefore we apply LayerNorm for both encoder and decoderfeatures before similarity computation.

Original: Indeed we observe that the loss does not decrease withoutnormalization.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, our implementation is similar to the previous dynamic upsampling operators,which first generates the upsampling kernels and then assembles the decoder feature conditioned onthe kernels.
Revised: 3, our implementation is similar to the previous dynamic upsampling operators,which ﬁrst generates the upsampling kernels and then assembles the decoder feature conditionedon the kernels.

Original: We highlight the kernel generation module.
Revised: We highlight the kernel generation module.

Original: By setting a kernel size of K , for each encoder [Table] feature point, we compute the similarity scores between this point and eachof K × K neighbors inthe decoder.
Revised: By setting a kernel size of K , for eachencoder feature point, we compute the similarity scores between this point and each K × K neighborin the decoder.

Original: Then, the softmax normalization is applied to generate the upsampling kernels.
Revised: Then, the softmax normalization is applied to generate the upsampling kernels.

Original: SAPAis lightweight and can even work without additional parameters (with inner-product similarity).
Revised: SAPA is lightweight and can even work without additional parameters (with inner-product similarity).

Original: Tointuitively understand its lightweight property, we compare the computational complexity and numberof parameters of different dynamic upsampling operators in Table 1. 
Revised: To intuitively understand its lightweight property, we compare the computational complexity andnumber of parameters of different dynamic upsampling operators in Table 1. For example, when

Original: For example, when C = 256and d = 64 , the FLOPs are H ∗ W ∗ 199 K , H ∗ W ∗ 17 M , H ∗ W ∗ 28 K , and H ∗ W ∗ 228 K for CARAFE, IndexNet, A2U [Table] and SAPA-B, respectively.
Revised: C = 256 and d = 64 , the FLOPs are H ∗ W ∗ 199 K , H ∗ W ∗ 17 M , H ∗ W ∗ 28 K , and H ∗ W ∗ 228 Kfor CARAFE, IndexNet, A2U, and SAPA-B, respectively.

Original: We visualize the feature maps of upsampling kernels and upsampled features in Fig.
Revised: We visualize the feature maps of upsampling kernels and upsampled features in Fig.

Original: Our upsampling kernels show more favorable responses than other upsampling operators, with weightshighlighted on boundaries and noise suppressed in regions, which visually supports our propositionand is a concrete embodiment of Eq.
Revised: Our upsampling kernels show more favorable responses than other upsampling operators, with weightshighlighted on boundaries and noise suppressed in regions, which visually supports our propositionand is a concrete embodiment of Eq.

Original:  Experiments
Revised:  Experiments

Original: We first focus our experiments on semantic segmentation to justify the effectiveness of SAPA.
Revised: We ﬁrst focus our experiments on semantic segmentation to justify the effectiveness of SAPA.

Original: We thenshowcase its universality across three additional dense prediction tasks, including object detection,depth estimation, and image matting.
Revised: We thenshowcase its universality across three additional dense prediction tasks, including object detection,depth estimation, and image matting.

Original: All our experiments are run on a server with 8 NVIDIA GeForce
Revised: All our experiments are run on a server with 8 NVIDIA GeForce

Original: RTX 3090 GPUs.
Revised: RTX 3090 GPUs.

Original: 5.1 Data Sets, Metrics, Baselines, and Protocols
Revised: 5.1 Data Sets, Metrics, Baselines, and Protocols

Original: For semantic segmentation, we conduct experiments on the ADE20K dataset [4] and report the mIoUmetric.
Revised: For semantic segmentation, we conduct experiments on the ADE20K dataset [4] and report the mIoUmetric.

Original: Three strong transformer-based models are adopted as the baselines, including SegFormerB1 [5], MaskFormer-Swin-Base [6] and Mask2Former-Swin-Base [7].
Revised: Three strong transformer-based models are adopted as the baselines, including SegFormerB1 [5], MaskFormer-Swin-Base [6] and Mask2Former-Swin-Base [7].

Original: All training settings andimplementation details are kept the same as the original papers.
Revised: All training settings andimplementation details are kept the same as the original papers.

Original: We only modify the upsamplingstages with specific upsampling operators.
Revised: We only modify the upsamplingstages with speciﬁc upsampling operators.

Original: For object detection, we use the MS COCO [8] dataset, which involves 80 object categories.
Revised: For object detection, we use the MS COCO [8] dataset, which involves 80 object categories.

Original: We use
Revised: We use

Original: AP as the evaluation metric.
Revised: AP as the evaluation metric.

Original: Faster RCNN [23] with ResNet-50 [24] is adopted as the baseline.
Revised: Faster RCNN [23] with ResNet-50 [24] is adopted as the baseline.

Original: Weuse mmdetection [25] and follow its training configurations.
Revised: Weuse mmdetection [25] and follow its training conﬁgurations. [Table] [Table] CARAFE [1]

Original: For depth estimation, we use NYU Depth V2 dataset [10] and its default train/test split.
Revised: [Table] [Table] [Table] [Table]

Revised: SAPA-

Revised: [Table]

Revised: For depth estimation, we use NYU Depth V2 dataset [10] and its default train/test split.

Original: We choose
Revised: We choose

Original: BTS [9] with ResNet-50 as the baseline and follow its training configurations.
Revised: BTS [9] with ResNet-50 as the baseline and follow its training conﬁgurations.

Original: The inlier measureδ 1 < 1 .
Revised: The inlier measureδ 1 < 1 .

Original: 25 and RMSE are reported as the evaluation metrics.
Revised: 25 and RMSE are reported as the evaluation metrics.

Original: We replace all upsampling stages butthe last one for SAPA, due to no available high-res feature map for the last stage.
Revised: We replace all upsampling stages butthe last one for SAPA, due to no available high-res feature map for the last stage.

Original: For image matting, we train the model on the Adobe Image Matting dataset [11] and report fourmetrics on the Composition-1k test set, including SAD, MSE, Gradient, and Connectivity [26].A2U [Table] [Table]
Revised: For image matting, we train the model on the Adobe Image Matting dataset [11] and report fourmetrics on the Composition-1k test set, including SAD, MSE, Gradient, and Connectivity [26].

Original: matting [3] is adopted as the baseline.
Revised: A2Umatting [3] is adopted as the baseline.

Original: We use the code provided by the authors and follow the sametraining settings as in the original paper.
Revised: We use the code provided by the authors and follow the sametraining settings as in the original paper.

Original: 5.2 Main Results
Revised: 5.2 Main Results

Original: We compare SAPA and its variants against different upsampling operators on the three strongsegmentation baselines.
Revised: We compare SAPA and its variants against different upsampling operators on the three strongsegmentation baselines.

Original: Results are shown in Table 2, from which we can see that SAPA consistentlyoutperforms other upsampling operators.
Revised: Results are shown in Table 2, from which we can see that SAPA consistentlyoutperforms other upsampling operators.

Original: Note that SAPA can work well even without parametersand achieves the best performance with only few additional #Params and #Flops.
Revised: Note that SAPA can work well even without parametersand achieves the best performance with only few additional #Params and #Flops.

Original: Results on other three dense prediction tasks are shown in Table 3.
Revised: Results on other three dense prediction tasks are shown in Table 3.

Original: SAPA outperforms otherupsampling operators on depth estimation and image matting, but falls behind CARAFE on objectdetection.
Revised: SAPA outperforms otherupsampling operators on depth estimation and image matting, but falls behind CARAFE on objectdetection.

Original: One plausible explanation is that the demand of details in object detection is low (Section 6presents a further in-depth analysis).
Revised: One plausible explanation is that the demand of details in object detection is low (Section 6presents a further in-depth analysis).

Original: In detail-sensitive tasks like image matting, SAPA significantlyoutperforms CARAFE.
Revised: In detail-sensitive tasks like image matting, SAPA signiﬁcantlyoutperforms CARAFE.

Original: Qualitative comparisons are provided in supplementary material.
Revised: Qualitative comparisons are provided in supplementary material.

Original: 5.3 Ablation Study
Revised: 5.3 Ablation Study

Original: Here we conduct ablation studies to compare the choices of similarity function and normalizationfunction, the effect of different kernel sizes, and the number of embedding dimension.
Revised: Here we conduct ablation studies to compare the choices of similarity function and normalizationfunction, the effect of different kernel sizes, and the number of embedding dimension.

Original: For the defaultsetting, we use the bilinear similarity function, apply the normalization function h ( x ) =
Revised: For the defaultsetting, we use the bilinear similarity function, apply the normalization function h ( x ) =

Original: e x , set thekernel size K = 5 and the embedding dimension d = 64 .
Revised: e x , set thekernel size K = 5 and the embedding dimension d = 64 .

Original: Quantitative results are shown in Table 4. 
Revised: Quantitative results are shown in Table 4. [Table]

Original: Similarity Function We investigate three types of similarity function aforementioned and anadditional ‘plain addition’ baseline.
Revised: Similarity Function We investigate three types of similarity function aforementioned and anadditional ‘plain addition’ baseline.

Original: It ablates the gating mechanism from the gated bilinear similarity.
Revised: It ablates the gating mechanism from the gated bilinear similarity.

Original: Among them, gated bilinear similarity generates the best performance, which highlights thecomplementarity of semantic regions and boundary details in kernel generation.
Revised: Among them, gated bilinear similarity generates the best performance, which highlights thecomplementarity of semantic regions and boundary details in kernel generation.

Original: [Table]

Original: Normalization Function We also investigate different normalization functions.
Revised: Normalization Function We also investigate different normalization functions.

Original: ‘None’ indicatesno normalization is used.

Original: Among validated functions, h ( x ) =
Revised: Among validatedfunctions, h ( x ) =

Original: e x works the best, which meansnormalization matters.
Revised: e x works the best, which means normalization matters.

Original: The other three have to play with the epsilon trick to prevent zero division.
Revised: However, we thinknormalization mainly affects the optimization of the network, the other three have to play with theepsilon trick to prevent zero division, which may affect performance.

Original: Kernel Size
Revised: Kernel Size

Original: The kernel size controls the region that each point in the upsampled feature can attendto.
Revised: The kernel size controls the region that each point in the upsampled feature can attendto.

Original: Results show that, compared with a large kernel, a small local window is sufficient to distinguishthe semantic clusters.
Revised: Results show that, compared with a large kernel, a small local window is sufﬁcient to distinguishbetween regions and edges.

Original: Embedding Dimension We further study the influence of embedding dimension in the range of 16 ,32 , 64 , and 128 .
Revised: Embedding Dimension We further study the inﬂuence of embedding dimension in the range of 16 ,32 , 64 , and 128 .

Original: Interestingly, results suggest that SAPA is not sensitive to the embedding dimension.
Revised: Interestingly, results suggest that SAPA is not sensitive to the embedding dimension.

Original: This also verifies that SAPA extracts existing knowledge rather than learn unknown knowledge.
Revised: This also veriﬁes that SAPA extracts existing knowledge rather than learn unknown knowledge.

Original:  Discussion
Revised:  Discussion

Original: Understanding SAPA from a Backward Perspective We have explained SAPA in the forwardpass, here we further discuss how it may work during training.
Revised: Understanding SAPA from a Backward Perspective We have explained SAPA in the forwardpass, here we further discuss how it may work during training.

Original: In fact, originally the model doesnot know to which the semantic cluster an encoder point should belong.
Revised: In fact, originally the model doesnot know to which the semantic cluster an encoder point should belong.

Original: The working mechanismof SAPA assigns each encoder feature point a possibility to choose a cluster.
Revised: The working mechanismof SAPA assigns each encoder feature point a possibility to choose a cluster.

Original: During training, theground truths produce gradients, thus changes the assignment possibility.
Revised: During training, theground truths produce gradients, thus changes the assignment possibility.

Original: In SAPA, for every encoderpoint, the semantic clusters in the corresponding local window in decoder features serve as implicitlabels and cooperate with the ground truths.
Revised: In SAPA, for every encoderpoint, the semantic clusters in the corresponding local window in decoder features serve as implicitlabels and cooperate with the ground truths.

Original: The correct cluster is the positive label, while the wrongone is negative.
Revised: The correct cluster is the positive label, while the wrongone is negative.

Original: In the preliminary stage of training, if an encoder feature point is more similar to awrong cluster, it will be punished by gradients and engender large losses, and vice versa.
Revised: In the preliminary stage of training, if an encoder feature point is more similar to awrong cluster, it will be punished by gradients and engender large losses, and vice versa.

Original: Therefore,the encoder feature points can gradually learn its affiliation.
Revised: Therefore,the encoder feature points can gradually learn its afﬁliation.

Original: We find this process is fast by visualizingthe epoch-to-epoch feature maps.
Revised: We ﬁnd this process is fast by visualizingthe epoch-to-epoch feature maps.

Original: Limitations compared with CARAFE CARAFE is a purely semantic-driven upsampling operatorable to mend semantic information with a single-input flow.
Revised: Limitations compared with CARAFE CARAFE is a purely semantic-driven upsampling operatorable to mend semantic information with a single-input ﬂow.

Original: Such a mechanism in CARAFE bringsadvantages in smoothing semantic regions.
Revised: Such a mechanism in CARAFE bringsadvantages in smoothing semantic regions.

Original: E.g. , we observe it mends holes in a continuous region.
Revised: E.g. , we observe it mends holes in a continuous region.

Original: However, due to its single-input flow, it cannot compensate the details lost in downsampling.
Revised: However, due to its single-input ﬂow, it cannot compensate the details lost in downsampling.

Original: Our
Revised: Our

Original: SAPA, by contrast, mainly aims to compensate details such as textures and boundaries.
Revised: SAPA, by contrast, mainly aims to compensate details such as textures and boundaries.

Original: SAPAcharacters in two aspects: semantic preservation and detail delineation.
Revised: SAPAcharacters in two aspects: semantic preservation and detail delineation.

Original: However, as Eq.
Revised: However, as Eq.

Original: (4) suggests,we do not add any semantic mending mechanism in SAPA.
Revised: (4) suggests,we do not add any semantic mending mechanism in SAPA.

Original: This explains why SAPA is worsethan CARAFE on object detection, because detection has less demand for details but more forregional integrity.
Revised: This explains why SAPA is worsethan CARAFE on object detection, because detection has less demand for details but more forregional integrity.

Original: In short, CARAFE highlights semantic mending , while SAPA highlights semanticpreserving and detail delineation .
Revised: In short, CARAFE highlights semantic mending , while SAPA highlights semanticpreserving and detail delineation .

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we introduce similarity-aware point affiliation, i.e. , SAPA.
Revised: In this paper, we introduce similarity-aware point afﬁliation, i.e. , SAPA.

Original: It not only indicates alightweight but effective upsampling operator suitable for tasks like semantic segmentation, but alsoexpresses a high-level concept that characterizes feature upsampling.
Revised: It not only indicates alightweight but effective upsampling operator suitable for tasks like semantic segmentation, but alsoexpresses a high-level concept that characterizes feature upsampling.

Original: SAPA can serve as an universalsubstitution for conventional upsampling operators.
Revised: SAPA can serve as an universalsubstitution for conventional upsampling operators.

Original: Experiments show the effectiveness of SAPAand also indicate its limitation: it is more suitable for tasks that favor detail delineation.
Revised: Experiments show the effectiveness of SAPAand also indicate its limitation: it is more suitable for tasks that favor detail delineation.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] Please see Section 6.
Revised: [Yes] Please see Section 6.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A](b)

Original: (b) Did you include complete proofs of all theoretical results?
Revised: Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-
Revised: g., with respect to the random seed after running experi

Original: ments multiple times
Revised: -ments multiple times)?

Original: [No] Our method generates stable results.
Revised: [No] Our method generates stable results.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] For each of thedatasets used, we cite the creators, as shown in Section 5.1.
Revised: [Yes] For each of the datasets used, we cite the creators, as shown in Section 5.1.

Original: (b) Did you mention the license of the assets?
Revised: b) Did you mention the license of the assets?

Original: [Yes] The data used in our work is opensource and can be used for academic research.
Revised: [Yes] The data used in our work is open source and can be used for academic research.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: Our code for this work will appear in the supplementary material.
Revised: Our code for this work will appear in the supplementary material

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using curating?

Original: [N/A]
Revised: [N/A]

Original: The data used in our work is open source and can be used for
Revised: The data used in our work is open source and can be used for

Original: academic research.
Revised: academic research

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: .(e ) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: The data do not contain personally identifiable
Revised: The data do not contain personally identiﬁable

Original: information or offensive content.
Revised: information or offensive content.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]