Original: SAPA: Similarity-Aware Point Afﬁliation for Feature Upsampling
Revised: SAPA: Similarity-Aware Point Affiliation for Feature Upsampling

Revised: Anonymous Author s

Revised: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: We introduce point afﬁliation into feature upsampling, a notion that describes theafﬁliation of each upsampled point to a semantic cluster formed by local decoderfeature points with semantic similarity.
Revised: We introduce point affiliation into feature upsampling, a notion that describes theaffiliation of each upsampled point to a semantic cluster formed by local decoderfeature points with semantic similarity.

Original: By rethinking point afﬁliation, we present ageneric formulation for generating upsampling kernels.
Revised: By rethinking point affiliation, we present ageneric formulation for generating upsampling kernels.

Original: The kernels encourage notonly semantic smoothness but also boundary sharpness in the upsampled featuremaps.
Revised: The kernels encourage notonly semantic smoothness but also boundary sharpness in the upsampled featuremaps.

Original: Such properties are particularly useful for some dense prediction tasks suchas semantic segmentation.
Revised: Such properties are particularly useful for some dense prediction tasks suchas semantic segmentation.

Original: The key idea of our formulation is to generate similarityaware kernels by comparing the similarity between each encoder feature point andthe spatially associated local region of decoder features.
Revised: The key idea of our formulation is to generate similarityaware kernels by comparing the similarity between each encoder feature point andthe spatially associated local region of decoder features.

Original: In this way, the encoderfeature point can function as a cue to inform the semantic cluster of upsampledfeature points.
Revised: In this way, the encoderfeature point can function as a cue to inform the semantic cluster of upsampledfeature points.

Original: To embody the formulation, we further instantiate a lightweightupsampling operator, termed Similarity-Aware Point Afﬁliation (SAPA), and investigate its variants.
Revised: To embody the formulation, we further instantiate a lightweightupsampling operator, termed Similarity-Aware Point Affiliation (SAPA), and investigate its variants.

Original: SAPA invites consistent performance improvements on a numberof dense prediction tasks, including semantic segmentation, object detection, depthestimation, and image matting.
Revised: SAPA invites consistent performance improvements on a numberof dense prediction tasks, including semantic segmentation, object detection, depthestimation, and image matting.

Original: Code will be available online.
Revised: Code will be available online.

Original:  Introduction
Revised:  Introduction

Original: We introduce the notion of point afﬁliation into feature upsampling.
Revised: We introduce the notion of point affiliation into feature upsampling.

Original: Point afﬁliation deﬁnes a relationbetween each upsampled point and a semantic cluster 1 to which the point should belong.
Revised: Point affiliation defines a relationbetween each upsampled point and a semantic cluster 1 to which the point should belong.

Original: It highlightsthe spatial arrangement of upsampled points at the semantic level.
Revised: It highlightsthe spatial arrangement of upsampled points at the semantic level.

Original: Considering an example shownin Fig.
Revised: Considering an example shownin Fig.

Original: 1 w.r.t.
Revised: 1 w.r.t.

Original: × 2 upsampling in semantic segmentation, the orange point of low resolution willcorrespond to 4 upsampled points of high resolution, in which the red and yellow ones should beassigned the ‘picture’ cluster and the ‘wall’ cluster, respectively.
Revised: × 2 upsampling in semantic segmentation, the orange point of low resolution willcorrespond to 4 upsampled points of high resolution, in which the red and yellow ones should beassigned the ‘picture’ cluster and the ‘wall’ cluster, respectively.

Original: Designating point afﬁliation isdifﬁcult and sometimes can be erroneous, however.
Revised: Designating point affiliation isdifficult and sometimes can be erroneous, however.

Original: In × 2 upsampling, nearest neighbor (NN) interpolation directly copies 4 identical points from thelow-res one, which assigns the same semantic cluster to the 4 points.
Revised: In × 2 upsampling, nearest neighbor (NN) interpolation directly copies 4 identical points from thelow-res one, which assigns the same semantic cluster to the 4 points.

Original: On regions in need of details,the 4 points probably do not share the same cluster but are forced to share.
Revised: On regions in need of details,the 4 points probably do not share the same cluster but are forced to share.

Original: Bilinear interpolationassigns point afﬁliation with distance priors.
Revised: Bilinear interpolationassigns point affiliation with distance priors.

Original: Yet, when tackling points of different semantic clusters,it not only cannot inform clear point afﬁliation, but also blurs the boundary between differentsemantic clusters.
Revised: Yet, when tackling points of different semantic clusters,it not only cannot inform clear point affiliation, but also blurs the boundary between differentsemantic clusters.

Original: Recent dynamic upsampling operators have similar issues.
Revised: Recent dynamic upsampling operators have similar issues.

Original: CARAFE [1] judges theafﬁliation of an upsampled point with content-aware kernels.
Revised: CARAFE [1] judges theaffiliation of an upsampled point with content-aware kernels.

Original: Certain semantic clusters will receivelarger weights than the rest and therefore dominate the afﬁliation of upsampled points.
Revised: Certain semantic clusters will receivelarger weights than the rest and therefore dominate the affiliation of upsampled points.

Original: However, theafﬁliation near boundaries or on regions full of details can still be ambiguous.
Revised: However, theaffiliation near boundaries or on regions full of details can still be ambiguous.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, theboundaries are unclear in the feature maps upsampled by CARAFE.
Revised: 2, theboundaries are unclear in the feature maps upsampled by CARAFE.

Original: The reason is that the kernelsare conditioned on decoder features alone; the decoder features carry little useful information about Upsample Kernel
Revised: The reason is that the kernelsare conditioned on decoder features alone; the decoder features carry little useful information abouthigh-res

Original: Kernel (𝑖 2𝑝 ,𝑗 2𝑝 ) (𝑖 1 ,𝑗 1 ) (⌊𝑖 2 2⌋,⌊j 2 2⌋) (⌊𝑖 1 2⌋,⌊j 1 2⌋) small large

Original: Encoder feature

Original: Decoder feature

Original: Upsampled feature (𝑖 2𝑤 ,𝑗 2𝑤 )

Original: Upsample

Original: Kernel (𝑖 2𝑤 ,𝑗 2𝑤 ) (𝑖 1 ,𝑗 1 ) (𝑖 2𝑝 ,𝑗 2𝑝 ) 𝐶𝐶𝐶𝐶

Original: [Figure]
Revised: [Figure]

Original: Inferring structure requires high-res encoder features.
Revised: Inferring structure requires high-res encoder features.

Original: For instance, if the orange point in Fig.
Revised: For instance, if the orange point in Fig.

Original: 1 lieson the low-res boundary, it is difﬁcult to judge to which cluster the 4 upsampled points should belong.
Revised: 1 lieson the low-res boundary, it is difficult to judge to which cluster the 4 upsampled points should belong.

Original: However, the encoder feature in Fig.
Revised: However, the encoder feature in Fig.

Original: 1 actually tells that, the yellow point is on the wall, and the redone is on the picture, which suggests one may extract useful information from the encoder feature toassist the designation of point afﬁliation.
Revised: 1 actually tells that, the yellow point is on the wall, and the redone is on the picture, which suggests one may extract useful information from the encoder feature toassist the designation of point affiliation.

Original: Indeed IndexNet [2] and A2U [3] have attempted to encodesuch information to improve detail delineation in encoder-dependent upsampling kernels; however,the encoder feature can easily introduce noise into kernels, engendering discontinuous feature mapsshown in Fig.
Revised: Indeed IndexNet [2] and A2U [3] have attempted to encodesuch information to improve detail delineation in encoder-dependent upsampling kernels; however,the encoder feature can easily introduce noise into kernels, engendering discontinuous feature mapsshown in Fig.

Original: Hence, the key problem seems to be how to extract only required information intothe upsampling kernels from the encoder feature while ﬁltering out the rest.
Revised: Hence, the key problem seems to be how to extract only required information intothe upsampling kernels from the encoder feature while filtering out the rest.

Original: To use encoder features effectively, an important assumption of this paper is that, an encoder featurepoint is most similar to the semantic cluster into which the point will be classiﬁed.
Revised: To use encoder features effectively, an important assumption of this paper is that, an encoder featurepoint is most similar to the semantic cluster into which the point will be classified.

Original: Per the left of
Revised: Per the left of

Original: Fig. 1, suppose that the encoder point in the red box is assigned into the cluster C 1 by its semanticmeaning, then it is similar to C 1 , while not similar to C 2 .
Revised: Fig. 1, suppose that the encoder point in the red box is assigned into the cluster C 1 by its semanticmeaning, then it is similar to C 1 , while not similar to C 2 .

Original: As a result, by comparing the similaritybetween the encoder feature point and different semantic clusters in the decoder feature, the afﬁliationof the upsampled point can be informed according to the similarity scores.
Revised: As a result, by comparing the similaritybetween the encoder feature point and different semantic clusters in the decoder feature, the affiliationof the upsampled point can be informed according to the similarity scores.

Original: In particular, we proposeto generate upsampling kernels with local mutual similarity between encoder and decoder features.
Revised: In particular, we proposeto generate upsampling kernels with local mutual similarity between encoder and decoder features.

Original: For every encoder feature point, we compute the similarity score between this point and each decoderfeature point in the spatially associated local window.
Revised: For every encoder feature point, we compute the similarity score between this point and each decoderfeature point in the spatially associated local window.

Original: For the green point in Fig.
Revised: For the green point in Fig.

Original: 1, since every pointin the window shares the same semantic cluster, the encoder feature point is as similar as every pointin the window.
Revised: 1, since every pointin the window shares the same semantic cluster, the encoder feature point is as similar as every pointin the window.

Original: In this case we expect an ‘average kernel’ which is the key characteristic to ﬁlternoise, and the upsampled 4 points would have the same semantic cluster as before.
Revised: In this case we expect an ‘average kernel’ which is the key characteristic to filternoise, and the upsampled 4 points would have the same semantic cluster as before.

Original: For the yellowpoint in the encoder, since it belongs to the ‘wall’ cluster, it is more similar to the points on the wallthan those on the picture.
Revised: For the yellowpoint in the encoder, since it belongs to the ‘wall’ cluster, it is more similar to the points on the wallthan those on the picture.

Original: In this case we expect a kernel with larger weights on points related to the‘wall’ cluster.
Revised: In this case we expect a kernel with larger weights on points related to the‘wall’ cluster.

Original: This can help to assign the afﬁliation of the yellow point to be in the ’wall’ cluster.
Revised: This can help to assign the affiliation of the yellow point to be in the ’wall’ cluster.

Original: By modeling the local mutual similarity, we derive a generic form of upsampling kernels and showthat this form implements our expected upsampling behaviors: encouraging both semantic smoothnessand boundary sharpness.
Revised: By modeling the local mutual similarity, we derive a generic form of upsampling kernels and showthat this form implements our expected upsampling behaviors: encouraging both semantic smoothnessand boundary sharpness.

Original: Following our formulation, we further instantiate a lightweight upsamplingoperator, termed Similarity-Aware Point Afﬁliation (SAPA), and investigate its variants.
Revised: Following our formulation, we further instantiate a lightweight upsamplingoperator, termed Similarity-Aware Point Affiliation (SAPA), and investigate its variants.

Original: We evaluate
Revised: We evaluate

Original: SAPA across a number of mainstream dense prediction tasks, for example: i) semantic segmentation :we test SAPA on several transformer-based segmentation baselines on the ADE20K dataset [4], suchas SegFormer [5], MaskFormer [6], and Mask2Former [7], improving the baselines by 1% ∼ 2 .
Revised: SAPA across a number of mainstream dense prediction tasks, for example: i) semantic segmentation :we test SAPA on several transformer-based segmentation baselines on the ADE20K dataset [4], suchas SegFormer [5], MaskFormer [6], and Mask2Former [7], improving the baselines by 1% ∼ 2 .

Original: 7%mIoU; ii) object detection : SAPA improves the performance of Faster RCNN by 0 .
Revised: 7%mIoU; ii) object detection : SAPA improves the performance of Faster RCNN by 0 .

Original: 4% AP on MS
Revised: 4% AP on MS

Original: COCO [8]; iii) monocular depth estimation : SAPA reduces the rmse metric of BTS [9] from 0 .
Revised: COCO [8]; iii) monocular depth estimation : SAPA reduces the rmse metric of BTS [9] from 0 .

Original: 419to 0 .
Revised: 419to 0 .

Original: 408 on NYU Depth V2 [10]; and iv) image matting
Revised: 408 on NYU Depth V2 [10]; and iv) image matting

Original: : SAPA outperforms a strong A2U mattingbaseline [3] on the Adobe Composition-1k testing set [11] with a further 3 .
Revised: : SAPA outperforms a strong A2U mattingbaseline [3] on the Adobe Composition-1k testing set [11] with a further 3 .

Original: 8% relative error reductionin the SAD metric.
Revised: 8% relative error reductionin the SAD metric.

Original: SAPA also outperforms or at least is on par with other state-of-the-art dynamicupsampling operators.
Revised: SAPA also outperforms or at least is on par with other state-of-the-art dynamicupsampling operators.

Original: Particularly, even without additional parameters, SAPA outperforms theprevious best upsampling operator CARAFE on semantic segmentation.
Revised: Particularly, even without additional parameters, SAPA outperforms theprevious best upsampling operator CARAFE on semantic segmentation.

Original: [Figure]
Revised: [Figure]

Original: U p s a m p l e d f ea t u r e m a p s
Revised: U p s a m p l e d f ea t u r e m a p s

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: K e r n e l
Revised: K e r n e l

Revised: The visualization is produced with SegNet [12] as

Original: For each  operator,  choose the  three channels  the feature maps of the last 
Revised: For each upsampling operator, we choose the first three channels from the feature maps of the last upsampling stage for visualization.

Original: Only SAPA  both smooth regions and 
Revised: Only SAPA shows both smooth regions and sharp boundaries.

Original: The kernel map of CARAFE is coarse and lacking in details, while IndexNet and A2U generate kernels withtoo much detail from encoder to preserve semantics.
Revised: The kernel map of CARAFE is coarse and lacking in details, while IndexNet and A2U generate kernels with much details from the encoder.

Original: Please refer to supplementary materials for additional visualizations.
Revised: See the supplementary materials for additional visualizations.

Original:  Related Work
Revised:  Related Work

Original: We review work related to feature upsampling.
Revised: We review work related to feature upsampling.

Original: Feature upsampling is a fundamental procedure inencoder-decoder architectures used to recover the spatial resolution of low-res decoder feature mapsand has been extensively used in dense prediction tasks such as semantic segmentation [12, 5, 6, 7]and depth estimation [14, 15, 9].
Revised: Feature upsampling is a fundamental procedure inencoder-decoder architectures used to recover the spatial resolution of low-res decoder feature mapsand has been extensively used in dense prediction tasks such as semantic segmentation [12, 5, 6, 7]and depth estimation [14, 15, 9].

Original: Standard upsampling operators are hand-crafted.
Revised: Standard upsampling operators are hand-crafted.

Original: NN and bilinear interpolation measure the semanticafﬁliation in terms of relative distances in upsampling, which follows ﬁxed rules to designate pointafﬁliation, even if the true afﬁliation may be different.
Revised: NN and bilinear interpolation measure the semanticaffiliation in terms of relative distances in upsampling, which follows fixed rules to designate pointaffiliation, even if the true affiliation may be different.

Original: Max unpooling [12] stores the indices ofmax-pooled feature points in encoder features and uses the sparse indices to guide upsampling.
Revised: Max unpooling [12] stores the indices ofmax-pooled feature points in encoder features and uses the sparse indices to guide upsampling.

Original: Whileit executes location-speciﬁc point afﬁliation which beneﬁts detail recovery, most upsampled pointsare assigned with null afﬁliation due to zero ﬁlling.
Revised: Whileit executes location-specific point affiliation which benefits detail recovery, most upsampled pointsare assigned with null affiliation due to zero filling.

Original: Pixel Shufﬂe [16] is widely-used in image/videosuper-resolution.
Revised: Pixel Shuffle [16] is widely-used in image/videosuper-resolution.

Original: Its upsampling only includes memory operation – reshaping depth channels tospace ones.
Revised: Its upsampling only includes memory operation – reshaping depth channels tospace ones.

Original: The notion of point afﬁliation does not apply to this operator, however.
Revised: The notion of point affiliation does not apply to this operator, however.

Original: Another stream of upsampling operators implement learning-based upsampling.
Revised: Another stream of upsampling operators implement learning-based upsampling.

Original: Among them,transposed convolution or deconvolution [17] is known as an inverse convolutional operator.
Revised: Among them,transposed convolution or deconvolution [17] is known as an inverse convolutional operator.

Original: Basedon a novel interpretation of deconvolution, PixelTCL [18] is proposed to alleviate the checkerboardartifacts [19] of standard deconvolution.
Revised: Basedon a novel interpretation of deconvolution, PixelTCL [18] is proposed to alleviate the checkerboardartifacts [19] of standard deconvolution.

Original: In addition, bilinear additive upsampling [20] attemptsto combine learnable convolutional kernels with hand-crafted upsampling operators to achievecomposited upsampling.
Revised: In addition, bilinear additive upsampling [20] attemptsto combine learnable convolutional kernels with hand-crafted upsampling operators to achievecomposited upsampling.

Original: Recently, DUpsample [21] seeks to reconstruct the upsampled feature mapwith pre-learned projection matrices, expecting to achieve a data-dependent upsampling behavior.
Revised: Recently, DUpsample [21] seeks to reconstruct the upsampled feature mapwith pre-learned projection matrices, expecting to achieve a data-dependent upsampling behavior.

Original: While these operators are learnable, the upsampling kernels are ﬁxed once learned, still resulting inﬁxed designation of point afﬁliation.
Revised: While these operators are learnable, the upsampling kernels are fixed once learned, still resulting infixed designation of point affiliation.

Original: In learning-based upsampling, some recent work introduces the idea of generating content-awaredynamic kernels.
Revised: In learning-based upsampling, some recent work introduces the idea of generating content-awaredynamic kernels.

Original: Instead of learning the parameters of the kernels, they learn how to predict thekernels.
Revised: Instead of learning the parameters of the kernels, they learn how to predict thekernels.

Original: In particular, CARAFE [1] predicts dynamic kernels conditioned on the decoder features.
Revised: In particular, CARAFE [1] predicts dynamic kernels conditioned on the decoder features.

Original: IndexNet [2] and A2U [3], by contrast, generate encoder-dependent kernels.
Revised: IndexNet [2] and A2U [3], by contrast, generate encoder-dependent kernels.

Original: While they signiﬁcantlyoutperform previous upsampling operators in various tasks, they still can cause uncertain pointafﬁliation, resulting in either unclear predictions near boundaries or fragile predictions in regions.
Revised: While they significantlyoutperform previous upsampling operators in various tasks, they still can cause uncertain pointaffiliation, resulting in either unclear predictions near boundaries or fragile predictions in regions.

Original: Our work is closely related to dynamic kernel-based upsampling.
Revised: Our work is closely related to dynamic kernel-based upsampling.

Original: We also seek to predict dynamickernels; however, we aim to address the uncertain point afﬁliation in prior arts to achieve simultaneousregion smoothness and boundary sharpness.
Revised: We also seek to predict dynamickernels; however, we aim to address the uncertain point affiliation in prior arts to achieve simultaneousregion smoothness and boundary sharpness.

Original:  Dynamic Upsampling Revisited
Revised:  Dynamic Upsampling Revisited

Original: We ﬁrst revisit two key components shared by existing dynamic upsampling operators: kernelgeneration and feature assembly.
Revised: We first revisit two key components shared by existing dynamic upsampling operators: kernelgeneration and feature assembly.

Original: Kernel Generation Given the decoder feature X  , if we upsample it to the target feature X(cid:48) , whichis often twice the size of X , then for any point at the location l (cid:48)
Revised: Kernel Generation Given the decoder feature X ∈ R H × W × C , if we upsample it to the targetfeature X

Original: =
Revised: ′ ∈

Original:  ( i (cid:48) , j (cid:48) ) in X (cid:48) , we generate akernel W l (cid:48) based on the neighborhood feature N l (cid:48) that spatially corresponds to l (cid:48) .
Revised: R 2 H × 2 W × C , then for any point at the location l ′ = ( i ′ , j ′ ) in X ′ , we generate a kernel W l ′ based on the neighborhood feature N l ′ that spatially corresponds to l ′ .

Original: In this way, kernelgeneration can be deﬁned by
Revised: In this way, kernelgeneration can be defined by

Original: [Equation]
Revised: [Equation]

Original: 1) where ψ refers to a kernel generation module, which is often implemented by a sub-network used topredict the kernel conditioned on N l (cid:48) , and norm is a normalization function.
Revised: where ψ refers to a kernel generation module, which is often implemented by a sub-network usedto predict the kernel conditioned on N l ′ , and norm is a normalization function.

Original: The feature N l (cid:48) canoriginate from two sources, to be speciﬁc, from the encoder feature Y  or from the decoder feature X .
Revised: The feature N l ′can originate from two sources, to be specific, from the encoder feature Y ∈ R 2 H × 2 W × C or fromthe decoder feature X .

Original: If the encoder feature Y is chosen as the source, then a local region of Y centered at l (cid:48) is extractedto be N l (cid:48) .
Revised: If the encoder feature Y is chosen as the source, then a local region of Ycentered at l ′ is extracted to be N l ′ .

Original: If the decoder feature X is chosen, one ﬁrst needs to compute the projective locationof l (cid:48) , i.e. , l = ( (cid:98) i (cid:48) 2 (cid:99) , (cid:98) j (cid:48) 2 (cid:99) ) according to the spatial location correspondence, then a neighborhoodregion extracted from X  l is regarded as N l (cid:48) .
Revised: If the decoder feature X is chosen, one first needs to computethe projective location of l ′ , i.e. , l = ( i, j ) = ⌊ l ′ 2 ⌋ = ( ⌊ i ′ 2 ⌋ , ⌊ j ′ 2 ⌋ ) according to the spatial locationcorrespondence, then a local region of X centered at l is regarded as N l ′ .

Original: The softmax function is often used as the normalizationfunction such that relevant points can be softly selected to compute the value of the target point usingthe weight W l (cid:48) .
Revised: The softmax function isoften used as the normalization function such that relevant points can be softly selected to computethe value of the target point using the weight W l ′ .

Original: As illustrated in Fig. 2, the source of N l (cid:48) can affect the predicted kernel.
Revised: According to Fig. 2, the source of N l ′ can affect the predicted kernel.

Original: The kernels predicted by
Revised: The kernels predicted by

Original: CARAFE, IndexNet, and A2U show signiﬁcantly distinct characteristics.
Revised: CARAFE, IndexNet, and A2U show significantly distinct characteristics.

Original: With the decoder featurealone, the kernel map is coarse and lacking in details.
Revised: With the decoder featurealone, the kernel map is coarse and lacking in details.

Original: Beneﬁted from the encoder feature, the kernelmaps generated by IndexNet and A2U have rich details; however, they manifest large similarity tothe original encoder feature, which means noise is introduced into the kernel.
Revised: Benefiting from the encoder feature, the kernelmaps generated by IndexNet and A2U have rich details; however, they manifest high similarity to theencoder feature, which means noise is introduced into the kernel.

Original: Feature Assembly In this step, for each target feature point at l (cid:48) , we assemble the correspondingsub-region of decode feature with the predictedkernel W l (cid:48) and assign a value to this target point. Ifthe predicted kernel has a kernel size of K × K  , then the sub-region is also of size K × K centeredat l  ,where l = ( i, j )  =
Revised: Feature Assembly For each target feature point at l ′ , we assemble the corresponding sub-regionof decoder feature with the predicted K × K kernel W l ′ , whose weight is denoted by W l ′ , ( u,v ) ,u, v = − r, ..., r, r =

Original: ( (cid:98) i (cid:48) 2 (cid:99) ,(cid:98) j (cid:48) 2 (cid:99) ) . We compute the weighted sum between the normalized kerneland the sub-region of decoder feature to obtain the value of  X (cid:48) l (cid:48) by 
Revised: ⌊ K 2 ⌋ , to obtain the value of the target point X ′ l ′ by a weighted sum

Original: [Equation]
Revised: [Equation]

Original: By executing feature assembly on every target feature point, we can obtain the target upsampledfeature map.
Revised: By executing feature assembly on every target feature point, we can obtain the target upsampledfeature map.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, the upsampled feature has a close relation to the kernel.
Revised: 2, the upsampled feature has a close relation to the kernel.

Original: A wellpredicted kernel can encourge both semantic smoothness and boundary sharpness; a kernel withoutencoding details or with too many details encoded can introduce noise.
Revised: A wellpredicted kernel can encourge both semantic smoothness and boundary sharpness; a kernel withoutencoding details or with too many details encoded can introduce noise.

Original: We consider an ideal kernelshould only response at the position in need of details, while do not response (appearing as an averagevalue over an area) at good semantic regions.
Revised: We consider an ideal kernelshould only response at the position in need of details, while do not response (appearing as an averagevalue over an area) at good semantic regions.

Original: More importantly, an ideal kernel should assign weightsreasonably so that each point can be designated to a correct semantic cluster.
Revised: More importantly, an ideal kernel should assign weightsreasonably so that each point can be designated to a correct semantic cluster.

Original:  Rethinking Point Afﬁliation with Local Mutual Similarity
Revised:  Rethinking Point Affiliation with Local Mutual Similarity

Original: To obtain an expected upsampling kernel mentioned above, we ﬁrst derive a generic formulation forgenerating  upsampling kernels by exploiting local mutual similarity, then explain why the formulation encourages semantic smoothness and boundary sharpness, and ﬁnally present an upsampling operator, SAPA, that embodies our formulation.
Revised: To obtain an expected upsampling kernel mentioned above, we first derive a generic formulation forgenerating the upsampling kernel by exploiting local mutual similarity, then explain why the formulation encourages semantic smoothness and boundary sharpness, and finally present an upsamplingoperator, SAPA, that embodies our formulation.

Original: 4.1 Local Mutual Similarity
Revised: 4.1 Local Mutual Similarity

Original: We rethink point afﬁliation from the view of local mutual similarity between encoder and decoderfeatures.
Revised: We rethink point affiliation from the view of local mutual similarity between encoder and decoderfeatures.

Original: With a detailed analysis, we explain why such similarity can assist point afﬁliation.
Revised: With a detailed analysis, we explain why such similarity can assist point affiliation.

Original: We ﬁrst deﬁne a generic similarity function sim( x , y ) :
Revised: We first define a generic similarity function sim( x , y ) :

Original: R C × R C → R .
Revised: R C × R C → R .

Original: It scores the similaritybetween a vector x and a vector y of the same dimension C .
Revised: It scores the similaritybetween a vector x and a vector y of the same dimension C .

Original: We also deﬁne a normalizationfunction involving n real numbers x 1 , x 2 , ..., x n by norm( x i : x 1 , x 2 , ..., x n )
Revised: We also define a normalizationfunction involving n real numbers x 1 , x 2 , ..., x n by norm( x i : x 1 , x 2 , ..., x n )

Original: =
Revised: =

Original: h( x i ) (cid:80) nj =1 h ( x j ) , whereh( x ) :
Revised: h( x i ) (cid:80) nj =1 h ( x j ) , whereh( x ) :

Original: R → R is an arbitrary function, ignoring zero division.
Revised: R → R is an arbitrary function, ignoring zero division.

Original: Given sim( x , y ) and h( x ) , we candeﬁne a generic formulation for generating the upsampling kernel
Revised: Given sim( x , y ) and h( x ) , we candefine a generic formulation for generating the upsampling kernel

Original: [Equation]
Revised: [Equation]

Original: where w is the kernel value speciﬁc to x and y .
Revised: where w is the kernel value specific to x and y .

Original: To analyze the upsampling behavior of the kernel,we further deﬁne the following notations.
Revised: To analyze the upsampling behavior of the kernel,we further define the following notations.

Original: Let Y ∈ R 2 H × 2 W × C and X ∈

Original: R

Original: H × W × C be the encoder and decoder feature maps, respectively. Let y i,j = Y ( i,j ) ∈ R C, ( i, j ) ∈ { 0 , 1 , ..., 2 H − 1 } × { 0 , 1 , ..., 2 W − 1 } denote the encoder featurevector and x k,l = X ( k,l ) ∈ R C, ( k, l ) ∈ { 0 , 1 , ..., H − 1 } × { 0 , 1 , ..., W − 1 } be the decoder featurevector, where C is the number of channels.Let K be the upsampling kernel size.
Revised: Let y l ′ ∈ R C denote the encoder feature vector at position l ′ and x l ∈ R C be the decoder featurevector at position l , where C is the number of channels.

Original: Our operation will be done within a local window of size K × K , between each encoder vector y i,j and all its spatially associated decoder feature vectors,x (cid:98) i 2 (cid:99) + m, (cid:98) j 2 (cid:99) + n ’s, where m, n =−(cid:98)
Revised: Our operation will be done within a localwindow of size K × K , between each encoder vector y l ′ and all its spatially associated decoderfeature vectors, x l +( u,v ) ’s, where u, v = − r, ..., r, r =

Original:  K 2 (cid:99) , ..., (cid:98) K 2 (cid:99) .
Revised: ⌊ K 2 ⌋ .

Original: To simplify our analysis, we also assume local smoothness.
Revised: To simplify our analysis, we also assume local smoothness.

Original: That is, points with the same semanticcluster will have a similar value, which means a local region will share the same value on everychannel of the feature map.
Revised: That is, points with the same semanticcluster will have a similar value, which means a local region will share the same value on everychannel of the feature map.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, we deﬁne x w =
Revised: 1, we define x a =

Original: ( R w 1 , R w 2 , ..., R wC ) T and x p =( R p 1 , R p 2 , ..., R pC ) T as the semantic clusters related to ‘wall’ and ‘picture’, respectively, where R wcand R pc are constants, and c = 1 , 2 , ..., C .
Revised: ( R a 1 , R a 2 , ..., R aC ) T and x p =( R p 1 , R p 2 , ..., R pC ) T as the semantic clusters related to ‘wall’ and ‘picture’, respectively, where R acand R pc are constants, and c = 1 , 2 , ..., C .

Original: For ease of analysis, we deﬁne two types of windowsdistinguished by their contents.
Revised: For ease of analysis, we define two types of windowsdistinguished by their contents.

Original: When all the points inside a window belong to the same semanticcluster, it is called a smooth window; while different semantic clusters appear in a window, it isdeﬁned as a detail window.
Revised: When all the points inside a window belong to the same semanticcluster, it is called a smooth window; while different semantic clusters appear in a window, it isdefined as a detail window.

Original: Next, we explain why the kernel can ﬁlter noise and can encourage semantic smoothness in a smoothwindow, and why it can help to recover details when dealing with boundaries/textures in a detailwindow.
Revised: Next, we explain why the kernel can filter out noise, why it encourages semantic smoothness in asmooth window, and why it can help to recover details when dealing with boundaries/textures in adetail window.

Original: Upsampling in a Smooth Window Without loss of generality, we consider the point y i 1 ,j 1 in Fig.1 in the encoder feature.
Revised: Upsampling in a Smooth Window Without loss of generality, we consider an encoder point atthe position l ′ in Fig.

Original: Its corresponding window is a smooth window ofwith the semanticcluster ‘picture’, thus x (cid:98) i 12 (cid:99) + m, (cid:98) j 12 (cid:99) + n = x p , m, n =−(cid:98)
Revised: Its corresponding window is a smooth window of the semantic cluster‘picture’, thus x l +( u,v ) =

Original: K 2 (cid:99) ,  ..., (cid:98) K 2 (cid:99) .
Revised: x p , u, v = − r, ..., r, r = ⌊ K 2 ⌋ .

Original: Then the upsampling kernelweight for the upsampled point ( i 1 , j 1 ) at the position ( m, n ) takes the form
Revised: Then the upsampling kernel weight w.r.t.the upsampled point l ′ at the position ( u, v ) takes the form

Original: [Equation]
Revised: [Equation]

Original: which has nothing to do with i 1 , j 1 , m , and n .
Revised: which has nothing to do with l , u , and v .

Original: Eq.
Revised: Eq.

Original: (4) reveals a key characteristic of local mutualsimilarity in a smooth window: the kernel weight is a constant regardless of y .
Revised: (4) reveals a key characteristic of local mutualsimilarity in a smooth window: the kernel weight is a constant regardless of y .

Original: Therefore, the kernelfundamentally can ﬁlter out noise from encoder features with an ‘average’ kernel.[Figure]
Revised: Therefore, the kernelfundamentally can filter out noise from encoder features with an ‘average’ kernel.

Original: 𝐻 ×𝑊 ×𝐶 𝐻×𝑊×𝐶 softmaxreshape matrix multiplication 𝑑 𝐾

Original: [Figure]

Original: 𝑑 𝑑 𝐶 → 𝑑 𝐶 →𝑑 2𝐻×2𝑊×𝐶

Original: [Figure]

Original: fusion

Original: [Figure]

Original: The scores are then transformed into upsampling kernel weights after normalization.

Original: In this way each of the 4 upsampling kernels can select some decoder points from the 3 × 3 decoder window, conditioned on mutual similarity, and assigns the value to the corresponding upsampled point.

Original: Note that, in the derivation above the necessary conditions include: i) x is from a local window in thedecoder feature map; ii) a normalization function in the form of
Revised: Note that, in the derivation above the necessary conditions include: i) x is from a local window in thedecoder feature map; ii) a normalization function in the form of

Original: [Figure]
Revised: [Equation]

Original:  Upsampling in a Detail Window Again we consider two  points y i 2 w ,j 2 w and y i 2 p ,j 2 p in Fig.1 inthe encoder feature.
Revised: j h ( x j ) . Upsampling in a Detail Window Again we consider two encoder points at the position m ′ and n ′in Fig.

Original: Ideally y i 2 w ,j 2 w and y i 2 p ,j 2 p should be classiﬁed into the semantic cluster of‘wall’ and ’picture’, respectively.
Revised: Ideally y m ′ and y n ′ should be classified into the semantic cluster of ‘wall’ and ’picture’,respectively.

Original: Taking y i 2 w ,j 2 w as an example, following our assumption, it is182 more similar to points of the ‘wall’ cluster rather than the ‘picture’ cluster.
Revised: Taking y m ′ as an example, following our assumption, it is more similar to points ofthe ‘wall’ cluster rather than the ‘picture’ cluster.

Original: From Eq.
Revised: From Eq.

Original: (4), we cantell that sim( x(cid:98) i 22 (cid:99) + mw , (cid:98) j 22 (cid:99) + n w , y i 2 w ,j 2 w )is larger than sim( x (cid:98)
Revised: (4), we can tell that sim( x m + a , y m ′ )

Original: i 22 (cid:99) + mp , (cid:98) j 22 (cid:99) +n p , y i 2 w ,j 2 w ) , wherex (cid:98) i 22 (cid:99) + mw , (cid:98) j 22 (cid:99) + n w andx (cid:98) i 22 (cid:99) + mp , (cid:98) j 22 (cid:99) + n p belong to the ‘wall’cluster and the ‘picture’ cluster,respectively.
Revised: islarger than sim( x m + p , y m ′ ) , where a and p are the offsets in the window such that m + a and m + pare within the ‘wall’ and the ‘picture’ cluster, respectively.

Original: Therefore, after computing similarity scores and normalization, one can acquire a kernelwith signiﬁcantly larger weights on points with the semantic cluster of ‘wall’ than that of ‘picture’,i.e. , w i 2 w ,j 2 w ,m w ,n w >> w i 2 w ,j 2 w ,m p ,n p .
Revised: Therefore, after computing similarityscores and normalization, one can acquire a kernel with significantly larger weights on points with thesemantic cluster of ‘wall’ than that of ‘picture’, i.e. , W m ′ ,a >> W m ′ ,p .

Original: After applying the kernel to the corresponding window,the upsampled point at ( i 2 w , j 2 w ) will be revalued and assigned to the semantic cluster of ‘wall’.
Revised: After applying the kernel tothe corresponding window, the upsampled point at m ′ will be revalued and assigned to the semanticcluster of ‘wall’.

Original: Similarly, the upsampled point at ( i 2 p , j 2 p ) will be assigned into ‘picture’. 
Revised: Similarly, the upsampled point at n ′ will be assigned into ‘picture’. normalization 𝑑 𝐾 !

Revised: 𝑑 kernels Encoder feature

Revised: Decoder feature 2𝐻×2𝑊×𝐶 𝐻×𝑊×𝐶

Revised: P r e - p r o ce ss i ng ( 𝑑=𝐶 ) 𝐺 𝐵 𝐼 Pre-processing

Revised: Computing similarity 2𝐻×2𝑊×𝐶

Original:  Note that, in Eq.
Revised: Upsampledfeature Kernel generation matrix multiplication linear embedding gating mechanism Feature assembly ( 𝑑<𝐶 ) ( 𝑑<𝐶 ) Note that, in Eq.

Original: (4) we have no constraint on y .
Revised: (4) we have no constraint on y .

Original: But here in a detail window, y as an encoderfeature vector can play a vital role for designating correct point afﬁliation.
Revised: But here in a detail window, y as an encoderfeature vector can play a vital role for designating correct point affiliation.

Original: Next in our concreteimplementation, we discuss how to appropriately use y in the similarity function.
Revised: Next in our concreteimplementation, we discuss how to appropriately use y in the similarity function.

Original: 4.2 SAPA: Similarity-Aware Point Afﬁliation
Revised: 4.2 SAPA: Similarity-Aware Point Affiliation

Original: Here we embody Eq.
Revised: Here we embody Eq.

Original: (3) by choosing appropriate similarity and normalization functions.
Revised: (3) by investigating different similarity and normalization functions.

Original: Normalization Function Though we do not constrain h ( x ) in theory, in reality it must be carefullychosen.
Revised: Normalization Function Though we do not constrain h ( x ) in theory, in reality it must be carefullychosen.

Original: For example, to avoid zero division, the scope for the choice of h ( x ) is narrowed.
Revised: For example, to avoid zero division, the scope for the choice of h ( x ) is narrowed.

Original: Followingexisting practices [1, 2, 3], we choose h ( x ) =
Revised: Followingexisting practices [1, 2, 3], we choose h ( x ) =

Original: e x by default, which is equivalent to softmaxnormalization.
Revised: e x by default, which is equivalent to softmaxnormalization.

Original: We also test some other h ( x ) ’s, such as h ( x ) =
Revised: We also test some other h ( x ) ’s, such as h ( x ) =

Original: ReLU ( x ) , h ( x ) = sigmoid ( x ) , andh ( x ) = softplus ( x ) .
Revised: ReLU ( x ) , h ( x ) = sigmoid ( x ) , andh ( x ) = softplus ( x ) .

Original: Their performance comparisons will be given in ablation studies.
Revised: Their performance comparisons will be given in ablation studies.

Original: Similarity Function Wefocus on the choices of similarity functions and study three types offunctions:
Revised: Similarity Function We study three types of similarity functions:

Original: • Inner-product similarity: sim( x , y ) =
Revised: • Inner-product similarity: sim( x , y ) =

Original: x T y ,
Revised: x T y ,

Original: • (Low-rank) bilinear similarity [22]: sim( x , y ) =
Revised: • (Low-rank) bilinear similarity [22]: sim( x , y ) =

Original: x T P T x P y y ,
Revised: x T P T x P y y ,

Original: • Gated (low-rank) bilinear similarity: sim( x , y ) =
Revised: • Gated (low-rank) bilinear similarity: sim( x , y ) =

Original: g x T P T x P xy y + (1 − g ) xT P T x
Revised: g x T P T x P xy y + (1 − g ) x

Original:  P xx x ,
Revised: T P T x P xx x ,

Original: where P x ∈ R d × C , P y ∈ R d × C , P xy ∈ R d × C , and P xx ∈ R d × C are the linear projection matrices,d is the embedding dimension, and g ∈ (0 , 1) is a gate unit learned by linear projection.
Revised: where P x ∈ R d × C , P y ∈ R d × C , P xy ∈ R d × C , and P xx ∈ R d × C are the linear projection matrices,d is the embedding dimension, and g ∈ (0 , 1) is a gate unit learned by linear projection.

Revised: We model theprojection matrices to be low-rank, i.e. , d < C , to reduce the number of parameters and computationalcomplexity.

Original: The gatemodulated bilinear similarity is designed to further ﬁlter out the encoder noise.
Revised: The gate-modulated bilinear similarity is designed to further filter out the encoder noise.

Original: The gate is generatedby learning a linear projection matrix used to project the decoder feature X to a single-channel mask,and then it is normalized to (0 , 1) by sigmoid function.
Revised: The gate is generated by learning a linear projection matrix used to project the decoder feature Xto a single-channel mask, and then it is normalized to (0 , 1) by sigmoid function.

Original: Then, we have Y =
Revised: Finally, we have Y =

Original: G Y
Revised: G Y

Original: + (1 − G )
Revised: + (1 − G )

Original: X(nearest neighbor interpolation is used for matching the resolution),where G is the matrix form of the
Revised: X (nearest neighbor interpolation is used for matching the resolution),

Original: [Table]
Revised: where

Original:  gate unit.
Revised: G is the matrix form of the gate unit.

Original: The use of the gate implies we reduce noise in some spatial regions by replacing encoderfeatures Y with decoder features X .
Revised: The use of the gate implies we reduce noise in some spatialregions by replacing encoder features Y with decoder features X .

Original: Its vector-form explains that in the area of noise it tends to switchto self-similarity mode.
Revised: Its vector form explains that in thearea of noise it tends to switch to the self-similarity mode.

Original: We will prove the effectiveness of the gating mechanism by comparing thegate-modulated bilinear similarity with a baseline without the gating mechanism.
Revised: We will prove the effectiveness of thegating mechanism by comparing the gate-modulated bilinear similarity with a baseline without thegating mechanism.

Revised: In practice, encoder and decoder features may have different data distributions, which is not suitableto compare similarity directly.

Revised: Therefore we apply LayerNorm for both encoder and decoderfeatures before similarity computation.

Revised: Indeed we observe that the loss does not decrease withoutnormalization.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, our implementation is similar to the previous dynamic upsampling operators,which ﬁrst generates the upsampling kernels and then assembles the decoder feature conditionedon the kernels.
Revised: 3, our implementation is similar to the previous dynamic upsampling operators,which first generates the upsampling kernels and then assembles the decoder feature conditioned onthe kernels.

Original: We highlight the kernel generation module.
Revised: We highlight the kernel generation module.

Original: By setting a kernel size of K , for eachencoder feature point, we compute the similarity scores between this point and each  K × K neighborin the decoder.
Revised: By setting a kernel size of K , for each encoder [Table] feature point, we compute the similarity scores between this point and each of K × K neighbors inthe decoder.

Original: Then, the softmax normalization is applied to generate the upsampling kernels.
Revised: Then, the softmax normalization is applied to generate the upsampling kernels.

Original: SAPA is lightweight and can even work without additional parameters (with inner-product similarity).
Revised: SAPAis lightweight and can even work without additional parameters (with inner-product similarity).

Original: To intuitively understand its lightweight property, we compare the computational complexity andnumber of parameters of different dynamic upsampling operators in Table 1.For example, when
Revised: Tointuitively understand its lightweight property, we compare the computational complexity and numberof parameters of different dynamic upsampling operators in Table 1.

Original:  C = 256 and d = 64 , the FLOPs are H ∗ W ∗ 199 K , H ∗ W ∗ 17 M , H ∗ W ∗ 28 K , and H ∗ W ∗ 228 Kfor CARAFE, IndexNet, A2U, and SAPA-B, respectively.
Revised: For example, when C = 256and d = 64 , the FLOPs are H ∗ W ∗ 199 K , H ∗ W ∗ 17 M , H ∗ W ∗ 28 K , and H ∗ W ∗ 228 K for CARAFE, IndexNet, A2U [Table] and SAPA-B, respectively.

Original: We visualize the feature maps of upsampling kernels and upsampled features in Fig.
Revised: We visualize the feature maps of upsampling kernels and upsampled features in Fig.

Original: Our upsampling kernels show more favorable responses than other upsampling operators, with weightshighlighted on boundaries and noise suppressed in regions, which visually supports our propositionand is a concrete embodiment of Eq.
Revised: Our upsampling kernels show more favorable responses than other upsampling operators, with weightshighlighted on boundaries and noise suppressed in regions, which visually supports our propositionand is a concrete embodiment of Eq.

Original:  Experiments
Revised:  Experiments

Original: We ﬁrst focus our experiments on semantic segmentation to justify the effectiveness of SAPA.
Revised: We first focus our experiments on semantic segmentation to justify the effectiveness of SAPA.

Original: We thenshowcase its universality across three additional dense prediction tasks, including object detection,depth estimation, and image matting.
Revised: We thenshowcase its universality across three additional dense prediction tasks, including object detection,depth estimation, and image matting.

Original: All our experiments are run on a server with 8 NVIDIA GeForce
Revised: All our experiments are run on a server with 8 NVIDIA GeForce

Original: RTX 3090 GPUs.
Revised: RTX 3090 GPUs.

Original: 5.1 Data Sets, Metrics, Baselines, and Protocols
Revised: 5.1 Data Sets, Metrics, Baselines, and Protocols

Original: For semantic segmentation, we conduct experiments on the ADE20K dataset [4] and report the mIoUmetric.
Revised: For semantic segmentation, we conduct experiments on the ADE20K dataset [4] and report the mIoUmetric.

Original: Three strong transformer-based models are adopted as the baselines, including SegFormerB1 [5], MaskFormer-Swin-Base [6] and Mask2Former-Swin-Base [7].
Revised: Three strong transformer-based models are adopted as the baselines, including SegFormerB1 [5], MaskFormer-Swin-Base [6] and Mask2Former-Swin-Base [7].

Original: All training settings andimplementation details are kept the same as the original papers.
Revised: All training settings andimplementation details are kept the same as the original papers.

Original: We only modify the upsamplingstages with speciﬁc upsampling operators.
Revised: We only modify the upsamplingstages with specific upsampling operators.

Original: For object detection, we use the MS COCO [8] dataset, which involves 80 object categories.
Revised: For object detection, we use the MS COCO [8] dataset, which involves 80 object categories.

Original: We use
Revised: We use

Original: AP as the evaluation metric.
Revised: AP as the evaluation metric.

Original: Faster RCNN [23] with ResNet-50 [24] is adopted as the baseline.
Revised: Faster RCNN [23] with ResNet-50 [24] is adopted as the baseline.

Original: Weuse mmdetection [25] and follow its training conﬁgurations. [Table] [Table] CARAFE [1]
Revised: Weuse mmdetection [25] and follow its training configurations.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table] SAPA- [Table] For depth estimation, we use NYU Depth V2 dataset [10] and its default train/test split.
Revised: For depth estimation, we use NYU Depth V2 dataset [10] and its default train/test split.

Original: We choose
Revised: We choose

Original: BTS [9] with ResNet-50 as the baseline and follow its training conﬁgurations.
Revised: BTS [9] with ResNet-50 as the baseline and follow its training configurations.

Original: The inlier measureδ 1 < 1 .
Revised: The inlier measureδ 1 < 1 .

Original: 25 and RMSE are reported as the evaluation metrics.
Revised: 25 and RMSE are reported as the evaluation metrics.

Original: We replace all upsampling stages butthe last one for SAPA, due to no available high-res feature map for the last stage.
Revised: We replace all upsampling stages butthe last one for SAPA, due to no available high-res feature map for the last stage.

Original: For image matting, we train the model on the Adobe Image Matting dataset [11] and report fourmetrics on the Composition-1k test set, including SAD, MSE, Gradient, and Connectivity [26]. 
Revised: For image matting, we train the model on the Adobe Image Matting dataset [11] and report fourmetrics on the Composition-1k test set, including SAD, MSE, Gradient, and Connectivity [26]. A2U [Table] [Table]

Original: A2Umatting [3] is adopted as the baseline.
Revised: matting [3] is adopted as the baseline.

Original: We use the code provided by the authors and follow the sametraining settings as in the original paper.
Revised: We use the code provided by the authors and follow the sametraining settings as in the original paper.

Original: 5.2 Main Results
Revised: 5.2 Main Results

Original: We compare SAPA and its variants against different upsampling operators on the three strongsegmentation baselines.
Revised: We compare SAPA and its variants against different upsampling operators on the three strongsegmentation baselines.

Original: Results are shown in Table 2, from which we can see that SAPA consistentlyoutperforms other upsampling operators.
Revised: Results are shown in Table 2, from which we can see that SAPA consistentlyoutperforms other upsampling operators.

Original: Note that SAPA can work well even without parametersand achieves the best performance with only few additional #Params and #Flops.
Revised: Note that SAPA can work well even without parametersand achieves the best performance with only few additional #Params and #Flops.

Original: Results on other three dense prediction tasks are shown in Table 3.
Revised: Results on other three dense prediction tasks are shown in Table 3.

Original: SAPA outperforms otherupsampling operators on depth estimation and image matting, but falls behind CARAFE on objectdetection.
Revised: SAPA outperforms otherupsampling operators on depth estimation and image matting, but falls behind CARAFE on objectdetection.

Original: One plausible explanation is that the demand of details in object detection is low (Section 6presents a further in-depth analysis).
Revised: One plausible explanation is that the demand of details in object detection is low (Section 6presents a further in-depth analysis).

Original: In detail-sensitive tasks like image matting, SAPA signiﬁcantlyoutperforms CARAFE.
Revised: In detail-sensitive tasks like image matting, SAPA significantlyoutperforms CARAFE.

Original: Qualitative comparisons are provided in supplementary material.
Revised: Qualitative comparisons are provided in supplementary material.

Original: 5.3 Ablation Study
Revised: 5.3 Ablation Study

Original: Here we conduct ablation studies to compare the choices of similarity function and normalizationfunction, the effect of different kernel sizes, and the number of embedding dimension.
Revised: Here we conduct ablation studies to compare the choices of similarity function and normalizationfunction, the effect of different kernel sizes, and the number of embedding dimension.

Original: For the defaultsetting, we use the bilinear similarity function, apply the normalization function h ( x ) =
Revised: For the defaultsetting, we use the bilinear similarity function, apply the normalization function h ( x ) =

Original: e x , set thekernel size K = 5 and the embedding dimension d = 64 .
Revised: e x , set thekernel size K = 5 and the embedding dimension d = 64 .

Original: Quantitative results are shown in Table 4.[Table]
Revised: Quantitative results are shown in Table 4.

Original: Similarity Function We investigate three types of similarity function aforementioned and anadditional ‘plain addition’ baseline.
Revised: Similarity Function We investigate three types of similarity function aforementioned and anadditional ‘plain addition’ baseline.

Original: It ablates the gating mechanism from the gated bilinear similarity.
Revised: It ablates the gating mechanism from the gated bilinear similarity.

Original: Among them, gated bilinear similarity generates the best performance, which highlights thecomplementarity of semantic regions and boundary details in kernel generation.
Revised: Among them, gated bilinear similarity generates the best performance, which highlights thecomplementarity of semantic regions and boundary details in kernel generation.

Revised: [Table]

Original: Normalization Function We also investigate different normalization functions.
Revised: Normalization Function We also investigate different normalization functions.

Revised: ‘None’ indicatesno normalization is used.

Original: Among validatedfunctions, h ( x ) =
Revised: Among validated functions, h ( x ) =

Original: e x works the best, which means normalization matters.
Revised: e x works the best, which meansnormalization matters.

Original: However, we thinknormalization mainly affects the optimization of the network, the other three have to play with theepsilon trick to prevent zero division, which may affect performance.
Revised: The other three have to play with the epsilon trick to prevent zero division.

Original: Kernel Size
Revised: Kernel Size

Original: The kernel size controls the region that each point in the upsampled feature can attendto.
Revised: The kernel size controls the region that each point in the upsampled feature can attendto.

Original: Results show that, compared with a large kernel, a small local window is sufﬁcient to distinguishbetween regions and edges.
Revised: Results show that, compared with a large kernel, a small local window is sufficient to distinguishthe semantic clusters.

Original: Embedding Dimension We further study the inﬂuence of embedding dimension in the range of 16 ,32 , 64 , and 128 .
Revised: Embedding Dimension We further study the influence of embedding dimension in the range of 16 ,32 , 64 , and 128 .

Original: Interestingly, results suggest that SAPA is not sensitive to the embedding dimension.
Revised: Interestingly, results suggest that SAPA is not sensitive to the embedding dimension.

Original: This also veriﬁes that SAPA extracts existing knowledge rather than learn unknown knowledge.
Revised: This also verifies that SAPA extracts existing knowledge rather than learn unknown knowledge.

Original:  Discussion
Revised:  Discussion

Original: Understanding SAPA from a Backward Perspective We have explained SAPA in the forwardpass, here we further discuss how it may work during training.
Revised: Understanding SAPA from a Backward Perspective We have explained SAPA in the forwardpass, here we further discuss how it may work during training.

Original: In fact, originally the model doesnot know to which the semantic cluster an encoder point should belong.
Revised: In fact, originally the model doesnot know to which the semantic cluster an encoder point should belong.

Original: The working mechanismof SAPA assigns each encoder feature point a possibility to choose a cluster.
Revised: The working mechanismof SAPA assigns each encoder feature point a possibility to choose a cluster.

Original: During training, theground truths produce gradients, thus changes the assignment possibility.
Revised: During training, theground truths produce gradients, thus changes the assignment possibility.

Original: In SAPA, for every encoderpoint, the semantic clusters in the corresponding local window in decoder features serve as implicitlabels and cooperate with the ground truths.
Revised: In SAPA, for every encoderpoint, the semantic clusters in the corresponding local window in decoder features serve as implicitlabels and cooperate with the ground truths.

Original: The correct cluster is the positive label, while the wrongone is negative.
Revised: The correct cluster is the positive label, while the wrongone is negative.

Original: In the preliminary stage of training, if an encoder feature point is more similar to awrong cluster, it will be punished by gradients and engender large losses, and vice versa.
Revised: In the preliminary stage of training, if an encoder feature point is more similar to awrong cluster, it will be punished by gradients and engender large losses, and vice versa.

Original: Therefore,the encoder feature points can gradually learn its afﬁliation.
Revised: Therefore,the encoder feature points can gradually learn its affiliation.

Original: We ﬁnd this process is fast by visualizingthe epoch-to-epoch feature maps.
Revised: We find this process is fast by visualizingthe epoch-to-epoch feature maps.

Original: Limitations compared with CARAFE CARAFE is a purely semantic-driven upsampling operatorable to mend semantic information with a single-input ﬂow.
Revised: Limitations compared with CARAFE CARAFE is a purely semantic-driven upsampling operatorable to mend semantic information with a single-input flow.

Original: Such a mechanism in CARAFE bringsadvantages in smoothing semantic regions.
Revised: Such a mechanism in CARAFE bringsadvantages in smoothing semantic regions.

Original: E.g. , we observe it mends holes in a continuous region.
Revised: E.g. , we observe it mends holes in a continuous region.

Original: However, due to its single-input ﬂow, it cannot compensate the details lost in downsampling.
Revised: However, due to its single-input flow, it cannot compensate the details lost in downsampling.

Original: Our
Revised: Our

Original: SAPA, by contrast, mainly aims to compensate details such as textures and boundaries.
Revised: SAPA, by contrast, mainly aims to compensate details such as textures and boundaries.

Original: SAPAcharacters in two aspects: semantic preservation and detail delineation.
Revised: SAPAcharacters in two aspects: semantic preservation and detail delineation.

Original: However, as Eq.
Revised: However, as Eq.

Original: (4) suggests,we do not add any semantic mending mechanism in SAPA.
Revised: (4) suggests,we do not add any semantic mending mechanism in SAPA.

Original: This explains why SAPA is worsethan CARAFE on object detection, because detection has less demand for details but more forregional integrity.
Revised: This explains why SAPA is worsethan CARAFE on object detection, because detection has less demand for details but more forregional integrity.

Original: In short, CARAFE highlights semantic mending , while SAPA highlights semanticpreserving and detail delineation .
Revised: In short, CARAFE highlights semantic mending , while SAPA highlights semanticpreserving and detail delineation .

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we introduce similarity-aware point afﬁliation, i.e. , SAPA.
Revised: In this paper, we introduce similarity-aware point affiliation, i.e. , SAPA.

Original: It not only indicates alightweight but effective upsampling operator suitable for tasks like semantic segmentation, but alsoexpresses a high-level concept that characterizes feature upsampling.
Revised: It not only indicates alightweight but effective upsampling operator suitable for tasks like semantic segmentation, but alsoexpresses a high-level concept that characterizes feature upsampling.

Original: SAPA can serve as an universalsubstitution for conventional upsampling operators.
Revised: SAPA can serve as an universalsubstitution for conventional upsampling operators.

Original: Experiments show the effectiveness of SAPAand also indicate its limitation: it is more suitable for tasks that favor detail delineation.
Revised: Experiments show the effectiveness of SAPAand also indicate its limitation: it is more suitable for tasks that favor detail delineation.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] Please see Section 6.
Revised: [Yes] Please see Section 6.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b)
Revised: [N/A]

Original:  Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi
Revised: g., with respect to the random seed after running experi-

Original: -ments multiple times)?
Revised: ments multiple times

Original: [No] Our method generates stable results.
Revised: [No] Our method generates stable results.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] For each of the datasets used, we cite the creators, as shown in Section 5.1.
Revised: [Yes] For each of thedatasets used, we cite the creators, as shown in Section 5.1.

Original: b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] The data used in our work is open source and can be used for academic research.
Revised: [Yes] The data used in our work is opensource and can be used for academic research.

Original: c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: Our code for this work will appear in the supplementary material
Revised: Our code for this work will appear in the supplementary material.

Original: d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: The data used in our work is open source and can be used for
Revised: The data used in our work is open source and can be used for

Original: academic research
Revised: academic research.

Original: .(e ) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: The data do not contain personally identiﬁable
Revised: The data do not contain personally identifiable

Original: information or offensive content.
Revised: information or offensive content.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]