Original: Deep Fourier Up-Sampling
Revised: Deep Fourier Up-Sampling

Original: Abstract
Revised: Abstract

Original: Existing convolutional neural networks widely adopt spatial down-/up-sampling for multi-scale modeling.
Revised: Existing convolutional neural networks widely adopt spatial down-/up-sampling for multi-scale modeling.

Original: However, spatial up-sampling operators ( e.g. , interpolation, transposed convolution, and un-pooling) heavily depend on local pixel attention, incapably exploring the global dependency.
Revised: However, spatial up-sampling operators ( e.g. , interpolation, transposed convolution, and un-pooling) heavily depend on local pixel attention, incapably exploring the global dependency.

Original: In contrast, the Fourier domain obeys the nature of global modeling according to the spectral convolution theorem.
Revised: In contrast, the Fourier domain obeys the nature of global modeling according to the spectral convolution theorem.

Original: Unlike the spatial domain that performs up-sampling with the property of local similarity, up-sampling in the Fourier domain is more challenging as it does not follow such a local property.
Revised: Unlike the spatial domain that performs up-sampling with the property of local similarity, up-sampling in the Fourier domain is more challenging as it does not follow such a local property.

Original: In this study, we propose a theoretically sound Deep Fourier Up-Sampling (FourierUp) to solve these issues.
Revised: In this study, we propose a theoretically sound Deep Fourier UpSampling (FourierUp) to solve these issues.

Original: We revisit the relationships between spatial and Fourier domains and reveal the transform rules on the features of different resolutions in the Fourier domain, which provide key insights for FourierUp’s designs.
Revised: We revisit the relationships between spatial and Fourier domains and reveal the transform rules on the features of different resolutions in the Fourier domain, which provide key insights for FourierUp’s designs.

Original: FourierUp as a generic operator consists of three key components: 2D discrete Fourier transform, Fourier dimension increase rules, and 2D inverse Fourier transform, which can be directly integrated with existing networks.
Revised: FourierUp as a generic operator consists of three key components: 2D discrete Fourier transform, Fourier dimension increase rules, and 2D inverse Fourier transform, which can be directly integrated with existing networks.

Original: Extensive experiments across multiple computer vision tasks, including object detection, image segmentation, image de-raining, image dehazing, and guided image super-resolution, demonstrate the consistent performance gains obtained by introducing our FourierUp.
Revised: Extensive experiments across multiple computer vision tasks, including object detection, image segmentation, image de-raining, image dehazing, and guided image superresolution, demonstrate the consistent performance gains obtained by introducing our FourierUp.

Revised: Code is available at https://manman1995.github.io/ .

Original:  Introduction
Revised:  Introduction

Original: Spatial down-/up-sampling has been widely used in convolutional neural networks for multi-scale modeling.
Revised: Spatial down-/up-sampling has been widely used in convolutional neural networks for multi-scale modeling.

Original: For example, U-Net [1], a variation of encoder-decoder, employs pooling layers to reduce the feature resolution in the encoder and then recovers the resolution using up-sampling operations in the decoder.
Revised: For example, U-Net [1], a variation of encoder-decoder, employs pooling layers to reduce the feature resolution in the encoder and then recovers the resolution using up-sampling operations in the decoder.

Original: In addition, the feature pyramid [2–5] and image pyramid [6–9] driven multi-scale neural networks rely on the down-/up-sampling operation to obtain multi-scale property and improve modeling capability.
Revised: In addition, the feature pyramid [2–5] and image pyramid [6–9] driven multi-scale neural networks rely on the down-/up-sampling operation to obtain multi-scale property and improve modeling capability.

Original: However, spatial up-sampling operators ( e.g. , interpolation, transposed convolution, and un-pooling) heavily depend on local pixel attention, and thus cannot explore the global dependency that is indispensable for many computer vision tasks [10, 11, 1, 12– 20].
Revised: However, spatial up-sampling operators ( e.g. , interpolation, transposed convolution, and un-pooling) heavily depend on local pixel attention, and thus cannot explore the global dependency that is indispensable for many computer vision tasks [10, 11, 1, 12– 20].

Original: According to the spectral convolution theorem, the Fourier domain obeys the nature of global modeling, providing an alternative solution for multi-scale modeling.
Revised: According to the spectral convolution theorem, the Fourier domain obeys the nature of global modeling, providing an alternative solution for multi-scale modeling.

Original: However, unlike the spatial domain with local similarity property, up-sampling in the Fourier domain is more challenging as it does not follow such a local property.
Revised: However, unlike the spatial domain with local similarity property, up-sampling in the Fourier domain is more challenging as it does not follow such a local property.

Original: The observation encourages us to explore deep Fourier up-sampling.
Revised: The observation encourages us to explore deep Fourier up-sampling.

Original: Recent studies have explored information interaction in both spatial and Fourier domains.
Revised: Recent studies have explored information interaction in both spatial and Fourier domains.

Original: FFC [21], for instance, replaces the conventional convolution with a spatial-Fourier interaction, which consists of a spatial (or local) path that performs conventional convolution on a portion of input feature channels and a spectral (or global) path that operates in the Fourier domain.
Revised: FFC [21], for instance, replaces the conventional convolution with a spatial-Fourier interaction, which consists of a spatial (or local) path that performs conventional convolution on a portion of input feature channels and a spectral (or global) path that operates in the Fourier domain.

Original: DFT [22] devises a
Revised: DFT [22] devises a

Original: Residual Fast Fourier Transform Block to integrate both low- and high-frequency residual information by performing the interaction between a regular spatial residual stream and a channel-wise Fourier transform stream.
Revised: Residual Fast Fourier Transform Block to integrate both low- and high-frequency residual information by performing the interaction between a regular spatial residual stream and a channel-wise Fourier transform stream.

Original: However, the aforementioned methodologies only interact at a single resolution scale, and the spatial-Fourier interaction potential of multiple scales in the Fourier domain has not been investigated.
Revised: However, the aforementioned methodologies only interact at a single resolution scale, and the spatial-Fourier interaction potential of multiple scales in the Fourier domain has not been investigated.

Original: The key to solving this problem lies in how to implement deep Fourier up-sampling for multi-scale Fourier pattern modeling.
Revised: The key to solving this problem lies in how to implement deep Fourier up-sampling for multi-scale Fourier pattern modeling.

Original: Challenges.
Revised: Challenges.

Original: Owing to the local similarity and cross-scale position invariant properties of the spatial domain, the various spatial up-sampling operations including transposed convolution, un-pooling, and interpolation techniques are capable of using the pixel neighboring relationship to interpolate the unknown pixel values at local regions, increasing the spatial resolution of the features, as shown in Figure 1(a).
Revised: Owing to the local similarity and cross-scale position invariant properties of the spatial domain, the various spatial up-sampling operations including transposed convolution, un-pooling, and interpolation techniques are capable of using the pixel neighboring relationship to interpolate the unknown pixel values at local regions, increasing the spatial resolution of the features, as shown in Figure 1(a).

Original: In contrast to the spatial domain, the Fourier domain does not share the same scaleinvariant property and local texture similarity, and hence cannot implement up-sampling using the same techniques as the spatial domain, as illustrated in Figure 1(b).
Revised: In contrast to the spatial domain, the Fourier domain does not share the same scaleinvariant property and local texture similarity, and hence cannot implement up-sampling using the same techniques as the spatial domain, as illustrated in Figure 1(b).

Original: Solutions.
Revised: Solutions.

Original: In this paper, we wish to investigate the possibility of devising a reliable up-sampling in the
Revised: In this paper, we wish to investigate the possibility of devising a reliable up-sampling in the

Original: Fourier domain in a theoretical sound manner.
Revised: Fourier domain in a theoretical sound manner.

Original: To answer this question, we first revisit the relationship between spatial and Fourier domains, revealing the transform rules on the features of different resolutions in the Fourier domain (see Section 3.1 and Section 3.2).
Revised: To answer this question, we first revisit the relationship between spatial and Fourier domains, revealing the transform rules on the features of different resolutions in the Fourier domain (see Section 3.1 and Section 3.2).

Original: On the basis of the above rules, we propose a theoretically feasible Deep Fourier Up-Sampling (FourierUp).
Revised: On the basis of the above rules, we propose a theoretically feasible Deep Fourier Up-Sampling (FourierUp).

Original: Specifically, we develop three variants (Periodic Padding, Area Interpolation/Cropping and Corner Interpolation) of FourierUp (see Section 3.3), as illustrated in Figure 1(c).
Revised: Specifically, we develop three variants (Periodic Padding, Area Interpolation/Cropping and Corner Interpolation) of FourierUp (see Section 3.3), as illustrated in Figure 1(c).

Original: Each variant consists of three key components: 2D discrete Fourier transform, Fourier dimension increase rules, and 2D inverse Fourier transform.
Revised: Each variant consists of three key components: 2D discrete Fourier transform, Fourier dimension increase rules, and 2D inverse Fourier transform.

Original: FourierUp is a generic operator that can be directly integrated with existing networks.
Revised: FourierUp is a generic operator that can be directly integrated with existing networks.

Original: Extensive experiments on multiple computer vision tasks, including object detection, image segmentation, image de-raining, image dehazing, and guided image super-resolution, demonstrate the consistent performance gains obtained by introducing our FourierUp.
Revised: Extensive experiments on multiple computer vision tasks, including object detection, image segmentation, image de-raining, image dehazing, and guided image super-resolution, demonstrate the consistent performance gains obtained by introducing our FourierUp.

Original: We believe that the proposed FourierUp could refresh the neural network designs where the spatial and Fourier information interaction at only a single resolution scale are mainstream choices.
Revised: We believe that the proposed FourierUp could refresh the neural network designs where the spatial and Fourier information interaction at only a single resolution scale are mainstream choices.

Original: Contributions.
Revised: Contributions.

Original: We propose Deep Fourier Up-Sampling, a novel method that enables the integration of the features of different resolutions in the Fourier domain.
Revised: We propose Deep Fourier Up-Sampling, a novel method that enables the integration of the features of different resolutions in the Fourier domain.

Original: This is the first thorough effort to explore the Fourier up-sampling for multi-scale modeling. 2)
Revised: This is the first thorough effort to explore the Fourier up-sampling for multi-scale modeling. 2)

Original: The proposed FourierUp is a generic operator that can be directly integrated with the existing networks in a plug-and-play manner.
Revised: The proposed FourierUp is a generic operator that can be directly integrated with the existing networks in a plug-and-play manner.

Original: Equipped with the theoretically sound FourierUp, we show that existing networks could achieve consistent performance improvement across multiple computer vision tasks.
Revised: Equipped with the theoretically sound FourierUp, we show that existing networks could achieve consistent performance improvement across multiple computer vision tasks.

Original:  Related Work
Revised:  Related Work

Original: Spatial Up-Sampling.
Revised: Spatial Up-Sampling.

Original: Convolutional neural networks with spatial down-/up-sampling have become the de facto structures in many computer vision tasks [23–31].
Revised: Convolutional neural networks with spatial down-/up-sampling have become the de facto structures in many computer vision tasks [23–31].

Original: Typically, U-Net [1] builds multi-scale feature maps using the encoder with down-sampling and then utilizes the up-sampling operation to fuse the multi-scale features in the decoder.
Revised: Typically, U-Net [1] builds multi-scale feature maps using the encoder with down-sampling and then utilizes the up-sampling operation to fuse the multi-scale features in the decoder.

Original: Additionally, the feature pyramid [2–5] and image pyramid[6–9] are commonly used to obtain the multi-scale property in neural networks [6–9].
Revised: Additionally, the feature pyramid [2–5] and image pyramid[6–9] are commonly used to obtain the multi-scale property in neural networks [6–9].

Original: Among them, spatial up-sampling plays a significant role in multi-scale modeling.
Revised: Among them, spatial up-sampling plays a significant role in multi-scale modeling.

Original: However, existing upsampling operations only work in the spatial domain and current studies rarely explore the potential ( e.g. , the global modeling capability) of up-sampling in the frequency domain.
Revised: However, existing upsampling operations only work in the spatial domain and current studies rarely explore the potential ( e.g. , the global modeling capability) of up-sampling in the frequency domain.

Original: Spatial-Fourier Interaction.
Revised: Spatial-Fourier Interaction.

Original: Recently, several studies attempt to employ Fourier transform in deep models [32–35, 21].
Revised: Recently, several studies attempt to employ Fourier transform in deep models [32–35, 21].

Original: Some of these efforts use discrete Fourier transform to transfer the spatial features to the Fourier domain and then use frequency information to improve the performance of particular tasks [32, 34].
Revised: Some of these efforts use discrete Fourier transform to transfer the spatial features to the Fourier domain and then use frequency information to improve the performance of particular tasks [32, 34].

Original: Another line is to use convolution theorem to speed up the models, such as using fast Fourier transform (FFT) [35, 21].
Revised: Another line is to use convolution theorem to speed up the models, such as using fast Fourier transform (FFT) [35, 21].

Original: For example, FFC [21] replaces the convolution with the spatial-Fourier interaction.
Revised: For example, FFC [21] replaces the convolution with the spatial-Fourier interaction.

Original: The work proposed in [36] uses spectral pooling to reduce feature resolution by truncating the frequency domain representation.
Revised: The work proposed in [36] uses spectral pooling to reduce feature resolution by truncating the frequency domain representation.

Original: However, all the techniques only interact with each other at a single spatial resolution and have not explored the interaction potential at multiple resolutions in both spatial and frequency domains as performing the frequency up-sampling is non-trivial.
Revised: However, all the techniques only interact with each other at a single spatial resolution and have not explored the interaction potential at multiple resolutions in both spatial and frequency domains as performing the frequency up-sampling is non-trivial.

Original: As a tentative exploration, we study the relationship between the spatial domain and
Revised: As a tentative exploration, we study the relationship between the spatial domain and

Original: Fourier domain and reveal the transform rules over the feature of different resolutions in the Fourier domain.
Revised: Fourier domain and reveal the transform rules over the feature of different resolutions in the Fourier domain.

Original: This delivers the underlying insights for the designs of multi-scale Fourier modeling patterns, which has the potential of versatility for different network architectures.
Revised: This delivers the underlying insights for the designs of multi-scale Fourier modeling patterns, which has the potential of versatility for different network architectures.

Original:  Deep Fourier Up-Sampling
Revised:  Deep Fourier Up-Sampling

Original: We first explore the mapping relationship between the spatial and Fourier domains, and then present three Deep Fourier up-sampling variants, including i) periodic padding of magnitude and phase, ii) area up-sampling of magnitude and phase, and iii) corner interpolation of magnitude and phase, based on the explored transform rules.
Revised: We first explore the mapping relationship between the spatial and Fourier domains, and then present three Deep Fourier up-sampling variants, including i) periodic padding of magnitude and phase, ii) area up-sampling of magnitude and phase, and iii) corner interpolation of magnitude and phase, based on the explored transform rules.

Original: In terms of the first two variants, we provide two theorems and their proofs as follows while the third is reported in supplementary materials.
Revised: In terms of the first two variants, we provide two theorems and their proofs as follows while the third is reported in supplementary materials.

Original: Definitions.
Revised: Definitions.

Original: f ( x, y ) ∈ R 2 M × 2 N is the 2 -times zero-inserted up-sampled version of g (
Revised: f ( x, y ) ∈ R 2 M × 2 N is the 2 -times zero-inserted up-sampled version of g (

Original: x, y ) ∈ R M × N in spatial domain, and F ( u, v ) ∈
Revised: x, y ) ∈ R M × N in spatial domain, and F ( u, v ) ∈

Original: R 2 M × 2 N , G ( u, v ) ∈ R M × N denote their Fourier transforms.
Revised: R 2 M × 2 N , G ( u, v ) ∈ R M × N denote their Fourier transforms.

Original: H ( u, v ) ∈ R 2 M × 2 N is the 2 -
Revised: H ( u, v ) ∈ R 2 M × 2 N is the 2 -

Original: Theorem-1.
Revised: Theorem-1.

Original: F ( u, v ) =
Revised: F ( u, v ) =

Original: F ( u + M, v ) =
Revised: F ( u + M, v ) =

Original: F ( u, v + N ) =
Revised: F ( u, v + N ) =

Original: F ( u + M, v + N ) and G ( u, v )
Revised: F ( u + M, v + N ) and G ( u, v )

Original: =
Revised: =

Original: F ( u,v )
Revised: F ( u,v )

Original: where u = 0 , 1 , 2 , . .
Revised: where u = 0 , 1 , 2 , . .

Original: , N − 1 and v = 0 , 1 , 2 , . .
Revised: , N − 1 and v = 0 , 1 , 2 , . .

Original: , M − 1 .
Revised: , M − 1 .

Original: F ( u, v ) is exactly the periodic padding of
Revised: F ( u, v ) is exactly the periodic padding of

Original: G ( u, v ) where G ( u, v ) is exactly the quarter of F ( u, v ) with the value being 14 times decay.
Revised: G ( u, v ) where G ( u, v ) is exactly the quarter of F ( u, v ) with the value being 14 times decay.

Original: Theorem-2.
Revised: Theorem-2.

Original: H (2 u, 2 v ) =
Revised: H (2 u, 2 v ) =

Original: H (2 u + 1 , 2 v ) = H (2 u, 2 v + 1) =
Revised: H (2 u + 1 , 2 v ) = H (2 u, 2 v + 1) =

Original: H (2 u + 1 , 2 v + 1) =
Revised: H (2 u + 1 , 2 v + 1) =

Original: G ( u, v ) with u = 0 , 1 , . .
Revised: G ( u, v ) with u = 0 , 1 , . .

Original: , M − 1 and v = 0 , 1 , . . .
Revised: , M − 1 and v = 0 , 1 , . . .

Original: , N − 1 and
Revised: , N − 1 and

Original: [Equation]
Revised: [Equation]

Original: where A ( x, y ) = 

Revised: where A ( x, y ) =

Original: + e jπxM
Revised: + e jπxM

Original: + e jπyN + e jπ
Revised: + e jπyN + e jπ

Original: ( xM + yN ) and x = 0 , 1 , . . .
Revised: ( xM + yN ) and x = 0 , 1 , . . .

Original: M − 1 , y = 0 , 1 , . . .
Revised: M − 1 , y = 0 , 1 , . . .

Original: N − 1 .
Revised: N − 1 .

Original: Theorem-3.
Revised: Theorem-3.

Original: Suppose the corner interpolated F corG ( u, v ) of the Fourier map G ( u, v )
Revised: Suppose the corner interpolated F corG ( u, v ) of the Fourier map G ( u, v )

Original: ∈ R M × N , it holds that the inverse Fourier transform f corg ( x, y ) of F corG ( u, v )
Revised: ∈ R M × N , it holds that the inverse Fourier transform f corg ( x, y ) of F corG ( u, v )

Original: [Equation]
Revised: [Equation]

Original: where x ′ =
Revised: where x ′ =

Original: 2 x and y ′
Revised: 2 x and y ′

Original: =
Revised: =

Original: 2 y , x = 0 , 1 , . . .
Revised: 2 y , x = 0 , 1 , . . .

Original: , M − 1 and y = 0 , 1 , . . .
Revised: , M − 1 and y = 0 , 1 , . . .

Original: , N − 1 .
Revised: , N − 1 .

Original: 3.1 Proof-1 of Theorem-1: Periodic Padding of Magnitude and Phase
Revised: 3.1 Proof-1 of Theorem-1: Periodic Padding of Magnitude and Phase

Original: Note that f
Revised: Note that f

Original: ( x, y ) ∈
Revised: ( x, y ) ∈

Original: R 2 M × 2 N is up-sampled over g (
Revised: R 2 M × 2 N is up-sampled over g (

Original: x, y ) ∈ R M × N by a factor of 2 .
Revised: x, y ) ∈ R M × N by a factor of 2 .

Original: The relationship between g ( x, y ) and f ( x, y ) can be written as
Revised: The relationship between g ( x, y ) and f ( x, y ) can be written as

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where m = 1 , 2 , . .
Revised: where m = 1 , 2 , . .

Original: , M − 1 and n = 1 , 2 , . . .
Revised: , M − 1 and n = 1 , 2 , . . .

Original: , N − 1 , the Fourier transform F ( u, v ) of f ( x, y ) is expressed as
Revised: , N − 1 , the Fourier transform F ( u, v ) of f ( x, y ) is expressed as

Original: [Equation]
Revised: [Equation]

Original: Then, we show the periodicity of F ( u, v ) ∈
Revised: Then, we show the periodicity of F ( u, v ) ∈

Original: R 2 M × 2 N with M and N .
Revised: R 2 M × 2 N with M and N .

Original: It means F ( u, v ) =
Revised: It means F ( u, v ) =

Original: F ( u +
Revised: F ( u +

Original: M, v ) =
Revised: M, v ) =

Original: F ( u, v + N ) =
Revised: F ( u, v + N ) =

Original: F ( u + M, v + N ) with u = 0 , 1 , 2 , . .
Revised: F ( u + M, v + N ) with u = 0 , 1 , 2 , . .

Original: , N − 1 and v = 0 , 1 , 2 , . .
Revised: , N − 1 and v = 0 , 1 , 2 , . .

Original: , M − 1 .
Revised: , M − 1 .

Original: We take the F ( u, v ) =
Revised: We take the F ( u, v ) =

Original: F ( u + M, v ) for example and recall Eq.
Revised: F ( u + M, v ) for example and recall Eq.

Original: (4) as
Revised: (4) as

Original: [Equation]
Revised: [Equation]

Original: where e − 2 πjx = 1 for any integer x .
Revised: where e − 2 πjx = 1 for any integer x .

Original: Similarly, we can proof the periodicity of F ( u, v ) as well.
Revised: Similarly, we can proof the periodicity of F ( u, v ) as well.

Original: Based on the above proof, the DFT of g ( x, y ) can be formulated as:
Revised: Based on the above proof, the DFT of g ( x, y ) can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: Revising Eq.
Revised: Revising Eq.

Original: (4), we can figure out that G ( u, v ) =
Revised: (4), we can figure out that G ( u, v ) =

Original: F ( u,v ).
Revised: F ( u,v ).

Original: 3.2 Proof-2 of Theorem-2: Area Interpolation of Magnitude and Phase
Revised: 3.2 Proof-2 of Theorem-2: Area Interpolation of Magnitude and Phase

Original: The 2D Inverse Discrete Fourier transform (IDFT) of G ( u, v ) can be written as:
Revised: The 2D Inverse Discrete Fourier transform (IDFT) of G ( u, v ) can be written as:

Original: [Equation]
Revised: [Equation]

Original: We up-sample G ( u, v ) with a size of M × N to get H ( u, v ) with a size of 2 M × 2 N .
Revised: We up-sample G ( u, v ) with a size of M × N to get H ( u, v ) with a size of 2 M × 2 N .

Original: Specifically, the area interpolation shown in Figure 3(b) is used for interpolation and then the interpolated pixels are the same as the original pixel in the 2 × 2 local regions.
Revised: Specifically, the area interpolation shown in Figure 3(b) is used for interpolation and then the interpolated pixels are the same as the original pixel in the 2 × 2 local regions.

Original: Namely, H (2 u, 2 v ) =
Revised: Namely, H (2 u, 2 v ) =

Original: H (2 u + 1 , 2 v ) =
Revised: H (2 u + 1 , 2 v ) =

Original: H (2 u, 2 v + 1) = H (2 u + 1 , 2 v + 1) =
Revised: H (2 u, 2 v + 1) = H (2 u + 1 , 2 v + 1) =

Original: G ( u, v ) with u = 0 , 1 , . .
Revised: G ( u, v ) with u = 0 , 1 , . .

Original: , M − 1 and v = 0 , 1 , . . .
Revised: , M − 1 and v = 0 , 1 , . . .

Original: , N − 1 .
Revised: , N − 1 .

Original: Similar to Eq.
Revised: Similar to Eq.

Original: (4), we can infer
Revised: (4), we can infer

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: N − 1 (cid:88) v =
Revised: N − 1 (cid:88) v =

Original: H (2 u, 2 v ) e j 2 π ( 2 ux 2 M + 2 vy 2 N ) + 4 MN
Revised: H (2 u, 2 v ) e j 2 π ( 2 ux 2 M + 2 vy 2 N ) + 4 MN

Original: M − 1 (cid:88) u =
Revised: M − 1 (cid:88) u =

Original: N − 1 (cid:88) v =
Revised: N − 1 (cid:88) v =

Original: H (2 u + 1 , 2 v ) e j 2 π ( (2 u +1)
Revised: H (2 u + 1 , 2 v ) e j 2 π ( (2 u +1)

Original: x 2 M + 2 vy 2 N ) + 4 MN
Revised: x 2 M + 2 vy 2 N ) + 4 MN

Original: M − 1 (cid:88) u =
Revised: M − 1 (cid:88) u =

Original: N − 1 (cid:88) v =
Revised: N − 1 (cid:88) v =

Original: H (2 u, 2 v + 1) e j 2 π ( 2 ux 2 M + (2 v +1)
Revised: H (2 u, 2 v + 1) e j 2 π ( 2 ux 2 M + (2 v +1)

Original: y 2 N ) + 4 MN
Revised: y 2 N ) + 4 MN

Original: M − 1 (cid:88) u =
Revised: M − 1 (cid:88) u =

Original: N − 1 (cid:88) v =
Revised: N − 1 (cid:88) v =

Original: H (2 u + 1 , 2 v + 1) e j 2 π ( (2 u +1) x 2 M + (2 v +1)
Revised: H (2 u + 1 , 2 v + 1) e j 2 π ( (2 u +1) x 2 M + (2 v +1)

Original: y 2 N )
Revised: y 2 N )

Original: = 4 MN
Revised: = 4 MN

Original: M − 1 (cid:88) u =
Revised: M − 1 (cid:88) u =

Original: N − 1 (cid:88) v =
Revised: N − 1 (cid:88) v =

Original: G ( u,v ) e j 2 π ( uxM + vyN ) [1 + e jπxM + e jπyN + e jπ ( xM + yN ) ] .
Revised: G ( u,v ) e j 2 π ( uxM + vyN ) [1 + e jπxM + e jπyN + e jπ ( xM + yN ) ] .

Original: Similarly, we can write g ( x, y ) as g ( x, y ) =
Revised: Similarly, we can write g ( x, y ) as g ( x, y ) =

Original: MN
Revised: MN

Original: M − 1 (cid:88) u =
Revised: M − 1 (cid:88) u =

Original: N − 1 (cid:88) v =
Revised: N − 1 (cid:88) v =

Original: G ( u, v ) e j 2 π ( uxM + vyN ) .
Revised: G ( u, v ) e j 2 π ( uxM + vyN ) .

Original: Recalling Eq.
Revised: Recalling Eq.

Original: (8) and Eq.
Revised: (8) and Eq.

Original: (10), we can infer
Revised: (10), we can infer

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: (10) where x = 0 , 1 , . . .
Revised: (10) where x = 0 , 1 , . . .

Original: M − 1 , y = 0 , 1 , . . .
Revised: M − 1 , y = 0 , 1 , . . .

Original: N − 1 .
Revised: N − 1 .

Original: We remark 1+ e jπxM
Revised: We remark 1+ e jπxM

Original: + e jπyN + e jπ
Revised: + e jπyN + e jπ

Original: ( xM
Revised: ( xM

Original: [Equation]
Revised: [Equation]

Original: We can find that the variable x shares the same operations as the variable y .
Revised: We can find that the variable x shares the same operations as the variable y .

Original: For brevity, we only take the operation of variable x as example
Revised: For brevity, we only take the operation of variable x as example

Original: [Equation]
Revised: [Equation]

Original: Equally, we have
Revised: Equally, we have

Original: [Equation]
Revised: [Equation]

Original: We prove that the partial derivative of | A ( x, y ) | on both x and y is negative for x ∈ [0 , M − 1] and y ∈ [0 , N − 1] .
Revised: We prove that the partial derivative of | A ( x, y ) | on both x and y is negative for x ∈ [0 , M − 1] and y ∈ [0 , N − 1] .

Original: Besides, we have
Revised: Besides, we have

Original: [Equation]
Revised: [Equation]

Original: That is to say, the intensity drops from the side to the center, shown in Figure 4.
Revised: That is to say, the intensity drops from the side to the center, shown in Figure 4.

Original: Specifically, the intensity drops to zero at the position of x = M or y = N .
Revised: Specifically, the intensity drops to zero at the position of x = M or y = N .

Original: def DFU_Padding(X): # X: input with shape [N, C, H, W] # A and P are the amplitude and phase A, P = FFT(X)
Revised: def DFU_Padding(X): # X: input with shape [N, C, H, W] # A and P are the amplitude and phase A, P = FFT(X)

Original: # Fourier up-sampling transform
Revised: # Fourier up-sampling transform

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Return Y #[N, C, 2H, 2W] def DFU_AreaInterpolation(X): # X: input with shape [N, C, H, W] # A and P are the amplitude and phase A, P = FFT(X)
Revised: Return Y #[N, C, 2H, 2W] def DFU_AreaInterpolation(X): # X: input with shape [N, C, H, W] # A and P are the amplitude and phase A, P = FFT(X)

Original: # Fourier up-sampling transform rules A_aip
Revised: # Fourier up-sampling transform rules A_aip

Original: = Area-Interpolation(A)
Revised: = Area-Interpolation(A)

Original: P_aip =
Revised: P_aip =

Original: Area-Interpolation(P)
Revised: Area-Interpolation(P)

Original: A_aip
Revised: A_aip

Original: = Convs_1x1(A_aip)
Revised: = Convs_1x1(A_aip)

Original: P_aip = Convs_1x1(P_aip)
Revised: P_aip = Convs_1x1(P_aip)

Original: # Inverse Fourier transform Y =
Revised: # Inverse Fourier transform Y =

Original: iFFT(A_aip, P_aip)
Revised: iFFT(A_aip, P_aip)

Original: #Area Cropping Y =
Revised: #Area Cropping Y =

Original: Area-Cropping(Y)
Revised: Area-Cropping(Y)

Original: Y =
Revised: Y =

Original: Resize(Y)
Revised: Resize(Y)

Original: Return Y #[N, C, 2H, 2W]
Revised: Return Y #[N, C, 2H, 2W]

Original: 3.3 Architectural Design
Revised: 3.3 Architectural Design

Original: Recall the Theorem-1 and Theorem-2 , we propose two deep
Revised: Recall the Theorem-1 and Theorem-2 , we propose two deep

Original: Fourier up-sampling variants: Periodic Padding and Area
Revised: Fourier up-sampling variants: Periodic Padding and Area

Original: Periodic Padding Up-Sampling.
Revised: Periodic Padding Up-Sampling.

Original: The pseudo-code of Periodic
Revised: The pseudo-code of Periodic

Original: Padding Up-Sampling is shown in the left of Figure 2.
Revised: Padding Up-Sampling is shown in the left of Figure 2.

Original: Given an image X ∈ R H
Revised: Given an image X ∈ R H

Original: × W × C , we first adopt the Fourier transform
Revised: × W × C , we first adopt the Fourier transform

Original: FFT(X) to obtain its amplitude component A and phase component P .
Revised: FFT(X) to obtain its amplitude component A and phase component P .

Original: We then perform the periodic padding over A and
Revised: We then perform the periodic padding over A and

Original: P two times in both the H and W dimensions, as illustrated in
Revised: P two times in both the H and W dimensions, as illustrated in

Original: Figure 3(a).
Revised: Figure 3(a).

Original: The padded A _ pep and P _ pep are then fed into two independent convolution module with 1 × 1 kernel and followed by the inverse Fourier transform iFFT( . ) to project the padded ones back to spatial domain.
Revised: The padded A _ pep and P _ pep are then fed into two independent convolution module with 1 × 1 kernel and followed by the inverse Fourier transform iFFT( . ) to project the padded ones back to spatial domain.

Original: Area Interpolation-Cropping Up-Sampling.
Revised: Area Interpolation-Cropping Up-Sampling.

Original: The pseudo-code of Area interpolation-Cropping Up-Sampling is shown in the right of Figure 2.
Revised: The pseudo-code of Area interpolation-Cropping Up-Sampling is shown in the right of Figure 2.

Original: We first conduct the Area interpolation over the phase and amplitude by 2 × 2 area interpolation with the same pixel, as illustrated in Figure 3(b).
Revised: We first conduct the Area interpolation over the phase and amplitude by 2 × 2 area interpolation with the same pixel, as illustrated in Figure 3(b).

Original: We then employ the inverse Fourier transform to project the interpolated ones back to spatial domain.
Revised: We then employ the inverse Fourier transform to project the interpolated ones back to spatial domain.

Original: As described in Section 3.2, the inverse spatial representation will be periodic while the pixel value will be decay.
Revised: As described in Section 3.2, the inverse spatial representation will be periodic while the pixel value will be decay.

Original: The degree of decay of the pixel increases when the pixel is closer to the center.
Revised: The degree of decay of the pixel increases when the pixel is closer to the center.

Original: To better maintain the information, we perform the area cropping operation (detailed in Figure 1) in the four corners with the W 2 × Hsize and then merge them together as a whole at spatial dimension, finally resize them to the size of 2 H × 2 W .
Revised: To better maintain the information, we perform the area cropping operation (detailed in Figure 1) in the four corners with the W 2 × Hsize and then merge them together as a whole at spatial dimension, finally resize them to the size of 2 H × 2 W .

Original: Note that albeit being designed on the basis of strict theories, both constructed spectral up-sampling modules contain certain approximations, like a learnable 1 × 1 convolution operator instead of strictly 1 / 4 as described in Theorem-1 of main manuscript, and an approximation cropping to preserve the map corners instead of accurate A mapping as proved in Theorem-2 of main manuscript and
Revised: Note that albeit being designed on the basis of strict theories, both constructed spectral up-sampling modules contain certain approximations, like a learnable 1 × 1 convolution operator instead of strictly 1 / 4 as described in Theorem-1 of main manuscript, and an approximation cropping to preserve the map corners instead of accurate A mapping as proved in Theorem-2 of main manuscript and

Original: Theorem-3 of supplementary materials.
Revised: Theorem-3 of supplementary materials.

Original: Such strategy makes the proposed modules able to be more easily implemented and more flexibly represent real data spectral structures.
Revised: Such strategy makes the proposed modules able to be more easily implemented and more flexibly represent real data spectral structures.

Original: It is worth noting that this should be the first attempt for constructing easy equitable spectral upsampling modules, and hope it would inspire more effective and rational ones from more spectral perspectives.
Revised: It is worth noting that this should be the first attempt for constructing easy equitable spectral upsampling modules, and hope it would inspire more effective and rational ones from more spectral perspectives.

Original:  Experiments
Revised:  Experiments

Original: To demonstrate the efficacy of our proposed deep Fourier up-sampling, we conduct extensive experiments on multiple computer vision tasks, including object detection, image segmentation, image de-raining, image dehazing, and guided image super-resolution.
Revised: To demonstrate the efficacy of our proposed deep Fourier up-sampling, we conduct extensive experiments on multiple computer vision tasks, including object detection, image segmentation, image de-raining, image dehazing, and guided image super-resolution.

Original: We provide more experimental results in the supplementary material.
Revised: We provide more experimental results in the supplementary material.

Original: 4.1 Experimental Settings
Revised: 4.1 Experimental Settings

Original: Object Detection.
Revised: Object Detection.

Original: Following [12], the PASCAL VOC 2007 and 2012 training sets [37] are used as training data.
Revised: Following [12], the PASCAL VOC 2007 and 2012 training sets [37] are used as training data.

Original: The PASCAL VOC 2007 testing set is used for evaluations as the ground truth annotations of VOC 2012 testing set are not publicly available.
Revised: The PASCAL VOC 2007 testing set is used for evaluations as the ground truth annotations of VOC 2012 testing set are not publicly available.

Original: We employ the FPN-based Faster
Revised: We employ the FPN-based Faster

Original: RCNN [12] with ResNet50 backbone and YOLO-v3 with Darknet53 [38] as baselines.
Revised: RCNN [12] with ResNet50 backbone and YOLO-v3 with Darknet53 [38] as baselines.

Original: Image Segmentation.
Revised: Image Segmentation.

Original: Following [39, 40], Synapse Dataset and CANDI Dataset are used as the testbed of medical image segmentation.
Revised: Following [39, 40], Synapse Dataset and CANDI Dataset are used as the testbed of medical image segmentation.

Original: We adopt the two representative image segmentation algorithms, U-Net [1] and Att-UNet [41], as the base models.
Revised: We adopt the two representative image segmentation algorithms, U-Net [1] and Att-UNet [41], as the base models.

Original: Image De-raining.
Revised: Image De-raining.

Original: Following [42], we choose two widely-used standard benchmark datasets, including Rain200H and Rain200L, for evaluations.
Revised: Following [42], we choose two widely-used standard benchmark datasets, including Rain200H and Rain200L, for evaluations.

Original: we employ two representative de-raining methods, LPNet with up-sampling [11] and PReNet without up-sampling [42], as baselines.
Revised: we employ two representative de-raining methods, LPNet with up-sampling [11] and PReNet without up-sampling [42], as baselines.

Original: Image Dehazing.
Revised: Image Dehazing.

Original: Following [10], we employ RESIDE[43] dataset [44] for evaluations.
Revised: Following [10], we employ RESIDE[43] dataset [44] for evaluations.

Original: We also use two different network designs AODNet [45] without up-sampling operator and MSBDN [10] with up-sampling operator, for validation.
Revised: We also use two different network designs AODNet [45] without up-sampling operator and MSBDN [10] with up-sampling operator, for validation.

Original: Guided Image Super-resolution.
Revised: Guided Image Super-resolution.

Original: Following [14, 46], we adopt the pan-sharpening, the representative task of guided image super-resolution, for evaluations.
Revised: Following [14, 46], we adopt the pan-sharpening, the representative task of guided image super-resolution, for evaluations.

Original: The WorldView II, WorldView III, and
Revised: The WorldView II, WorldView III, and

Original: GaoFen2 in [14, 46] are used for evaluations.
Revised: GaoFen2 in [14, 46] are used for evaluations.

Original: We employ two different network designs for validation, including PANNET [47] without up-sampling operator and DCFNET [48] with up-sampling operator.
Revised: We employ two different network designs for validation, including PANNET [47] without up-sampling operator and DCFNET [48] with up-sampling operator.

Original: Several widely-used image quality assessment (IQA) metrics are employed to evaluate the performance, including the relative dimensionless global error in synthesis (ERGAS) [49], the peak signal-to-noise ratio (PSNR), the spectral angle mapper (SAM) [50], DSC, and HD95.
Revised: Several widely-used image quality assessment (IQA) metrics are employed to evaluate the performance, including the relative dimensionless global error in synthesis (ERGAS) [49], the peak signal-to-noise ratio (PSNR), the spectral angle mapper (SAM) [50], DSC, and HD95.

Original: 4.2 Implementation Details
Revised: 4.2 Implementation Details

Original: Regarding the above competitive baselines, they can be divided into two categories: one with spatial up-sampling ([1], Att-UNet [41], DCFNET [48], LPNet [11], MSBDN [10]) and another one without spatial up-sampling (PReNet [42], AODNet[45], PANNET [47]).
Revised: Regarding the above competitive baselines, they can be divided into two categories: one with spatial up-sampling ([1], Att-UNet [41], DCFNET [48], LPNet [11], MSBDN [10]) and another one without spatial up-sampling (PReNet [42], AODNet[45], PANNET [47]).

Original: The purpose of the exploration on the baselines without spatial up-sampling is to show the versatility of our FourierUp for different network structures.
Revised: The purpose of the exploration on the baselines without spatial up-sampling is to show the versatility of our FourierUp for different network structures.

Original: Different from directly replacing the spatial up-sampling with the FourierUp in the baselines with spatial up-sampling, we need to encapsulate the FourierUp for the baselines without spatial up-sampling, in which a down-sampling operation is introduced to first reduce the resolution of features.
Revised: Different from directly replacing the spatial up-sampling with the FourierUp in the baselines with spatial up-sampling, we need to encapsulate the FourierUp for the baselines without spatial up-sampling, in which a down-sampling operation is introduced to first reduce the resolution of features.

Original: We provide the detailed structures of the encapsulated FourierUp and the baselines with the FourierUp in the Figure 5 and supplementary material.
Revised: We provide the detailed structures of the encapsulated FourierUp and the baselines with the FourierUp in the Figure 5 and supplementary material.

Original: For the baselines with spatial up-sampling, we perform the comparison over four configurations:
Revised: For the baselines with spatial up-sampling, we perform the comparison over four configurations:

Original: 1) Original : the baseline without any changes; 2) FourierUp-AreaUp : replacing the original model’s spatial up-sampling with the union of the Area-Interpolation variant of our FourierUp and the spatial up-sampling itself; 3) FourierUp-Padding : replacing the original model’s spatial up-sampling operator with the union of the Periodic-Padding variant of our FourierUp and the spatial up-sampling itself; 4) Spatial-Up : replacing the variants of FourierUp in the settings of 2) / 3) with the spatial up-sampling.
Revised: 1) Original : the baseline without any changes; 2) FourierUp-AreaUp : replacing the original model’s spatial up-sampling with the union of the Area-Interpolation variant of our FourierUp and the spatial up-sampling itself; 3) FourierUp-Padding : replacing the original model’s spatial up-sampling operator with the union of the Periodic-Padding variant of our FourierUp and the spatial up-sampling itself; 4) Spatial-Up : replacing the variants of FourierUp in the settings of 2) / 3) with the spatial up-sampling.

Original: For a fair comparison, we use the same number of trainable parameters as 2) / 3) .
Revised: For a fair comparison, we use the same number of trainable parameters as 2) / 3) .

Original: For the baselines without spatial up-sampling, we perform the comparison over four configurations:
Revised: For the baselines without spatial up-sampling, we perform the comparison over four configurations:

Original: 1) Original : the baseline without any changes; 2) FourierUp-AreaUp : replacing the original model’s convolution with the encapsulated FourierUp that is equipped with the Area-Interpolation variant; 3) FourierUp-Padding : replacing the original model’s convolution with the encapsulated FourierUp that is equipped with the Periodic-Padding variant; 4) Spatial-Up : replacing the the encapsulated FourierUp of the settings of 2) / 3) with the spatial up-sampling.
Revised: 1) Original : the baseline without any changes; 2) FourierUp-AreaUp : replacing the original model’s convolution with the encapsulated FourierUp that is equipped with the Area-Interpolation variant; 3) FourierUp-Padding : replacing the original model’s convolution with the encapsulated FourierUp that is equipped with the Periodic-Padding variant; 4) Spatial-Up : replacing the the encapsulated FourierUp of the settings of 2) / 3) with the spatial up-sampling.

Original: For a fair comparison, we use the same number of trainable parameters as 2) / 3) .
Revised: For a fair comparison, we use the same number of trainable parameters as 2) / 3) .

Original: 4.3 Comparison and Analysis
Revised: 4.3 Comparison and Analysis

Original: Quantitative Comparison.
Revised: Quantitative Comparison.

Original: We perform the model performance comparison over different configurations, as described in implementation details.
Revised: We perform the model performance comparison over different configurations, as described in implementation details.

Original: The quantitative results are presented in Tablesto 5 where the best and second best results are highlighted in bold and Underline.
Revised: The quantitative results are presented in Tablesto 5 where the best and second best results are highlighted in bold and Underline.

Original: From the results, by integrating with our proposed two FourierUp variants, we can observe performance gain against the baselines across all the datasets in all tested tasks, suggesting the effectiveness of our approach.
Revised: From the results, by integrating with our proposed two FourierUp variants, we can observe performance gain against the baselines across all the datasets in all tested tasks, suggesting the effectiveness of our approach.

Original: For example, for the PReNet of Table 1, “FourierUp-padding” and “FourierUp-AreaUp” obtain 0.83dB/0.65dB and 2.1dB/1.9dB PSNR gains than the “Original”, 0.52dB/0.34dB and 1.7dB/1.5dB
Revised: For example, for the PReNet of Table 1, “FourierUp-padding” and “FourierUp-AreaUp” obtain 0.83dB/0.65dB and 2.1dB/1.9dB PSNR gains than the “Original”, 0.52dB/0.34dB and 1.7dB/1.5dB

Original: PSNR gains than “Spatial-Up” on the Rain200H and Rain200L datasets, respectively.
Revised: PSNR gains than “Spatial-Up” on the Rain200H and Rain200L datasets, respectively.

Original: Such results validate the effectiveness of our proposed FourierUp.
Revised: Such results validate the effectiveness of our proposed FourierUp.

Original: The corresponding visualization consistently supports the analysis in Figure 6, where the FourierUp is capable of better maintaining the details.
Revised: The corresponding visualization consistently supports the analysis in Figure 6, where the FourierUp is capable of better maintaining the details.

Original: Qualitative Comparison.
Revised: Qualitative Comparison.

Original: Due to the limited space, we only report the visual results of the deraining/dehazing task in Figures 6 and 7 that can more clearly show the effectiveness of FourierUp.
Revised: Due to the limited space, we only report the visual results of the deraining/dehazing task in Figures 6 and 7 that can more clearly show the effectiveness of FourierUp.

Original: More results can be found in the supplementary materials.
Revised: More results can be found in the supplementary materials.

Original: As shown, integrating the FourierUp with the original baseline achieves more visually pleasing results.
Revised: As shown, integrating the FourierUp with the original baseline achieves more visually pleasing results.

Original: Specifically, zooming-in the red box region of Figures 6 and 7, the model equipped with the FourierUp is capable of better recovering the texture details while removing the rain/hazy effect.
Revised: Specifically, zooming-in the red box region of Figures 6 and 7, the model equipped with the FourierUp is capable of better recovering the texture details while removing the rain/hazy effect.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  Limitations
Revised:  Limitations

Original: First, the more comprehensive experiments on broader computer vision tasks ( e.g. , image de-noising and image de-blurring) have not been explored.
Revised: First, the more comprehensive experiments on broader computer vision tasks ( e.g. , image de-noising and image de-blurring) have not been explored.

Original: Second, the deep Fourier Up-sampling integrated with spatial up-sampling will increase the model parameter numbers.
Revised: Second, the deep Fourier Up-sampling integrated with spatial up-sampling will increase the model parameter numbers.

Original: This is negligible at the significant performance gain at fewer parameter increase.
Revised: This is negligible at the significant performance gain at fewer parameter increase.

Original: Note that, the focus of this work is beyond designing a plug-and-play module to integrate it into existing networks for further performance gain.
Revised: Note that, the focus of this work is beyond designing a plug-and-play module to integrate it into existing networks for further performance gain.

Original: This work also provides a powerful scale change choice of the up-sampling operator pool when developing a new model from start.
Revised: This work also provides a powerful scale change choice of the up-sampling operator pool when developing a new model from start.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we have proposed a deep Fourier up-sampling to explore the possibility of the upsampling in the Fourier domain, which provides the key insight for the multi-scale Fourier pattern modeling.
Revised: In this paper, we have proposed a deep Fourier up-sampling to explore the possibility of the upsampling in the Fourier domain, which provides the key insight for the multi-scale Fourier pattern modeling.

Original: We theoretically demonstrate that our designs of Fourier up-sampling are feasible.
Revised: We theoretically demonstrate that our designs of Fourier up-sampling are feasible.

Original: It is appealing that the proposed FourierUp is a generic operator, thus being directly integrated with existing networks.
Revised: It is appealing that the proposed FourierUp is a generic operator, thus being directly integrated with existing networks.

Original: Extensive experiments demonstrate the effectiveness of our method.
Revised: Extensive experiments demonstrate the effectiveness of our method.

Original: We believe the FourierUp has the potential to advance broader computer vision tasks, e.g. , image/video superresolution and image/video in-painting.
Revised: We believe the FourierUp has the potential to advance broader computer vision tasks, e.g. , image/video superresolution and image/video in-painting.

Original: Broader Impact
Revised: Broader Impact

Original: Our work shows the promising capability of up-sampling in the Fourier domain for computer vision algorithms through two novel designs with theoretical proofs.
Revised: Our work shows the promising capability of up-sampling in the Fourier domain for computer vision algorithms through two novel designs with theoretical proofs.

Original: Using our deep Fourier up-sampling with negligible computational cost will improve the performance of neural networks and facilitate the development of AI in real-world applications.
Revised: Using our deep Fourier up-sampling with negligible computational cost will improve the performance of neural networks and facilitate the development of AI in real-world applications.

Original: However, the efficacy of our method may raise potential concerns when it is improperly used.
Revised: However, the efficacy of our method may raise potential concerns when it is improperly used.

Original: For example, the safety of the applications of our method in real-world applications may not be guaranteed.
Revised: For example, the safety of the applications of our method in real-world applications may not be guaranteed.

Original: We will investigate the robustness and effectiveness of our method in broader real-world applications.
Revised: We will investigate the robustness and effectiveness of our method in broader real-world applications.

Original: Acknowledgements
Revised: Acknowledgements

Original: We gratefully acknowledge the support of the Major Key Project of PCL (PCL2021A12), MindSpore,
Revised: We gratefully acknowledge the support of the Major Key Project of PCL (PCL2021A12), MindSpore,

Original: CANN, the JKW Research Funds under Grant 20-163-14-LZ-001-004-01 and Ascend AI Processor used for this research.
Revised: CANN, the JKW Research Funds under Grant 20-163-14-LZ-001-004-01 and Ascend AI Processor used for this research.

Original: Chongyi Li and Chen Change Loy are supported by the RIE2020 Industry
Revised: Chongyi Li and Chen Change Loy are supported by the RIE2020 Industry

Original: Alignment Fund – Industry Collaboration Projects (IAF-ICP)
Revised: Alignment Fund – Industry Collaboration Projects (IAF-ICP)

Original: Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
Revised: Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] , see Section 5.
Revised: [Yes] , see Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] , see Section 6.
Revised: [Yes] , see Section 6.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] , See Section 3.1 and Section 3.2.
Revised: [Yes] , See Section 3.1 and Section 3.2.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] , See Section 3.1 and Section 3.2.
Revised: [Yes] , See Section 3.1 and Section 3.2.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e
Revised: [Yes] (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how they were chosen)?
Revised: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] , the effect of random seed could almost be negligible since we set the same initiation seed during experiments.
Revised: [No] , the effect of random seed could almost be negligible since we set the same initiation seed during experiments.

Original: Reproducibility can be guaranteed.
Revised: Reproducibility can be guaranteed.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] , the data creators claim that they allow and encourage the data for scientific research.
Revised: [Yes] , the data creators claim that they allow and encourage the data for scientific research.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] , We train and test the framework using the publicly accessible datasets.
Revised: [N/A] , We train and test the framework using the publicly accessible datasets.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]