Original: Deep Fourier Up-Sampling
Revised: Deep Fourier Up-Sampling

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Existing convolutional neural networks widely adopt spatial down-/up-sampling formulti-scale modeling.
Revised: Existing convolutional neural networks widely adopt spatial down-/up-sampling for multi-scale modeling.

Original: However, spatial up-sampling operators ( e.g. , interpolation,transposed convolution, and un-pooling) heavily depend on local pixel attention,incapably exploring the global dependency.
Revised: However, spatial up-sampling operators ( e.g. , interpolation, transposed convolution, and un-pooling) heavily depend on local pixel attention, incapably exploring the global dependency.

Original: In contrast, the Fourier domain is inaccordance with the nature of global modeling according to the spectral convolutiontheorem.
Revised: In contrast, the Fourier domain obeys the nature of global modeling according to the spectral convolution theorem.

Original: Unlike the spatial domain thateasily performs up-sampling with theproperty of local similarity, up-sampling in the Fourier domain is more challengingas it does not follow such a local property.
Revised: Unlike the spatial domain that performs up-sampling with the property of local similarity, up-sampling in the Fourier domain is more challenging as it does not follow such a local property.

Original: In this study, we propose a theoreticallyfeasible Deep Fourier Up-Sampling (FourierUp) to solve these issues.
Revised: In this study, we propose a theoretically sound Deep Fourier Up-Sampling (FourierUp) to solve these issues.

Original: We revisitthe relationships between spatial and Fourier domains and reveal the transformrules on the features of different resolutions in the Fourier domain, which providekey insights for FourierUp’s designs.
Revised: We revisit the relationships between spatial and Fourier domains and reveal the transform rules on the features of different resolutions in the Fourier domain, which provide key insights for FourierUp’s designs.

Original: FourierUp as a generic operator consists ofthree key components: 2D discrete Fourier transform, Fourier dimension increaserules, and 2D inverse Fourier transform, which can be directly integrated withexisting networks.
Revised: FourierUp as a generic operator consists of three key components: 2D discrete Fourier transform, Fourier dimension increase rules, and 2D inverse Fourier transform, which can be directly integrated with existing networks.

Original: Extensive experiments across multiple computer vision tasks,including object detection, image segmentation, image de-raining, image dehazing,and guided image super-resolution, demonstrate the consistent performance gainsobtained by introducing our FourierUp.Code will be publicly available.
Revised: Extensive experiments across multiple computer vision tasks, including object detection, image segmentation, image de-raining, image dehazing, and guided image super-resolution, demonstrate the consistent performance gains obtained by introducing our FourierUp.

Original:  Introduction
Revised:  Introduction

Original: Motivation. The spatial down-/up-sampling has been widely used in convolutional neural networksfor multi-scale modeling.
Revised: Spatial down-/up-sampling has been widely used in convolutional neural networks for multi-scale modeling.

Original: For example, U-Net [1], a variation of encoder-decoder, utilizes poolinglayers to reduce the feature resolution in the encoder and then recovers the resolution using upsampling operations in the decoder.
Revised: For example, U-Net [1], a variation of encoder-decoder, employs pooling layers to reduce the feature resolution in the encoder and then recovers the resolution using up-sampling operations in the decoder.

Original: In addition, the feature pyramid [2–5] and image pyramid [6–9]driven multi-scale neural networks rely on the down-/up-sampling operation to obtain multi-scaleproperty and improve modeling capability.
Revised: In addition, the feature pyramid [2–5] and image pyramid [6–9] driven multi-scale neural networks rely on the down-/up-sampling operation to obtain multi-scale property and improve modeling capability.

Original: However, spatial up-sampling operators ( e.g. , interpolation,transposed convolution and un-pooling) heavily depend on local pixel attention, and thus cannotexplore the global dependency that is indispensable for many computer vision tasks [10, 11, 1, 12].
Revised: However, spatial up-sampling operators ( e.g. , interpolation, transposed convolution, and un-pooling) heavily depend on local pixel attention, and thus cannot explore the global dependency that is indispensable for many computer vision tasks [10, 11, 1, 12– 20].

Original: According to the spectral convolution theorem, the Fourier domain is in accordance with the natureof global modeling, which provides an alternative solution for multi-scale modeling.
Revised: According to the spectral convolution theorem, the Fourier domain obeys the nature of global modeling, providing an alternative solution for multi-scale modeling.

Original: However,unlike the spatial domain with local similarity property, up-sampling in the Fourier domain is morechallenging as it does not follow such a local property.
Revised: However, unlike the spatial domain with local similarity property, up-sampling in the Fourier domain is more challenging as it does not follow such a local property.

Original: This encourages us to explore deep Fourierup-sampling.
Revised: The observation encourages us to explore deep Fourier up-sampling.

Original: Recent studies have exploredthe information interaction in both spatial and Fourier domains.
Revised: Recent studies have explored information interaction in both spatial and Fourier domains.

Original: FFC[13], for instance, replaces the conventional convolution with a spatial-Fourier interaction, whichconsists of a spatial (or local) path that performs conventional convolution on a portion of input featurechannels and a spectral (or global) path that operates in the Fourier domain.
Revised: FFC [21], for instance, replaces the conventional convolution with a spatial-Fourier interaction, which consists of a spatial (or local) path that performs conventional convolution on a portion of input feature channels and a spectral (or global) path that operates in the Fourier domain.

Original: DFT [14] devises a
Revised: DFT [22] devises a

Original: Residual Fast Fourier Transform Block to integrate both low- and high-frequency residual informationby performing the interaction between a regular spatial residual stream and a channel-wise Fourier transform stream.
Revised: Residual Fast Fourier Transform Block to integrate both low- and high-frequency residual information by performing the interaction between a regular spatial residual stream and a channel-wise Fourier transform stream.

Original: However, the aforementioned methodologies only interact at a single resolutionscale, and the spatial-Fourier interaction potential of multiple scales in the Fourier domain has notbeen investigated.
Revised: However, the aforementioned methodologies only interact at a single resolution scale, and the spatial-Fourier interaction potential of multiple scales in the Fourier domain has not been investigated.

Original: The key to solving this problem lies in how to implement deep Fourier up-samplingfor multi-scale Fourier pattern modeling.
Revised: The key to solving this problem lies in how to implement deep Fourier up-sampling for multi-scale Fourier pattern modeling.

Original: Challenges.
Revised: Challenges.

Original: We thus seek a potential solution for multi-scale Fourier pattern modeling by designingdeep Fourier up-sampling.

Original: Owing to the local similarity and cross-scale position invariant propertiesof the spatial domain, the various spatial up-sampling operations including transposed convolution,un-pooling, and interpolation techniques are capable of using the pixel neighboring relationship tointerpolate the unknown pixel values at local regions, increasing the spatial resolution of the features,as shown in Fig.1(a).
Revised: Owing to the local similarity and cross-scale position invariant properties of the spatial domain, the various spatial up-sampling operations including transposed convolution, un-pooling, and interpolation techniques are capable of using the pixel neighboring relationship to interpolate the unknown pixel values at local regions, increasing the spatial resolution of the features, as shown in Figure 1(a).

Original: In contrast to the spatial domain, the Fourier domain does not share the samescale-invariant property and local texture similarity, and hence cannot implement up-sampling usingthe same techniques as the spatial domain, as illustrated in Fig. 1(b).
Revised: In contrast to the spatial domain, the Fourier domain does not share the same scaleinvariant property and local texture similarity, and hence cannot implement up-sampling using the same techniques as the spatial domain, as illustrated in Figure 1(b).

Original: We therefore wonder “Can weprovide a reliable up-sampling in the Fourier domain in a theoretical sound manner?”.

Original: Solutions.
Revised: Solutions.

Revised: In this paper, we wish to investigate the possibility of devising a reliable up-sampling in the

Original:  To answer this question, we first revisit the relationship between spatial and Fourierdomains, revealing the transform rules on the features of different resolutions in the Fourier domain(see Section 3.2 and Section 3.3).
Revised: Fourier domain in a theoretical sound manner. To answer this question, we first revisit the relationship between spatial and Fourier domains, revealing the transform rules on the features of different resolutions in the Fourier domain (see Section 3.1 and Section 3.2).

Original: On the basis of the above rules, we propose a theoretically feasible Deep Fourier Up-Sampling (FourierUp).
Revised: On the basis of the above rules, we propose a theoretically feasible Deep Fourier Up-Sampling (FourierUp).

Original: Specifically, we develop two variants (Periodic Padding and Area Interpolation/Cropping) of FourierUp (see Section 3.4), as illustrated in Fig. 1(c).
Revised: Specifically, we develop three variants (Periodic Padding, Area Interpolation/Cropping and Corner Interpolation) of FourierUp (see Section 3.3), as illustrated in Figure 1(c).

Original: Each variantconsists of three key components: 2D discrete Fourier transform, Fourier dimension increase rules,and 2D inverse Fourier transform.
Revised: Each variant consists of three key components: 2D discrete Fourier transform, Fourier dimension increase rules, and 2D inverse Fourier transform.

Original: FourierUp is a generic operator that can be directly integratedwith existing networks.
Revised: FourierUp is a generic operator that can be directly integrated with existing networks.

Original: Extensive experiments on multiple computer vision tasks, including objectdetection, image segmentation, image de-raining, image dehazing, and guided image super-resolution,demonstrate the consistent performance gains obtained by introducing our FourierUp.
Revised: Extensive experiments on multiple computer vision tasks, including object detection, image segmentation, image de-raining, image dehazing, and guided image super-resolution, demonstrate the consistent performance gains obtained by introducing our FourierUp.

Original: We believethat the proposed FourierUp could refresh the neural network designs where the spatial and Fourierinformation interaction at only a single resolution scale previously reigned supreme.
Revised: We believe that the proposed FourierUp could refresh the neural network designs where the spatial and Fourier information interaction at only a single resolution scale are mainstream choices.

Original: Contributions.
Revised: Contributions.

Original: We proposea novel Deep Fourier Up-Sampling, which enables the integration ofthe features of different resolutions in the Fourier domain.
Revised: We propose Deep Fourier Up-Sampling, a novel method that enables the integration of the features of different resolutions in the Fourier domain.

Original: To the best of our knowledge, this is thefirst thorough effort to explore the Fourier up-sampling for multi-scale modeling. 
Revised: This is the first thorough effort to explore the Fourier up-sampling for multi-scale modeling. 2)

Original: The proposed FourierUp is a generic operator that can be directly integrated with the existing networks, whichprovides flexible plug-and-play.
Revised: The proposed FourierUp is a generic operator that can be directly integrated with the existing networks in a plug-and-play manner.

Original: Equipped with the theoretically feasible FourierUp, the networks  achieve consistent performance improvement across multiple computer vision tasks, which suggeststhe potential of our method for refreshing the neural network designs in the Fourier domain.
Revised: Equipped with the theoretically sound FourierUp, we show that existing networks could achieve consistent performance improvement across multiple computer vision tasks.

Original:  Related Work
Revised:  Related Work

Original: Spatial Up-Sampling.
Revised: Spatial Up-Sampling.

Original: Convolutional neural networks with spatial down-/up-sampling have becomethe de facto structures in many computer vision tasks [15–23].
Revised: Convolutional neural networks with spatial down-/up-sampling have become the de facto structures in many computer vision tasks [23–31].

Original: Typically, U-Net [1] builds multi-scalefeature maps using the encoder with down-sampling and then utilizes the up-sampling operationto fuse the multi-scale features in the decoder.
Revised: Typically, U-Net [1] builds multi-scale feature maps using the encoder with down-sampling and then utilizes the up-sampling operation to fuse the multi-scale features in the decoder.

Original: Additionally, the feature pyramid [2–5] and imagepyramid[6–9] are commonly used to obtain the multi-scale property in neural networks [6–9].
Revised: Additionally, the feature pyramid [2–5] and image pyramid[6–9] are commonly used to obtain the multi-scale property in neural networks [6–9].

Original: Amongthem, spatial up-sampling plays a significant role in multi-scale modeling.
Revised: Among them, spatial up-sampling plays a significant role in multi-scale modeling.

Original: However, existing upsampling operations only work in the spatial domain and current studies rarely explore the potential( e.g. , the global modeling capability) of up-sampling in the frequency domain.
Revised: However, existing upsampling operations only work in the spatial domain and current studies rarely explore the potential ( e.g. , the global modeling capability) of up-sampling in the frequency domain.

Original: Spatial-Fourier Interaction.
Revised: Spatial-Fourier Interaction.

Original: Recently, several studies attempt to employ Fourier transform in deepmodels [24–27, 13].
Revised: Recently, several studies attempt to employ Fourier transform in deep models [32–35, 21].

Original: Some of these efforts use discrete Fourier transform to transfer the spatialfeatures to the Fourier domain and then use frequency information to improve the performance ofparticular tasks [24, 26].
Revised: Some of these efforts use discrete Fourier transform to transfer the spatial features to the Fourier domain and then use frequency information to improve the performance of particular tasks [32, 34].

Original: Another line is to use convolution theorem to speed up the models, such asusing fast Fourier transform (FFT) [27, 13].
Revised: Another line is to use convolution theorem to speed up the models, such as using fast Fourier transform (FFT) [35, 21].

Original: For example, FFC [13] replaces the convolution withthe spatial-Fourier interaction.
Revised: For example, FFC [21] replaces the convolution with the spatial-Fourier interaction.

Original: The work proposed in [28] uses spectral pooling to reduce featureresolution by truncating the frequency domain representation.
Revised: The work proposed in [36] uses spectral pooling to reduce feature resolution by truncating the frequency domain representation.

Original: However, all the techniques onlyinteract with each other at a single spatial resolution and have not explored the interaction potential atmultiple resolutions in both spatial and frequency domains as performing the frequency up-samplingis non-trivial.
Revised: However, all the techniques only interact with each other at a single spatial resolution and have not explored the interaction potential at multiple resolutions in both spatial and frequency domains as performing the frequency up-sampling is non-trivial.

Original: As a tentative exploration, we study the relationship between the spatial domain and
Revised: As a tentative exploration, we study the relationship between the spatial domain and

Original: Fourier domain and reveal the transform rules over the feature of different resolutions in the Fourierdomain.
Revised: Fourier domain and reveal the transform rules over the feature of different resolutions in the Fourier domain.

Original: This delivers the underlying insights for the designs of multi-scale Fourier modeling patterns,which has the potential of versatility for different network architectures.
Revised: This delivers the underlying insights for the designs of multi-scale Fourier modeling patterns, which has the potential of versatility for different network architectures.

Original:  Deep Fourier Up-Sampling
Revised:  Deep Fourier Up-Sampling

Original: We first explore the mapping relationship between the spatial and Fourier domains, and then presenttwo Deep Fourier Up-sampling modules, which can be easily equipped into a general networkarchitecture to deliver the spectral knowledge underlying data. The construction of these two FourierUp modules are inspired from the following theoretical evidences.
Revised: We first explore the mapping relationship between the spatial and Fourier domains, and then present three Deep Fourier up-sampling variants, including i) periodic padding of magnitude and phase, ii) area up-sampling of magnitude and phase, and iii) corner interpolation of magnitude and phase, based on the explored transform rules.

Original: For a spacial map g ( x, y ) ∈ R M × N , we denote its 2 -times up-sampled dilated interpolation asf dilg ( x, y ) ∈ R 2 M × 2 N , with expression as 
Revised: In terms of the first two variants, we provide two theorems and their proofs as follows while the third is reported in supplementary materials.

Original: [Equation]
Revised: Definitions.

Original: others ( [Equation] where m = 0 , 1 , 2, . . , M − 1 and n = 0 , 1 , 2 , . . .
Revised: f ( x, y ) ∈ R 2 M × 2 N is the 2 -times zero-inserted up-sampled version of g (

Original: , N − 1 . Denote G ( u, v ) ∈ R M × N , Fdilg ( u, v ) ∈
Revised: x, y ) ∈ R M × N in spatial domain, and F ( u, v ) ∈

Original: R 2 M × 2 Nas the corresponding Fourier transforms of g and f dilg ,respectively. In the spectral domain, denote the two-times up-sampled Periodic Padding interpolation of Gas F PPG ( u + pM, v+ qN ) =
Revised: R 2 M × 2 N , G ( u, v ) ∈ R M × N denote their Fourier transforms.

Original: G ( u, v ) , p, q = 0 , 1 , u = 0 , 1 , · · · , M, v = 0 , 1 , · · · , N, (2) and denote its two-times up-sampled Area Interpolation as F AIG (2 u + p, 2v + q ) = G ( u, v ) , p, q = 0 , 1 , u = 0 , 1 , · · · , M − 1 , v = 0 , 1 , · · · , N − 1 . Denote the inverse Fourier transform of F AIG as f AIG .
Revised: H ( u, v ) ∈ R 2 M × 2 N is the 2 -

Original: Theorem-1.
Revised: Theorem-1.

Original: For up-sampled Periodic Padding interpolation F PPG of the Fourier map G , it holds that
Revised: F ( u, v ) = F ( u + M, v ) = F ( u, v + N ) = F ( u + M, v + N ) and G ( u, v )

Original: F PPG / 4 = F dilg .
Revised: = F ( u,v )

Original: This theorem implies that the spectral interpolation F PPG of G be exactly transformed to a type ofspacial interpolation of the original map g (under a proportional constant decay).
Revised: where u = 0 , 1 , 2 , . . , N − 1 and v = 0 , 1 , 2 , . .

Original: This naturallyinspires to design Periodic Padding up-sampling scheme in spectral domain.
Revised: , M − 1 . F ( u, v ) is exactly the periodic padding of G ( u, v ) where G ( u, v ) is exactly the quarter of F ( u, v ) with the value being 14 times decay.

Original: Theorem-2.
Revised: Theorem-2.

Original: For up-sampled Area Interpolation F AIG of G , for any p, q = 0 , 1 , it holds that
Revised: H (2 u, 2 v ) = H (2 u + 1 , 2 v ) = H (2 u, 2 v + 1) =

Original: [Equation]
Revised: H (2 u + 1 , 2 v + 1) =

Original: g
Revised: G ( u, v ) with u = 0 , 1 , . . , M − 1 and v = 0 , 1 , . . . , N − 1 and

Original: [Equation]
Revised: [Equation]

Original: where A ( x, y ) = 

Revised: where A ( x, y ) =

Original: + e jπxM
Revised: + e jπxM

Original: + e jπyN + e jπ
Revised: + e jπyN + e jπ

Original: ( xM + yN ) and x = 0 , 1 , . . .
Revised: ( xM + yN ) and x = 0 , 1 , . . .

Original: M − 1 , y = 0 , 1 , . . .
Revised: M − 1 , y = 0 , 1 , . . .

Original: N − 1 .
Revised: N − 1 .

Original: This theorem implies that the spatial transform of the spectral interpolation F AIG ofG is with anexplicit relationship with the original spatial map g by a mapping A .
Revised: Theorem-3. Suppose the corner interpolated F corG ( u, v ) of the Fourier map G ( u, v ) ∈ R M × N , it holds that the inverse Fourier transform f corg ( x, y ) of F corG ( u, v )

Original: A very good property of A , as depicted in Fig. 2, is that its intensity quickly drops to zero from the four map corders toits center. This inspires us to first crop these corners and then merge them to get the final Area Interpolation-Cropping Up-sampling scheme.
Revised: [Equation]

Original: We first provide the proofs of the two theorems, and then introduce the design details of these two
Revised: where x ′ = 2 x and y ′ = 2 y , x = 0 , 1 , . . .

Original: FourierUp modules.
Revised: , M − 1 and y = 0 , 1 , . . . , N − 1 .

Original: 3.2 Proof-1 of Theorem-1: Periodic Padding of Magnitude and Phase
Revised: 3.1 Proof-1 of Theorem-1: Periodic Padding of Magnitude and Phase

Original: Note thatthe relationship between fdilg (
Revised: Note that f

Original:  x, y ) ∈
Revised: ( x, y ) ∈

Original: R 2 M × 2 N and g ( x, y ) can be written asf dil g ( x, y ) = (cid:26) g ( x 2 , y2 ) , x = 2 m, y = 2 n 0 , others (5) where m = 0 , 1 , 2 , . . .
Revised: R 2 M × 2 N is up-sampled over g ( x, y ) ∈ R M × N by a factor of 2 . The relationship between g ( x, y ) and f ( x, y ) can be written as [Equation]

Revised: [Equation]

Original:  , M − 1 and n =0 , 1 , 2 , . . ., N − 1 .
Revised: where m = 1 , 2 , . . , M − 1 and n = 1 , 2 , . . .

Original: And the Fourier transform Fdilg ( u, v ) off dilg ( x, y ) isthus expressed as
Revised: , N − 1 , the Fourier transform F ( u, v ) of f ( x, y ) is expressed as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Then, we show the periodicity of Fdilg ( u, v ) ∈ R 2 M × 2 N with M and N .
Revised: Then, we show the periodicity of F ( u, v ) ∈ R 2 M × 2 N with M and N .

Original: It means Fdilg ( u, v ) =
Revised: It means F ( u, v ) =

Original: Fdilg ( u +M, v ) = F dilg ( u, v + N ) = F dilg ( u + M, v + N ) with u = 0 , 1 , 2 , . . .
Revised: F ( u +

Original: , M − 1 andv =0 , 1 , 2 , . . .
Revised: M, v ) =

Original: , N − 1 . We take the Fdilg ( u, v) = F dilg ( u + M, v ) for example and recall Eq. (6) as
Revised: F ( u, v + N ) =

Original: [Equation] ( u + M, v  )  = [Equation] M
Revised: F ( u + M, v + N ) with u = 0 , 1 , 2 , . .

Original: [Equation] (cid:88) x = N − [Equation]
Revised: , N − 1 and v = 0 , 1 , 2 , . .

Original: (cid:88)
Revised: , M − 1 .

Original: y =[Equation]
Revised: We take the F ( u, v ) =

Original: M [Equation] (cid:88) x = N − 1 (cid:88) y
Revised: F ( u + M, v ) for example and recall Eq.

Original: = f [Equation]
Revised: (4) as

Original: [Equation]
Revised: [Equation]

Original: where e − 2 πjx = 1 for any integer x .
Revised: where e − 2 πjx = 1 for any integer x .

Original: Similarly, we caneasily proof the periodicity of Fdilg ( u, v ) .
Revised: Similarly, we can proof the periodicity of F ( u, v ) as well.

Original: Based on the above proof, the DFT of g ( x, y ) can be formulated as
Revised: Based on the above proof, the DFT of g ( x, y ) can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: Revising Eq.
Revised: Revising Eq.

Original: (6), we can figure out that G ( u, v ) =
Revised: (4), we can figure out that G ( u, v ) =

Original: Fdilg ( u,v ).
Revised: F ( u,v ).

Revised: 3.2 Proof-2 of Theorem-2: Area Interpolation of Magnitude and Phase

Original: The 2D Inverse Discrete Fourier transform (IDFT) of G ( u, v ) can be written as
Revised: The 2D Inverse Discrete Fourier transform (IDFT) of G ( u, v ) can be written as:

Original: [Equation]
Revised: [Equation]

Original: We up-sample G ( u, v ) ∈ R M × N to get F AIG ( u, v ) ∈ R 2 M × 2 N .
Revised: We up-sample G ( u, v ) with a size of M × N to get H ( u, v ) with a size of 2 M × 2 N .

Original: Specifically, the area interpolation shown in Fig. 4(b) is used for interpolation and then the interpolatedpixels are the same as the original pixel in the 2 × 2 localregions.
Revised: Specifically, the area interpolation shown in Figure 3(b) is used for interpolation and then the interpolated pixels are the same as the original pixel in the 2 × 2 local regions.

Original: Namely, F AIG (2 u, 2 v ) = F AIG (2 u + 1 , 2 v ) =
Revised: Namely, H (2 u, 2 v ) = H (2 u + 1 , 2 v ) =

Original: F AIG (2 u, 2 v + 1) =
Revised: H (2 u, 2 v + 1) =

Original: F AIG (2 u + 1 , 2 v + 1) =
Revised: H (2 u + 1 , 2 v + 1) =

Original: G ( u, v ) withu = 0 , 1 , . .
Revised: G ( u, v ) with u = 0 , 1 , . .

Original: , M − 1 and v = 0 , 1 , . . .
Revised: , M − 1 and v = 0 , 1 , . . .

Original: , N − 1 .
Revised: , N − 1 .

Original: Similar to Eq.
Revised: Similar to Eq.

Original: (6), we can inferf AIG ( x,y ) = 4 MN [Figure]
Revised: (4), we can infer

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: = N − 1 (cid:88) v = H (2 u, 2 v ) e j 2 π ( 2 ux 2 M + 2 vy 2 N ) + 4 MN

Original: [Equation]
Revised: M − 1 (cid:88) u =

Original: [Figure]
Revised: N − 1 (cid:88) v =

Original: [Equation] [Figure] [Equation]
Revised: H (2 u + 1 , 2 v ) e j 2 π ( (2 u +1)

Original: [Figure]
Revised: x 2 M + 2 vy 2 N ) + 4 MN

Original: [Equation]
Revised: M − 1 (cid:88) u =

Original: [Figure]
Revised: N − 1 (cid:88) v =

Original: [Equation] [Figure] [Equation]
Revised: H (2 u, 2 v + 1) e j 2 π ( 2 ux 2 M + (2 v +1)

Original: [Figure]
Revised: y 2 N ) + 4 MN

Original: [Equation]
Revised: M − 1 (cid:88) u =

Original: [Figure]
Revised: N − 1 (cid:88) v = H (2 u + 1 , 2 v + 1) e j 2 π ( (2 u +1) x 2 M + (2 v +1) y 2 N )

Original:  4 MN
Revised: = 4 MN

Original: M − [Equation] u 
Revised: M − 1 (cid:88) u =

Original: [Equation] N 
Revised: N − 1 (cid:88) v =

Original: [Equation] v
Revised: G ( u,v ) e j 2 π ( uxM + vyN ) [1 + e jπxM + e jπyN + e jπ ( xM + yN ) ] .

Original: [Equation] [Equation]
Revised: Similarly, we can write g ( x, y ) as g ( x, y ) =

Original: MN
Revised: MN

Original: M [Equation]
Revised: M − 1 (cid:88) u =

Original: N [Equation] =[Equation]
Revised: N − 1 (cid:88) v =

Original: [Equation] Similarly, we can write g ( x, y ) as [Equation]
Revised: G ( u, v ) e j 2 π ( uxM + vyN ) .

Original: Recalling Eq.
Revised: Recalling Eq.

Original: (10) and Eq.
Revised: (8) and Eq.

Original: (12), we can infer, for any p, q = 0 , 1 , it holds that
Revised: (10), we can infer

Original: [Equation]
Revised: [Equation]

Revised: g

Original: [Equation]
Revised: [Equation]

Original: (12) where x = 0 , 1 , . . .
Revised: (10) where x = 0 , 1 , . . .

Original: M − 1 , y = 0 , 1 , . . .
Revised: M − 1 , y = 0 , 1 , . . .

Original: N − 1 .
Revised: N − 1 .

Original: Take the p, q = 0 for example, we can obtain1 + e jπxM
Revised: We remark 1+ e jπxM

Original: + e jπyN + e jπ
Revised: + e jπyN + e jπ

Original: ( xM+ yN ) = A ( x, y ) , where [Equation]
Revised: ( xM

Original: [Equation]
Revised: [Equation]

Original: and then [Equation] [Equation] We can find that the variable x shares the same operations as the variable y .
Revised: We can find that the variable x shares the same operations as the variable y .

Original: For brevity, we only takethe operation of variable x as example
Revised: For brevity, we only take the operation of variable x as example

Original: [Equation]
Revised: [Equation]

Original: Equally, we have
Revised: Equally, we have

Original: [Equation]
Revised: [Equation]

Original: We prove that the partial derivative of | A ( x, y ) | on both x and y are negative for x = 0 ,1 , . . . M −1 , y = 0 , 1 ,. . . N − 1 .
Revised: We prove that the partial derivative of | A ( x, y ) | on both x and y is negative for x ∈ [0 , M − 1] and y ∈ [0 , N − 1] .

Original: Besides, we have
Revised: Besides, we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: That is to say, the intensity drops from the side to the center, shown in Fig.
Revised: That is to say, the intensity drops from the side to the center, shown in Figure 4.

Original: Specifically, theintensity drops to zero at the position of x = M or y = N .
Revised: Specifically, the intensity drops to zero at the position of x = M or y = N .

Original: def FourierUp_Padding(X): # X: input with shape [N, C, H, W] # A and P are the amplitude and phase A, P = FFT(X)
Revised: def DFU_Padding(X): # X: input with shape [N, C, H, W] # A and P are the amplitude and phase A, P = FFT(X)

Original: # Fourier up-sampling transformrules
Revised: # Fourier up-sampling transform

Original: [Equation] [Figure] 
Revised: [Figure] [Equation]

Original: =
Revised: [Figure] [Equation] [Figure] [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original:  Return Y #[N, C, 2H, 2W] def FourierUp_AreaInterpolation(X): # X: input with shape [N, C, H, W] # A and P are the amplitude and phase A, P = FFT(X)
Revised: [Equation] [Figure] [Figure] Return Y #[N, C, 2H, 2W] def DFU_AreaInterpolation(X): # X: input with shape [N, C, H, W] # A and P are the amplitude and phase A, P = FFT(X)

Original: # Fourier up-sampling transform rules A_aip
Revised: # Fourier up-sampling transform rules A_aip

Original: = Area-Interpolation(A)
Revised: = Area-Interpolation(A)

Original: P_aip =
Revised: P_aip =

Original: Area-Interpolation(P)
Revised: Area-Interpolation(P)

Original: A_aip
Revised: A_aip

Original: = Convs_1x1(A_aip)
Revised: = Convs_1x1(A_aip)

Original: P_aip = Convs_1x1(P_aip)
Revised: P_aip = Convs_1x1(P_aip)

Original: # Inverse Fourier transform Y =
Revised: # Inverse Fourier transform Y =

Original: iFFT(A_aip, P_aip)
Revised: iFFT(A_aip, P_aip)

Original: #Area Cropping Y =
Revised: #Area Cropping Y =

Original: Area-Cropping(Y)
Revised: Area-Cropping(Y)

Original: Y =
Revised: Y =

Original: Resize(Y)
Revised: Resize(Y)

Original: Return Y #[N, C, 2H, 2W]
Revised: Return Y #[N, C, 2H, 2W]

Original: 3.4 FourierUp Module Design
Revised: 3.3 Architectural Design

Original: Recalling Theorem-1 and Theorem-2 , wecorrespondingly propose two deep Fourier up-samplingmodels, called Periodic Padding and AreaInterpolation-Cropping, respectively.
Revised: Recall the Theorem-1 and Theorem-2 , we propose two deep Fourier up-sampling variants: Periodic Padding and Area

Original: Periodic Padding Up-Sampling.
Revised: Periodic Padding Up-Sampling.

Original: The pseudo-code ofthe Periodic Padding Up-Sampling schemeis shown in the leftpart of Fig.
Revised: The pseudo-code of Periodic Padding Up-Sampling is shown in the left of Figure 2.

Original: Specifically, given an image X ∈ R H
Revised: Given an image X ∈ R H

Original: × W × C , we first adopt the 
Revised: × W × C , we first adopt the Fourier transform

Original: Fourier transform FFT(X) to obtain its amplitude component A and phase component P .
Revised: FFT(X) to obtain its amplitude component A and phase component P .

Original: We thenperform the periodic padding over A and P two times in both the H and W dimensions, as illustratedin Fig. 4(a), to form the padded A _ pep and P _ pep .
Revised: We then perform the periodic padding over A and P two times in both the H and W dimensions, as illustrated in Figure 3(a).

Original: Such up-sampling maps are then fed into twoindependent convolution module witha kernel size of 1 × 1  and followed by the inverse Fouriertransform iFFT( . ) to project the padded ones back tothe spatial domain.
Revised: The padded A _ pep and P _ pep are then fed into two independent convolution module with 1 × 1 kernel and followed by the inverse Fourier transform iFFT( . ) to project the padded ones back to spatial domain.

Original: Area Interpolation-Cropping Up-Sampling.
Revised: Area Interpolation-Cropping Up-Sampling.

Original: The pseudo-code of Area interpolation-Cropping Up-Samplingmodule is shown in the rightpart of Fig.
Revised: The pseudo-code of Area interpolation-Cropping Up-Sampling is shown in the right of Figure 2.

Original: We first conduct the Area interpolationover the phase and amplitude by 2 × 2 area interpolation with the same pixel, as illustrated in Fig.4(b).
Revised: We first conduct the Area interpolation over the phase and amplitude by 2 × 2 area interpolation with the same pixel, as illustrated in Figure 3(b).

Original: We then employ the inverse Fourier transform to project the interpolated ones back to the spatialdomain.
Revised: We then employ the inverse Fourier transform to project the interpolated ones back to spatial domain.

Original: As described in section 3.3, the inverse spatial representation is periodic while the pixelvalue tends to be decayed quickly.
Revised: As described in Section 3.2, the inverse spatial representation will be periodic while the pixel value will be decay.

Original: The degree of decay of the pixel increases when the pixel is closerto the center.
Revised: The degree of decay of the pixel increases when the pixel is closer to the center.

Original: To better maintain the information, we perform the area cropping operation (detailed in Fig. 1) in the four corners with the W 2 × H 2 size, then merge them together as a whole at the spatialdimension, and finally resize them to the expected up-sampling scale of 2 H × 2 W .
Revised: To better maintain the information, we perform the area cropping operation (detailed in Figure 1) in the four corners with the W 2 × Hsize and then merge them together as a whole at spatial dimension, finally resize them to the size of 2 H × 2 W .

Revised: Note that albeit being designed on the basis of strict theories, both constructed spectral up-sampling modules contain certain approximations, like a learnable 1 × 1 convolution operator instead of strictly 1 / 4 as described in Theorem-1 of main manuscript, and an approximation cropping to preserve the map corners instead of accurate A mapping as proved in Theorem-2 of main manuscript and

Revised: Theorem-3 of supplementary materials.

Revised: Such strategy makes the proposed modules able to be more easily implemented and more flexibly represent real data spectral structures.

Revised: It is worth noting that this should be the first attempt for constructing easy equitable spectral upsampling modules, and hope it would inspire more effective and rational ones from more spectral perspectives.

Original:  Experiments
Revised:  Experiments

Original: To demonstrate the efficacy of our proposed deep Fourier up-sampling, we conduct extensive experiments on multiple computer vision tasks, including object detection, image segmentation, imagede-raining, image dehazing, and guided image super-resolution.
Revised: To demonstrate the efficacy of our proposed deep Fourier up-sampling, we conduct extensive experiments on multiple computer vision tasks, including object detection, image segmentation, image de-raining, image dehazing, and guided image super-resolution.

Original: We provide more experimentalresults in the supplementary material.
Revised: We provide more experimental results in the supplementary material.

Original: 4.1 Experimental Settings
Revised: 4.1 Experimental Settings

Original: Object Detection.
Revised: Object Detection.

Original: Following [12], the PASCAL VOC 2007 and 2012 training sets [29] are usedas training data.
Revised: Following [12], the PASCAL VOC 2007 and 2012 training sets [37] are used as training data.

Original: The PASCAL VOC 2007 testing set is used for evaluations as the ground truthannotations of VOC 2012 testing set are not publicly available.
Revised: The PASCAL VOC 2007 testing set is used for evaluations as the ground truth annotations of VOC 2012 testing set are not publicly available.

Original: We employ the FPN-based Faster
Revised: We employ the FPN-based Faster

Original: RCNN [12] with ResNet50 backbone and YOLO-v3 with Darknet53 [30] as baselines.
Revised: RCNN [12] with ResNet50 backbone and YOLO-v3 with Darknet53 [38] as baselines.

Original: Image Segmentation.
Revised: Image Segmentation.

Original: Following [31, 32], Synapse Dataset and CANDI Dataset are used as thetestbed of medical image segmentation.
Revised: Following [39, 40], Synapse Dataset and CANDI Dataset are used as the testbed of medical image segmentation.

Original: We adopt the two representative image segmentationalgorithms, U-Net [1] and Att-UNet [33], as the base models.
Revised: We adopt the two representative image segmentation algorithms, U-Net [1] and Att-UNet [41], as the base models.

Original: Image De-raining.
Revised: Image De-raining.

Original: Following [34], we choose two widely-used standard benchmark datasets,including Rain200H and Rain200L, for evaluations.
Revised: Following [42], we choose two widely-used standard benchmark datasets, including Rain200H and Rain200L, for evaluations.

Original: we employ two representative de-rainingmethods, LPNet with up-sampling [11] and PReNet without up-sampling [34], as baselines.
Revised: we employ two representative de-raining methods, LPNet with up-sampling [11] and PReNet without up-sampling [42], as baselines.

Original: Image Dehazing.
Revised: Image Dehazing.

Original: Following [10], we employ RESIDE[35] dataset [36] for evaluations.
Revised: Following [10], we employ RESIDE[43] dataset [44] for evaluations.

Original: We also usetwo different network designs AODNet [37] without up-sampling operator and MSBDN [10] withup-sampling operator, for validation.
Revised: We also use two different network designs AODNet [45] without up-sampling operator and MSBDN [10] with up-sampling operator, for validation.

Original: Guided Image Super-resolution.
Revised: Guided Image Super-resolution.

Original: Following [38, 39], we adopt the pan-sharpening, the representativetask of guided image super-resolution, for evaluations.
Revised: Following [14, 46], we adopt the pan-sharpening, the representative task of guided image super-resolution, for evaluations.

Original: The WorldView II, WorldView III, and
Revised: The WorldView II, WorldView III, and

Original: GaoFen2 in [38, 39] are used for evaluations.
Revised: GaoFen2 in [14, 46] are used for evaluations.

Original: We employ two different network designs for validation,including PANNET [40] without up-sampling operator and DCFNET [41] with up-sampling operator.
Revised: We employ two different network designs for validation, including PANNET [47] without up-sampling operator and DCFNET [48] with up-sampling operator.

Original: Several widely-used image quality assessment (IQA) metrics are employed to evaluate the performance, including the relative dimensionless global error in synthesis (ERGAS) [42], the peaksignal-to-noise ratio (PSNR), the spectral angle mapper (SAM) [43], DSC, and HD95.
Revised: Several widely-used image quality assessment (IQA) metrics are employed to evaluate the performance, including the relative dimensionless global error in synthesis (ERGAS) [49], the peak signal-to-noise ratio (PSNR), the spectral angle mapper (SAM) [50], DSC, and HD95.

Original: 4.2 Implementation Details
Revised: 4.2 Implementation Details

Original: Regarding the above competitive baselines, they can be divided into two categories: one with spatialup-sampling ([1], Att-UNet[33], DCFNET [41],  LPNet [11], MSBDN [10]) and another one withoutspatial up-sampling (PReNet [34], AODNet[37], PANNET [40]).
Revised: Regarding the above competitive baselines, they can be divided into two categories: one with spatial up-sampling ([1], Att-UNet [41], DCFNET [48], LPNet [11], MSBDN [10]) and another one without spatial up-sampling (PReNet [42], AODNet[45], PANNET [47]).

Original: The purpose of the exploration onthe baselines without spatial up-sampling is to show the versatility of our FourierUp for differentnetwork structures.
Revised: The purpose of the exploration on the baselines without spatial up-sampling is to show the versatility of our FourierUp for different network structures.

Original: Different from directly replacing the spatial up-sampling with the FourierUpin the baselines with spatial up-sampling, we need to encapsulate the FourierUp for the baselineswithout spatial up-sampling, in which a down-sampling operation is introduced to first reduce theresolution of features.
Revised: Different from directly replacing the spatial up-sampling with the FourierUp in the baselines with spatial up-sampling, we need to encapsulate the FourierUp for the baselines without spatial up-sampling, in which a down-sampling operation is introduced to first reduce the resolution of features.

Original: We provide the detailed structures of the encapsulated FourierUp and thebaselines with the FourierUp in the  supplementary material.
Revised: We provide the detailed structures of the encapsulated FourierUp and the baselines with the FourierUp in the Figure 5 and supplementary material.

Original: For the baselines with spatial up-sampling, we perform the comparison over four configurations:
Revised: For the baselines with spatial up-sampling, we perform the comparison over four configurations:

Original: 1) Original : the baseline without any changes;

Original:  2) FourierUp-AreaUp : replacing the original model’s spatial up-sampling with the union ofthe Area-Interpolation variant of our FourierUp and the spatial up-sampling itself; 3) FourierUp-Padding : replacing the original model’s spatial up-sampling operator with the union of the Periodic-Padding variant of our FourierUp and the spatial up-sampling itself; 4) Spatial-Up : replacing the variants of FourierUp in the settings of 2) / 3) with the spatial 
Revised: 1) Original : the baseline without any changes; 2) FourierUp-AreaUp : replacing the original model’s spatial up-sampling with the union of the Area-Interpolation variant of our FourierUp and the spatial up-sampling itself; 3) FourierUp-Padding : replacing the original model’s spatial up-sampling operator with the union of the Periodic-Padding variant of our FourierUp and the spatial up-sampling itself; 4) Spatial-Up : replacing the variants of FourierUp in the settings of 2) / 3) with the spatial up-sampling.

Original: up-sampling.

Original: For a fair comparison, we use the same number of trainable parameters as 
Revised: For a fair comparison, we use the same number of trainable parameters as 2) / 3) .

Original: For the baselines without spatial up-sampling, we perform the comparison over four configurations:
Revised: For the baselines without spatial up-sampling, we perform the comparison over four configurations:

Original: 1) Original : the baseline without any changes;

Original: 2) FourierUp-AreaUp : replacing the original model’s

Original: convolution with the encapsulated

Original:  3) FourierUp-Padding : replacing the original model’s convolution with the encapsulated FourierUp that is equipped with the Periodic-Padding variant; 4) Spatial-Up : replacing the the encapsulated FourierUp of the settings of 2) / 3) with the spatial up-sampling.
Revised: 1) Original : the baseline without any changes; 2) FourierUp-AreaUp : replacing the original model’s convolution with the encapsulated FourierUp that is equipped with the Area-Interpolation variant; 3) FourierUp-Padding : replacing the original model’s convolution with the encapsulated FourierUp that is equipped with the Periodic-Padding variant; 4) Spatial-Up : replacing the the encapsulated FourierUp of the settings of 2) / 3) with the spatial up-sampling.

Original: For a fair comparison, we use the same number of trainable parameters as 2) / 3) .[Table] [Table]
Revised: For a fair comparison, we use the same number of trainable parameters as 2) / 3) .

Original: [Table]

Original: [Table]

Original: [Table]

Original: 4.3 Comparison and Analysis
Revised: 4.3 Comparison and Analysis

Original: Quantitative Comparison.
Revised: Quantitative Comparison.

Original: We perform the model performance comparison over different configurations, as described in implementation details.
Revised: We perform the model performance comparison over different configurations, as described in implementation details.

Original: The quantitative results are presented in Tables 1to 5 where the best and second best results are highlighted in bold and Underline.
Revised: The quantitative results are presented in Tablesto 5 where the best and second best results are highlighted in bold and Underline.

Original: From the results,by integrating with our proposed two FourierUp variants, we can observe performance gain againstthe baselines across all the datasets in all tested tasks, suggesting the effectiveness of our approach.
Revised: From the results, by integrating with our proposed two FourierUp variants, we can observe performance gain against the baselines across all the datasets in all tested tasks, suggesting the effectiveness of our approach.

Original: For example, for the PReNet of Table 1, “FourierUp-padding” and “FourierUp-AreaUp” obtain0.83dB/0.65dB and 2.1dB/1.9dB PSNR gains than the “Original”, 0.52dB/0.34dB and 1.7dB/1.5dB
Revised: For example, for the PReNet of Table 1, “FourierUp-padding” and “FourierUp-AreaUp” obtain 0.83dB/0.65dB and 2.1dB/1.9dB PSNR gains than the “Original”, 0.52dB/0.34dB and 1.7dB/1.5dB

Original: PSNR gains than “Spatial-Up” on the Rain200H and Rain200L datasets, respectively.
Revised: PSNR gains than “Spatial-Up” on the Rain200H and Rain200L datasets, respectively.

Original: Such results validate the effectiveness of our proposed FourierUp.
Revised: Such results validate the effectiveness of our proposed FourierUp.

Original: The corresponding visualization consistentlysupports the analysis in Fig. 5, where the FourierUp is capable of better maintaining the details.
Revised: The corresponding visualization consistently supports the analysis in Figure 6, where the FourierUp is capable of better maintaining the details.

Original: Qualitative Comparison.
Revised: Qualitative Comparison.

Original: Due to the limited space, we only report the visual results of the de- 238 raining/dehazing task in Fig. 5 and 6  that can more clearly show the effectiveness of FourierUp.
Revised: Due to the limited space, we only report the visual results of the deraining/dehazing task in Figures 6 and 7 that can more clearly show the effectiveness of FourierUp.

Original: Moreresults can be found in the supplementary materials.
Revised: More results can be found in the supplementary materials.

Original: As shown, integrating the FourierUp with theoriginal baseline achieves more visually pleasing results.
Revised: As shown, integrating the FourierUp with the original baseline achieves more visually pleasing results.

Original: Specifically, zooming-in the red box regionof Fig. 5 and 6, the model equipped with the FourierUp is capable of better recovering the texturedetails while removing the rain/hazy effect. 
Revised: Specifically, zooming-in the red box region of Figures 6 and 7, the model equipped with the FourierUp is capable of better recovering the texture details while removing the rain/hazy effect. [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original:  Limitations
Revised:  Limitations

Original: First, the more comprehensive experiments on broader computer vision tasks ( e.g. , image de-noisingand image classification) have not been explored.
Revised: First, the more comprehensive experiments on broader computer vision tasks ( e.g. , image de-noising and image de-blurring) have not been explored.

Original: Second, the deep Fourier Up-sampling integratedwith spatial up-sampling will increase the model parameter numbers.
Revised: Second, the deep Fourier Up-sampling integrated with spatial up-sampling will increase the model parameter numbers.

Original: This is negligible at thesignificant performance gain at fewer parameter increase.
Revised: This is negligible at the significant performance gain at fewer parameter increase.

Original: Note that, the focus of this work is beyonddesigning a plug-and-play module to integrate it into existing networks for further performance gain.
Revised: Note that, the focus of this work is beyond designing a plug-and-play module to integrate it into existing networks for further performance gain.

Original: This work also provides a powerful scale change choice of the up-sampling operator pool whendeveloping a new model from start.
Revised: This work also provides a powerful scale change choice of the up-sampling operator pool when developing a new model from start.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we have proposed a deep Fourier up-sampling to explore the possibility of the upsampling in the Fourier domain, which provides the key insight for the multi-scale Fourier patternmodeling.
Revised: In this paper, we have proposed a deep Fourier up-sampling to explore the possibility of the upsampling in the Fourier domain, which provides the key insight for the multi-scale Fourier pattern modeling.

Original: We theoretically demonstrate that our designs of Fourier up-sampling are feasible.
Revised: We theoretically demonstrate that our designs of Fourier up-sampling are feasible.

Original: Itis appealing that the proposed FourierUp is a generic operator, thus being directly integrated withexisting networks.
Revised: It is appealing that the proposed FourierUp is a generic operator, thus being directly integrated with existing networks.

Original: Extensive experiments demonstrate the effectiveness of our method.
Revised: Extensive experiments demonstrate the effectiveness of our method.

Original: We believethe FourierUp has the potential to advance broader computer vision tasks, e.g. , image/video superresolution and image/video in-painting.
Revised: We believe the FourierUp has the potential to advance broader computer vision tasks, e.g. , image/video superresolution and image/video in-painting.

Original: Broader Impact
Revised: Broader Impact

Original: Our work shows the promising capability of up-sampling in the Fourier domain for computer visionalgorithms through two novel designs with theoretical proofs.
Revised: Our work shows the promising capability of up-sampling in the Fourier domain for computer vision algorithms through two novel designs with theoretical proofs.

Original: Using our deep Fourier up-samplingwith negligible computational cost will improve the performance of neural networks and facilitatethe development of AI in real-world applications.
Revised: Using our deep Fourier up-sampling with negligible computational cost will improve the performance of neural networks and facilitate the development of AI in real-world applications.

Original: However, the efficacy of our method may raisepotential concerns when it is improperly used.
Revised: However, the efficacy of our method may raise potential concerns when it is improperly used.

Original: For example, the safety of the applications of ourmethod in real-world applications may not be guaranteed.
Revised: For example, the safety of the applications of our method in real-world applications may not be guaranteed.

Original: We will investigate the robustness andeffectiveness of our method in broader real-world applications.
Revised: We will investigate the robustness and effectiveness of our method in broader real-world applications.

Revised: Acknowledgements

Revised: We gratefully acknowledge the support of the Major Key Project of PCL (PCL2021A12), MindSpore,

Revised: CANN, and Ascend AI Processor used for this research.

Revised: Chongyi Li and Chen Change Loy are supported by the RIE2020 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP)

Revised: Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).

Original: Checklist
Revised: Checklist

Original: For

Original:  all authors (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s 
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: contributions

Original: and scope? [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] , see Section 5.
Revised: [Yes] , see Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] , see Section 6.
Revised: [Yes] , see Section 6.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] , See Section3.2 and Section 3.3.
Revised: [Yes] , See Section 3.1 and Section 3.2.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] , See Section 3.2 and Section 3.
Revised: [Yes] , See Section 3.1 and Section 3.2.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: [Yes] (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] , the effect of random seed could almost be negligible since we set the same initiation seed during experiments.
Revised: [No] , the effect of random seed could almost be negligible since we set the same initiation seed during experiments.

Original: Reproducibility can beguaranteed.
Revised: Reproducibility can be guaranteed.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] , the data creators claim that they allow and encourage the data for scientific research.
Revised: [Yes] , the data creators claim that they allow and encourage the data for scientific research.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] , We train and test the framework using the publicly accessible datasets.
Revised: [N/A] , We train and test the framework using the publicly accessible datasets.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated 
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [Table]
Revised: [N/A]