Original: A DAPTIVE
Revised: A DAPTIVE

Original: C ROSS -L AYER
Revised: C ROSS -L AYER

Original: A TTENTION
Revised: A TTENTION

Original: FOR
Revised: FOR

Original: I MAGE
Revised: I MAGE

Original: R ESTORATION
Revised: R ESTORATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Non-local attention module has been proven to be crucial for image restoration.
Revised: Non-local attention module has been proven to be crucial for image restoration.

Original: Conventional non-local attention processes features of each layer separately, so it risks missing correlation between features among different layers.
Revised: Conventional non-local attention processes features of each layer separately, so it risks missing correlation between features among different layers.

Original: To address this problem, we propose Cross-Layer Attention (CLA) module in this paper.
Revised: To address this problem, we propose Cross-Layer Attention (CLA) module in this paper.

Original: Instead of ﬁnding correlated key pixels within the same layer, each query pixel is allowed to attend to key pixels at previous layers of the network.
Revised: Instead of ﬁnding correlated key pixels within the same layer, each query pixel is allowed to attend to key pixels at previous layers of the network.

Original: In order to mitigate the expensive computational cost of such hierarchical attention design, only a small ﬁxed number of keys can be selected for each query from a previous layer.
Revised: In order to mitigate the expensive computational cost of such hierarchical attention design, only a small ﬁxed number of keys can be selected for each query from a previous layer.

Original: We further propose a variant of CLA termed Adaptive Cross-Layer Attention (ACLA).
Revised: We further propose a variant of CLA termed Adaptive Cross-Layer Attention (ACLA).

Original: In ACLA, the number of keys to be aggregated for each query is dynamically selected.
Revised: In ACLA, the number of keys to be aggregated for each query is dynamically selected.

Original: A neural architecture search method is used to ﬁnd the insert positions of
Revised: A neural architecture search method is used to ﬁnd the insert positions of

Original: ACLA modules to render a compact neural network with compelling performance.
Revised: ACLA modules to render a compact neural network with compelling performance.

Original: Extensive experiments on image restoration tasks including single image superresolution, image denoising, image demosaicing and image compression artifacts reduction validate the effectiveness and efﬁciency of ACLA.
Revised: Extensive experiments on image restoration tasks including single image superresolution, image denoising, image demosaicing and image compression artifacts reduction validate the effectiveness and efﬁciency of ACLA.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Image restoration algorithms aim to recover a high-quality image from a contaminated input image by solving an ill-posed image restoration problem.
Revised: Image restoration algorithms aim to recover a high-quality image from a contaminated input image by solving an ill-posed image restoration problem.

Original: There are various image restoration tasks depending on the type of corruptions, such as image denoising (Zhang et al., 2019; Liu et al., 2018b), demosaicing (Zhang et al., 2017b; 2019), single image super-resolution (Fan et al., 2019; Lai et al., 2017; Tai et al., 2017), and image compression artifacts reduction(Zhang et al., 2017a).
Revised: There are various image restoration tasks depending on the type of corruptions, such as image denoising (Zhang et al., 2019; Liu et al., 2018b), demosaicing (Zhang et al., 2017b; 2019), single image super-resolution (Fan et al., 2019; Lai et al., 2017; Tai et al., 2017), and image compression artifacts reduction(Zhang et al., 2017a).

Original: To restore corrupted information from the contaminated image, a variety of image priors (Buades et al., 2005; Zoran &
Revised: To restore corrupted information from the contaminated image, a variety of image priors (Buades et al., 2005; Zoran &

Original: Weiss, 2011; Zontak et al., 2013) were proposed.
Revised: Weiss, 2011; Zontak et al., 2013) were proposed.

Original: Recently, image restoration methods based on deep neural networks have achieved great success.
Revised: Recently, image restoration methods based on deep neural networks have achieved great success.

Original: Inspired by the widely used non-local prior, most recent approaches based on neural networks (Zhang et al., 2019; Liu et al., 2018b) adapt non-local attention into their neural network to enhance the representation learning, following the non-local neural networks (Wang et al., 2018).
Revised: Inspired by the widely used non-local prior, most recent approaches based on neural networks (Zhang et al., 2019; Liu et al., 2018b) adapt non-local attention into their neural network to enhance the representation learning, following the non-local neural networks (Wang et al., 2018).

Original: In a nonlocal block, a response is calculated as a weighted sum over all pixel-wise features on the feature map to account for long-range information.
Revised: In a nonlocal block, a response is calculated as a weighted sum over all pixel-wise features on the feature map to account for long-range information.

Original: Such a module was initially designed for high-level recognition tasks such as image classiﬁcation, and it has been proven to be beneﬁcial for low-level vision tasks (Zhang et al., 2019;
Revised: Such a module was initially designed for high-level recognition tasks such as image classiﬁcation, and it has been proven to be beneﬁcial for low-level vision tasks (Zhang et al., 2019;

Original: Liu et al., 2018b).
Revised: Liu et al., 2018b).

Original: Though attention modules have been shown to be effective in boosting the performance.
Revised: Though attention modules have been shown to be effective in boosting the performance.

Original: Most attention modules only explore the correlation among features at the same layer.
Revised: Most attention modules only explore the correlation among features at the same layer.

Original: Actually, features at different intermediate layers encode variant information at different scales, and might be helpful to augment the information used in recovering the high-quality image.
Revised: Actually, features at different intermediate layers encode variant information at different scales, and might be helpful to augment the information used in recovering the high-quality image.

Original: Motivated by the potential beneﬁt of exploring feature correlation across intermediate layers, Holistic Attention Network (HAN) (Niu et al., 2020) is proposed to ﬁnd interrelationship among features at hierarchical levels with a Layer
Revised: Motivated by the potential beneﬁt of exploring feature correlation across intermediate layers, Holistic Attention Network (HAN) (Niu et al., 2020) is proposed to ﬁnd interrelationship among features at hierarchical levels with a Layer

Original: Attention Module (LAM).
Revised: Attention Module (LAM).

Original: However, LAM assigns a single importance weight to all features at the same layer and neglects the difference of spatial positions of these features.
Revised: However, LAM assigns a single importance weight to all features at the same layer and neglects the difference of spatial positions of these features.

Original: Recent research in omnidirectional representation (Tay et al., 2021) suggests that exploring the relationship among features at different layers can beneﬁt the representation learning of neural networks.
Revised: Recent research in omnidirectional representation (Tay et al., 2021) suggests that exploring the relationship among features at different layers can beneﬁt the representation learning of neural networks.

Original: Nevertheless, calculating correlation among features at hierarchical layers is computationally expensive due to the quadratic complexity of dot product attention.
Revised: Nevertheless, calculating correlation among features at hierarchical layers is computationally expensive due to the quadratic complexity of dot product attention.

Original: The complexity of such cross-layer attention design is increased from ( HW ) 2 L to ( HWL ) 2 , where H, W are the height and width of the feature map and
Revised: The complexity of such cross-layer attention design is increased from ( HW ) 2 L to ( HWL ) 2 , where H, W are the height and width of the feature map and

Original: L is the number of layers.
Revised: L is the number of layers.

Original: 1. C ONTRIBUTIONS
Revised: 1. C ONTRIBUTIONS

Original: Our contributions are listed as follows.
Revised: Our contributions are listed as follows.

Original: First, in order to address the problem caused by only referring to keys within the same layer in most attention modules, we propose a novel attention module termed Cross-Layer Attention (CLA), which searches for keys across different layers for each query feature.
Revised: First, in order to address the problem caused by only referring to keys within the same layer in most attention modules, we propose a novel attention module termed Cross-Layer Attention (CLA), which searches for keys across different layers for each query feature.

Original: With the help of the deformation mechanism, CLA only attends to a small set of keys at different layers for each query feature.
Revised: With the help of the deformation mechanism, CLA only attends to a small set of keys at different layers for each query feature.

Original: Second, we propose an improved CLA termed Adaptive Cross-Layer Attention (ACLA) which selects an adaptive number of keys at each layer for each query, and searches for the optimal insert positions of ACLA modules.
Revised: Second, we propose an improved CLA termed Adaptive Cross-Layer Attention (ACLA) which selects an adaptive number of keys at each layer for each query, and searches for the optimal insert positions of ACLA modules.

Original: We deploy ACLA modules on commonly used neural network models, e.g. EDSR (Lim et al., 2017a), for image restoration.
Revised: We deploy ACLA modules on commonly used neural network models, e.g. EDSR (Lim et al., 2017a), for image restoration.

Original: Extensive experiments on single image super-resolution, image denoising, image compression artifacts reduction, and image demosaicing demonstrate the effectiveness of our approach.
Revised: Extensive experiments on single image super-resolution, image denoising, image compression artifacts reduction, and image demosaicing demonstrate the effectiveness of our approach.

Original:  R ELATED W ORKS 2. N EURAL N ETWORKS FOR I MAGE R ESTORATION
Revised:  R ELATED W ORKS 2. N EURAL N ETWORKS FOR I MAGE R ESTORATION

Original: Adopting neural networks for image restoration has achieved great success by utilizing their power in representation.
Revised: Adopting neural networks for image restoration has achieved great success by utilizing their power in representation.

Original: ARCNN (Dong et al., 2015a) was ﬁrst proposed to use CNN for compression artifacts reduction.
Revised: ARCNN (Dong et al., 2015a) was ﬁrst proposed to use CNN for compression artifacts reduction.

Original: Later, DnCNN (Zhang et al., 2017a) uses residual learning and batch normalization to boost performance of CNN for image denoising.
Revised: Later, DnCNN (Zhang et al., 2017a) uses residual learning and batch normalization to boost performance of CNN for image denoising.

Original: In IRCNN (Zhang et al., 2017b), a learned set of CNNs are used as denoising prior for other image restoration tasks.
Revised: In IRCNN (Zhang et al., 2017b), a learned set of CNNs are used as denoising prior for other image restoration tasks.

Original: For single image superresolution(Lai et al., 2017; Zhang et al., 2018d; Haris et al., 2018; Fan et al., 2019), even more efforts have been devoted to designing advanced architectures and learning methods.
Revised: For single image superresolution(Lai et al., 2017; Zhang et al., 2018d; Haris et al., 2018; Fan et al., 2019), even more efforts have been devoted to designing advanced architectures and learning methods.

Original: For example, RDN (Zhang et al., 2018d) and CARN (Ahn et al., 2018) fuse low-level and high-level features with dense connection to provide richer information and details for reconstructing.
Revised: For example, RDN (Zhang et al., 2018d) and CARN (Ahn et al., 2018) fuse low-level and high-level features with dense connection to provide richer information and details for reconstructing.

Original: Recently, non-local attention (Liu et al., 2018b; Dai et al., 2019; Mei et al., 2020) is also used to further boost the peformance of
Revised: Recently, non-local attention (Liu et al., 2018b; Dai et al., 2019; Mei et al., 2020) is also used to further boost the peformance of

Original: CNN for image restoration.
Revised: CNN for image restoration.

Original: 2. A TTENTION M ECHANISM
Revised: 2. A TTENTION M ECHANISM

Original: Attention mechanism has been applied to many computer vision tasks, such as image captioning (Xu et al., 2015; Chen et al., 2017) and image classiﬁcation (Hu et al., 2018; Wang et al., 2017).
Revised: Attention mechanism has been applied to many computer vision tasks, such as image captioning (Xu et al., 2015; Chen et al., 2017) and image classiﬁcation (Hu et al., 2018; Wang et al., 2017).

Original: Non-local attention (Wang et al., 2018) was ﬁrst proposed to capture long-range dependencies for high-level recognition tasks.
Revised: Non-local attention (Wang et al., 2018) was ﬁrst proposed to capture long-range dependencies for high-level recognition tasks.

Original: Recently, several works propose to leverage non-local attention for low-level vision tasks.
Revised: Recently, several works propose to leverage non-local attention for low-level vision tasks.

Original: In NLRN (Liu et al., 2018b) a recurrent neural network is proposed to incorporate non-local attention.
Revised: In NLRN (Liu et al., 2018b) a recurrent neural network is proposed to incorporate non-local attention.

Original: RNAN (Zhang et al., 2019) proposed a residual local and non-local mask branch to obtain non-local mixed attention.
Revised: RNAN (Zhang et al., 2019) proposed a residual local and non-local mask branch to obtain non-local mixed attention.

Original: RCAN (Zhang et al., 2018c) exploits the interdependencies among feature channels by generating different attention for each channel-wise feature.
Revised: RCAN (Zhang et al., 2018c) exploits the interdependencies among feature channels by generating different attention for each channel-wise feature.

Original: HAN (Niu et al., 2020) is proposed to ﬁnd interrelationship among features at hierarchical levels with a layer attention module.
Revised: HAN (Niu et al., 2020) is proposed to ﬁnd interrelationship among features at hierarchical levels with a layer attention module.

Original: Besides, some recent works attempt to explore the beneﬁts of transformer based models for image restoration.
Revised: Besides, some recent works attempt to explore the beneﬁts of transformer based models for image restoration.

Original: IPT (Chen et al., 2021) is proposed to solve various restoration problems in a multi-task learning framework based on visual Transformer.
Revised: IPT (Chen et al., 2021) is proposed to solve various restoration problems in a multi-task learning framework based on visual Transformer.

Original: SwinIR (Liang et al., 2021) adopt the architecture of Swin Transformer.
Revised: SwinIR (Liang et al., 2021) adopt the architecture of Swin Transformer.

Original: However, compared with methods using CNN architecture, transformer-based image restoration methods usually use large datasets for training.
Revised: However, compared with methods using CNN architecture, transformer-based image restoration methods usually use large datasets for training.

Original: Speciﬁcally, IPT uses ImageNet to pretrain the model.
Revised: Speciﬁcally, IPT uses ImageNet to pretrain the model.

Original: SwinIR adapts a combination of four datasets consisting of over 8000 high-quality images as training set for the tasks of denoising and compression artifact reduction.
Revised: SwinIR adapts a combination of four datasets consisting of over 8000 high-quality images as training set for the tasks of denoising and compression artifact reduction.

Original: 2. N EURAL A RCHITECTURE S EARCH
Revised: 2. N EURAL A RCHITECTURE S EARCH

Original: Neural Architecture Search (NAS) has attracted lots of attention recently.
Revised: Neural Architecture Search (NAS) has attracted lots of attention recently.

Original: Early works of NAS adopt heuristic methods such as reinforcement learning (Zoph & Le, 2016) and evolutionary algorithm (Xie & Yuille, 2017).
Revised: Early works of NAS adopt heuristic methods such as reinforcement learning (Zoph & Le, 2016) and evolutionary algorithm (Xie & Yuille, 2017).

Original: The search process with such methods requires huge computational resources.
Revised: The search process with such methods requires huge computational resources.

Original: Recently, various strategies are designed to reduce the expensive costs including weight sharing (Pham et al., 2018), progressive search (Liu et al., 2018a) and one-shot search (Liu et al., 2018c; Xie et al., 2018).
Revised: Recently, various strategies are designed to reduce the expensive costs including weight sharing (Pham et al., 2018), progressive search (Liu et al., 2018a) and one-shot search (Liu et al., 2018c; Xie et al., 2018).

Original: For example, DARTS (Liu et al., 2018c) ﬁrstly relaxes the search space to be continuous and conducts differentiable search.
Revised: For example, DARTS (Liu et al., 2018c) ﬁrstly relaxes the search space to be continuous and conducts differentiable search.

Original: The architecture parameters and network weights are trained simultaneously by gradient descent to reduce the search time.
Revised: The architecture parameters and network weights are trained simultaneously by gradient descent to reduce the search time.

Original: Despite the success of NAS methods for classiﬁcation, dense prediction tasks such as semantic image segmentation and image restoration, usually demand more complicate network architectures.
Revised: Despite the success of NAS methods for classiﬁcation, dense prediction tasks such as semantic image segmentation and image restoration, usually demand more complicate network architectures.

Original: Some recent works have been devoted to explore hierarchical search space for dense prediction tasks.
Revised: Some recent works have been devoted to explore hierarchical search space for dense prediction tasks.

Original: For example, Auto-DeepLab (Liu et al., 2019) introduces a hierarchical search space for semantic image segmentation.
Revised: For example, Auto-DeepLab (Liu et al., 2019) introduces a hierarchical search space for semantic image segmentation.

Original: DCNAS (Zhang et al., 2021) build a densely connected search space to extract multi-level information.
Revised: DCNAS (Zhang et al., 2021) build a densely connected search space to extract multi-level information.

Original: HNAS (Guo et al., 2020) also adopts a hierarchical search space for single image super-resolution.
Revised: HNAS (Guo et al., 2020) also adopts a hierarchical search space for single image super-resolution.

Original:  M ETHOD 3. CLA: C ROSS -L AYER A TTENTION
Revised:  M ETHOD 3. CLA: C ROSS -L AYER A TTENTION

Original: Non-Local Attention.
Revised: Non-Local Attention.

Original: Non-local attention (Wang et al., 2018) is designed to integrate self-attention mechanism into convolutional neural networks for computer vision tasks.
Revised: Non-local attention (Wang et al., 2018) is designed to integrate self-attention mechanism into convolutional neural networks for computer vision tasks.

Original: It is usually applied on an input feature map x ∈ R H × W × C to explore self similarities among all spatial positions of that feature map.
Revised: It is usually applied on an input feature map x ∈ R H × W × C to explore self similarities among all spatial positions of that feature map.

Original: For illustration purpose, we reshape x to N × C , where N = H × W .
Revised: For illustration purpose, we reshape x to N × C , where N = H × W .

Original: A generic non-local attention can be formulated as
Revised: A generic non-local attention can be formulated as

Original: [Equation]
Revised: [Equation]

Original: where i indexes the spatial position of feature maps.
Revised: where i indexes the spatial position of feature maps.

Original: y is the output of non-local attention with the same size as x . f
Revised: y is the output of non-local attention with the same size as x . f

Original: ( x i , x n ) outputs the pairwise afﬁnity between the query feature x i and its key feature x n .
Revised: ( x i , x n ) outputs the pairwise afﬁnity between the query feature x i and its key feature x n .

Original: g ( x n ) computes an embedding of feature x n .
Revised: g ( x n ) computes an embedding of feature x n .

Original: C ( x ) is a normalization term.
Revised: C ( x ) is a normalization term.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Aggregate Gather
Revised: Aggregate Gather

Original: Gather
Revised: Gather

Original: Output Feature Map
Revised: Output Feature Map

Original: [Figure]
Revised: [Figure]

Original: Feature Map of
Revised: Feature Map of

Original: Referred Layer
Revised: Referred Layer

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Non-local attention in Equation (1) is usually wrapped into a non-local block (Wang et al., 2018) with a residual connection from the input feature x .
Revised: Non-local attention in Equation (1) is usually wrapped into a non-local block (Wang et al., 2018) with a residual connection from the input feature x .

Original: The mathematical formulation is given as
Revised: The mathematical formulation is given as

Original: [Equation]
Revised: [Equation]

Original: where h denotes a learnable feature transformation.
Revised: where h denotes a learnable feature transformation.

Original: Cross-Layer Attention.
Revised: Cross-Layer Attention.

Original: To search for keys from different layers for each query feature, we propose
Revised: To search for keys from different layers for each query feature, we propose

Original: Cross-Layer Attention (CLA).
Revised: Cross-Layer Attention (CLA).

Original: To derive the formulation of CLA, we start by adapting non-local attention in Equation (1) to a cross-layer design.
Revised: To derive the formulation of CLA, we start by adapting non-local attention in Equation (1) to a cross-layer design.

Original: Suppose that x i is the output of the i -th layer in a CNN backbone for image restoration, where i ∈ { 1 , . . .
Revised: Suppose that x i is the output of the i -th layer in a CNN backbone for image restoration, where i ∈ { 1 , . . .

Original: , L } and L is the number of layers.
Revised: , L } and L is the number of layers.

Original: A non-local attention with cross-layer design can be formulated as
Revised: A non-local attention with cross-layer design can be formulated as

Original: [Equation]
Revised: [Equation]

Original: where the superscripts j, l index the layer and the subscripts i, n index the spatial locations of features.
Revised: where the superscripts j, l index the layer and the subscripts i, n index the spatial locations of features.

Original: y, x denote the output feature and input feature respectively.
Revised: y, x denote the output feature and input feature respectively.

Original: Query feature in this formulation can thus refer to features in previous layers.
Revised: Query feature in this formulation can thus refer to features in previous layers.

Original: However, given the quadratic complexity of correlation computation, the complexity of such adaption of non-local attention is increased from N 2 L to ( NL ) 2 .
Revised: However, given the quadratic complexity of correlation computation, the complexity of such adaption of non-local attention is increased from N 2 L to ( NL ) 2 .

Original: In order to mitigate the expensive computation cost, we leverage deformable mechanism proposed in deformable convolution (DCN) (Dai et al., 2017) to design our proposed CLA as
Revised: In order to mitigate the expensive computation cost, we leverage deformable mechanism proposed in deformable convolution (DCN) (Dai et al., 2017) to design our proposed CLA as

Original: [Equation]
Revised: [Equation]

Original: where k indexes the sampled keys.
Revised: where k indexes the sampled keys.

Original: p i represents the 2-d spatial position of the query in the feature map, and ∆ p ik is the 2-d offset from the position of query p i to the corresponding sampled key.
Revised: p i represents the 2-d spatial position of the query in the feature map, and ∆ p ik is the 2-d offset from the position of query p i to the corresponding sampled key.

Original: As p i + ∆ p ik can be fractional, bilinear interpolation is used as in (Dai et al., 2017) to compute x ( p i +∆ p ik ) .
Revised: As p i + ∆ p ik can be fractional, bilinear interpolation is used as in (Dai et al., 2017) to compute x ( p i +∆ p ik ) .

Original: To further reduce the computation complexity in computing attention weights, attention weights can be generated from query feature alone.
Revised: To further reduce the computation complexity in computing attention weights, attention weights can be generated from query feature alone.

Original: Thus, our proposed CLA can be simpliﬁed as
Revised: Thus, our proposed CLA can be simpliﬁed as

Original: [Equation]
Revised: [Equation]

Original: where f is the function to generate attention weights from the query feature.
Revised: where f is the function to generate attention weights from the query feature.

Original: Similar to the design for non-local attention module in (Wang et al., 2018), when deploying CLA in CNN backbones, we also wrap it into a non-local block with residual connection.
Revised: Similar to the design for non-local attention module in (Wang et al., 2018), when deploying CLA in CNN backbones, we also wrap it into a non-local block with residual connection.

Original: With our proposed CLA, each query feature now can refer to a ﬁxed number of keys from each previous layer.
Revised: With our proposed CLA, each query feature now can refer to a ﬁxed number of keys from each previous layer.

Original: However, query features at different spatial positions may have different preferences on keys sampled from different layers.
Revised: However, query features at different spatial positions may have different preferences on keys sampled from different layers.

Original: Especially for image restoration tasks, the restoration process at different spatial position may vary signiﬁcantly due to the diversity of textures in an image.
Revised: Especially for image restoration tasks, the restoration process at different spatial position may vary signiﬁcantly due to the diversity of textures in an image.

Original: Besides, when deploying CLA in CNN backbones for image restoration, we ﬁnd that increasing the number of inserted CLA will not constantly improve the performance, but result in much heavier computation cost.
Revised: Besides, when deploying CLA in CNN backbones for image restoration, we ﬁnd that increasing the number of inserted CLA will not constantly improve the performance, but result in much heavier computation cost.

Original: This observation motivates us to search for an optimal conﬁguration of deploying CLA in neural networks to efﬁciently explore its representation learning capacity.
Revised: This observation motivates us to search for an optimal conﬁguration of deploying CLA in neural networks to efﬁciently explore its representation learning capacity.

Original: 3. ACLA: A DAPTIVE C ROSS -L AYER A TTENTION
Revised: 3. ACLA: A DAPTIVE C ROSS -L AYER A TTENTION

Original: To achieve adaptive key selection and the search for the optimal conﬁguration of CLA, we propose
Revised: To achieve adaptive key selection and the search for the optimal conﬁguration of CLA, we propose

Original: Adaptive Cross-Layer Attention (ACLA).
Revised: Adaptive Cross-Layer Attention (ACLA).

Original: Speciﬁcally, for each query feature, we dynamically search for the keys sampled from previous layers with ACLA.
Revised: Speciﬁcally, for each query feature, we dynamically search for the keys sampled from previous layers with ACLA.

Original: Besides, when deploying ACLA in CNN backbones, a neural architecture search method is used to search for the insert positions of ACLA.
Revised: Besides, when deploying ACLA in CNN backbones, a neural architecture search method is used to search for the insert positions of ACLA.

Original: An objective based on the computation cost of inserted ACLA modules is used to supervise the search procedure.
Revised: An objective based on the computation cost of inserted ACLA modules is used to supervise the search procedure.

Original: Adaptive Cross-Layer Attention.
Revised: Adaptive Cross-Layer Attention.

Original: To adaptively search for the informative sampled keys for a query feature from its previous layers, we ﬁrst follow the method in CLA to get a ﬁxed number of sampled keys from each layer.
Revised: To adaptively search for the informative sampled keys for a query feature from its previous layers, we ﬁrst follow the method in CLA to get a ﬁxed number of sampled keys from each layer.

Original: Next, a hard gating mask is applied on each of the sampled key as
Revised: Next, a hard gating mask is applied on each of the sampled key as

Original: [Equation]
Revised: [Equation]

Original: 6) where m ki,l is the hard gating mask for the k -th sampled key from x l for query feature x ji , whose value is either 1 or 0 .
Revised: 6) where m ki,l is the hard gating mask for the k -th sampled key from x l for query feature x ji , whose value is either 1 or 0 .

Original: We further relax the hard gating mask to continuous domain with the simpliﬁed binary
Revised: We further relax the hard gating mask to continuous domain with the simpliﬁed binary

Original: Gumbel-Softmax (Verelst & Tuytelaars, 2020).
Revised: Gumbel-Softmax (Verelst & Tuytelaars, 2020).

Original: Therefore, the hard gating mask m ki,l is approximated by
Revised: Therefore, the hard gating mask m ki,l is approximated by

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: with sampling parameter β ki,l , Gumbel noise (cid:15) ki,l, 1 , (cid:15) ki,l, 2 , and temperature τ .
Revised: with sampling parameter β ki,l , Gumbel noise (cid:15) ki,l, 1 , (cid:15) ki,l, 2 , and temperature τ .

Original: σ is the sigmoid function.
Revised: σ is the sigmoid function.

Original: The sampling parameter β ki,l can be regarded as a soft gating mask.
Revised: The sampling parameter β ki,l can be regarded as a soft gating mask.

Original: To make the gating decisions on sampled keys input-dependent, a mask unit M is used to generate the soft gating mask β from the features of the sampled keys as
Revised: To make the gating decisions on sampled keys input-dependent, a mask unit M is used to generate the soft gating mask β from the features of the sampled keys as

Original: [Equation]
Revised: [Equation]

Original: Following the design in (Verelst & Tuytelaars, 2020), a 1 × 1 convolution layer is used as the mask unit M in our model.
Revised: Following the design in (Verelst & Tuytelaars, 2020), a 1 × 1 convolution layer is used as the mask unit M in our model.

Original: To train the mask unit M , the straight-through estimator from (Bengio et al., 2013; Verelst & Tuytelaars, 2020) is used for m ki,l .
Revised: To train the mask unit M , the straight-through estimator from (Bengio et al., 2013; Verelst & Tuytelaars, 2020) is used for m ki,l .

Original: In particular, during backward pass, we set m ki,l = ˆ m ki,l , while in forward pass we set
Revised: In particular, during backward pass, we set m ki,l = ˆ m ki,l , while in forward pass we set

Original: [Equation]
Revised: [Equation]

Original: Besides, Gumbel noise (cid:15) ki,l, 1 and (cid:15) ki,l, 2 are set to 0 during inference.
Revised: Besides, Gumbel noise (cid:15) ki,l, 1 and (cid:15) ki,l, 2 are set to 0 during inference.

Original: To demonstrate the effectiveness of adaptive key selection in ACLA, we visualized keys selected by ACLA at different layers for a query feature in Figure
Revised: To demonstrate the effectiveness of adaptive key selection in ACLA, we visualized keys selected by ACLA at different layers for a query feature in Figure

Original: Insert Positions.
Revised: Insert Positions.

Original: To search for the insert positions of ACLA, we ﬁrst densely insert ACLA after each layer of the CNN backbone as in Figure 3 to build the supernet.
Revised: To search for the insert positions of ACLA, we ﬁrst densely insert ACLA after each layer of the CNN backbone as in Figure 3 to build the supernet.

Original: Similar to the gating formulation in ACLA, we deﬁne a hard decision parameter s j ∈ { 0 , 1 } for the j -th inserted ACLA in the supernet.
Revised: Similar to the gating formulation in ACLA, we deﬁne a hard decision parameter s j ∈ { 0 , 1 } for the j -th inserted ACLA in the supernet.

Original: s j = 1 indicates that a ACLA module is inserted after the j -th layer, and otherwise for s j = 0 .
Revised: s j = 1 indicates that a ACLA module is inserted after the j -th layer, and otherwise for s j = 0 .

Original: The simpliﬁed binary Gumbel-Softmax (Verelst & Tuytelaars, 2020) can also be used here to approximate the hard decision parameter s j by
Revised: The simpliﬁed binary Gumbel-Softmax (Verelst & Tuytelaars, 2020) can also be used here to approximate the hard decision parameter s j by

Original: [Equation]
Revised: [Equation]

Original: with sampling parameter α j , Gumbel noise (cid:15) , and temperature τ .
Revised: with sampling parameter α j , Gumbel noise (cid:15) , and temperature τ .

Original: Thus, the output of ACLA in the supernet can be expressed as
Revised: Thus, the output of ACLA in the supernet can be expressed as

Original: [Equation]
Revised: [Equation]

Original: Different from data-dependent design of the gating mask in ACLA, α j here is deﬁned as architecture parameter and can be directly optimized by stochastic gradient descent (SGD) during the search process.
Revised: Different from data-dependent design of the gating mask in ACLA, α j here is deﬁned as architecture parameter and can be directly optimized by stochastic gradient descent (SGD) during the search process.

Original: By gradually decrease the value of temperature, α j will be optimized such that s j will approach 1 or 0 .
Revised: By gradually decrease the value of temperature, α j will be optimized such that s j will approach 1 or 0 .

Original: [Figure]
Revised: [Figure]

Original: Search Procedure.
Revised: Search Procedure.

Original: We propose to optimize both the accuracy and the computation cost (FLOPs) of the ACLA modules inserted.
Revised: We propose to optimize both the accuracy and the computation cost (FLOPs) of the ACLA modules inserted.

Original: Thus, the cost of the ACLA modules inserted needs to be estimated during the search phase.
Revised: Thus, the cost of the ACLA modules inserted needs to be estimated during the search phase.

Original: Following the formulation of the ACLA in the supernet, we can estimate the cost of the ACLA inserted after the j -th residual block as
Revised: Following the formulation of the ACLA in the supernet, we can estimate the cost of the ACLA inserted after the j -th residual block as

Original: [Equation]
Revised: [Equation]

Original: (12) where N is the number of spatial positions.
Revised: (12) where N is the number of spatial positions.

Original: C is the number of channels.
Revised: C is the number of channels.

Original: K is the maximal number of sampled keys.
Revised: K is the maximal number of sampled keys.

Original: 2 m kj,l NC 2 is the FLOPs for the convolution on generating the gating masks.
Revised: 2 m kj,l NC 2 is the FLOPs for the convolution on generating the gating masks.

Original: 2 NCis the FLOPs for the convolution on sampling keys.
Revised: 2 NCis the FLOPs for the convolution on sampling keys.

Original: 6 KNC is the FLOPs for the aggregation process.
Revised: 6 KNC is the FLOPs for the aggregation process.

Original: Then we can get the estimation on the computation cost of all inserted ACLA modules as
Revised: Then we can get the estimation on the computation cost of all inserted ACLA modules as

Original: [Equation]
Revised: [Equation]

Original: which is the summation of the cost of each layer, weighted by corresponding decision parameters.
Revised: which is the summation of the cost of each layer, weighted by corresponding decision parameters.

Original: As mentioned before, with the help of the continuously relaxed representation in the supernet, we can search for the architecture by updating the architecture parameters using SGD.
Revised: As mentioned before, with the help of the continuously relaxed representation in the supernet, we can search for the architecture by updating the architecture parameters using SGD.

Original: To supervise the search process, we design a loss function with the cost-based regularization to achieve the multi-objective optimization:
Revised: To supervise the search process, we design a loss function with the cost-based regularization to achieve the multi-objective optimization:

Original: [Equation]
Revised: [Equation]

Original: where λ is the hyper-parameters that controls the magnitude of the cost term.
Revised: where λ is the hyper-parameters that controls the magnitude of the cost term.

Original: We ﬁnd that at the beginning of the search process, ACLA modules inserted at shallow layers are more likely to be strengthened.
Revised: We ﬁnd that at the beginning of the search process, ACLA modules inserted at shallow layers are more likely to be strengthened.

Original: To solve this problem, we split our search procedure into two stages.
Revised: To solve this problem, we split our search procedure into two stages.

Original: In the ﬁrst stage, we only optimize the parameters of the network for enough epochs to get network weights sufﬁciently trained.
Revised: In the ﬁrst stage, we only optimize the parameters of the network for enough epochs to get network weights sufﬁciently trained.

Original: In the second stage, we activate the architecture optimization.
Revised: In the second stage, we activate the architecture optimization.

Original: We alternatively optimize the network weights by descending ∇ w L train ( w, α ) on the training set, and optimize the architecture parameters by descending ∇ α L val ( w, α ) on the validation set.
Revised: We alternatively optimize the network weights by descending ∇ w L train ( w, α ) on the training set, and optimize the architecture parameters by descending ∇ α L val ( w, α ) on the validation set.

Original: When the search procedure terminates, we derive the insert positions based on the architecture parameters α .
Revised: When the search procedure terminates, we derive the insert positions based on the architecture parameters α .

Original:  E XPERIMENTS 4. I MPLEMENTATION D ETAILS
Revised:  E XPERIMENTS 4. I MPLEMENTATION D ETAILS

Original: We select DIV2K (Timofte et al., 2017) as the training set for our experiments as like in (Dai et al., 2019; Zhang et al., 2018d), which include 800 images for training and 100 images for validation.
Revised: We select DIV2K (Timofte et al., 2017) as the training set for our experiments as like in (Dai et al., 2019; Zhang et al., 2018d), which include 800 images for training and 100 images for validation.

Original: We augment the training images by randomly rotating 90 ◦ , 180 ◦ , 270 ◦ and horizontally ﬂipping.
Revised: We augment the training images by randomly rotating 90 ◦ , 180 ◦ , 270 ◦ and horizontally ﬂipping.

Original: In each
Revised: In each

Original: [Table]
Revised: [Table]

Original: mini-
Revised: mini-

Original: [Table]
Revised: [Table]

Original: 16 low-quality patches with size 48 × 48 are provided as inputs.
Revised: 16 low-quality patches with size 48 × 48 are provided as inputs.

Original: ADAM optimizer is used for both the search phase and training phase.
Revised: ADAM optimizer is used for both the search phase and training phase.

Original: Default values of β 1 and β 2 are set as 0 . 9 and 0 .
Revised: Default values of β 1 and β 2 are set as 0 . 9 and 0 .

Original: 999 , respectively, and we set (cid:15)
Revised: 999 , respectively, and we set (cid:15)

Original: =
Revised: =

Original: 10 − 8 .
Revised: 10 − 8 .

Original: Before the search, we perform a cross-validation on 20% of the training data to decide the value of λ .
Revised: Before the search, we perform a cross-validation on 20% of the training data to decide the value of λ .

Original: In search phase, the learning rate is initialized as 10 − 4 and cosine learning rate schedule is used.
Revised: In search phase, the learning rate is initialized as 10 − 4 and cosine learning rate schedule is used.

Original: The search process takes 200 epochs.
Revised: The search process takes 200 epochs.

Original: In the training phase, the learning rate is initialized as 10 − 4 and then reduced to half every 200 epochs.
Revised: In the training phase, the learning rate is initialized as 10 − 4 and then reduced to half every 200 epochs.

Original: The model is trained for 1000 epochs in total.
Revised: The model is trained for 1000 epochs in total.

Original: K = 8 is sef for all CLA modules by default.
Revised: K = 8 is sef for all CLA modules by default.

Original: For all ACLA modules, the value of K are initialized as 16 .
Revised: For all ACLA modules, the value of K are initialized as 16 .

Original: 4. S INGLE I MAGE S UPER - RESOLUTION
Revised: 4. S INGLE I MAGE S UPER - RESOLUTION

Original: We test our methods on top of the widely used super-resolution backbone EDSR (Lim et al., 2017b) and RCAN (Zhang et al., 2018b).
Revised: We test our methods on top of the widely used super-resolution backbone EDSR (Lim et al., 2017b) and RCAN (Zhang et al., 2018b).

Original: The LR images are obtained by the bicubic downsampling of HR images.
Revised: The LR images are obtained by the bicubic downsampling of HR images.

Original: Our methods are evaluated on ﬁve standard datasets: Set5 (Bevilacqua et al., 2012), Set14 (Zeyde et al., 2010), B100 (Martin et al., 2001), Urban100 (Huang et al., 2015), and
Revised: Our methods are evaluated on ﬁve standard datasets: Set5 (Bevilacqua et al., 2012), Set14 (Zeyde et al., 2010), B100 (Martin et al., 2001), Urban100 (Huang et al., 2015), and

Original: Manga109 (Matsui et al., 2017).
Revised: Manga109 (Matsui et al., 2017).

Original: The reconstructed results by our model are ﬁrst converted to YCbCr space, PSNR and SSIM in the luminance channel is calculated in our experiments.
Revised: The reconstructed results by our model are ﬁrst converted to YCbCr space, PSNR and SSIM in the luminance channel is calculated in our experiments.

Original: We compare our methods with 6 other methods: SRCNN (Dong et al., 2015b), VDSR (Kim et al., 2016), MemNet (Tai et al., 2017), SRMDNF (Zhang et al., 2018a), RDN (Zhang et al., 2018d), and HAN (Niu et al., 2020).
Revised: We compare our methods with 6 other methods: SRCNN (Dong et al., 2015b), VDSR (Kim et al., 2016), MemNet (Tai et al., 2017), SRMDNF (Zhang et al., 2018a), RDN (Zhang et al., 2018d), and HAN (Niu et al., 2020).

Original: The quantitative results are shown in Table 1.
Revised: The quantitative results are shown in Table 1.

Original: Some visual results can be found in Appendix A.6.
Revised: Some visual results can be found in Appendix A.6.

Original: Our methods greatly improves the performance of EDSR and RCAN on all benchmarks and all upsampling scales.
Revised: Our methods greatly improves the performance of EDSR and RCAN on all benchmarks and all upsampling scales.

Original: Besides, our methods outperform current state-of-the-art method HAN, which is also an attention based method, demonstrating the superiority of our proposed attention modules.
Revised: Besides, our methods outperform current state-of-the-art method HAN, which is also an attention based method, demonstrating the superiority of our proposed attention modules.

Original: 4. I MAGE D ENOISING
Revised: 4. I MAGE D ENOISING

Original: We evaluate our proposed CLA and ACLA module on standard benchmarks for image denoising:
Revised: We evaluate our proposed CLA and ACLA module on standard benchmarks for image denoising:

Original: KCLDAk24, BSD68 (Martin et al., 2001), and Urban100 (Huang et al., 2015).
Revised: KCLDAk24, BSD68 (Martin et al., 2001), and Urban100 (Huang et al., 2015).

Original: The noisy images are created by adding AWGN noises with σ = 10 , 30 , 50 , 70 .
Revised: The noisy images are created by adding AWGN noises with σ = 10 , 30 , 50 , 70 .

Original: We compare our approach withmethods, namely DnCNN (Zhang et al., 2017a), MemNet (Tai et al., 2017), RNAN (Zhang et al., 2019), and PANet (Mei et al., 2020).
Revised: We compare our approach withmethods, namely DnCNN (Zhang et al., 2017a), MemNet (Tai et al., 2017), RNAN (Zhang et al., 2019), and PANet (Mei et al., 2020).

Original: For fair comparisons, a 16-layer EDSR is used as the baseline
Revised: For fair comparisons, a 16-layer EDSR is used as the baseline

Original: CNN backbone.
Revised: CNN backbone.

Original: As shown in Table 2, our methods are also beneﬁcial for representation learning with neural network on image denoising.
Revised: As shown in Table 2, our methods are also beneﬁcial for representation learning with neural network on image denoising.

Original: [Table]
Revised: [Table]

Original: 4. I MAGE C OMPRESSION A RTIFACTS R EDUCTION
Revised: 4. I MAGE C OMPRESSION A RTIFACTS R EDUCTION

Original: For the task of image compression artifacts reduction (CAR)
Revised: For the task of image compression artifacts reduction (CAR)

Original: [Table]
Revised: [Table]

Original: we compare our method withapproaches: DnCNN (Zhang et al., 2017a), RNAN (Zhang et al., 2019), and PANet (Mei et al., 2020).
Revised: we compare our method withapproaches: DnCNN (Zhang et al., 2017a), RNAN (Zhang et al., 2019), and PANet (Mei et al., 2020).

Original: All methods are evaluated on LIVE1 (Sheikh et al., 2005) and Classic5 (Foi et al., 2007).
Revised: All methods are evaluated on LIVE1 (Sheikh et al., 2005) and Classic5 (Foi et al., 2007).

Original: To obtain the low-quality compressed images, we follow the standard JPEG compression process and use Matlab JPEG encoder with quality q = 10 , 20 , 30 , 40 .
Revised: To obtain the low-quality compressed images, we follow the standard JPEG compression process and use Matlab JPEG encoder with quality q = 10 , 20 , 30 , 40 .

Original: For fair comparison, the results are only evaluated on Y channel in YCbCr Space.
Revised: For fair comparison, the results are only evaluated on Y channel in YCbCr Space.

Original: The quantitative results are shown in Table 3.
Revised: The quantitative results are shown in Table 3.

Original: 16-layer
Revised: 16-layer

Original: EDSR is used as our baseline CNN backbone.
Revised: EDSR is used as our baseline CNN backbone.

Original: Both CLA and ACLA improves the performance of the neural network baseline.
Revised: Both CLA and ACLA improves the performance of the neural network baseline.

Original: [Table]
Revised: [Table]

Original: 4. A BLATION S TUDY AND D ISCUSSION
Revised: 4. A BLATION S TUDY AND D ISCUSSION

Original: ACLA vs. Non-local attention To verify the effectiveness of our proposed methods, we compare
Revised: ACLA vs. Non-local attention To verify the effectiveness of our proposed methods, we compare

Original: CLA and ACLA with Non-Local attention (Wang et al., 2018) and vanilla Cross-Layer Non-Local attention in terms of computational efﬁciency and performance.
Revised: CLA and ACLA with Non-Local attention (Wang et al., 2018) and vanilla Cross-Layer Non-Local attention in terms of computational efﬁciency and performance.

Original: The vanilla Cross-Layer Non-Local attention follows the formulation in Equation (3).
Revised: The vanilla Cross-Layer Non-Local attention follows the formulation in Equation (3).

Original: The comparison is performed on Set5 for single image super-resolution with EDSR backbone.
Revised: The comparison is performed on Set5 for single image super-resolution with EDSR backbone.

Original: The Non-Local attention modules and vanilla CrossLayer Non-Local attention modules are inserted evenly after every 8th residual blocks.
Revised: The Non-Local attention modules and vanilla CrossLayer Non-Local attention modules are inserted evenly after every 8th residual blocks.

Original: All the FLOPs in our ablation study are calculated for input of size 48 × 48 .
Revised: All the FLOPs in our ablation study are calculated for input of size 48 × 48 .

Original: Results are presented in Table 4, where
Revised: Results are presented in Table 4, where

Original: NL stands for Non-Local attention and CLNL stands for vanilla Cross-Layer Non-Local.
Revised: NL stands for Non-Local attention and CLNL stands for vanilla Cross-Layer Non-Local.

Original: As shown in Table 4, with less computation cost, CLA and ACLA achieve much better performance compared to standard Non-Local attention module.
Revised: As shown in Table 4, with less computation cost, CLA and ACLA achieve much better performance compared to standard Non-Local attention module.

Original: [Table]
Revised: [Table]

Original: Number of inserted CLA modules As stated before, the computation cost of the cross-layer design in CLA is quadratic to the number of inserted CLA modules.
Revised: Number of inserted CLA modules As stated before, the computation cost of the cross-layer design in CLA is quadratic to the number of inserted CLA modules.

Original: To verify that dense insertion of CLA modules is not necessary, we perform an ablation study on the number of inserted CLA modules on
Revised: To verify that dense insertion of CLA modules is not necessary, we perform an ablation study on the number of inserted CLA modules on

Original: Set5 ( × 2 ) for single image super-resolution.
Revised: Set5 ( × 2 ) for single image super-resolution.

Original: We use EDSR as our backbone where L CLA modules are evenly inserted.
Revised: We use EDSR as our backbone where L CLA modules are evenly inserted.

Original: As shown in Table 5, with more CLA modules inserted, the performance can be slightly improved.
Revised: As shown in Table 5, with more CLA modules inserted, the performance can be slightly improved.

Original: However, the computation cost and parameter size of the model are also greatly increased.
Revised: However, the computation cost and parameter size of the model are also greatly increased.

Original: While with insert positions searched as shown in Table 6, our model can reach comparable performance with much less computational resources.
Revised: While with insert positions searched as shown in Table 6, our model can reach comparable performance with much less computational resources.

Original: [Table]
Revised: [Table]

Original: Ablation study on ACLA In our paper, we propose ACLA, which is a variant of CLA. Two adaptive designs are applied in  ACLA to select the sampled keys to be aggregated and search for the insert positions in CNN backbones.
Revised: Ablation study on ACLA As explained in Section 3.2, ACLA further improves CLA by two adaptive designs: selecting an adaptive number of keys at each layer for non-local attention and searching for optimal insert positions of ACLA modules.

Original: To verify the effectiveness of the two adaptive designs in ACLA, we perform an ablation study on their inﬂuence on top of CLA.
Revised: To verify the effectiveness of the two adaptive designs in ACLA [Table] we perform an ablation study on their inﬂuence on top of CLA.

Original: The comparison is also performed on Set5 ( × 2 ) for single image super-resolution with EDSR backbone.
Revised: The comparison is also performed on Set5 ( × 2 ) for single image super-resolution with EDSR backbone.

Original: The results are shown in Table 6.
Revised: The results are shown in Table 6.

Original: CLA-I stands for CLA is deployed with the search for insert positions as in ACLA.
Revised: CLA-I stands for CLA is deployed with the search for insert positions as in ACLA.

Original: CLA-K stands for CLA that adaptively select sampled keys as in ACLA.
Revised: CLA-K stands for CLA that adaptively select sampled keys as in ACLA.

Original: We can clearly see that both adaptive designs are beneﬁcial to the performance of CLA.
Revised: We can clearly see that both adaptive designs are beneﬁcial to the performance of CLA.

Original: [Table]
Revised: [Table]

Original: Number of sampled keys K for CLA and ACLA As discussed before, the key point sampling strategy in CLA plays a vital role to reduce the computation cost.
Revised: Number of sampled keys K for CLA and ACLA As discussed before, the key point sampling strategy in CLA plays a vital role to reduce the computation cost.

Original: To verify that a small number of sampled keys K can be sufﬁcient, we perform experiments on CLA with different value of K .
Revised: To verify that a small number of sampled keys K can be sufﬁcient, we perform experiments on CLA with different value of K .

Original: The comparison is performed on Set5 ( × 2 ) for single image super-resolution with EDSR backbone.
Revised: The comparison is performed on Set5 ( × 2 ) for single image super-resolution with EDSR backbone.

Original: Besides, we also compare ACLA with different value K , which is the maximal number of sampled keys.
Revised: Besides, we also compare ACLA with different value K , which is the maximal number of sampled keys.

Original: The results are displayed in Table 7 and Table 8.
Revised: The results are displayed in Table 7 and Table 8.

Original: With increased value of K , the performance of CLA and ACLA does not constantly improve.
Revised: With increased value of K , the performance of CLA and ACLA does not constantly improve.

Original: CLA with K = 8 and ACLA with K = 16 can already achieve comparable performance to those with larger K .
Revised: CLA with K = 8 and ACLA with K = 16 can already achieve comparable performance to those with larger K .

Original: This is also consistent with studies (Zhang et al., 2015; Elad & Aharon, 2006) on the power of sparse representation learning for image restoration.
Revised: This is also consistent with studies (Zhang et al., 2015; Elad & Aharon, 2006) on the power of sparse representation learning for image restoration.

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper [Table] we ﬁrst propose a novel attention module Cross-Layer Attention (CLA) to search for informative keys across different layers for each query feature.
Revised: In this paper, we ﬁrst propose a novel attention module Cross-Layer Attention (CLA) to search for informative keys across different layers for each query feature.

Original: We further proposea variant of CLA, termed as Adaptive Cross-Layer Attention (ACLA).
Revised: We further propose Adaptive CLA, or

Original: In the proposed ACLA, each query dynamically select keys from different layers conditioned on the input. A NAS method is used to search forthe insert positions of ACLA to achieve effective and efﬁcient deployment.
Revised: ACLA, which improves CLA by two adaptive designs: selecting adaptive number of keys at each layer and searching for insert positions of ACLA modules.

Original: Experiments on image restoration tasks including single-image super resolution, image denoising, image compression artifacts reduction and image demosaicing validate the effectiveness and efﬁciency of CLA and ACLA.
Revised: Experiments on image restoration tasks including single-image super resolution, image denoising, image compression artifacts reduction and image demosaicing validate the effectiveness and efﬁciency of CLA and ACLA.

Original: A A PPENDIX A. M ORE IMPLEMENTATION DETAILS
Revised: A A PPENDIX A. M ORE IMPLEMENTATION DETAILS

Original: We summarize the value of λ , i.e. hyper-parameter that controls the magnitude of the cost term, for different experiments in Table 9.
Revised: We summarize the value of λ , i.e. hyper-parameter that controls the magnitude of the cost term, for different experiments in Table 9.

Original: The insert positions of ACLA in the searched models are also displayed in the same table.
Revised: The insert positions of ACLA in the searched models are also displayed in the same table.

Original: Since RCAN constitutes 10 residual groups, we insert ACLA module after each of the residual group in the corresponding super network.
Revised: Since RCAN constitutes 10 residual groups, we insert ACLA module after each of the residual group in the corresponding super network.

Original: For experiments with EDSR,
Revised: For experiments with EDSR,

Original: ACLA modules are inserted after each residual block in the super network.
Revised: ACLA modules are inserted after each residual block in the super network.

Original: Note that 16-layer EDSR is used for Image Denoising, Image Demosaicing
Revised: Note that 16-layer EDSR is used for Image Denoising, Image Demosaicing

Original: [Table]
Revised: [Table]

Original: and Image Compression Artifacts Reduction.
Revised: and Image Compression Artifacts Reduction.

Original: [Table]
Revised: [Table]

Original: A. I MAGE D EMOSAICING
Revised: A. I MAGE D EMOSAICING

Original: To demonstrate the effectiveness of our proposed CLA and ACLA on various image restoration tasks
Revised: To demonstrate the effectiveness of our proposed CLA and ACLA on various image restoration tasks

Original: [Table]
Revised: [Table]

Original: we add an experiment on image demosaicing.
Revised: we add an experiment on image demosaicing.

Original: The evaluation is conducted on Kodak
Revised: The evaluation is conducted on Kodak

Original: [Table]
Revised: [Table]

Original: McMaster (Zhang et al., 2017b), BSD
Revised: McMaster (Zhang et al., 2017b), BSD

Original: [Table]
Revised: [Table]

Original: and Urban100, following the settings in RNAN (Zhang et al., 2019).
Revised: and Urban100, following the settings in RNAN (Zhang et al., 2019).

Original: We compare our approach with IRCNN (Zhang et al., 2017b), RNAN (Zhang et al., 2019)
Revised: We compare our approach with IRCNN (Zhang et al., 2017b), RNAN (Zhang et al., 2019)

Original: [Table]
Revised: [Table]

Original: and PANet (Mei et al., 2020).
Revised: and PANet (Mei et al., 2020).

Original: We also use 16-layer EDSR as the baseline CNN model.
Revised: We also use 16-layer EDSR as the baseline CNN model.

Original: As shown in Table
Revised: As shown in Table

Original: [Table]
Revised: [Table]

Original: our approach yields the best reconstruction result for image demosaicing
Revised: our approach yields the best reconstruction result for image demosaicing

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. C OMPARISON WITH OTHER ATTENTION METHODS
Revised: A. C OMPARISON WITH OTHER ATTENTION METHODS

Original: In our paper
Revised: In our paper

Original: [Table]
Revised: [Table]

Original: we have compared CLA and ACLA with non-local attention in section 4.5.
Revised: we have compared CLA and ACLA with non-local attention in section 4.5.

Original: Here
Revised: Here

Original: [Table]
Revised: [Table]

Original: we further compare our proposed ACLA against other forms of attention modules that are widely used in the CV community, including Squeeze-and-Excitation (SE) (Hu et al., 2018) attention and
Revised: we further compare our proposed ACLA against other forms of attention modules that are widely used in the CV community, including Squeeze-and-Excitation (SE) (Hu et al., 2018) attention and

Original: Multi-Head Attention (MHA) (Bello et al., 2019).
Revised: Multi-Head Attention (MHA) (Bello et al., 2019).

Original: SE aim at reweighting the channel-wise responses by using soft self-attention to model interdependencies between the channels of the convolutional features.
Revised: SE aim at reweighting the channel-wise responses by using soft self-attention to model interdependencies between the channels of the convolutional features.

Original: MHA is actually a variant of self-attention from the NLP domain.
Revised: MHA is actually a variant of self-attention from the NLP domain.

Original: Speciﬁcally, MHA can be regarded as a special kind of non-local attention that takes account of the relative position information.
Revised: Speciﬁcally, MHA can be regarded as a special kind of non-local attention that takes account of the relative position information.

Original: For the experiments with SE and MHA, we insert four SE blocks and MHA blocks evenly in the
Revised: For the experiments with SE and MHA, we insert four SE blocks and MHA blocks evenly in the

Original: EDSR backbone, respectively.
Revised: EDSR backbone, respectively.

Original: The comparison is performed on 2x single-image super-resolution following the settings in section 4.2.
Revised: The comparison is performed on 2x single-image super-resolution following the settings in section 4.2.

Original: The results are displayed in Table 12.
Revised: The results are displayed in Table 12.

Original: Although MHA and SE bring improvements over the EDSR baseline.
Revised: Although MHA and SE bring improvements over the EDSR baseline.

Original: The best results are achieved by our proposed ACLA.
Revised: The best results are achieved by our proposed ACLA.

Original: Besides, we also perform an experiment that combines ACLA with SE, where SE blocks are inserted after each ACLA module.
Revised: Besides, we also perform an experiment that combines ACLA with SE, where SE blocks are inserted after each ACLA module.

Original: [Table]
Revised: [Table]

Original: A. P ERFORMANCE COMPARISON WITH OTHER ATTENTION - BASED METHODS
Revised: A. P ERFORMANCE COMPARISON WITH OTHER ATTENTION - BASED METHODS

Original: As image restoration is regarded as an ill-posed problem, improving the performance of CNN backbones for image restoration has always been a challenging task.
Revised: As image restoration is regarded as an ill-posed problem, improving the performance of CNN backbones for image restoration has always been a challenging task.

Original: Recently, attention methods have been widely used to improve performance.
Revised: Recently, attention methods have been widely used to improve performance.

Original: Here, we compare our ACLA with HAN and
Revised: Here, we compare our ACLA with HAN and

Original: SAN, which are also attention-based methods for single image super-resolution.
Revised: SAN, which are also attention-based methods for single image super-resolution.

Original: Both HAN and
Revised: Both HAN and

Original: SAN are based on the previous state-of-the-arts (SOTA) method RCAN.
Revised: SAN are based on the previous state-of-the-arts (SOTA) method RCAN.

Original: In the table below, we compared the improvements of ACLA, HAN
Revised: In the table below, we compared the improvements of ACLA, HAN

Original: [Table]
Revised: [Table]

Original: and SAN.
Revised: and SAN.

Original: The improvements over RCAN are listed in the parentheses after the PSNR results.
Revised: The improvements over RCAN are listed in the parentheses after the PSNR results.

Original: For our ALCA, the percentage comparisons of improvement are also calculated.
Revised: For our ALCA, the percentage comparisons of improvement are also calculated.

Original: For example, the improvement of ACLA over RCAN on B100 is 0.04, which is 400% of the improvement of SAN over RCAN.
Revised: For example, the improvement of ACLA over RCAN on B100 is 0.04, which is 400% of the improvement of SAN over RCAN.

Original: Besides, we have also calculated the improvement by ACLA on the EDSR backbone.
Revised: Besides, we have also calculated the improvement by ACLA on the EDSR backbone.

Original: As shown in the table, compared to HAN and SAN which are competitive baselines representing the recent progress of this literature, our method makes signiﬁcant improvements in this literature.
Revised: As shown in the table, compared to HAN and SAN which are competitive baselines representing the recent progress of this literature, our method makes signiﬁcant improvements in this literature.

Original: [Table]
Revised: [Table]

Original: Besides
Revised: Besides

Original: [Table]
Revised: [Table]

Original: we also compare the inference time between our proposed ACLA, HAN, and SAN.
Revised: we also compare the inference time between our proposed ACLA, HAN, and SAN.

Original: The running time is the average of 1000 runs on input of size 48 × 48 .
Revised: The running time is the average of 1000 runs on input of size 48 × 48 .

Original: The running time is evaluated on a single 16G Tesla V100.
Revised: The running time is evaluated on a single 16G Tesla V100.

Original: As shown Table 12, our method EDSR+ACLA achieves even better performance than HAN with much less inference time.
Revised: As shown Table 12, our method EDSR+ACLA achieves even better performance than HAN with much less inference time.

Original: A. V ISUALIZATION OF SELECTED KEYS BY ACLA
Revised: A. V ISUALIZATION OF SELECTED KEYS BY ACLA

Original: We present more examples on visualization of selected keys by ACLA in Figure 4 to demonstrate the superiority of our method in searching for informative keys for query feature.
Revised: We present more examples on visualization of selected keys by ACLA in Figure 4 to demonstrate the superiority of our method in searching for informative keys for query feature.

Original: The visualization is based on our results for 2 × image super-resolution.
Revised: The visualization is based on our results for 2 × image super-resolution.

Original: Similar to Figure 2, the ﬁrst row shows the positions of the keys selected by ACLA with K = 16 .
Revised: Similar to Figure 2, the ﬁrst row shows the positions of the keys selected by ACLA with K = 16 .

Original: For comparison, the positions of keys with top- 16 attention weights following the vanilla CLNL attention formulation in Equation (3) is displayed in the second row.
Revised: For comparison, the positions of keys with top- 16 attention weights following the vanilla CLNL attention formulation in Equation (3) is displayed in the second row.

Original: From left to the right are the sampled key positions from the 3rd, 12nd, 26th, and 31st resblock.
Revised: From left to the right are the sampled key positions from the 3rd, 12nd, 26th, and 31st resblock.

Original: The visualization results show that ACLA adaptively selects semantically similar keys for the query feature, and its vanilla counterpart CLNL lacks such capability.
Revised: The visualization results show that ACLA adaptively selects semantically similar keys for the query feature, and its vanilla counterpart CLNL lacks such capability.

Original: For instance, in Figure 2, the query is from the ear of the elephant on the right side.
Revised: For instance, in Figure 2, the query is from the ear of the elephant on the right side.

Original: With ACLA, 60% of the selected keys across are also from the ear of the same elephant.
Revised: With ACLA, 60% of the selected keys across are also from the ear of the same elephant.

Original: Besides, among the keys selected outside the ear of the same elephant, 5 out of 11 are from the ear of the elephant on the left, which have similar textures as the ear of the elephant on the right.
Revised: Besides, among the keys selected outside the ear of the same elephant, 5 out of 11 are from the ear of the elephant on the left, which have similar textures as the ear of the elephant on the right.

Original: While with CLNL, only 39% of the selected keys are from the the ear of the elephant on the right.
Revised: While with CLNL, only 39% of the selected keys are from the the ear of the elephant on the right.

Original: Similar observations can also be found in Figure 4.
Revised: Similar observations can also be found in Figure 4.

Original: In Figure 4 (a), we pick a query point from the frame structure at the top of a gate.
Revised: In Figure 4 (a), we pick a query point from the frame structure at the top of a gate.

Original: With ACLA, 90% of the keys selected are distributed on the frame structures at the top of gates.
Revised: With ACLA, 90% of the keys selected are distributed on the frame structures at the top of gates.

Original: While with CLNL, positions from the gates and the frame structure at the balcony are also given high attention weights.
Revised: While with CLNL, positions from the gates and the frame structure at the balcony are also given high attention weights.

Original: Only 61% of the selected keys are distributed on the frame structures at the top of gates, which may limit the power of attention modules.
Revised: Only 61% of the selected keys are distributed on the frame structures at the top of gates, which may limit the power of attention modules.

Original: Similar observations can also be found Figure 4 (b) and Figure 4 (c).
Revised: Similar observations can also be found Figure 4 (b) and Figure 4 (c).

Original: In Figure 4 (b), the query is from the bridge in the middle of the image.
Revised: In Figure 4 (b), the query is from the bridge in the middle of the image.

Original: All the keys selected by ACLA are also from the bridge.
Revised: All the keys selected by ACLA are also from the bridge.

Original: In Figure 4 (c), the query is from the back of a yak.
Revised: In Figure 4 (c), the query is from the back of a yak.

Original: Most of the keys selected by
Revised: Most of the keys selected by

Original: ACLA are also located on the body of yaks.
Revised: ACLA are also located on the body of yaks.

Original: While as shown in the second row, CLNL even assigns large attention weights to positions from the grass and the background.
Revised: While as shown in the second row, CLNL even assigns large attention weights to positions from the grass and the background.

Original: Such observations strongly demonstrate the power of ACLA in searching for informative keys across different layers.
Revised: Such observations strongly demonstrate the power of ACLA in searching for informative keys across different layers.

Original: [Figure]
Revised: [Figure]

Original: Low
Revised: Low

Original: Resblock
Revised: Resblock

Original: Resblock Resblock
Revised: Resblock Resblock

Original: Resblock(
Revised: Resblock(

Original: [Figure]
Revised: [Figure]

Original: Resblock Resblock
Revised: Resblock Resblock

Original: Resblock
Revised: Resblock

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Resblock Resblock
Revised: Resblock Resblock

Original: Resblock(
Revised: Resblock(

Original: [Figure]
Revised: [Figure]

Original: A. V ISUAL R ESULTS
Revised: A. V ISUAL R ESULTS

Original: We present some visual results of our proposed methods for 4 × SR with BI degradation model in
Revised: We present some visual results of our proposed methods for 4 × SR with BI degradation model in

Original: Figure 5.
Revised: Figure 5.

Original: The visual results of CLA and ACLA are from our experiments with EDSR backbone.
Revised: The visual results of CLA and ACLA are from our experiments with EDSR backbone.

Original: Bicubic
Revised: Bicubic

Original: VDSR
Revised: VDSR

Original: EDSR
Revised: EDSR

Original: RDN
Revised: RDN

Original: RCAN
Revised: RCAN

Original: CLA
Revised: CLA

Original: ACLA
Revised: ACLA

Original: HR
Revised: HR

Original: SRCNN
Revised: SRCNN

Original: FSRCNN
Revised: FSRCNN

Original: EDSR
Revised: EDSR

Original: RDN
Revised: RDN

Original: RCAN
Revised: RCAN

Original: CLA
Revised: CLA

Original: ACLA
Revised: ACLA

Original: Bicubic
Revised: Bicubic

Original: VDSR HR
Revised: VDSR HR

Original: SRCNN
Revised: SRCNN

Original: FSRCNN
Revised: FSRCNN

Original: Bicubic
Revised: Bicubic

Original: VDSR HR
Revised: VDSR HR

Original: SRCNN
Revised: SRCNN

Original: FSRCNN
Revised: FSRCNN

Original: EDSR
Revised: EDSR

Original: RDN
Revised: RDN

Original: RCAN
Revised: RCAN

Original: CLA
Revised: CLA

Original: ACLA
Revised: ACLA