Original: B ASIC B INARY C ONVOLUTION U NIT FOR B INARIZED I MAGE R ESTORATION N ETWORK
Revised: B ASIC B INARY C ONVOLUTION U NIT FOR B INARIZED I MAGE R ESTORATION N ETWORK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Lighter and faster image restoration (IR) models are crucial for the deployment on resource-limited devices.
Revised: Lighter and faster image restoration (IR) models are crucial for the deployment on resource-limited devices.

Original: Binary neural network (BNN), one of the most promising model compression methods, can dramatically reduce the computations and parameters of full-precision convolutional neural networks (CNN).
Revised: Binary neural network (BNN), one of the most promising model compression methods, can dramatically reduce the computations and parameters of full-precision convolutional neural networks (CNN).

Original: However, there are different properties between BNN and full-precision CNN and we can hardly use the experience of designing CNN to develop BNN.
Revised: However, there are different properties between BNN and full-precision CNN, and we can hardly use the experience of designing CNN to develop BNN.

Original: In this study, we reconsider components in binary convolution, such as residual connection, BatchNorm, activation function, and structure, for IR tasks.
Revised: In this study, we reconsider components in binary convolution, such as residual connection, BatchNorm, activation function, and structure, for IR tasks.

Original: We conduct systematic analyses to explain each component’s role in binary convolution and discuss the pitfalls.
Revised: We conduct systematic analyses to explain each component’s role in binary convolution and discuss the pitfalls.

Original: Specifically, we find that residual connection can reduce the information loss caused by binarization; BatchNorm can solve the value range gap between residual connection and binary convolution; The position of the activation function dramatically affects the performance of BNN.
Revised: Specifically, we find that residual connection can reduce the information loss caused by binarization; BatchNorm can solve the value range gap between residual connection and binary convolution; The position of the activation function dramatically affects the performance of BNN.

Original: Based on our findings and analyses, we design a simple yet efficient basic binary convolution unit (BBCU).
Revised: Based on our findings and analyses, we design a simple yet efficient basic binary convolution unit (BBCU).

Original: Furthermore, we divide IR networks into four parts and specially design variants of BBCU for each part to explore the benefit of binarizing these parts.
Revised: Furthermore, we divide IR networks into four parts and specially design variants of BBCU for each part to explore the benefit of binarizing these parts.

Original: We conduct experiments on different IR tasks and our BBCU significantly outperforms other
Revised: We conduct experiments on different IR tasks, and our BBCU significantly outperforms other

Original: BNNs and lightweight models, which shows that BBCU can serve as a basic unit for binarized IR networks.
Revised: BNNs and lightweight models, which shows that BBCU can serve as a basic unit for binarized IR networks.

Original: All codes and models will be released.
Revised: All codes and models will be released.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Image restoration (IR) aims to restore a high-quality (HQ) image from its low-quality (LQ) counterpart corrupted by various degradation factors.
Revised: Image restoration (IR) aims to restore a high-quality (HQ) image from its low-quality (LQ) counterpart corrupted by various degradation factors.

Original: Typical IR tasks include image denoising, superresolution (SR), and compression artifacts reduction.
Revised: Typical IR tasks include image denoising, superresolution (SR), and compression artifacts reduction.

Original: Due to its ill-posed nature and high practical values, image restoration is an active yet challenging research topic in computer vision.
Revised: Due to its ill-posed nature and high practical values, image restoration is an active yet challenging research topic in computer vision.

Original: Recently, the deep convolutional neural network (CNN) has achieved excellent performance by learning a mapping from LQ to HQ image patches for image restoration (Chen & Pock, 2016; Zhang et al., 2018a; Tai et al., 2017; Isobe et al., 2020).
Revised: Recently, the deep convolutional neural network (CNN) has achieved excellent performance by learning a mapping from LQ to HQ image patches for image restoration (Chen & Pock, 2016; Zhang et al., 2018a; Tai et al., 2017; Isobe et al., 2020).

Original: However, most IR tasks require dense pixel prediction and the powerful performance of CNN-based models usually relies on increasing model size and computational complexity.
Revised: However, most IR tasks require dense pixel prediction and the powerful performance of CNN-based models usually relies on increasing model size and computational complexity.

Original: That requires extensive computing and memory resources.
Revised: That requires extensive computing and memory resources.

Original: While, most hand-held devices and small drones are not equipped with GPUs and enough memory to store and run the computationally expensive CNN models.
Revised: While, most hand-held devices and small drones are not equipped with GPUs and enough memory to store and run the computationally expensive CNN models.

Original: Thus, it is quite essential to largely reduce its computation and memory cost while preserving model performance to promote IR models.
Revised: Thus, it is quite essential to largely reduce its computation and memory cost while preserving model performance to promote IR models.

Original: Binary neural network (Courbariaux et al., 2016) (BNN, also known as 1-bit CNN) has been recognized as one of the most promising neural network compression methods (
Revised: Binary neural network (Courbariaux et al., 2016) (BNN, also known as 1-bit CNN) has been recognized as one of the most promising neural network compression methods (

Original: He et al., 2017; Jacob et al., 2018; Zoph & Le, 2016) for deploying models onto resource-limited devices.
Revised: He et al., 2017; Jacob et al., 2018; Zoph & Le, 2016) for deploying models onto resource-limited devices.

Original: BNN could achieve 32 × memory compression ratio and up to 64 × computational reductions on specially designed processors (Rastegari et al., 2016).
Revised: BNN could achieve 32 × memory compression ratio and up to 64 × computational reductions on specially designed processors (Rastegari et al., 2016).

Original: Nowadays, the researches of BNN mainly concentrate on high-level tasks, especially classification (Liu et al., 2018; 2020), but do not fully explored in lowlevel vision, like image denoising.
Revised: Nowadays, the researches of BNN mainly concentrate on high-level tasks, especially classification (Liu et al., 2018; 2020), but do not fully explored in lowlevel vision, like image denoising.

Original: Considering the great significance of BNN for the deployment of IR deep networks and the difference between high-level and low-level vision tasks, there is an urgent need to explore the property of BNN on low-level vision tasks and provide a simple, strong, universal, and extensible baseline for latter researches and deployment.
Revised: Considering the great significance of BNN for the deployment of IR deep networks and the difference between high-level and low-level vision tasks, there is an urgent need to explore the property of BNN on low-level vision tasks and provide a simple, strong, universal, and extensible baseline for latter researches and deployment.

Original: Recently, there have been several works exploring the application of BNN on image SR networks.
Revised: Recently, there have been several works exploring the application of BNN on image SR networks.

Original: Specifically, Ma et al.
Revised: Specifically, Ma et al.

Original: (Ma et al., 2019) tried to binarize the convolution kernel weight to decrease[Figure] the SR network model size.
Revised: (Ma et al., 2019) tried to binarize the convolution kernel weight to decrease the SR network model size.

Original: But, the computational complexity is still high due to the preservation  of full-precision activations.
Revised: But, the computational complexity is still high due to the preservation [Figure] of full-precision activations.

Original: Then BAM (Xin et al., 2020) adopted the bit accumulation mechanism to approximate the full-precision convolution for the SR network.
Revised: Then BAM (Xin et al., 2020) adopted the bit accumulation mechanism to approximate the full-precision convolution for the SR network.

Original: Zhang et al.
Revised: Zhang et al.

Original: (Zhang et al., 2021b) designed a compact uniform prior to constrain the convolution weight into a highly narrow range centered at zero for better optimization.
Revised: (Zhang et al., 2021b) designed a compact uniform prior to constrain the convolution weight into a highly narrow range centered at zero for better optimization.

Original: BTM (Jiang et al., 2021) further introduced knowledge distillation (Hinton et al., 2015) to boost the performance of binarized SR networks.
Revised: BTM (Jiang et al., 2021) further introduced knowledge distillation (Hinton et al., 2015) to boost the performance of binarized SR networks.

Original: However, the above-mentioned binarized SR networksstill can hardly achieve the potential of BNN.
Revised: However, the above-mentioned binarized SR networks can hardly achieve the potential of BNN.

Original: In this work [Figure] we explore the property of three essential components of BNN, including residual connection (He et al., 2016), BatchNorm (BN)layer (Ioffe & Szegedy, 2015), and activation function (Glorot et al., 2011). We develop an efficient and  strong basic binary Conv unit (BBCU) based on our analyses.
Revised: In this work, we explore the properties of three key components of BNN, including residual connection (He et al., 2016), BatchNorm (BN) (Ioffe & Szegedy, 2015), and activation function (Glorot et al., 2011) and design a strong basic binary Conv unit (BBCU) based on our analyses.

Original: (1) For the IR tasks, we find that residual connection is quite important for binarized IR networks.
Revised: (1) For the IR tasks, we observe that residual connection is quite important for binarized IR networks That is because that BNN will binarize the input

Revised: full

Revised: -precision

Original: That is because that BNN will binarize the input full-precision activations toor -1 before binary convolution (BC).
Revised: activations to 1 or - before binary convolution (BC).

Original: It means that BNN would lose a large amount of information about the value range of activations.
Revised: It means that BNN would lose a large amount of information about the value range of activations.

Original: By adding the full-precision residual connection for each binary convolution (BC), BNN can reduce the effect of value range information loss.
Revised: By adding the full-precision residual connection for each binary convolution (BC), BNN can reduce the effect of value range information loss.

Original: (2) Then, we explore the BathcNorm for BBCU.
Revised: (2) Then, we explore the BN for BBCU.

Original: BNN methods (Liu et al., 2020) for classification always adopt a BatchNorm in BC.
Revised: BNN methods (Liu et al., 2020) for classification always adopt a BN in BC.

Original: However, in IR tasks, EDSR (Lim et al., 2017) has demonstrated that BatchNorm is harmful to SR performance.
Revised: However, in IR tasks, EDSR (Lim et al., 2017) has demonstrated that BN is harmful to SR performance.

Original: We find that BatchNorm in BNN for IR tasks is useful and can be used to balance the value range of residual connection and BC.
Revised: We find that BN in BNN for IR tasks is useful and can be used to balance the value range of residual connection and BC.

Original: Specifically, as shown in Fig.
Revised: Specifically, as shown in Fig.

Original: 2 (b), the values of the full-precision residual connection are mostly in the range of - 1 to 1 because the value range of input images is around 0 to 1 or -1 to 1, while the values of the BC without BN are large and ranges from -15 to 15 for the bit counting operation.
Revised: 2 (b), the values of the full-precision residual connection are mostly in the range of - 1 to 1 because the value range of input images is around 0 to 1 or -1 to 1, while the values of the BC without BN are large and ranges from -15 to 15 for the bit counting operation (Fig. 7).

Original: The value range of BC is larger than that of the residual connection, which covers the preserved full-precision image information in the residual connection and limits the performance of BNN.
Revised: The value range of BC is larger than that of the residual connection, which covers the preserved full-precision image information in the residual connection and limits the performance of BNN.

Original: In Fig.
Revised: In Fig.

Original: 2 (a), the BatchNorm in BNN for image restoration can realize the value range alignment of the residual connection and BC.
Revised: 2 (a), the BN in BNN for image restoration can realize the value range alignment of the residual connection and BC.

Original: (3) Based on these findings, to remove BatchNorm, we propose a residual alignment (RA) scheme by multiplying the input image by an amplification factor k to increase the value range of the residual connection rather than using BatchNorm to narrow the value range of the BC (Fig.
Revised: (3) Based on these findings, to remove BN, we propose a residual alignment (RA) scheme by multiplying the input image by an amplification factor k to increase the value range of the residual connection rather than using BN to narrow the value range of the BC (Fig.

Original: 2 (c)).
Revised: 2 (c)).

Original: Using this scheme to remove BatchNorm not only further improves the performance of binarized IR networks but also simplifies the BBCU structure (Sec. 4.5).
Revised: Using this scheme can improve the performance of binarized IR networks and simplify the BNN structure (Sec. 4.5).

Original: (4) In Fig.
Revised: (4) In Fig.

Original: 2 (d), different from BNNs (Liu et al., 2020; 2018) for classification, we further move the activation function into the residual connection and  improve performance (Sec. 4.5).
Revised: 2 (d), different from BNNs (Liu et al., 2020; 2018) for classification, we further move the activation function into the residual connection and can improve performance (Sec. 4.5).

Original: That is because activation function would narrow the negative value ranges of residual connection.
Revised: That is because activation function would narrow the negative value ranges of residual connection.

Original: Its information would be covered by the next BC with large negative value ranges (Fig.
Revised: Its information would be covered by the next BC with large negative value ranges (Fig.

Original: 2 (c)).
Revised: 2 (c)).

Original:  Furthermore, we divide IR networks into four parts: head, body, upsampling, and tail (Fig. 3 (a)).
Revised: (5) Furthermore, we divide IR networks into four parts: head, body, upsampling, and tail (Fig.(a)).

Original: These four parts have different input and output channel numbers.
Revised: These four parts have different input and output channel numbers.

Original: Previous binarized SR networks (Xin et al., 2020; Jiang et al., 2021) merely binarize the body part.
Revised: Previous binarized SR networks (Xin et al., 2020; Jiang et al., 2021) merely binarize the body part.

Original: However, upsampling part accounts for 52 .
Revised: However, upsampling part accounts for 52 .

Original: 3% total calculations and needs to be binarized.
Revised: 3% total calculations and needs to be binarized.

Original: Besides, the binarized head and tail parts are also worth exploring.
Revised: Besides, the binarized head and tail parts are also worth exploring.

Original: Thus, wespecially design different variants of the basic binary Conv unit (BBCU) scheme for these four parts (Fig.(b)).
Revised: Thus, we design different variants of the BBCU to binarize these four parts (Fig. 3 (b)).

Original: Overall, our contributions can be mainly summarized in threefold:
Revised: Overall, our contributions can be mainly summarized in threefold:

Original: • We believe our work is timely.
Revised: • We believe our work is timely.

Original: The high computational and memory cost of IR networks hinder their application on resource-limited devices.
Revised: The high computational and memory cost of IR networks hinder their application on resource-limited devices.

Original: BNN, as one of the most promising compression methods, can help IR networks to solve this dilemma.
Revised: BNN, as one of the most promising compression methods, can help IR networks to solve this dilemma.

Original: Since the BNN-based networks have different properties from full-precision CNN networks, we reconsider, analyze, and visualize some essential components of BNN to explore their functions.
Revised: Since the BNN-based networks have different properties from full-precision CNN networks, we reconsider, analyze, and visualize some essential components of BNN to explore their functions.

Original: • According to our findings and analyses on BNN, we specially develop a simple, strong, universal, and extensible basic binary Conv unit (BBCU) for IR networks.
Revised: • According to our findings and analyses on BNN, we specially develop a simple, strong, universal, and extensible basic binary Conv unit (BBCU) for IR networks.

Original: Furthermore, we develop variants of BBCU and adapt it to different parts of IR networks.
Revised: Furthermore, we develop variants of BBCU and adapt it to different parts of IR networks.

Original: • Extensive experiments on different IR tasks show that BBCU can outperform the SOTA BNN methods (Fig. 1).
Revised: • Extensive experiments on different IR tasks show that BBCU can outperform the SOTA BNN methods (Fig. 1).

Original: BBCU can serve as a strong basic binary convolution unit for future binarized IR networks, which is meaningful to academic research and industry.
Revised: BBCU can serve as a strong basic binary convolution unit for future binarized IR networks, which is meaningful to academic research and industry.

Revised: The remainder of the paper is organized as follows.

Revised: In Sec. 2, we review related recent literature in image restoration and binary neural network.

Revised: We describe our key observations, analyses, and structure of BBCU in Sec. 3.

Revised: In Sec. 4, we evaluate the performance of BBCU on various IR tasks (e.g., SR, denoising, deblocking) and further validate BBCU.

Revised: We conclude in Sec. 5.

Original:  R ELATED W ORK 2. I MAGE R ESTORATION
Revised:  R ELATED W ORK 2. I MAGE R ESTORATION

Original: As pioneer works, SRCNN (Dong et al., 2015b), DnCNN (Zhang et al., 2017), and ARCNN (Dong et al., 2015a) use the compact networks for image super-resolution, image denoising, and compression artifacts reduction, respectively.
Revised: As pioneer works, SRCNN (Dong et al., 2015b), DnCNN (Zhang et al., 2017), and ARCNN (Dong et al., 2015a) use the compact networks for image super-resolution, image denoising, and compression artifacts reduction, respectively.

Original: Since then, researchers had carried out its study with different perspectives and obtained more elaborate neural network architecture designs and learning strategies, such as residual block (Kim et al., 2016; Zhang et al., 2021a; Cavigelli et al., 2017), dense block (Zhang et al., 2018c; 2020; Wang et al., 2018), attention mechanism (Zhang et al., 2018b; Xia et al., 2022a;
Revised: Since then, researchers had carried out its study with different perspectives and obtained more elaborate neural network architecture designs and learning strategies, such as residual block (Kim et al., 2016; Zhang et al., 2021a; Cavigelli et al., 2017), dense block (Zhang et al., 2018c; 2020; Wang et al., 2018), attention mechanism (Zhang et al., 2018b; Xia et al., 2022a;

Original: Dai et al., 2019), GAN (Gulrajani et al., 2017; Wang et al., 2018), and others (Wei et al., 2021; Peng et al., 2019; Jia et al., 2019; Fu et al., 2019; Kim et al., 2019; Fu et al., 2021; Soh et al., 2020; Xia et al., 2022b), to improve model representation ability.
Revised: Dai et al., 2019), GAN (Gulrajani et al., 2017; Wang et al., 2018), and others (Wei et al., 2021; Peng et al., 2019; Jia et al., 2019; Fu et al., 2019; Kim et al., 2019; Fu et al., 2021; Soh et al., 2020; Xia et al., 2022b), to improve model representation ability.

Original: However, these IR networks require high computational and memory costs, which hinders practical application on edge devices.
Revised: However, these IR networks require high computational and memory costs, which hinders practical application on edge devices.

Original: To address this issue, we explore and design a BBCU for binarized IR networks.
Revised: To address this issue, we explore and design a BBCU for binarized IR networks.

Original: 2. B INARY N EURAL N ETWORKS
Revised: 2. B INARY N EURAL N ETWORKS

Original: Binary neural network (BNN) is the most extreme form of model quantization as it quantizes convolution weights and activations to only 1 bit, achieving great speed-up compared with its full-precision counterpart.
Revised: Binary neural network (BNN) is the most extreme form of model quantization as it quantizes convolution weights and activations to only 1 bit, achieving great speed-up compared with its full-precision counterpart.

Original: As a pioneer work, BNN (Courbariaux et al., 2016) directly applied binarization to a full-precision model with a pre-defined binarization function.
Revised: As a pioneer work, BNN (Courbariaux et al., 2016) directly applied binarization to a full-precision model with a pre-defined binarization function.

Original: Afterward, XNOR-Net (Rastegari et al., 2016) adopted a gain term to compensate for lost information to improve the performance of
Revised: Afterward, XNOR-Net (Rastegari et al., 2016) adopted a gain term to compensate for lost information to improve the performance of

Original: BNN (Courbariaux et al., 2016).
Revised: BNN (Courbariaux et al., 2016).

Original: After that, Bi-Real (Liu et al., 2018) introduced residual connection to preserve full-precision information in forward propagation.
Revised: After that, Bi-Real (Liu et al., 2018) introduced residual connection to preserve full-precision information in forward propagation.

Original: IRNet
Revised: IRNet

Original: (Qin et al., 2020) developed a scheme to retain the information in forward and backward propagations.
Revised: (Qin et al., 2020) developed a scheme to retain the information in forward and backward propagations.

Original: Recently, ReActNet (Liu et al., 2020) proposed generalized activation functions to learn more representative features.
Revised: Recently, ReActNet (Liu et al., 2020) proposed generalized activation functions to learn more representative features.

Original: For super-resolution task, Ma et al.
Revised: For super-resolution task, Ma et al.

Original: (2019) binarized convolution weights to save model size.
Revised: (2019) binarized convolution weights to save model size.

Original: However, it can hardly speed up inference enough for retaining full-precision activations.
Revised: However, it can hardly speed up inference enough for retaining full-precision activations.

Original: Then, Xin et al.
Revised: Then, Xin et al.

Original: (2020) further binarized activations and used the bit accumulation mechanism to approximate the full-precision convolution for SR networks.
Revised: (2020) further binarized activations and used the bit accumulation mechanism to approximate the full-precision convolution for SR networks.

Original: Besides, Zhang et al.
Revised: Besides, Zhang et al.

Original: (2021b) introduced a compact uniform prior for better optimization.
Revised: (2021b) introduced a compact uniform prior for better optimization.

Original: Subsequently, Jiang et al.
Revised: Subsequently, Jiang et al.

Original: (2021) designed a binary training mechanism by adjusting the feature distribution.
Revised: (2021) designed a binary training mechanism by adjusting the feature distribution.

Original: In this paper, we reconsider the function of each basic component in BC and develop a strong, simple, and efficient BBCU.
Revised: In this paper, we reconsider the function of each basic component in BC and develop a strong, simple, and efficient BBCU.

Original:  P ROPOSED M ETHOD 3. B ASIC B INARY C ONV U NIT D ESIGN
Revised:  P ROPOSED M ETHOD 3. B ASIC B INARY C ONV U NIT D ESIGN

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2(a), we first construct the BBCU-V1.
Revised: 2(a), we first construct the BBCU-V1.

Original: Specifically, the full-precision convolution X fj ⊗W fj  is approximated by the binary convolution X bj ⊗W bj .
Revised: Specifically, the full-precision convolution X fj ⊗ W fj ( X fj , W fj , and ⊗ are full-precision activations, weights, and Conv respectively) is approximated by the binary convolution X bj ⊗ W bj .

Original: For binary convolution, both weights and activations are binarized to -

Revised: For binary convolution, both weights and activations are binarized to -

Original: and +1.
Revised: and +1.

Original: Efficient bitwise XNOR and bit counting operations can replace computationally heavy floating-point matrix multiplication, which can be defined as:
Revised: Efficient bitwise XNOR and bit counting operations can replace computationally heavy floating-point matrix multiplication, which can be defined as:

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Original: where X fj ∈ R C
Revised: where X fj ∈ R C

Original: × H × W and W fj ∈ R C out × C in ×
Revised: × H × W and W fj ∈ R C out × C in ×

Original: K h
Revised: K h

Original: × K w are full-precision activations and convolution weights in j -th layer, respectively.
Revised: × K w are full-precision activations and convolution weights in j -th layer, respectively.

Original: Similarly, X bj ∈ R C × H × W and W bj ∈ R C out × C in × K
Revised: Similarly, X bj ∈ R C × H × W and W bj ∈ R C out × C in × K

Original: h
Revised: h

Original: ×
Revised: ×

Original: K w are binarized activations and convolution weights in j -th layer separately.
Revised: K w are binarized activations and convolution weights in j -th layer separately.

Original: x f i,j , w f i,j , x b i,j , and w b i,j [Figure] are the elements of i -th channel of X fj , W fj , X bj , and W bj respectively.
Revised: x fi,j , w fi,j , x bi,j , and w bi,j are the elements of i -th channel of X fj , W fj , X bj , and W bj respectively.

Original: α i,j is the learnable coefficient controlling the threshold of sign function for i -th channel of X fj .
Revised: α i,j is the learnable coefficient controlling the threshold of sign function for i -th channel of X fj .

Original: It is notable that the weight binarization method is inherited from XONR-Net
Revised: It is notable that the weight binarization method is inherited from XONR-Net

Original: (Rastegari et al., 2016), of which ∥ W fj ∥ 1 n is the average of absolute weight values and acts as a scaling factor to minimize the difference between binary and full-precision convolution weights.
Revised: (Rastegari et al., 2016), of which ∥ W fj ∥ 1 n is the average of absolute weight values and acts as a scaling factor to minimize the difference between binary and full-precision convolution weights.

Original: Then, we use the RPReLU (Liu et al., 2020) as our activation function, which is defined as follows:
Revised: Then, we use the RPReLU (Liu et al., 2020) as our activation function, which is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where Y j ∈ R C × H × W is the input feature maps of RPReLU function f ( . ) in j -th layer.
Revised: where Y j ∈ R C × H × W is the input feature maps of RPReLU function f ( . ) in j -th layer.

Original: y i,j is the element of i -th channel of Y j .
Revised: y i,j is the element of i -th channel of Y j .

Original: γ i,j and ζ i,j are learnable shifts for moving the distribution.
Revised: γ i,j and ζ i,j are learnable shifts for moving the distribution.

Original: β i,j is a learnable coefficient controlling the slope of the negative part, which acts on i -th channel of Y j .
Revised: β i,j is a learnable coefficient controlling the slope of the negative part, which acts on i -th channel of Y j .

Original: Different from the common residual block, which consists of two convolutions used in full-precision
Revised: Different from the common residual block, which consists of two convolutions used in full-precision

Original: IR network (Lim et al., 2017), we find that residual connection is essential for binary convolution to supplement the information loss caused by binarization.
Revised: IR network (Lim et al., 2017), we find that residual connection is essential for binary convolution to supplement the information loss caused by binarization.

Original: Thus, we set a residual connection for each binary convolution.
Revised: Thus, we set a residual connection for each binary convolution.

Original: Therefore, the BBCU-V1 can be expressed mathematically as:
Revised: Therefore, the BBCU-V1 can be expressed mathematically as:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where κ j , τ j ∈ R C are learnable parameters of BatchNorm in j -th layer.
Revised: where κ j , τ j ∈ R C are learnable parameters of BatchNorm in j -th layer.

Original: In BNNs, the derivative of the sign function in Eq.
Revised: In BNNs, the derivative of the sign function in Eq.

Original: 2 is an impulse function that cannot be utilized in training.
Revised: 2 is an impulse function that cannot be utilized in training.

Original: Thus, we adopt the approximated derivative function as the derivative of the sign function.
Revised: Thus, we adopt the approximated derivative function as the derivative of the sign function.

Original: It can be expressed mathematically as:
Revised: It can be expressed mathematically as:

Original: [Equation]
Revised: [Equation]

Original: otherwise .
Revised: However, EDSR (Lim et al., 2017) demonstrated that BN changes the distribution of image, which is harmful to accurate pixel prediction in SR.

Original: [Equation] [Equation] [Figure]
Revised: So, can we also directly remove the BN in BBCUV1 to obtain BBCU-V2 (Fig.

Original: (Fig. 2 (b))?
Revised: 2 (b))?

Original: In BBCU-V1 (Fig. 2 (a)), we can see that the bit counting operation of binary convolution tends to output large value ranges (from -15 to 15).
Revised: In BBCU-V1 (Fig. 2 (a)), we can see that the bit counting operation of binary convolution tends to output large value ranges (from -15 to 15).

Original: While, residual connection preserves the full-precision information, which flows from the front end of IR network with a  value range from around -1 to 1.
Revised: In contrast, residual connection preserves the full-precision information, which flows from the front end of IR network with a samll value range from around -1 to 1.

Original: By adding a BatchNorm, its learnable parameters can narrow the value range of binary convolution and make it close to the value range of residual connection(from around -1 to 1) to avoid full-precision information being covered.
Revised: By adding a BN, its learnable parameters can narrow the value range of binary convolution and make it close to the value range of residual connection to avoid full-precision information being covered.

Original: The process can be expressed as:
Revised: The process can be expressed as:

Original: [Equation]
Revised: [Equation]

Original:  where κ j , τ j ∈ R C are learnable parameters of BatchNorm in j -th layer.
Revised: [Equation] where κ j , τ j ∈ R C are learnable parameters of BN in j -th layer.

Original: Thus, compared with 
Revised: Thus, compared with BBCU-V1,

Original: BBCU-V1, BBCU-V2 simply removes BatchNorm and suffers a huge performance drop.
Revised: BBCU-V2 simply removes BN and suffers a huge performance drop.

Original: After the above exploration, we know that BatchNorm is essential for BBCU, because it can balance the value range of binary convolution and residual connection.
Revised: After the above exploration, we know that BN is essential for BBCU, because it can balance the value range of binary convolution and residual connection.

Original: Hence, as shown in Figs. 2(c) and 3(b), we propose residual alignment (RA) scheme by multiplying the value range of input image by an amplification factor k ( k > 1) to remove BatchNorm:
Revised: However, BN changes image distributions limiting restoration. In BBCU-V3, we propose residual alignment (RA) scheme by multiplying the value range of input image by an amplification factor k ( k > 1) to remove BN Figs. 2(c) and 3(b):

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We can see from the Eq. 8, since the residual connection flows from the amplified input image, the value range of X fj also is amplified, which we define as k X fj .
Revised: We can see from the Eq. 8, since the residual connection flows from the amplified input image, the value range of X fj also is amplified, which we define as k X fj .

Original: Meanwhile, the values of binary convolution are almost not affected by RA, because X bj filters amplitude information of k X fj .
Revised: Meanwhile, the values of binary convolution are almost not affected by RA, because X bj filters amplitude information of k X fj .

Original: Different from BatchNorm, RA makes the value range of residual connection close to binary convolution (- to 60).
Revised: Different from BatchNorm, RA makes the value range of residual connection close to binary convolution (- to 60).

Original: Besides, we find that using RA to remove the BatchNorm has two main benefits: (1) Similar to full-precision IR networks, the binarized IR networks would have better performance without
Revised: Besides, we find that using RA to remove the BatchNorm has two main benefits: (1) Similar to full-precision IR networks, the binarized IR networks would have better performance without

Original: BatchNorm.
Revised: BatchNorm.

Original: The structure of BBCU becomes more simple, efficient, and strong.
Revised: The structure of BBCU becomes more simple, efficient, and strong.

Original: Based on the above findings, we are aware that the activation function (Eq. 4) in BBCU-V (Fig. 2(c)) narrows the negative value range of residual connection, which means it loses negative full-precision information.
Revised: Based on the above findings, we are aware that the activation function (Eq. 4) in BBCU-V (Fig. 2(c)) narrows the negative value range of residual connection, which means it loses negative full-precision information.

Original: Thus, we further develop BBCU-V4 by moving the activation function into the residual connection to avoid losing the negative full-precision information.
Revised: Thus, we further develop BBCU-V4 by moving the activation function into the residual connection to avoid losing the negative full-precision information.

Original: The experiments (Sec. 4.5) show that our design is accurate.
Revised: The experiments (Sec. 4.5) show that our design is accurate.

Original: We then take BBCU-V4 as the final design of BBCU.
Revised: We then take BBCU-V4 as the final design of BBCU.

Original: 3. A RM IR N ETWORKS WITH BBCU
Revised: 3. A RM IR N ETWORKS WITH BBCU

Original: As shown in Fig. 3(a), the existing image restoration (IR) networks could be divided into four parts: head H , body B , upsampling U , and tail T .
Revised: As shown in Fig. 3(a), the existing image restoration (IR) networks could be divided into four parts: head H , body B , upsampling U , and tail T .

Original: If the IR networks do not need increasing resolution (Zhang et al., 2017), it can remove the upsampling U part.
Revised: If the IR networks do not need increasing resolution (Zhang et al., 2017), it can remove the upsampling U part.

Original: Specifically, given an LQ input I LQ , the process of IR network restoring HQ output ˆ I HQ can be formulated as:
Revised: Specifically, given an LQ input I LQ , the process of IR network restoring HQ output ˆ I HQ can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: Previous BNN SR works (Xin et al., 2020; Jiang et al., 2021;
Revised: Previous BNN SR works (Xin et al., 2020; Jiang et al., 2021;

Original: Zhang et al., 2021b) concentrate on binarizing the body B part.
Revised: Zhang et al., 2021b) concentrate on binarizing the body B part.

Original: However, upsampling U part accounts for 52.3 % total calculations and is essential to be binarized.
Revised: However, upsampling U part accounts for 52.3 % total calculations and is essential to be binarized.

Original: Besides, the binarized head H and tail T parts are also worth exploring.
Revised: Besides, the binarized head H and tail T parts are also worth exploring.

Original: As shown in Fig. 3(b), we further design different variants of BBCU for these four parts.
Revised: As shown in Fig. 3(b), we further design different variants of BBCU for these four parts.

Original: (1) For the head H part, its input is I LQ ∈ R 3 × H × W and the binary convolution output with C channels.
Revised: (1) For the head H part, its input is I LQ ∈ R 3 × H × W and the binary convolution output with C channels.

Original: Thus, we cannot directly add I LQ to the binary convolution output for the difference in the number of channels.
Revised: Thus, we cannot directly add I LQ to the binary convolution output for the difference in the number of channels.

Original: To address the issue, we develop BBCU-H by repeating I LQ to have C channels.
Revised: To address the issue, we develop BBCU-H by repeating I LQ to have C channels.

Original: For body B part, since the input and output channels are the same, we develop BBCU-B by directly adding the input activation to the binary convolution output.
Revised: For body B part, since the input and output channels are the same, we develop BBCU-B by directly adding the input activation to the binary convolution output.

Original: (3) For upsampling U part, we develop
Revised: (3) For upsampling U part, we develop

Original: BBCU-U by repeating the channels of input activations to add with the binary convolution.
Revised: BBCU-U by repeating the channels of input activations to add with the binary convolution.

Original: (4) For the tail T part, we develop BBCU-T by adopting I LQ as the residual connection.
Revised: (4) For the tail T part, we develop BBCU-T by adopting I LQ as the residual connection.

Original: To better evaluate  the benefits of binarizing each part on computations and parameters, we define two metrics:[Equation]
Revised: To better evaluate [Figure] the benefits of binarizing each part on computations and parameters, we define two metrics:

Original: [Equation]
Revised: [Equation]

Original: where PSNR b , OPs b , and Parms b denote the performance, calculations, and parameters after binarizing one part of networks.
Revised: where PSNR b , OPs b , and Parms b denote the performance, calculations, and parameters after binarizing one part of networks.

Original: Similarly, PSNR f , OPs f , and Parms f measure full-precision networks.
Revised: Similarly, PSNR f , OPs f , and Parms f measure full-precision networks.

Original: We adopt reconstruction loss L rec to guide the image restoration training, which is defined as:
Revised: We adopt reconstruction loss L rec to guide the image restoration training, which is defined as:

Original: [Equation]
Revised: [Equation]

Original: where I HQ and ˆ I HQ are the real and restored HQ images, respectively.
Revised: where I HQ and ˆ I HQ are the real and restored HQ images, respectively.

Original: ∥ · ∥ 1 denotes the L 1 norm.
Revised: ∥ · ∥ 1 denotes the L 1 norm.

Original:  E XPERIMENTS 4. E XPERIMENT S ETTINGS
Revised:  E XPERIMENTS 4. E XPERIMENT S ETTINGS

Original: Training Strategy.
Revised: Training Strategy.

Original: We apply our method to three typical image restoration tasks: image superresolution, color image denoising, and image compression artifacts reduction.
Revised: We apply our method to three typical image restoration tasks: image superresolution, color image denoising, and image compression artifacts reduction.

Original: We train all models on
Revised: We train all models on

Original: DIV2K (Agustsson & Timofte, 2017), which contains 800 high-quality images.
Revised: DIV2K (Agustsson & Timofte, 2017), which contains 800 high-quality images.

Original: Besides, we adopt widely used test sets for evaluation and report PSNR and SSIM.
Revised: Besides, we adopt widely used test sets for evaluation and report PSNR and SSIM.

Original: For image super-resolution, we take simple and practical SRResNet
Revised: For image super-resolution, we take simple and practical SRResNet

Original: (Ledig et al., 2017) as the backbone.
Revised: (Ledig et al., 2017) as the backbone.

Original: The mini-batch containsimages with the size of 192 × 192 randomly cropped from training data.
Revised: The mini-batch containsimages with the size of 192 × 192 randomly cropped from training data.

Original: We set the initial learning rate to 1 × 10 − 4 , train models with 300 epochs, and perform halving every 200 epochs.
Revised: We set the initial learning rate to 1 × 10 − 4 , train models with 300 epochs, and perform halving every 200 epochs.

Original: For image denoising and compression artifacts reduction, we take DnCNN and DnCNN3 as backbone (Zhang et al., 2017).
Revised: For image denoising and compression artifacts reduction, we take DnCNN and DnCNN3 as backbone (Zhang et al., 2017).

Original: The mini-batch contains 32 images with the size of 64 × 64 randomly cropped from training data.
Revised: The mini-batch contains 32 images with the size of 64 × 64 randomly cropped from training data.

Original: We set the initial learning rate to 1 × 10 − 4 , train models with 300,000 iterations, and perform halving per 200,000 iterations.
Revised: We set the initial learning rate to 1 × 10 − 4 , train models with 300,000 iterations, and perform halving per 200,000 iterations.

Original: The amplification factor k in the residual alignment is set to 130.
Revised: The amplification factor k in the residual alignment is set to 130.

Original: We implement our models with a Tesla V100 GPU.
Revised: We implement our models with a Tesla V100 GPU.

Original: OPs and Parameters Calculation of BNN.
Revised: OPs and Parameters Calculation of BNN.

Original: Following Rastegari et al.
Revised: Following Rastegari et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Original: , the operations of BNN (OPs b ) is calculated by OPs b = OPs f / 64 (OPs f indicates FLOPs), and the parameters of BNN (Parms b ) is calculated by Parms b =
Revised: , the operations of BNN (OPs b ) is calculated by OPs b = OPs f / 64 (OPs f indicates FLOPs), and the parameters of BNN (Parms b ) is calculated by Parms b =

Original: Parms f / 32.
Revised: Parms f / 32.

Original: 4. E VALUATION ON I MAGE S UPER -R ESOLUTION
Revised: 4. E VALUATION ON I MAGE S UPER -R ESOLUTION

Original: Following BAM (Xin et al., 2020), we take SRResNet
Revised: Following BAM (Xin et al., 2020), we take SRResNet

Original: (Ledig et al., 2017) as backbone.
Revised: (Ledig et al., 2017) as backbone.

Original: We binarize body B part of SRResNet with some SOTA BNNs including BNN (Courbariaux et al., 2016),
Revised: We binarize body B part of SRResNet with some SOTA BNNs including BNN (Courbariaux et al., 2016),

Original: BiReal (Liu et al., 2018), IRNet (Qin et al., 2020), ReActNet (Liu et al., 2020), and BAM (Xin et al., 2020), BTM (Jiang et al., 2021).
Revised: BiReal (Liu et al., 2018), IRNet (Qin et al., 2020), ReActNet (Liu et al., 2020), and BAM (Xin et al., 2020), BTM (Jiang et al., 2021).

Original: For comparison, we adopt our BBCU to binarize body B part to obtain
Revised: For comparison, we adopt our BBCU to binarize body B part to obtain

Original: BBCU-L.
Revised: BBCU-L.

Original: Furthermore, we binarize body B and upsampling U parts simultaneously to develop
Revised: Furthermore, we binarize body B and upsampling U parts simultaneously to develop

Original: BBCU-M. Besides, we reduce the number of channels of SRResNet to 12, obtaining SRResNetLite.

Original: In addition, we use Set5 (Bevilacqua et al., 2012), Set14 (Zeyde et al., 2010), B100 (Martin et al., 2001), Urban100 (Huang et al., 2015), and Manga109 (Matsui et al., 2017) for evaluation.

Original: The quantitative results (PSNR and SSIM), the number of parameters, and operations of different methods are shown in Tab. 1.

Original: Compared with other binarized methods, our BBCU-L achieves the

Original: [Table]
Revised: [Table]

Original: [Figure] -M (Ours) 3.
Revised: 0.7603 29.25 0.8912 BBCU-M (Ours) 3.

Original: 0.7263 25.35 0.7602 29.22 0.8918 BBCU-L (Ours) 79.
Revised: 0.7263 25.35 0.7602 29.22 0.8918 BBCU-L (Ours) 79.

Original: HR
Revised: HR

Original: SRResNet-Lite
Revised: SRResNet-Lite

Original: Bi-Real
Revised: Bi-Real

Original: BBCU-M
Revised: BBCU-M

Original: Bicubic
Revised: Bicubic

Original: SRResNet
Revised: SRResNet

Original: ReActNet
Revised: ReActNet

Original: BBCU-L
Revised: BBCU-L

Revised: HR

Revised: SRResNet-Lite

Revised: Bi-Real

Revised: BBCU-M

Revised: Bicubic

Revised: SRResNet

Revised: ReActNet

Revised: BBCU-L

Original: HR
Revised: HR

Original: SRResNet-Lite 
Revised: SRResNet-Lite Bi-Real BBCU-M

Original: Bi-Real
Revised: Bicubic SRResNet ReActNet

Original: BBCU-M
Revised: BBCU-L

Original: Bicubic SRResNet ReActNet
Revised: HR

Original: BBCU-L HR SRResNet-Lite
Revised: SRResNet-Lite

Original: Bi-Real
Revised: Bi-Real

Original: BBCU-M
Revised: BBCU-M

Original: Bicubic
Revised: Bicubic

Original: SRResNet
Revised: SRResNet

Original: ReActNet
Revised: ReActNet

Original: BBCU-L
Revised: BBCU-L

Original: HR SRResNet-Lite Bi-Real BBCU-M
Revised: BBCU-M. Besides, we reduce the number of channels of SRResNet to 12, obtaining SRResNetLite.

Original: Bicubic
Revised: In addition, we use Set5 (Bevilacqua et al., 2012), Set14 (Zeyde et al., 2010), B100 (Martin et al., 2001), Urban100 (Huang et al., 2015), and Manga109 (Matsui et al., 2017) for evaluation.

Original: SRResNet
Revised: The quantitative results (PSNR and SSIM), the number of parameters, and operations of different methods are shown in Tab. 1.

Original: ReActNet BBCU-L  best results on all benchmarks and all scale factors.
Revised: Compared with other binarized methods, our BBCU-L achieves the best results on all benchmarks and all scale factors.

Original: Specifically, for 4 × SR, our BBCU-L surpasses
Revised: Specifically, for 4 × SR, our BBCU-L surpasses

Original: ReActNet by 0.25dB, 0.27dB, and 0.44dB on Set5, Urban100, and Manga109, respectively.
Revised: ReActNet by 0.25dB, 0.27dB, and 0.44dB on Set5, Urban100, and Manga109, respectively.

Original: For 2 ×
Revised: For 2 ×

Original: SR, BBCU-L also achieves 0.32dB, 0.27dB, and 0.58dB improvement on these three benchmarks compared with ReActNet.
Revised: SR, BBCU-L also achieves 0.32dB, 0.27dB, and 0.58dB improvement on these three benchmarks compared with ReActNet.

Original: Furthermore, our BBCU-M achieves the second best performance on most benchmarks consuming 5% of operations and 13 .
Revised: Furthermore, our BBCU-M achieves the second best performance on most benchmarks consuming 5% of operations and 13 .

Original: 7% parameters of ReActNet (Liu et al., 2020) on 4 × SR.
Revised: 7% parameters of ReActNet (Liu et al., 2020) on 4 × SR.

Original: Besides, BBCU-L significantly outperforms SRResNet-Lite by 0.16dB and 0.3dB with less computational cost, showing the superiority of BNN.
Revised: Besides, BBCU-L significantly outperforms SRResNet-Lite by 0.16dB and 0.3dB with less computational cost, showing the superiority of BNN.

Original: The qualitative results are shown in Fig. 4, and our BBCU-L has the best visual quality containing more realistic details close to respective ground-truth HQ images.
Revised: The qualitative results are shown in Fig. 4, and our BBCU-L has the best visual quality containing more realistic details close to respective ground-truth HQ images.

Original: More qualitative results are provided in appendix.
Revised: More qualitative results are provided in appendix.

Original: 4. E VALUATION ON I MAGE D ENOISING
Revised: 4. E VALUATION ON I MAGE D ENOISING

Original: For image denoising, we use DnCNN (Zhang et al., 2017) as the backbone and binarize its body B part with BNN methods, including BNN, Bi-Real, IRNet, BTM, ReActNet, and our BBCU.
Revised: For image denoising, we use DnCNN (Zhang et al., 2017) as the backbone and binarize its body B part with BNN methods, including BNN, Bi-Real, IRNet, BTM, ReActNet, and our BBCU.

Original: We also develop DnCNN-Lite by reducing the number of channels of DnCNN from 64 to 12.
Revised: We also develop DnCNN-Lite by reducing the number of channels of DnCNN from 64 to 12.

Original: The standard benchmarks: Urban100 (Huang et al., 2015), BSD68 (Martin et al., 2001), and Set12 (Shan et al.,  2019) are applied to evaluate each method.
Revised: The standard benchmarks: Urban100 (Huang et al., 2015), BSD68 (Martin et al., 2001), and Set12 (Shan et al., [Table] [Table] 2019) are applied to evaluate each method.

Original: Additive white Gaussian noises (AWGN) with different noise levels σ (15, 25, 50) are added to the clean images.
Revised: Additive white Gaussian noises (AWGN) with different noise levels σ (15, 25, 50) are added to the clean images.

Original: The quantitative results of image denoising are shown in Tab. 2, respectively.
Revised: The quantitative results of image denoising are shown in Tab. 2, respectively.

Original: As one can see, our
Revised: As one can see, our

Original: BBCU achieves the best performance among compared BNNs.
Revised: BBCU achieves the best performance among compared BNNs.

Original: In particular, our BBCU surpasses the state-of-the-art BNN model ReActNet by 0.82dB, 0.88dB, and 1dB on CBSD68, Kodak24, and[Table] [Table]
Revised: In particular, our BBCU surpasses the state-of-the-art BNN model ReActNet by 0.82dB, 0.88dB, and 1dB on CBSD68, Kodak24, and

Original: Urban100 datasets respectively as σ = 15 .
Revised: Urban100 datasets respectively as σ = 15 .

Original: Compared with DnCNN-Lite, our BBCU surpasses it by 0.92dB, 1.03dB, and 1.5dB on these three benchmarks as σ = 15 consuming 58% computations of
Revised: Compared with DnCNN-Lite, our BBCU surpasses it by 0.92dB, 1.03dB, and 1.5dB on these three benchmarks as σ = 15 consuming 58% computations of

Original: DnCNN-Lite.
Revised: DnCNN-Lite.

Original: Qualitative results are provided in appendix.
Revised: Qualitative results are provided in appendix.

Original: 4. E VALUATION ON JPEG C OMPRESSION A RTIFACT R EDUCTION
Revised: 4. E VALUATION ON JPEG C OMPRESSION A RTIFACT R EDUCTION

Original: For this JPEG compression deblocking, we use practical DnCNN-

Revised: For this JPEG compression deblocking, we use practical DnCNN-

Original: (Zhang et al., 2017) as the backbone and replace the full-precision body B part of DnCNN3 with some competitive BNN methods, including BNN, Bi-Real, IRNet, BTM, ReActNet, and our BBCU.
Revised: (Zhang et al., 2017) as the backbone and replace the full-precision body B part of DnCNN3 with some competitive BNN methods, including BNN, Bi-Real, IRNet, BTM, ReActNet, and our BBCU.

Original: The compressed images are generated by Matlab standard JPEG encoder with quality factors q ∈ { 10 , 20 , 30 , 40 } .
Revised: The compressed images are generated by Matlab standard JPEG encoder with quality factors q ∈ { 10 , 20 , 30 , 40 } .

Original: We take the widely used LIVE1 (Sheikh, 2005) and Classic

Revised: We take the widely used LIVE1 (Sheikh, 2005) and Classic

Original: (Foi et al., 2007) as test sets to evaluate the performance of each method.
Revised: (Foi et al., 2007) as test sets to evaluate the performance of each method.

Original: The quantitative results are presented in Tab. 3.
Revised: The quantitative results are presented in Tab. 3.

Original: As we can see, our
Revised: As we can see, our

Original: BBCU achieves the best performance on all test sets and quality factors among all compared BNNs.
Revised: BBCU achieves the best performance on all test sets and quality factors among all compared BNNs.

Original: Specifically, our BBCU surpasses the state-of-the-art BNN model ReActNet by 0.38dB and 0.34dB on the Live1 and Classic5 as q = 40 .
Revised: Specifically, our BBCU surpasses the state-of-the-art BNN model ReActNet by 0.38dB and 0.34dB on the Live1 and Classic5 as q = 40 .

Original: In addition, our BBCU surpasses DnCNN-3-Lite by 0.16dB and 0.2dB on benchmarks as q = 10 .
Revised: In addition, our BBCU surpasses DnCNN-3-Lite by 0.16dB and 0.2dB on benchmarks as q = 10 .

Original: The visual comparisons are provided in appendix.
Revised: The visual comparisons are provided in appendix.

Original:  A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: [Table]
Revised: [Table]

Original: Basic Binary Convolution Unit.
Revised: Basic Binary Convolution Unit.

Original: To show the effectiveness of BBCU for the binarized IR network, we binarize the body part of SRResNet with four types of BBCU (Fig. 2) separately.
Revised: To validate BBCU for the binarized IR network, we binarize the body part of SRResNet with four variants of BBCU (Fig. 2) separately.

Original: The results are shown in Tab. 4.
Revised: The results are shown in Tab. 4.

Original: Compared withthe performance of BBCU-V1,the figure for BBCU-V2 declines by 0.14dB and 0.24dB on Urban100 and Manga109.
Revised: Compared with BBCU-V1, BBCU-V2 declines by 0.14dB and 0.24dB on Urban100 and Manga109.

Original: That is becausethat BBCU-V2 simply removes the BatchNorm making the value range of binary Conv far larger than residual connection and cover full-precision information (Fig. 2).
Revised: This is because BBCU-V2 simply removes the BN making the value range of binary Conv far larger than residual connection and cover full-precision information (Fig. 2).

Original: (2) BBCU-V3further adds residual alignment schemebased on BBSCM-V2, which not only address the value range imbalance between binary Conv and residual connection but also remove the BatchNorm.
Revised: (2) BBCU-V3 adds residual alignment scheme on BBCU-V2, which addresses the value range imbalance between binary Conv and residual connection and removes the BN.

Original: Since the BatchNorm isactually harmful for IR network training, our BBCU-V3 significantly outperform BBCU-V1.
Revised: Since the BN is harmful for IR networks, BBCU-V3 surpasses BBCU-V1 by 0.17dB, 0.16dB, and 0.29dB on Set5,

Original: Specifically, BBCU-V3 surpasses BBCU-V1 by 0.17dB, 0.16dB, and 0.29dB on Set5, Urban100, and Manga109 respectively.
Revised: Urban100, and Manga109 respectively.

Original: (3) BBCU-V4 moves the activation function into the residual connection, which preserves the full-precision negative values of residual connection (Fig. 2).
Revised: (3) BBCU-V4 moves the activation function into the residual connection, which preserves the full-precision negative values of residual connection (Fig. 2).

Original: Thus, BBCU-V4 improves BBCU-V3.
Revised: Thus, BBCU-V4 outperforms BBCU-V3.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Residual Connection.
Revised: Residual Connection.

Original: We take SRResNet as the backbone with 32 Convs in body part and explore the relationship between performance and the number of binary convolution in residual connection.
Revised: We take SRResNet as the backbone with 32 Convs in body part and explore the relationship between performance and the number of binary convolution in residual connection.

Original: Specifically, for both full-precision and binarized SRResNet, we set 1,2,4,8,16, and 32 Convs with a residual connection as basic convolution units, respectively.
Revised: Specifically, for both full-precision and binarized SRResNet, we set 1,2,4,8,16, and 32 Convs with a residual connection as basic convolution units, respectively.

Original: We evaluate SRResNet equipped with these basic convolution units on 4 × Urban100 (see Fig. 5).
Revised: We evaluate SRResNet equipped with these basic convolution units on 4 × Urban100 (see Fig. 5).

Original: (1) For full-precision networks, it is best to have 2 Convs with a residual connection to form a basic convolution unit.
Revised: (1) For full-precision networks, it is best to have 2 Convs with a residual connection to form a basic convolution unit.

Original: Besides, if there are more than 4 Convs in a basic convolution unit, its performance would sharply drop.
Revised: Besides, if there are more than 4 Convs in a basic convolution unit, its performance would sharply drop.

Original: For binarized networks, it is best to equip each binary convolution with a residual connection.
Revised: For binarized networks, it is best to equip each binary convolution with a residual connection.

Original: In addition, compared with the full-precision network, the binarized network is more insensitive to the growth of the Convs number in the basic convolution unit.
Revised: In addition, compared with the full-precision network, the binarized network is more insensitive to the growth of the Convs number in the basic convolution unit.

Original: (2) For binarized SRResNet, we delete residual connection of a BBCU in a certain position.
Revised: (2) For binarized SRResNet, we delete residual connection of a BBCU in a certain position.

Original: In Tab. 5, it seems that once the residual connection is removed (the full-precision information is lost) at any position, the binarized SRResNet would suffer a severe performance drop.
Revised: In Tab. 5, it seems that once the residual connection is removed (the full-precision information is lost) at any position, the binarized SRResNet would suffer a severe performance drop.

Original: Amplification Factor.
Revised: Amplification Factor.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 6, the performance of full-precision remain basically unchanged with the variation of amplification factor k .
Revised: 6, the performance of full-precision remain basically unchanged with the variation of amplification factor k .

Original: However, the binarized network is sensitive to the k .
Revised: However, the binarized network is sensitive to the k .

Original: The best k ∗ is related to the number of channels n , which empirically fits k ∗ =
Revised: The best k ∗ is related to the number of channels n , which empirically fits k ∗ =

Original: 130 n/ 64 .
Revised: 130 n/ 64 .

Original: Intuitively, the larger number of channels makes binary convolution count more elements and have larger results, which needs larger k to increase the value of residual connection for balance.
Revised: Intuitively, the larger number of channels makes binary convolution count more elements and have larger results, which needs larger k to increase the value of residual connection for balance.

Original: The Binarization Benefit for Different Parts of IR Network.
Revised: The Binarization Benefit for Different Parts of IR Network.

Original: We binarize one part in SRResNet with BBCU in Fig.
Revised: We binarize one part in SRResNet with BBCU in Fig.

Original: 3 while keeping other parts full-precision.
Revised: 3 while keeping other parts full-precision.

Original: The results are shown in Tab. 6.
Revised: The results are shown in Tab. 6.

Original: We use V C (Eq. 10) and V P (Eq. 11) as metric to value binarization benefit of different parts.
Revised: We use V C (Eq. 10) and V P (Eq. 11) as metric to value binarization benefit of different parts.

Original: We can see that Upsampling part is most worth to be binarized.
Revised: We can see that Upsampling part is most worth to be binarized.

Original: However, it is ignored by previous works (Xin et al., 2020).
Revised: However, it is ignored by previous works (Xin et al., 2020).

Original: The binarization benefit of first and last convolution is relatively low.
Revised: The binarization benefit of first and last convolution is relatively low.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This work devotes attention to exploring the performance of BNN on low-level tasks and search of generic and efficient basic binary convolution unit.
Revised: This work devotes attention to exploring the performance of BNN on low-level tasks and search of generic and efficient basic binary convolution unit.

Original: Through decomposing and analyzing existing elements, we propose BBCU, a simple yet effective basic binary convolution unit, that outperforms existing state-of-the-arts with high efficiency.
Revised: Through decomposing and analyzing existing elements, we propose BBCU, a simple yet effective basic binary convolution unit, that outperforms existing state-of-the-arts with high efficiency.

Original: Furthermore, we divide IR networks to four parts, and specially develop the variants of BBCU for them to explore the binarization benefits.
Revised: Furthermore, we divide IR networks to four parts, and specially develop the variants of BBCU for them to explore the binarization benefits.

Original: Overall,
Revised: Overall,

Original: BBCU provide insightful analyses on BNN and can serve as strong basic binary convolution unit for future binarized IR networks, which is meaningful to academic research and industry.
Revised: BBCU provide insightful analyses on BNN and can serve as strong basic binary convolution unit for future binarized IR networks, which is meaningful to academic research and industry.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We provide the reproducibility statement of our proposed method in this section.
Revised: We provide the reproducibility statement of our proposed method in this section.

Original: We introduce the our finding about each basic component’s function in BNN and the design of basic binary convolution unit in Sec. 3.1.
Revised: We introduce the our finding about each basic component’s function in BNN and the design of basic binary convolution unit in Sec. 3.1.

Original: In addition, we introduce the realization in Sec. 3.2.
Revised: In addition, we introduce the realization in Sec. 3.2.

Original: We provide the implementation details and experiment settings in Sec. 4.
Revised: We provide the implementation details and experiment settings in Sec. 4.

Original: To ensure the reproducibility, we provide source code and pre-trained models in our supplementary materials.
Revised: To ensure the reproducibility, we provide source code and pre-trained models in our supplementary materials.

Original: Everyone can run our code to check results according to the given instructions
Revised: Everyone can run our code to check results according to the given instructions

Original: A A PPENDIX A. M ORE D ETAILED I NTRODUCTION ON BNN
Revised: A A PPENDIX A. M ORE D ETAILED I NTRODUCTION ON BNN

Original: Compared to the full-precision CNN model with the 32-bit weight parameter the Binary Neural
Revised: Compared to the full-precision CNN model with the 32-bit weight parameter the Binary Neural

Original: Network (BNN) obtains up to 32 × memory saving.
Revised: Network (BNN) obtains up to 32 × memory saving.

Original: In addition, as the activations are also binarized the binary convolution operation could be implemented by the bitwise XNOR operation and a bit counting operation (Rastegari et al., 2016).
Revised: In addition, as the activations are also binarized the binary convolution operation could be implemented by the bitwise XNOR operation and a bit counting operation (Rastegari et al., 2016).

Original: Fig.
Revised: Fig.

Original: 7 shows a simple example of binary convolution.
Revised: 7 shows a simple example of binary convolution.

Original: Different from full-precision convolution requiring float point operation, the binary convolution relies on the bitwise operation and could obtain up to 64 × computation saving (Rastegari et al., 2016).
Revised: Different from full-precision convolution requiring float point operation, the binary convolution relies on the bitwise operation and could obtain up to 64 × computation saving (Rastegari et al., 2016).

Original: -1 1 -1 1-1 -1 1 1 1 -1 1Binarized Activations 1 11 -1-1 1∗ Binarized Weights = -

Revised: -1 1 -1 1-1 -1 1 1 1 -1 1Binarized Activations 1 11 -1-1 1∗ Binarized Weights = -

Original: 1 -3 3 57 -5 1 3 -31 3 1 -7 5-1 3 1 1 -5-5 -1 3 1 -13 -1 1 -1 3Output -1 1 1 -1 -1 1 -1 -11 1 -1 1 -1 1 -1 

Revised: 1 -3 3 57 -5 1 3 -31 3 1 -7 5-1 3 1 1 -5-5 -1 3 1 -13 -1 1 -1 3Output -1 1 1 -1 -1 1 -1 -11 1 -1 1 -1 1 -1

Original: XNOR -1 

Revised: XNOR -1

Original: Bit Counting
Revised: Bit Counting

Original: A. M ORE V ISUAL C OMPARISON OF BNN S FOR I MAGE S UPER -R ESOLUTION
Revised: A. M ORE V ISUAL C OMPARISON OF BNN S FOR I MAGE S UPER -R ESOLUTION

Original: In this section, we provide more 4 × SR visual results of BNNs applied to SRResNet (Ledig et al., 2017) in Fig.
Revised: In this section, we provide more 4 × SR visual results of BNNs applied to SRResNet (Ledig et al., 2017) in Fig.

Original: We can see that our BBCU-L achieves the best visual quality among all comparing methods.
Revised: We can see that our BBCU-L achieves the best visual quality among all comparing methods.

Original: Besides, our the visual qualities of BBCU-M are also better than SRResNet-Lite
Revised: Besides, our the visual qualities of BBCU-M are also better than SRResNet-Lite

Original: . A. M ORE V ISUAL C OMPARISON OF BNN S FOR I MAGE D ENOISING
Revised: . A. M ORE V ISUAL C OMPARISON OF BNN S FOR I MAGE D ENOISING

Original: In this section, we provide additional denoising visual results of BNNs applied to DnCNN (Zhang et al., 2017) in Fig.
Revised: In this section, we provide additional denoising visual results of BNNs applied to DnCNN (Zhang et al., 2017) in Fig.

Original: We can see that our BBCU can remove heavy noise corruption and preserve high-frequency image details, resulting in sharper edges and more natural textures.
Revised: We can see that our BBCU can remove heavy noise corruption and preserve high-frequency image details, resulting in sharper edges and more natural textures.

Original: By contrast, other BNN or lightweight methods suffer from either over-smoothness or over-sharpness.
Revised: By contrast, other BNN or lightweight methods suffer from either over-smoothness or over-sharpness.

Original: A. M ORE V ISUAL C OMPARISON OF BNN S FOR I MAGE OR D EBLOCKING
Revised: A. M ORE V ISUAL C OMPARISON OF BNN S FOR I MAGE OR D EBLOCKING

Original: In this section, we provide additional deblocking visual results of BNNs applied to DnCNN3 (Zhang et al., 2017) in Fig.
Revised: In this section, we provide additional deblocking visual results of BNNs applied to DnCNN3 (Zhang et al., 2017) in Fig.

Original: We can see that our BBCU can better remove severe JPEG compression artifacts and restore more visual pleasant image textures compared with other BNN and lightweight methods.
Revised: We can see that our BBCU can better remove severe JPEG compression artifacts and restore more visual pleasant image textures compared with other BNN and lightweight methods.

Original: HR
Revised: HR

Original: SRResNet-Lite
Revised: SRResNet-Lite

Original: Bi-Real
Revised: Bi-Real

Original: BBCU-M
Revised: BBCU-M

Original: Bicubic
Revised: Bicubic

Original: SRResNet
Revised: SRResNet

Original: ReActNet
Revised: ReActNet

Original: BBCU-L
Revised: BBCU-L

Original: HR
Revised: HR

Original: SRResNet-Lite
Revised: SRResNet-Lite

Original: Bi-Real
Revised: Bi-Real

Original: BBCU-M
Revised: BBCU-M

Original: Bicubic
Revised: Bicubic

Original: SRResNet
Revised: SRResNet

Original: ReActNet
Revised: ReActNet

Original: BBCU-L
Revised: BBCU-L

Original: HR
Revised: HR

Original: SRResNet-Lite
Revised: SRResNet-Lite

Original: Bi-Real
Revised: Bi-Real

Original: BBCU-M
Revised: BBCU-M

Original: Bicubic
Revised: Bicubic

Original: SRResNet
Revised: SRResNet

Original: ReActNet
Revised: ReActNet

Original: BBCU-L
Revised: BBCU-L

Original: HR
Revised: HR

Original: SRResNet-Lite
Revised: SRResNet-Lite

Original: Bi-Real
Revised: Bi-Real

Original: BBCU-M
Revised: BBCU-M

Original: Bicubic
Revised: Bicubic

Original: SRResNet
Revised: SRResNet

Original: ReActNet
Revised: ReActNet

Original: BBCU-L
Revised: BBCU-L

Original: [Figure]
Revised: [Figure]

Original: ReActNet
Revised: ReActNet

Original: [Figure]
Revised: [Figure]

Original: DnCNN
Revised: DnCNN

Original: [Figure]
Revised: [Figure]

Original: ReActNet
Revised: ReActNet

Original: [Figure]
Revised: [Figure]

Original: DnCNN
Revised: DnCNN

Original: [Figure]
Revised: [Figure]

Original: DnCNN
Revised: DnCNN

Original: [Figure]
Revised: [Figure]

Original: Lite
Revised: Lite

Original: BNN
Revised: BNN

Original: ReActNet
Revised: ReActNet

Original: [Figure]
Revised: [Figure]

Original: DnCNN
Revised: DnCNN

Original: [Figure]
Revised: [Figure]

Original: DnCNN
Revised: DnCNN

Original: [Figure]
Revised: [Figure]

Original: ReActNet
Revised: ReActNet

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: ReActNet
Revised: ReActNet

Original: [Figure]
Revised: [Figure]

Original: DnCNN
Revised: DnCNN

Original: Bi-Real
Revised: Bi-Real

Original: BBCU
Revised: BBCU

Original: [Figure]
Revised: [Figure]

Original: DnCNN3-Lite
Revised: DnCNN3-Lite

Original: BNN
Revised: BNN

Original: ReActNet
Revised: ReActNet

Original: [Figure]
Revised: [Figure]

Original: DnCNN
Revised: DnCNN

Original: Bi-Real
Revised: Bi-Real

Original: BBCU
Revised: BBCU

Original: [Figure]
Revised: [Figure]

Original: DnCNN3-Lite
Revised: DnCNN3-Lite

Original: BNN
Revised: BNN

Original: ReActNet
Revised: ReActNet

Original: [Figure]
Revised: [Figure]

Original: DnCNN
Revised: DnCNN

Original: Bi-Real
Revised: Bi-Real

Original: BBCU
Revised: BBCU

Original: [Figure]
Revised: [Figure]

Original: DnCNN3-Lite
Revised: DnCNN3-Lite

Original: BNN
Revised: BNN

Original: ReActNet
Revised: ReActNet

Original: [Figure]
Revised: [Figure]

Original: DnCNN
Revised: DnCNN

Original: Bi-Real
Revised: Bi-Real

Original: [Figure]
Revised: [Figure]