Original: Learning Lipschitz-Controlled Activation Functions in Neural Networks for Plug-and-Play Image Reconstruction Methods
Revised: Learning Lipschitz-Controlled Activation Functions in Neural Networks for Plug-and-Play Image Reconstruction Methods

Original: Abstract
Revised: Abstract

Original: Ill-posed linear inverse problems are frequently encountered in image reconstruction tasks.
Revised: Ill-posed linear inverse problems are frequently encountered in image reconstruction tasks.

Original: Image reconstruction methods that combine the Plug-and-Play (PnP)priors framework with convolutional neural network (CNN) based denoisers haveshown impressive performances.
Revised: Image reconstruction methods that combine the Plug-and-Play (PnP) priors framework with convolutional neural network (CNN) based denoisers have shown impressive performances.

Original: However, it is non-trivial to guarantee the convergence of such algorithms, which is necessary for sensitive applications such asmedical imaging.
Revised: However, it is non-trivial to guarantee the convergence of such algorithms, which is necessary for sensitive applications such as medical imaging.

Original: It has been shown that PnP algorithms converge when deployedwith a certain class of averaged denoising operators.
Revised: It has been shown that PnP algorithms converge when deployed with a certain class of averaged denoising operators.

Original: While such averaged operatorscan be built from 1-Lipschitz CNNs, imposing such a constraint on CNNs usuallyleads to a severe drop in performance.
Revised: While such averaged operators can be built from 1-Lipschitz CNNs, imposing such a constraint on CNNs usually leads to a severe drop in performance.

Original: To mitigate this effect, we propose the use ofdeep spline neural networks which beneﬁt from learnable piecewise-linear splineactivation functions.
Revised: To mitigate this effect, we propose the use of deep spline neural networks which beneﬁt from learnable piecewise-linear spline activation functions.

Original: We introduce “slope normalization” to control the Lipschitzconstant of these activation functions.
Revised: We introduce “slope normalization” to control the Lipschitz constant of these activation functions.

Original: We show that averaged denoising operatorsbuilt from 1-Lipschitz deep spline networks consistently outperform those builtfrom 1-Lipschitz ReLU networks.
Revised: We show that averaged denoising operators built from 1-Lipschitz deep spline networks consistently outperform those built from 1-Lipschitz ReLU networks.

Original:  Introduction
Revised:  Introduction

Original: Linear inverse problems are ubiquitous in medical imaging.
Revised: Linear inverse problems are ubiquitous in medical imaging.

Original: There, the goal is to reconstruct animage x ∈ R K from measurements
Revised: There, the goal is to reconstruct an image x ∈ R K from measurements

Original: y =
Revised: y =

Original: Hx + n ∈ R M .
Revised: Hx + n ∈ R M .

Original: The linear operator H : R K → R Mmodels the acquisition system and n ∈ R M is an additive measurement noise.
Revised: The linear operator H : R K → R M models the acquisition system and n ∈ R M is an additive measurement noise.

Original: These problems aretypically ill-posed and additional information about x is required to obtain meaningful solutions.
Revised: These problems are typically ill-posed and additional information about x is required to obtain meaningful solutions.

Original: Inthe variational framework for solving an inverse problem, one formulates it as an optimization task
Revised: In the variational framework for solving an inverse problem, one formulates it as an optimization task

Original: [Equation]
Revised: [Equation]

Original: [Equation] where the data-ﬁdelity term E : R M × R M → R + imposes closeness between the solution estimateand the acquired measurements, the regularization term R : R K → R + imposes some prior knowledgeon the image of interest, and λ ∈ R + is a tunable hyperparameter.
Revised: where the data-ﬁdelity term E : R M × R M → R + imposes closeness between the solution estimate and the acquired measurements, the regularization term R : R K → R + imposes some prior knowledge on the image of interest, and λ ∈ R + is a tunable hyperparameter.

Original: The cost functional in (1) is thentypically minimized using proximal algorithms such as forward-backward splitting (FBS) [10] andthe alternating direction method of multipliers (ADMM) [5].
Revised: The cost functional in (1) is then typically minimized using proximal algorithms such as forward-backward splitting (FBS) [10] and the alternating direction method of multipliers (ADMM) [5].

Original: The main idea in the Plug-and-Play (PnP) priors framework [24, 8] is to replace the proximal operatorof R in the iterations of proximal algorithms with some denoiser, even though it might not correspondto an explicit regularization term.
Revised: The main idea in the Plug-and-Play (PnP) priors framework [24, 8] is to replace the proximal operator of R in the iterations of proximal algorithms with some denoiser, even though it might not correspond to an explicit regularization term.

Original: This implicit regularization approach has been shown to yield betterresults than conventional variational methods for a variety of inverse problems since it allows the useof powerful denoisers such as NLM [6], WNNM [13], BM3D [11], and neural networks [25, 17, 20],which have emerged as state-of-the-art.
Revised: This implicit regularization approach has been shown to yield better results than conventional variational methods for a variety of inverse problems since it allows the use of powerful denoisers such as NLM [6], WNNM [13], BM3D [11], and neural networks [25, 17, 20], which have emerged as state-of-the-art.

Original: However, the delicate point that remains is ensuring the convergence of these algorithms, which is non-trivial but essential for sensitive applications such asthe ones encountered in medical imaging.
Revised: However, the delicate point that remains is ensuring the convergence of these algorithms, which is non-trivial but essential for sensitive applications such as the ones encountered in medical imaging.

Original: There exist several works that analyze conditions on the denoiser under which PnP algorithms areguaranteed to converge [18, 21, 7, 19, 12].
Revised: There exist several works that analyze conditions on the denoiser under which PnP algorithms are guaranteed to converge [18, 21, 7, 19, 12].

Original: For example, Ryu et al.
Revised: For example, Ryu et al.

Original: [17] show that PnP-FBS and PnPADMM provably converge to ﬁxed points if the denoiser obeys an appropriate Lipschitz condition.
Revised: [17] show that PnP-FBS and PnPADMM provably converge to ﬁxed points if the denoiser obeys an appropriate Lipschitz condition.

Original: They then propose a practical way to enforce the derived Lipschitz constraint while training neuralnetwork denoisers.
Revised: They then propose a practical way to enforce the derived Lipschitz constraint while training neural network denoisers.

Original: However, their analysis requires the data-ﬁdelity term to be strongly convex andthis unfortunately rules out ill-posed inverse problems.
Revised: However, their analysis requires the data-ﬁdelity term to be strongly convex and this unfortunately rules out ill-posed inverse problems.

Original: In order to design convergent PnP schemesfor ill-posed problems, stricter conditions need to be enforced on the denoiser.
Revised: In order to design convergent PnP schemes for ill-posed problems, stricter conditions need to be enforced on the denoiser.

Original: More speciﬁcally, ithas been shown that averagedness (ﬁrm nonexpansiveness) of the denoiser is sufﬁcient to guaranteeﬁxed point convergence of PnP-FBS (PnP-ADMM)
Revised: More speciﬁcally, it has been shown that averagedness (ﬁrm nonexpansiveness) of the denoiser is sufﬁcient to guarantee ﬁxed point convergence of PnP-FBS (PnP-ADMM)

Original: The design and training of constrainedneural networks to satisfy the averagedness or ﬁrm nonexpansiveness conditions is a challenging taskand is an emerging direction of research [22, 14].
Revised: The design and training of constrained neural networks to satisfy the averagedness or ﬁrm nonexpansiveness conditions is a challenging task and is an emerging direction of research [22, 14].

Original: In this work, we look at the problem of training 1 -Lipschitz 1 (nonexpansive) neural networks inorder to construct averaged denoisers.
Revised: In this work, we look at the problem of training 1 -Lipschitz 1 (nonexpansive) neural networks in order to construct averaged denoisers.

Original: Several techniques have been proposed in the literature tocontrol the Lipschitz constant of ReLU networks such as spectral normalization [16] and Parsevalframes [9].
Revised: Several techniques have been proposed in the literature to control the Lipschitz constant of ReLU networks such as spectral normalization [16] and Parseval frames [9].

Original: However, it has been observed that constraining ReLU networks may lead to a drop indenoising performance.
Revised: However, it has been observed that constraining ReLU networks may lead to a drop in denoising performance.

Original: While the nonexpansiveness constraint reduces the capacity of the model,the performance drop could also be imputed to the choice of the model and the training scheme.
Revised: While the nonexpansiveness constraint reduces the capacity of the model, the performance drop could also be imputed to the choice of the model and the training scheme.

Original: Here, we propose to mitigate this drop in performance by using more adaptable models.
Revised: Here, we propose to mitigate this drop in performance by using more adaptable models.

Original: In particular,we consider deep spline neural networks [23] where the activation functions are learnable linearsplines.
Revised: In particular, we consider deep spline neural networks [23] where the activation functions are learnable linear splines.

Original: They are known to improve the performance relative to ReLU networks [4] and are amenableto a control of the Lipschitz constant [2].
Revised: They are known to improve the performance relative to ReLU networks [4] and are amenable to a control of the Lipschitz constant [2].

Original: We present an efﬁcient way to train 1-Lipschitz deepspline neural networks using “slope normalization”, which can be viewed as the counterpart ofspectral normalization for activation functions.
Revised: We present an efﬁcient way to train 1-Lipschitz deep spline neural networks using “slope normalization”, which can be viewed as the counterpart of spectral normalization for activation functions.

Original: We then apply our trained denoisers to the PnP-FBSframework for compressed sensing MRI and show the beneﬁts of our approach.
Revised: We then apply our trained denoisers to the PnP-FBS framework for compressed sensing MRI and show the beneﬁts of our approach.

Original: Contributions.
Revised: Contributions.

Original: We provide proof of concept that 1-Lipschitz deep spline neural network denoiserswith learnable activation functions improve the performance of provably convergent PnP algorithmsfor ill-posed linear inverse problems.
Revised: We provide proof of concept that 1-Lipschitz deep spline neural networks with learnable activation functions improve the performance of provably convergent PnP algorithms for ill-posed linear inverse problems.

Original:  Methods
Revised:  Methods

Original: In this work, we focus on the PnP-FBS algorithm.
Revised: In this work, we focus on the PnP-FBS algorithm.

Original: At each iteration, the estimate is updated as
Revised: At each iteration, the estimate is updated as

Original: [Equation]
Revised: [Equation]

Original: where D: R K → R K is the plugged-in denoiser and α ∈ R + is the gradient step size.
Revised: where D: R K → R K is the plugged-in denoiser and α ∈ R + is the gradient step size.

Original: We assumethat E : R M × R M → R + is convex in the second argument and is differentiable with L -Lipschitzcontinuous gradient.
Revised: We assume that E : R M × R M → R + is convex in the second argument and is differentiable with L -Lipschitz continuous gradient.

Original: It has been shown that for α ∈ (0 , 2 /L ) , PnP-FBS is guaranteed to converge to aﬁxed point if D: R K → R K is an averaged operator D = β R + (1 − β )Id , where R is a 1 -Lipschitz(nonexpansive) operator and β ∈ (0 , 1) [3, 14].
Revised: It has been shown that for α ∈ (0 , 2 /L ) , PnP-FBS is guaranteed to converge to a ﬁxed point if D: R K → R K is an averaged operator D = β R + (1 − β )Id , where R is a 1 -Lipschitz (nonexpansive) operator and β ∈ (0 , 1) [3, 14].

Original: Keeping this convergence result in mind, we now consider the problem of training an averageddenoiser where R is a neural network whose Lipschitz constant must be constrained to be at most one.
Revised: Keeping this convergence result in mind, we now consider the problem of training an averaged denoiser where R is a neural network whose Lipschitz constant must be constrained to be at most one.

Original: We take R to be a simple convolutional deep spline network [23] consisting of a series of alternatingconvolutional layers and pointwise nonlinear transformations.
Revised: We take R to be a simple convolutional deep spline network [23] consisting of a series of alternating convolutional layers and pointwise nonlinear transformations.

Original: In contrast to the standard ReLUnetwork, we have a learnable piecewise-linear spline activation function for each output channel ofthe preceding convolutional layer.
Revised: In contrast to the standard ReLU network, we have a learnable piecewise-linear spline activation function for each output channel of the preceding convolutional layer.

Original: The spline nonlinearities are represented using linear B-splinebasis functions that are compactly supported, allowing for an efﬁcient computation of the forwardand backward passes during training [4] (see Appendix for more details).
Revised: The spline nonlinearities are represented using linear B-spline basis functions that are compactly supported, allowing for an efﬁcient computation of the forward and backward passes during training [4] (see Appendix for more details).

Original: The Lipschitz constant of the deep spline network is upper-bounded by the product of the Lipschitzconstants of the individual layers.
Revised: The Lipschitz constant of the deep spline network is upper-bounded by the product of the Lipschitz constants of the individual layers.

Original: Therefore, in order to ensure that the network is 1 -Lipschitz, weconstrain the Lipschitz constants of each convolutional and spline activation functions to be at mostone.
Revised: Therefore, in order to ensure that the network is 1 -Lipschitz, we constrain the Lipschitz constants of each convolutional and spline activation functions to be at most one.

Original: To do this during the training process, we use the argument that if an operator T is L -Lipschitz, then the normalized operator (1 /L )T is 1-Lipschitz.
Revised: To do this during the training process, we use the argument that if an operator T is L -Lipschitz, then the normalized operator (1 /L )T is 1-Lipschitz.

Original: The convolution operation can be viewed as amatrix-vector product and its Lipschitz constant is the spectral norm (maximum singular value) ofthis matrix.
Revised: The convolution operation can be viewed as a matrix-vector product and its Lipschitz constant is the spectral norm (maximum singular value) of this matrix.

Original: We rely on the real spectral normalization (real-SN) method [17] to control the Lipschitzconstant of the convolutional layers.
Revised: We rely on the real spectral normalization (real-SN) method [17] to control the Lipschitz constant of the convolutional layers.

Original: Real-SN uses power iterations (without having to build theconvolution matrix explicitly) to efﬁciently compute the maximum singular value and normalizes theconvolution kernel accordingly.
Revised: Real-SN uses power iterations (without having to build the convolution matrix explicitly) to efﬁciently compute the maximum singular value and normalizes the convolution kernel accordingly.

Original: The Lipschitz constant of a linear spline function is the maximumabsolute value of its derivative.
Revised: The Lipschitz constant of a linear spline function is the maximum absolute value of its derivative.

Original: Since our learnable activation functions have a ﬁnite number ofpieces, we are able to compute the maximum absolute slope efﬁciently and normalize the function bythis constant to ensure that it is 1 -Lipschitz.
Revised: Since our learnable activation functions have a ﬁnite number of pieces, we are able to compute the maximum absolute slope efﬁciently and normalize the function by this constant to ensure that it is 1 -Lipschitz.

Original: We term this process “slope normalization”: it is theactivation-function counterpart of spectral normalization.
Revised: We term this process “slope normalization”: it is the activation-function counterpart of spectral normalization.

Original:  Experimental Results
Revised:  Experimental Results

Original: We now present some experimental results to demonstrate the advantages of the 1 -Lipschitz deepspline networks over the corresponding ReLU versions.
Revised: We now present some experimental results to demonstrate the advantages of the 1 -Lipschitz deep spline networks over the corresponding ReLU versions.

Original: We tackle the compressed sensing MRI(CS-MRI) problem [17] using the PnP-FBS algorithm with neural networks trained for Gaussiannoise suppression that are constrained to be averaged operators.
Revised: We tackle the compressed sensing MRI (CS-MRI) problem [17] using the PnP-FBS algorithm with neural networks trained for Gaussian noise suppression that are constrained to be averaged operators.

Original: We ﬁrst evaluate the performancesof the deep spline and ReLU based denoisers in Section 3.1.
Revised: We ﬁrst evaluate the performances of the deep spline and ReLU based denoisers in Section 3.1.

Original: The performances of these denoisers forthe CS-MRI problem are reported in Section 3.2.
Revised: The performances of these denoisers for the CS-MRI problem are reported in Section 3.2.

Original: 3.1 Evaluation of Gaussian Denoisers
Revised: 3.1 Evaluation of Gaussian Denoisers

Original: We train Gaussian denoisers of the form D = β R + (1 − β )Id , with R being a CNN.
Revised: We train Gaussian denoisers of the form D = β R + (1 − β )Id , with R being a CNN.

Original: The CNNis chosen to be of the form R( x ) =
Revised: The CNN is chosen to be of the form R( x ) =

Original: C Q ◦ · · · ◦ σ q ◦ C q ◦ · · · ◦ σ 1 ◦ C 1 ( x ) , where C q denotes alinear convolutional layer and σ q denotes a pointwise nonlinear activation function.
Revised: C Q ◦ · · · ◦ σ q ◦ C q ◦ · · · ◦ σ 1 ◦ C 1 ( x ) , where C q denotes a linear convolutional layer and σ q denotes a pointwise nonlinear activation function.

Original: We considerthree different denoisers characterized by three CNNs: an unconstrained ReLU network (ReLU-U),a 1 -Lipschitz ReLU network (ReLU-L), and a 1 -Lipschitz deep spline network (DS-L).
Revised: We consider three different denoisers characterized by three CNNs: an unconstrained ReLU network (ReLU-U), a 1 -Lipschitz ReLU network (ReLU-L), and a 1 -Lipschitz deep spline network (DS-L).

Original: Becausethe ReLU-U denoiser is not guaranteed to be an averaged operator (unconstrained CNN), it onlyserves as a baseline for comparing constrained cases (ReLU-L and DS-L).
Revised: Because the ReLU-U denoiser is not guaranteed to be an averaged operator (unconstrained CNN), it only serves as a baseline for comparing constrained cases (ReLU-L and DS-L).

Original: We ﬁx β = 0 .
Revised: We ﬁx β = 0 .

Original: 5 for allmodels.
Revised: 5 for all models.

Original: This value could have been optimized by grid search, but the goal is to compare the relativeperformance between ReLU-L and DS-L.
Revised: This value could have been optimized by grid search, but the goal is to compare the relative performance between ReLU-L and DS-L.

Original: The training dataset consists of 400 images from the BSD500 dataset [1] divided into 238 , 400 nonoverlapping patches of size 40 × 40 .
Revised: The training dataset consists of 400 images from the BSD500 dataset [1] divided into 238 , 400 nonoverlapping patches of size 40 × 40 .

Original: From the remaining images, we create a validation dataset of 12images and a test dataset of 68 images.
Revised: From the remaining images, we create a validation dataset ofimages and a test dataset of 68 images.

Original: The images take values between [0 , 1] .
Revised: The images take values between [0 , 1] .

Original: We consider threenoise levels σ = 5 / 255 , 10 / 255 and 20 / 255 , where σ is the standard deviation of the Gaussian noiseadded to the datasets.
Revised: We consider three noise levels σ = 5 / 255 , 10 / 255 and 20 / 255 , where σ is the standard deviation of the Gaussian noise added to the datasets.

Original: For each level of noise, we trained all denoising models considered (ReLU-U,
Revised: For each level of noise, we trained all denoising models considered (ReLU-U,

Original: ReLU-L, and DS-L) with Q = { 3 , 5 , 7 , 9 } (number of layers).
Revised: ReLU-L, and DS-L) with Q = { 3 , 5 , 7 , 9 } (number of layers).

Original: The kernels for the convolutionallayers are of size (3 × 3) and the number of channels in the intermediate convolutional layersis ﬁxed to 32 .
Revised: The kernels for the convolutional layers are of size (3 × 3) and the number of channels in the intermediate convolutional layers is ﬁxed to 32 .

Original: For DS-L, the grid size for the learnable nonlinearities is 0 .
Revised: For DS-L, the grid size for the learnable nonlinearities is 0 .

Original:  and the number ofknots is 51 (refer to Appendix A for details).
Revised: 2 and the number of knots is 51 (refer to Appendix A for details).

Original: All denoisers were trained for 200 epochs using an
Revised: All denoisers were trained for 200 epochs using an

Original: ADAM optimizer [15] with a learning rate of 10 − 5 and a batch size of 128 .
Revised: ADAM optimizer [15] with a learning rate of 10 − 5 and a batch size of 128 .

Original: The loss function is themean-square error (MSE).
Revised: The loss function is the mean-square error (MSE).

Original: For DS-L, the loss function has a regularization term that penalizes thesecond-order derivative of the learnable spline activation functions (refer to Appendix A for details).
Revised: For DS-L, the loss function has a regularization term that penalizes the second-order derivative of the learnable spline activation functions (refer to Appendix A for details).

Original: The corresponding regularization parameter is set to 10 − 5 .
Revised: The corresponding regularization parameter is set to 10 − 5 .

Original: The denoising results for σ = 5 / 255 are shown in Figure 1 while the results for σ = 10 / 255 and20 / 255 are shown in Appendix B (Figures 2 and 3).
Revised: The denoising results for σ = 5 / 255 are shown in Figure 1 while the results for σ = 10 / 255 and 20 / 255 are shown in Appendix B (Figures 2 and 3).

Original: As expected, we observe that for any valueof σ and Q , enforcing the 1 -Lipschitz condition on the ReLU CNN leads to a drop in performance(ReLU-U vs. ReLU-L in Figure 1).
Revised: As expected, we observe that for any value of σ and Q , enforcing the 1 -Lipschitz condition on the ReLU CNN leads to a drop in performance (ReLU-U vs. ReLU-L in Figure 1).

Original: The DS-L denoiser is a more adaptable model and improvesin performance over the ReLU-L denoiser.
Revised: The DS-L denoiser is a more adaptable model and improves in performance over the ReLU-L denoiser.

Original: Interestingly, the DS-L denoiser with only three layersconsistently outperforms the ReLU-L denoiser with nine layers even though it has fewer learnableparameters.
Revised: Interestingly, the DS-L denoiser with only three layers consistently outperforms the ReLU-L denoiser with nine layers even though it has fewer learnable parameters.

Original: This shows that increasing the capacity of the model by learning activation functionsseems to be more beneﬁcial than increasing the number of layers for such constrained training tasks.
Revised: This shows that increasing the capacity of the model by learning activation functions seems to be more beneﬁcial than increasing the number of layers for such constrained training tasks.

Original: 3.2 Compressed Sensing MRI
Revised: 3.2 Compressed Sensing MRI

Original: We now look at the CS-MRI problem of recovering an image x ∈ R K from its measurementsy = MFx + n ∈ C M , where M is a subsampling mask (identity matrix with some missing entries),
Revised: We now look at the CS-MRI problem of recovering an image x ∈ R K from its measurements y = MFx + n ∈ C M , where M is a subsampling mask (identity matrix with some missing entries),

Original: Number of layers
Revised: Number of layers

Original: PS N
Revised: PS N

Original: R ( [Figure]
Revised: R ( d B )

Original: Number of layers 0.
Revised: Number of layers 0.

Original: SS I M
Revised: SS I M

Original: ReLU-U
Revised: ReLU-U

Original: ReLU-L
Revised: ReLU-L

Original: DS-L
Revised: DS-L

Original: [Table] [Table]
Revised: [Table] [Table]

Original: F is the discrete Fourier transform and n is a complex Gaussian noise with variance σ 2 n (not to beconfused with σ , the noise variance used for training the denoisers) for both the real and imaginaryparts.
Revised: F is the discrete Fourier transform and n is a complex Gaussian noise with variance σ 2 n (not to be confused with σ , the noise variance used for training the denoisers) for both the real and imaginary parts.

Original: We use PnP-FBS with the trained denoisers described in the previous section to solve thisinverse problem.
Revised: We use PnP-FBS with the trained denoisers described in the previous section to solve this inverse problem.

Original: Since we are interested in a convergent PnP-FBS algorithm, we compare theperformances of the ReLU-L and DS-L denoisers only.
Revised: Since we are interested in a convergent PnP-FBS algorithm, we compare the performances of the ReLU-L and DS-L denoisers only.

Original: The data-ﬁdelity term is chosen as E ( y , MFx ) =
Revised: The data-ﬁdelity term is chosen as E ( y , MFx ) =

Original: (1 / 2) (cid:107) y − MFx (cid:107) 2 .
Revised: (1 / 2) (cid:107) y − MFx (cid:107) 2 .

Original: We run the CS-MRIexperiment on two images of size 256 × 256 and which take values between [0 , 1] .
Revised: We run the CS-MRI experiment on two images of size 256 × 256 and which take values between [0 , 1] .

Original: We consider threekinds of subsampling masks (random, radial and cartesian) [17], each with a subsampling ratio of0 .
Revised: We consider three kinds of subsampling masks (random, radial and cartesian) [17], each with a subsampling ratio of 0 . 3 .

Original: Further, we also consider three levels of noise σ n = 10 / 255 , 20 / 255 , and 30 / 255 .
Revised: Further, we also consider three levels of noise σ n = 10 / 255 , 20 / 255 , and 30 / 255 .

Original: The step sizeα for PnP-FBS is set to 10 − 5 .
Revised: The step size α for PnP-FBS is set to 10 − 5 .

Original: This value is chosen based on the Lipschitz constant of the gradient ofthe data-ﬁdelity term (Section 2).
Revised: This value is chosen based on the Lipschitz constant of the gradient of the data-ﬁdelity term (Section 2).

Original: The PnP-FBS algorithm is initialized with the zero-ﬁlled estimate,obtained by taking the inverse discrete Fourier transform of the zero-ﬁlled subsampled measurements.
Revised: The PnP-FBS algorithm is initialized with the zero-ﬁlled estimate, obtained by taking the inverse discrete Fourier transform of the zero-ﬁlled subsampled measurements.

Original: We use ReLU-L and DS-L denoisers with Q = 5 .
Revised: We use ReLU-L and DS-L denoisers with Q = 5 .

Original: For each experimental setup (subsampling maskand image type), we evaluate the performance of these denoisers in terms of PSNR for differentvalues of σ (denoising strength) and report the best models.
Revised: For each experimental setup (subsampling mask and image type), we evaluate the performance of these denoisers in terms of PSNR for different values of σ (denoising strength) and report the best models.

Original: The results for σ n = 10 / 255 arepresented in Table 1.
Revised: The results for σ n = 10 / 255 are presented in Table 1.

Original: The results for the other two noise levels as well the reconstructed imagesare provided in Appendix B (Tables 2 and 3 and Figures 4, 5 and 6).
Revised: The results for the other two noise levels as well the reconstructed images are provided in Appendix B (Tables 2 and 3 and Figures 4, 5 and 6).

Original: We observe that the DS-Ldenoiser systematically yields better image reconstructions than the ReLU-L denoiser.
Revised: We observe that the DS-L denoiser systematically yields better image reconstructions than the ReLU-L denoiser.

Original: Thus, theimprovement in denoising performance is translated to the CS-MRI problem, highlighting the beneﬁtsof the proposed approach.
Revised: Thus, the improvement in denoising performance is translated to the CS-MRI problem, highlighting the beneﬁts of the proposed approach.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we looked at the problem of building averaged (denoising) operators using 1-Lipschitz
Revised: In this work, we looked at the problem of building averaged (denoising) operators using 1-Lipschitz

Original: CNNs for provably convergent PnP algorithms that are used for solving ill-posed linear inverseproblems.
Revised: CNNs for provably convergent PnP algorithms that are used for solving ill-posed linear inverse problems.

Original: To mitigate the drop in performance resulting from such a constraint, we proposed theuse of deep spline networks whose piecewise-linear spline activation functions can be trained whileguaranteeing a controlled Lipschitz constant.
Revised: To mitigate the drop in performance resulting from such a constraint, we proposed the use of deep spline networks whose piecewise-linear spline activation functions can be trained while guaranteeing a controlled Lipschitz constant.

Original: To this end, we introduced “slope normalization”as the counterpart to spectral normalization.
Revised: To this end, we introduced “slope normalization” as the counterpart to spectral normalization.

Original: We showed that averaged denoising operators builtfrom 1-Lipschitz deep spline networks consistently outperformed those built from 1-Lipschitz ReLUnetworks for both denoising and CS-MRI.
Revised: We showed that averaged denoising operators built from 1-Lipschitz deep spline networks consistently outperformed those built from 1-Lipschitz ReLU networks for both denoising and CS-MRI.

Original: Our ﬁndings suggest that deep spline networks have higherexpressivity under Lipschitz constraints than ReLU networks, even with fewer trainable parameters.  
Revised: Our ﬁndings suggest that deep spline networks have higher expressivity under Lipschitz constraints than ReLU networks, even with fewer trainable parameters.  Acknowledgements

Revised: This work was supported in part by the Swiss National Science Foundation under Grant 200020_184646 / 1 and in part by the European Research Council (ERC Project FunLearn) under Grant

Original: A Deep Spline Neural Networks
Revised: A Deep Spline Neural Networks

Original: In deep spline neural networks, besides the linear weights or convolution kernels we also have learnable228 nonlinearities that are piecewise-linear splines.
Revised: In deep spline neural networks, besides the linear weights or convolution kernels, we also have learnable nonlinearities that are piecewise-linear splines.

Original: This choice stems from a functional formulation of the trainingof a neural network with free-form activation functions.
Revised: This choice stems from a functional formulation of the training of a neural network with free-form activation functions.

Original: If the cost functional for the training process includes a230 regularization term that penalizes the second-order total-variations of the activation functions, then the optimalactivation functions are known to be adaptive piecewise-linear splines [23].
Revised: If the cost functional for the training process includes a regularization term that penalizes the second-order total-variations of the activation functions, then the optimal activation functions are known to be adaptive piecewise-linear splines [23].

Original: A.1 B-Spline Representation
Revised: A.1 B-Spline Representation

Original: For each nonlinearity in the network, we consider a linear spline s with K knots on a ﬁnite grid of size T .
Revised: For each nonlinearity in the network, we consider a linear spline s with K knots on a ﬁnite grid of size T .

Original: We234 assume that K is odd.
Revised: We assume that K is odd.

Original: Let k min = − ( K − 1) / 2 and k max
Revised: Let k min = − ( K − 1) / 2 and k max

Original: = ( K − 1) / 2 .
Revised: = ( K − 1) / 2 .

Original: The function s is then represented as
Revised: The function s is then represented as

Original: [Equation]
Revised: [Equation]

Original: where ϕ T is the triangle-shaped B-spline
Revised: where ϕ T is the triangle-shaped B-spline

Original: [Equation]
Revised: [Equation]

Original: otherwise, (

Original: [Equation]

Original: and the B-spline coefﬁcients c = ( c k ) are the adjustable quantities during the training process.
Revised: and the B-spline coefﬁcients c = ( c k ) are the adjustable quantities during the training process.

Original: The last two237 coefﬁcients on either side - ( k min − 1 , k min ) and ( k max , k max +1 ) - are responsible for handling the linear extensionsbeyond the interval [ k min T, k max T ] .
Revised: The last two coefﬁcients on either side - ( k min − 1 , k min ) and ( k max , k max +1 ) - are responsible for handling the linear extensions beyond the interval [ k min T, k max T ] .

Original: The second-order total-variations for the regularization term in the cost function can be computed as TV (2) ( s ) =(cid:107) Lc (cid:107) 1 , where
Revised: The second-order total-variations for the regularization term in the cost function can be computed as TV (2) ( s ) = (cid:107) Lc (cid:107) 1 , where

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] [Equation]
Revised: This regularization term promotes sparsiﬁcation of the knots in the linear spline.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ReLU-U
Revised: ReLU-U

Original: ReLU-L
Revised: ReLU-L

Original: DS-L
Revised: DS-L

Original: [Figure]
Revised: [Figure]

Revised: ( d

Revised: [Figure]

Revised: [Figure]

Original: A.2 Computation of the Lipschitz Constant
Revised: A.2 Computation of the Lipschitz Constant

Original: The Lipschitz constant of the linear spline s is the maximum absolute value of its gradient and can be computed245 as s
Revised: The Lipschitz constant of the linear spline s is the maximum absolute value of its gradient and can be computed as s

Original: Lip = (cid:107)
Revised: Lip = (cid:107)

Original: Dc (cid:107) ∞ , where
Revised: Dc (cid:107) ∞ , where

Original: D = 1 T   −   · · · 0−  · · · ...
Revised: D = 1 T   −   · · · 0−  · · · ...

Original: 0· · · · · · −   .
Revised: 0· · · · · · −   .

Original: Thus, the Lipschitz constant of the linear spline activation functions can be computed in an efﬁcient manner in247 order to perform slope normalization.
Revised: Thus, the Lipschitz constant of the linear spline activation functions can be computed in an efﬁcient manner in order to perform slope normalization.

Original: B Additional Results
Revised: B Additional Results

Original: Here, we present additional results for both the Gaussian denoising and compressed sensing MRI tasks.
Revised: Here, we present additional results for both the Gaussian denoising and compressed sensing MRI tasks.

Original: B.1 Evaluation of Gaussian Denoisers
Revised: B.1 Evaluation of Gaussian Denoisers

Original: Figures 2 and 3 show the denoising results for Gaussian noises with variances σ = 10 / 255 and σ = 20 / 255 ,respectively.
Revised: Figures 2 and 3 show the denoising results for Gaussian noises with variances σ = 10 / 255 and σ = 20 / 255 , respectively.

Original: B.2 Compressed Sensing MRI
Revised: B.2 Compressed Sensing MRI

Original: In Tables 
Revised: In Tables 2 and 3, we present the results for CS-MRI

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: ound t r u t h
Revised: ound t r u t h

Original: Random
Revised: Random

Original: Radial
Revised: Radial

Original: Cartesian
Revised: Cartesian

Original: Random
Revised: Random

Original: Radial
Revised: Radial

Original: Cartesian
Revised: Cartesian

Original: [Figure]
Revised: [Figure]

Original: illi ng
Revised: illi ng

Original: [Figure]
Revised: [Figure]

Original: D
Revised: D

Original: [Figure]
Revised: [Figure]

Original: L
Revised: L

Original: G r ound t r u t h
Revised: G r ound t r u t h

Original: Random
Revised: Random

Original: Radial
Revised: Radial

Original: Cartesian
Revised: Cartesian

Original: Random
Revised: Random

Original: Radial
Revised: Radial

Original: Cartesian
Revised: Cartesian

Original: Z e r o f illi ng
Revised: Z e r o f illi ng

Original: R e L
Revised: R e L

Original: U - L
Revised: U - L

Original: D SL
Revised: D SL

Original: G r ound t r u t h
Revised: G r ound t r u t h

Original: Random
Revised: Random

Original: Radial
Revised: Radial

Original: Cartesian
Revised: Cartesian

Original: Random
Revised: Random

Original: Radial
Revised: Radial

Original: Cartesian
Revised: Cartesian

Original: Z e r o f illi ng
Revised: Z e r o f illi ng

Original: R e L
Revised: R e L

Original: [Figure]
Revised: [Figure]

Original: - L
Revised: - L

Original: D SL
Revised: D SL