Original: Revisiting Sparse Convolutional Model for Visual Recognition
Revised: Revisiting Sparse Convolutional Model for Visual Recognition

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Despite strong empirical performance for image classiﬁcation, deep neural networks are often regarded as “black boxes” and they are difﬁcult to interpret.
Revised: Despite strong empirical performance for image classiﬁcation, deep neural networks are often regarded as “black boxes” and they are difﬁcult to interpret.

Original: Onthe other hand, sparse convolutional models, which assume that a signal can beexpressed by a linear combination of a few elements from a convolutional dictionary, are powerful tools for analyzing natural images with good theoreticalinterpretability and biological plausibility.
Revised: Onthe other hand, sparse convolutional models, which assume that a signal can beexpressed by a linear combination of a few elements from a convolutional dictionary, are powerful tools for analyzing natural images with good theoreticalinterpretability and biological plausibility.

Original: However, such principled models havenot demonstrated competitive performance when compared with empirically designed deep networks.
Revised: However, such principled models havenot demonstrated competitive performance when compared with empirically designed deep networks.

Original: This paper revisits the sparse convolutional modeling forimage classiﬁcation and bridges the gap between good empirical performance (ofdeep learning) and good interpretability (of sparse convolutional models).
Revised: This paper revisits the sparse convolutional modeling forimage classiﬁcation and bridges the gap between good empirical performance (ofdeep learning) and good interpretability (of sparse convolutional models).

Original: Ourmethod uses differentiable optimization layers that are deﬁned from convolutionalsparse coding as drop-in replacements of standard convolutional layers in conventional deep neural networks.
Revised: Ourmethod uses differentiable optimization layers that are deﬁned from convolutionalsparse coding as drop-in replacements of standard convolutional layers in conventional deep neural networks.

Original: We show that such models have equally strongempirical performance on CIFAR-10, CIFAR-100 and ImageNet datasets whencompared to conventional neural networks.
Revised: We show that such models have equally strongempirical performance on CIFAR-10, CIFAR-100 and ImageNet datasets whencompared to conventional neural networks.

Original: By leveraging stable recovery propertyof sparse modeling, we further show that such models can be much more robust toinput corruptions as well as adversarial perturbations in testing through a simpleproper trade-off between sparse regularization and data reconstruction terms.
Revised: By leveraging stable recovery propertyof sparse modeling, we further show that such models can be much more robust toinput corruptions as well as adversarial perturbations in testing through a simpleproper trade-off between sparse regularization and data reconstruction terms.

Original:  Introduction
Revised:  Introduction

Original: In recent years, deep learning has been a dominant approach for image classiﬁcation and hassigniﬁcantly advanced the performance over previous shallow models.
Revised: In recent years, deep learning has been a dominant approach for image classiﬁcation and hassigniﬁcantly advanced the performance over previous shallow models.

Original: Despite the phenomenalempirical success, it has been increasingly realized as well as criticized that deep convolutionalnetworks (ConvNets) are “black boxes” for which we are yet to develop clear understanding [1].
Revised: Despite the phenomenalempirical success, it has been increasingly realized as well as criticized that deep convolutionalnetworks (ConvNets) are “black boxes” for which we are yet to develop clear understanding [1].

Original: Thelayer operations such as convolution, nonlinearity and normalization are geared towards minimizingan end-to-end training loss and do not have much data-speciﬁc meaning .
Revised: Thelayer operations such as convolution, nonlinearity and normalization are geared towards minimizingan end-to-end training loss and do not have much data-speciﬁc meaning .

Original: As such, the functionalityof each intermediate layer in a trained ConvNets is mostly unclear and the feature maps that theyproduce are hard to interpret.
Revised: As such, the functionalityof each intermediate layer in a trained ConvNets is mostly unclear and the feature maps that theyproduce are hard to interpret.

Original: The lack of interpretability also contributes to the notorious difﬁcultyin enhancing such learning systems for practical data which are usually corrupted by various formsof perturbation.
Revised: The lack of interpretability also contributes to the notorious difﬁcultyin enhancing such learning systems for practical data which are usually corrupted by various formsof perturbation.

Original: This paper presents a visual recognition framework by introducing layers that have explicit datamodeling to tackle shortcomings of current deep learning systems.
Revised: This paper presents a visual recognition framework by introducing layers that have explicit datamodeling to tackle shortcomings of current deep learning systems.

Original: We work under the assumptionthat the layer input can be represented by a few atoms from a dictionary shared by all data points.
Revised: We work under the assumptionthat the layer input can be represented by a few atoms from a dictionary shared by all data points.

Original: Thisis the classical sparse data modeling that, as shown in a pioneering work of [2], can easily discovermeaningful structures from natural image patches.
Revised: Thisis the classical sparse data modeling that, as shown in a pioneering work of [2], can easily discovermeaningful structures from natural image patches.

Original: Backed by its ability in learning interpretablerepresentations and strong theoretical guarantees  (e.g. for handling corrupted data), sparse modelinghas been used broadly in many signal and image processing applications[3].
Revised: Backed by its ability in learning interpretablerepresentations and strong theoretical guarantees [3, 4, 5, 6, 7, 8] (e.g. for handling corrupteddata), sparse modeling has been used broadly in many signal and image processing applications

Original: However, the empirical performance of sparse methods have been surpassed by deep learning methods for classiﬁcation ofmodern image datasets.
Revised: However, the empirical performance of sparse methods have been surpassed by deep learningmethods for classiﬁcation of modern image datasets.

Original: Because of the complementary beneﬁts of sparse modeling and deep learning, there exist manyefforts that leverage sparse modeling to gain theoretical insights into ConvNets and/or to developcomputational methods that further improve upon existing ConvNets.
Revised: Because of the complementary beneﬁts of sparse modeling and deep learning, there exist manyefforts that leverage sparse modeling to gain theoretical insights into ConvNets and/or to developcomputational methods that further improve upon existing ConvNets.

Original: One of the pioneering works is[4] which interpreted a ConvNet as approximately solving a multi-layer convolutional sparse codingmodel.
Revised: One of the pioneering works is[10] which interpreted a ConvNet as approximately solving a multi-layer convolutional sparse codingmodel.

Original: Based on this interpretation, the work [4] and its follow-ups [5, 6, 7, 8] presented alternativealgorithms and models in order to further enhance the practical performance of such learning systems.
Revised: Based on this interpretation, the work [10] and its follow-ups [11, 12, 13, 14] presentedalternative algorithms and models in order to further enhance the practical performance of suchlearning systems.

Original: However, there has been no empirical evidence that such systems can handle modern image datasetssuch as ImageNet and obtain comparable performance to deep learning.
Revised: However, there has been no empirical evidence that such systems can handle modernimage datasets such as ImageNet and obtain comparable performance to deep learning.

Original: The only exception to thebest of our knowledge is the work of [9, 10] which exhibited a performance on par to (on ImageNet)or better than (on CIFAR-10) ResNet.
Revised: The onlyexception to the best of our knowledge is the work of [15, 16] which exhibited a performance on parto (on ImageNet) or better than (on CIFAR-10) ResNet.

Original: However, the method in [9, 10] 1) requires a dedicated designof network architecture that may limit its applicability, 2) is computationally orders of magnitudeslower to train, and 3) does not demonstrate beneﬁts in terms of interpretability and robustness.
Revised: However, the method in [15, 16] 1) requires adedicated design of network architecture that may limit its applicability, 2) is computationally ordersof magnitude slower to train, and 3) does not demonstrate beneﬁts in terms of interpretability androbustness.

Original: In anutshell, sparse modeling is yet to demonstrate practicality that enables its broad applications.
Revised: In a nutshell, sparse modeling is yet to demonstrate practicality that enables its broadapplications.

Original: Paper contributions.
Revised: Paper contributions.

Original: In this paper, we revisit sparse
Revised: In this paper, we revisit sparse

Original: modeling for image classiﬁcation and demonstrate through a simple design that sparse modeling can be combined with deep learning to obtainperformance on par with standard ConvNets but with better  interpretability and stability.
Revised: modeling for image classiﬁcation and demonstrate through a simple design that sparse modeling can be combined with deep learning to obtainperformance on par with standard ConvNets but with better layer-wise interpretability and stability.

Original: Our methodencapsulates the sparse modeling into an implicit layer [11, 12, 13] and uses it as a drop-in replacement for any convolutional layer in standard ConvNets.
Revised: Our method encapsulates the sparse modeling into an implicit layer [17, 18, 19] and uses it as adrop-in replacement for any convolutional layer in standard ConvNets.

Original: The layer implements the convolutionalsparse coding (CSC) model of [14], and is referred to as a CSC-layer , where the input signal isapproximated by a sparse linear combination of atoms from a convolutional dictionary.
Revised: The layer implements theconvolutional sparse coding (CSC) model of [20], and is referred to as a CSC-layer , where the inputsignal is approximated by a sparse linear combination of atoms from a convolutional dictionary.

Original: Such aconvolutional dictionary is treated as the parameters of the CSC-layer that are amenable to trainingvia back-propagation.
Revised: Sucha convolutional dictionary is treated as the parameters of the CSC-layer that are amenable to trainingvia back-propagation.

Original: Then, the overall network with the CSC-layers may be trained in an end-to-endfashion from labeled data by minimizing the cross-entropy loss as usual.
Revised: Then, the overall network with the CSC-layers may be trained in an end-to-endfashion from labeled data by minimizing the cross-entropy loss as usual.

Original: This paper demonstratesthat such a learning framework has the following beneﬁts:
Revised: This paper demonstratesthat such a learning framework has the following beneﬁts:

Original: • Performance on standard datasets.
Revised: • Performance on standard datasets.

Original: We demonstrate that our network obtains better (on CIFAR100) or on par (on CIFAR-10 and ImageNet) performance with similar training time compared withstandard architectures such as ResNet [15].
Revised: We demonstrate that our network obtains better (on CIFAR100) or on par (on CIFAR-10 and ImageNet) performance with similar training time compared withstandard architectures such as ResNet [21].

Original: This provides the ﬁrst evidence on the strong empiricalperformance of sparse modeling for deep learning to the best of our knowledge.
Revised: This provides the ﬁrst evidence on the strong empirical performance of sparse modeling for deep learning to the best of our knowledge.

Original: Compared
Revised: Compared

Original: to previous sparse methods [9] that obtained similar performance, our method is of orders ofmagnitude faster.
Revised: to previous sparse methods [15] that obtained similar performance, our method is of orders of magnitude faster.

Original: • Robustness to input perturbations.
Revised: • Robustness to input perturbations.

Original: The stable recovery property of sparse convolution modelequips the CSC-layers with the ability to remove perturbation in the layer input and to recover cleansparse code.
Revised: The stable recovery property of sparse convolution modelequips the CSC-layers with the ability to remove perturbation in the layer input and to recover clean sparse code.

Original: As a result, our networks with CSC-layers are more robust to perturbations in theinput images compared with classical neural networks.
Revised: As a result, our networks with CSC-layers are more robust to perturbations in theinput images compared with classical neural networks.

Original: Unlike existing approaches for obtainingrobustness that require heavy data augmentation [16] or additional training techniques [17], ourmethod is light-weight and does not require modifying the training procedure at all
Revised: Unlike existing approaches for obtainingrobustness that require heavy data augmentation [22] or additional training techniques [23], ourmethod is light-weight and does not require modifying the training procedure at all.

Original:  Related Work
Revised:  Related Work

Original: Implicit layers.
Revised: Implicit layers.

Original: The idea of trainable layers deﬁned from implicit functions can be traced back at leastto the work of [18].
Revised: The idea of trainable layers deﬁned from implicit functions can be traced back atleast to the work of [24].

Original: Recently, there is a revival of interests in implicit layers [11, 12, 13, 19, 20, 21, 22]as an attractive alternative to explicit layers in existing neural networks.
Revised: Recently, there is a revival of interests in implicit layers [17, 18, 19, 25,26, 27, 28, 29] as an attractive alternative to explicit layers in existing neural networks.

Original: However, a majority ofthe cited works above deﬁne an implicit layer by a ﬁxed point iteration, typically motivated fromexisting explicit layers such as residual layers, therefore they do not have clear interpretation interms of modeling of the layer input.
Revised: However, amajority of the cited works above deﬁne an implicit layer by a ﬁxed point iteration, typically motivatedfrom existing explicit layers such as residual layers, therefore they do not have clear interpretationin terms of modeling of the layer input.

Original: Consequently, such models do not have the ability to dealwith input perturbations.
Revised: Consequently, such models do not have the ability to dealwith input perturbations.

Original: The only exceptions are differentiable optimization layers [23, 11, 12, 24]that incorporate complex dependencies between hidden layers through the formulation of convexoptimization.
Revised: The only exceptions are differentiable optimization layers [30, 17, 18, 31]that incorporate complex dependencies between hidden layers through the formulation of convexoptimization.

Original: Nevertheless, most of the above works focus on differentiating through the convexoptimization layers (such as disciplined parametrized programming [12]) without specializing in anyparticular signal models such as the sparse models considered in this paper nor demonstrating theirperformance when encapsulated in multi-layer neural networks.
Revised: Nevertheless, most of the above works focus on differentiating through the convexoptimization layers (such as disciplined parametrized programming [18]) without specializing in anyparticular signal models such as the sparse models considered in this paper nor demonstrating theirperformance when encapsulated in multi-layer neural networks.

Original: Sparse prior in deep learning.
Revised: Sparse prior in deep learning.

Original: Aside from image classiﬁcation, sparse modeling has been introduced to deep learning for many image processing tasks such as super resolution [25], denoising [26]and so on [27, 28, 29, 30].
Revised: Aside from image classiﬁcation, sparse modeling has been introduced to deep learning for many image processing tasks such as super resolution [32], denoising [33]and so on [34, 35, 36, 37].

Original: These works incorporate sparse modeling by using network architectures that are motivated by (but are not the same as) an unrolled sparse coding algorithmn LISTA [31].
Revised: These works incorporate sparse modeling by using network architectures that are motivated by (but are not the same as) an unrolled sparse coding algorithmn LISTA [38].

Original: In sharp contrast to ours, there is no guarantee that such architectures perform a sparse encodingwith respect to a particular (convolution) dictionary at all.
Revised: Insharp contrast to ours, there is no guarantee that such architectures perform a sparse encoding withrespect to a particular (convolution) dictionary at all.

Original: As a result, they lack the capability ofhandling input perturbations as in our method.
Revised: As a result, they lack the capability of handlinginput perturbations as in our method.

Original: A notable exception is the work of [9] where each layerperforms a precise sparse encoding and exhibits on par or better performance for image classiﬁcationover ResNet.
Revised: A notable exception is the work of [15] where each layerperforms a precise sparse encoding and exhibits on par or better performance for image classiﬁcationover ResNet.

Original: However, the practical beneﬁt of the sparse modeling in terms of robustness is notdemonstrated.
Revised: However, the practical beneﬁt of the sparse modeling in terms of robustness is notdemonstrated.

Original: Moreover, [9] adopts a patch-based sparse coding model for images and has a largecomputational burden.
Revised: Moreover, [15] adopts a patch-based sparse coding model for images and has a largecomputational burden.

Original: Robustness.
Revised: Robustness.

Original: It is known that modern neural networks are extremely vulnerable to small perturbationsin the input data.
Revised: It is known that modern neural networks are extremely vulnerable to small perturbationsin the input data.

Original: A plethora of techniques have been proposed to address this instability issue,including stability training [17], adversarial training [32, 33, 34], data augmentation [35, 36, 16],etc.
Revised: A plethora of techniques have been proposed to address this instability issue,including stability training [23], adversarial training [39, 40, 41], data augmentation [42, 43, 22],etc.

Original: Nevertheless, these techniques either need a computational and memory overhead, or requirea selection of appropriate augmentation strategies to cover all possible corruptions.
Revised: Nevertheless, these techniques either need a computational and memory overhead, or requirea selection of appropriate augmentation strategies to cover all possible corruptions.

Original: With standardtraining only, our model can be made robust to input perturbations in test data by simply adaptingsparse modeling to account for noise.
Revised: With standardtraining only, our model can be made robust to input perturbations in test data by simply adaptingsparse modeling to account for noise.

Revised: Closely related to our work are [44, 45, 46] which use sparsemodeling to improve adversarial robustness.

Revised: However, they either only demonstrate performanceon very simple networks [45, 46] or sacriﬁce natural accuracy for robustness [44].

Revised: In contrast, ourmethod is tested on realistic networks and does not affect natural accuracy.

Original:  Neural Networks with Sparse Modeling
Revised:  Neural Networks with Sparse Modeling

Original: In this section, we show how sparse modeling is incorporated into a deep network via a speciﬁc typeof network layer that we refer to as the convolutional sparse coding (CSC) layer.
Revised: In this section, we show how sparse modeling is incorporated into a deep network via a speciﬁc typeof network layer that we refer to as the convolutional sparse coding (CSC) layer.

Original: We describe the
Revised: We describe the

Original: CSC-layer in Sec.
Revised: CSC-layer in Sec.

Original: 3.1 and explain how we use them for deep learning in Sec. 3.2.
Revised: 3.1 and explain how we use them for deep learning in Sec. 3.2.

Original: Finally, Sec. 3.3explains how CSC enables robust inference with corrupted test data.
Revised: Finally, Sec. 3.3explains how CSC enables robust inference with corrupted test data.

Original: Notations.
Revised: Notations.

Original: Given a single-channel image
Revised: Given a single-channel image

Original: ξ ∈ R H
Revised: ξ ∈ R H

Original: × W represented as a matrix, we may treat it as a2D signal deﬁned on the discrete domain [1 , . . .
Revised: × W represented as a matrix, we may treat it as a2D signal deﬁned on the discrete domain [1 , . . .

Original: , H ] × [1 , . . .
Revised: , H ] × [1 , . . .

Original: , W ] and extended to Z × Z by paddingzeros.
Revised: , W ] and extended to Z × Z by paddingzeros.

Original: Given a 2D kernel α ∈ R k × k , we may treat it as a 2D signal deﬁned on the discrete domain[ − k 0 · · · , k 0 ] × [ − k 0 , · · · , k 0 ] with k = 2 k 0 + 1 and extended to Z × Z by padding zeros.
Revised: Given a 2D kernel α ∈ R k × k , we may treat it as a 2D signal deﬁned on the discrete domain[ − k 0 · · · , k 0 ] × [ − k 0 , · · · , k 0 ] with k = 2 k 0 + 1 and extended to Z × Z by padding zeros.

Original: Then, forconvenience, we use “ ∗ ” and “ (cid:63) ” to denote the convolution and correlation operators, respectively,between two 2D signals:
Revised: Then, forconvenience, we use “ ∗ ” and “ (cid:63) ” to denote the convolution and correlation operators, respectively,between two 2D signals:

Original: [Equation]
Revised: [Equation]

Original: 3.1 Convolutional Sparse Coding (CSC) Layer
Revised: 3.1 Convolutional Sparse Coding (CSC) Layer

Original: Sparse modeling is introduced in the form of an implicit layer of a neural network.
Revised: Sparse modeling is introduced in the form of an implicit layer of a neural network.

Original: Unlike classicalfully-connected or convolutional layers in which input-output relations are deﬁned by an explicitfunction, implicit layers are deﬁned from implicit functions.
Revised: Unlike classicalfully-connected or convolutional layers in which input-output relations are deﬁned by an explicitfunction, implicit layers are deﬁned from implicit functions.

Original: For our case, in particular, we introducean implicit layer that is deﬁned from an optimization problem involving the input to the layer as wellas a weight parameter, where the output of the layer is the solution to the optimization problem.
Revised: For our case, in particular, we introducean implicit layer that is deﬁned from an optimization problem involving the input to the layer as wellas a weight parameter, where the output of the layer is the solution to the optimization problem.

Original: A generative model via sparse convolution.
Revised: A generative model via sparse convolution.

Original: Concretely, given a multi-dimensional input signalx ∈ R M × H × W to the layer where H, W are spatial dimensions and M is the number of channels forx .
Revised: Concretely, given a multi-dimensional input signalx ∈ R M × H × W to the layer where H, W are spatial dimensions and M is the number of channels forx .

Original: We assume the signal x is generated by a multi-channel sparse code z ∈ R C × H × W convolutingwith a multi-dimensional kernel A ∈ R M × C × k × k , which is referred to as a convolution dictionary .
Revised: We assume the signal x is generated by a multi-channel sparse code z ∈ R C × H × W convolutingwith a multi-dimensional kernel A ∈ R M × C × k × k , which is referred to as a convolution dictionary .

Original: Here C is the number of channels for z and the convolution kernel A .
Revised: Here C is the number of channels for z and the convolution kernel A .

Original: To be more precise, we denotez as z .
Revised: To be more precise, we denotez as z .

Original: = ( ζ 1 , . .
Revised: = ( ζ 1 , . .

Original: . , ζ C ) where each ζ c ∈ R H × W (presumably sparse), and denote the kernel A as
Revised: . , ζ C ) where each ζ c ∈ R H × W (presumably sparse), and denote the kernel A as

Original: [Equation]
Revised: [Equation]

Original: where each α ij ∈ R k × k is a kernel of size k × k .
Revised: where each α ij ∈ R k × k is a kernel of size k × k .

Original: Then the signal x is generated via the followingoperator A ( · ) deﬁned by the kernel
Revised: Then the signal x is generated via the followingoperator A ( · ) deﬁned by the kernel

Original: A as:
Revised: A as:

Original: [Equation]
Revised: [Equation]

Original: A layer as convolutional sparse coding.
Revised: A layer as convolutional sparse coding.

Original: Given a multi-dimensional input signal x ∈ R M × H × W ,we deﬁne that the function of “a layer” is to perform an (inverse) mapping to a preferably sparseoutput
Revised: Given a multi-dimensional input signal x ∈ R M × H × W ,we deﬁne that the function of “a layer” is to perform an (inverse) mapping to a preferably sparseoutput

Original: z ∗ ∈ R C
Revised: z ∗ ∈ R C

Original: × H × W , where C is the number of output channels.
Revised: × H × W , where C is the number of output channels.

Original: Under the above sparse generativemodel, we can seek the optimal sparse solution z by solving the following Lasso type optimizationproblem:
Revised: Under the above sparse generativemodel, we can seek the optimal sparse solution z by solving the following Lasso type optimizationproblem:

Original: [Equation]
Revised: [Equation]

Original: The optimization problem in (4) is basedon the convolutional sparse coding (CSC)model [14] 2 .
Revised: The optimization problem in (4) is basedon the convolutional sparse coding (CSC)model [20] 1 .

Original: Hence, we refer to the implicit layer deﬁned by (4) as a CSC-layer .
Revised: Hence, we refer to the implicit layer deﬁned by (4) as a CSC-layer .

Original: The goal of the CSC model is to reconstructthe input x via A ( z ) , where the featuremap z speciﬁes the locations and magnitudes of the convolutional ﬁlters in A to belinearly combined (see Figure 1 for an illustration).
Revised: The goal of the CSC model is to reconstructthe input x via A ( z ) , where the featuremap z speciﬁes the locations and magnitudes of the convolutional ﬁlters in A to belinearly combined (see Figure 1 for an illustration).

Original: The reconstruction is not requiredto be exact in order to tolerate modelingdiscrepancies, and the difference betweenx and A ( z ) is penalized by its entry-wise(cid:96) 2 -norm (i.e., the (cid:96) 2 norm of x −A ( z ) ﬂattened into a vector).
Revised: The reconstruction is not requiredto be exact in order to tolerate modelingdiscrepancies, and the difference betweenx and A ( z ) is penalized by its entry-wise(cid:96) 2 -norm (i.e., the (cid:96) 2 norm of x −A ( z ) ﬂattened into a vector).

Original: Sparse modeling isintroduced by the entry-wise (cid:96) 1 -norm of zin the objective function, which enforces zto be sparse.
Revised: Sparse modeling isintroduced by the entry-wise (cid:96) 1 -norm of zin the objective function, which enforces z to be sparse.

Original: The parameter λ > 0 controls the tradeoff between the sparsity of z and the magnitude ofthe residual x − A ( z ) , and is treated as a hyper-parameter that subjects to tuning via cross-validation.
Revised: The parameter λ > 0 controls the tradeoffbetween the sparsity of z and the magnitude of the residual x − A ( z ) , and is treated as a hyperparameter that subjects to tuning via cross-validation.

Original: As we will show in Sec. 3.3, λ can be used to improve the performance of our model in the test phasewhen the input is corrupted.
Revised: As we will show in Sec. 3.3, λ can be used toimprove the performance of our model in the test phase when the input is corrupted.

Original: Based on the input-output mapping of the CSC-layer given in (4), one may perform forward propagation by solving the associated optimization, and perform backward propagation by deriving thegradient of z ∗ with respect to the input x and parameter A .
Revised: Based on the input-output mapping of the CSC-layer given in (4), one may perform forward propagation by solving the associated optimization, and perform backward propagation by deriving thegradient of z ∗ with respect to the input x and parameter A .

Original: In this paper, we adopt the fast iterativeshrinkage thresholding algorithm (FISTA) [39] for the forward propagation, which also produces anunrolled network architecture that can carry out automatic differentiation for backward propagation.
Revised: In this paper, we adopt the fast iterativeshrinkage thresholding algorithm (FISTA) [47] for the forward propagation, which also produces anunrolled network architecture that can carry out automatic differentiation for backward propagation.

Original: We defer a discussion of the implementation details of the CSC-layer to the Appendix.
Revised: We defer a discussion of the implementation details of the CSC-layer to the Appendix.

Original: 3.2 Sparse Dictionary Learning Network Architecture and Training
Revised: 3.2 Sparse Dictionary Learning Network Architecture and Training

Original: Convolution layers are basic ingredients of ConvNets that appear in many common network architectures such as LeNet [40] and ResNet [41].
Revised: Convolution layers are basic ingredients of ConvNets that appear in many common network architectures such as LeNet [48] and ResNet [49].

Original: In this paper, we incorporate sparse modeling into agiven existing/baseline network architecture by replacing certain / all convolution layers with the
Revised: In this paper, we incorporate sparse modeling into agiven existing/baseline network architecture by replacing certain / all convolution layers with the

Original: CSC-layer.
Revised: CSC-layer.

Original: Meanwhile, all other layers such as normalization, nonlinear, and fully connected layersare retained.
Revised: Meanwhile, all other layers such as normalization, nonlinear, and fully connected layersare retained.

Original: This simple design choice allows the CSC-layers to be broadly applicable.
Revised: This simple design choice allows the CSC-layers to be broadly applicable.

Original: We will referto our network with CSC-layers as Sparse Dictionary learning Network (SDNet).
Revised: We will referto our network with CSC-layers as Sparse Dictionary learning Network (SDNet).

Original: Give a set of training data { x i , y i } Ni =1 where x i denotes an image and y i is the corresponding label,we train a network with CSC-layers by solving the following optimization problem:
Revised: Give a set of training data { x i , y i } Ni =1 where x i denotes an image and y i is the corresponding label,we train a network with CSC-layers by solving the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where (cid:96) CE denotes the cross-entropy loss.
Revised: where (cid:96) CE denotes the cross-entropy loss.

Original: In above, we use f ( · ; θ ) to denote the mapping that isperformed by the neural network, where θ is a set of learnable parameters containing a subset ofkernel parameters { A s } s ∈S associated with the set S of CSC-layers.
Revised: In above, we use f ( · ; θ ) to denote the mapping that isperformed by the neural network, where θ is a set of learnable parameters containing a subset of kernel parameters { A s } s ∈S associated with the set S of CSC-layers.

Original: Following the convention in thesparse dictionary learning literature, we add the constraint A s ∈ N for the dictionaries in CSC-layers,where N denotes the set of normalized dictionaries:
Revised: Following the convention in thesparse dictionary learning literature, we add the constraint A s ∈ N for the dictionaries in CSC-layers,where N denotes the set of normalized dictionaries:

Original: [Equation]
Revised: [Equation]

Original: To handle such a constraint, we use the projected stochastic gradient descent (SGD) for solving theproblem in (5).
Revised: To handle such a constraint, we use the projected stochastic gradient descent (SGD) for solving theproblem in (5).

Original: That is, after each gradient update step for the parameters θ and { A s } s ∈S as in aregular SGD, an extra step is taken to project each A s onto the constraint set N .
Revised: That is, after each gradient update step for the parameters θ and { A s } s ∈S as in aregular SGD, an extra step is taken to project each A s onto the constraint set N .

Original: 3.3 Robust Inference
Revised: 3.3 Robust Inference

Original: The fundamental difference of the CSC-layer vis-a-vis a classical explicit layer (e.g., a convolutionallayer) is that the CSC-layer imposes an assumption on the input feature map.
Revised: The fundamental difference of the CSC-layer vis-a-vis a classical explicit layer (e.g., a convolutionallayer) is that the CSC-layer imposes an assumption on the input feature map.

Original: That is, it assumesthat the input feature map (or image) can be approximated by a superposition of a few atoms of adictionary, which is the layer parameter that is learnable and is shared across all data.
Revised: That is, it assumesthat the input feature map (or image) can be approximated by a superposition of a few atoms of adictionary, which is the layer parameter that is learnable and is shared across all data.

Original: In this section,we show that CSC-layers enables us to design a robust inference strategy to obtain robustness tocorruptions in ways that cannot be achieved by classical explicit layers.
Revised: In this section,we show that CSC-layers enables us to design a robust inference strategy to obtain robustness tocorruptions in ways that cannot be achieved by classical explicit layers.

Original: We leverage an attractive property of the CSC model is that it admits a stable recovery of thesparse signals with respect to input noise.
Revised: We leverage an attractive property of the CSC model is that it admits a stable recovery of thesparse signals with respect to input noise.

Original: In particular, [42, Theorem 19] shows that any boundedperturbation to the input of a CSC-layer produces a bounded perturbation to its output and doesnot change the support of the output, if one uses a λ in (4) that is proportional to the norm of theperturbation in the layer input, provided that certain technical conditions are satisﬁed. 
Revised: In particular, [50, Theorem 19] shows that any boundedperturbation to the input of a CSC-layer produces a bounded perturbation to its output and doesnot change the support of the output, if one uses a λ in (4) that is proportional to the norm of theperturbation in the layer input, provided that certain technical conditions are satisﬁed. Theorem 1.

Revised: (Informal version of [50, Theorem 19]) Suppose x (cid:92) has a representation A ( z (cid:92) ) as in(3) , and that it is contaminated by noise e to create the input x = x (cid:92) + e .

Revised: Then as long as z (cid:92) issufﬁciently sparse, the solution z ∗ to (4) with λ

Revised: = O ( (cid:107)

Original: {+e (cid:107) 2 ) satisﬁes ( i ) the support of z ∗ is containedin that of z (cid:92) and ( ii ) (cid:107) z ∗ − z (cid:92) (cid:107) 
+}
Revised: {+e (cid:107) 2 ) satisﬁes ( i ) the support of z ∗ is containedin that of z (cid:92) and ( ii ) (cid:107) z ∗ − z (cid:92) (cid:107) 
+}

Original: Intuitively,the parameter λ in (4) controls a balance between the sparsity regularization (cid:107) z (cid:107) 1 and the residualx − A ( z ) , the latter of which accounts for modeling discrepancies and increases when x is noisy.
Revised: = O ( (cid:107) e (cid:107) 2 ) . Intuitively, the parameter λ in (4) controls a balance between the sparsity regularization (cid:107) z (cid:107) 1 and theresidual x − A ( z ) , the latter of which accounts for modeling discrepancies and increases when x isnoisy.

Original: Therefore, using a larger value of λ helps the model to handle a larger residual.
Revised: Therefore, using a larger value of λ helps the model to handle a larger residual.

Original: We thus present a very simple approach to obtain model robustness.
Revised: We thus present a very simple approach to obtain model robustness.

Original: We consider the setting where amodel is trained on an uncorrupted dataset but is tested on data that is corrupted by random noise.
Revised: We consider the setting where amodel is trained on an uncorrupted dataset but is tested on data that is corrupted by random noise.

Original: Corruptions in the input image may propagate into deeper feature maps (hence corrupting the inputof all CSC-layers) during forward propagation.
Revised: Corruptions in the input image may propagate into deeper feature maps (hence corrupting the inputof all CSC-layers) during forward propagation.

Original: Therefore, instead of directly using the CSC-layersthat are obtained from the training phase, our method is to adjust the trade-off parameter λ in the
Revised: Therefore, instead of directly using the CSC-layersthat are obtained from the training phase, our method is to adjust the trade-off parameter λ in the

Original: CSC-layers for the test data.
Revised: CSC-layers for the test data.

Original: As we show in the experiments (see Sec. 4.2), an optimal value of λindeed increases with the variance of the noise in the input image, which is well-aligned with theresult in [42, Theorem 19].
Revised: As we show in the experiments (see Sec. 4.2), an optimal value of λindeed increases with the variance of the noise in the input image, which is well-aligned with theresult in [50, Theorem 19].

Revised: Input: A network architecture with CSC-layers f ( · ; θ , λ 0 ) , a (clean) training data T train , a (corrupted) test data T test , corruption type T , a set C of corruption levels, a set Λ of values for λ .

Revised: # Training the network 2: Train the network f ( · ; θ , λ 0 ) on T train as described in Sec. 3.

Original: Choosing the optimal λ .
Revised: Choosing the optimal λ .

Original: In practical applications the amount of noise in a given test dataset is oftenunknown.
Revised: In practical applications the amount of noise in a given test dataset is oftenunknown.

Original: Hence, the optimal choice of parameter λ becomes a nontrivial task.
Revised: Hence, the optimal choice of parameter λ becomes a nontrivial task.

Original: Here we present apractical technique for determining a proper choice of value λ , based on the simple observation thatthe amount of noise in the test data correlates with the magnitude of the residual x − A ( z ) .
Revised: Here we present apractical technique for determining a proper choice of value λ , based on the simple observation thatthe amount of noise in the test data correlates with the magnitude of the residual x − A ( z ) .

Original: Thatis, for test data corrupted by a larger amount of noise, we expect the magnitude of the residual in
Revised: Thatis, for test data corrupted by a larger amount of noise, we expect the magnitude of the residual in

Original: CSC-layers to become larger with such data fed into the network.
Revised: CSC-layers to become larger with such data fed into the network.

Original: Since the residual for any test datacan always be computed, the key question here is how we can ﬁnd a relationship between an optimalvalue λ and the magnitude of the residual.
Revised: Since the residual for any test datacan always be computed, the key question here is how we can ﬁnd a relationship between an optimalvalue λ and the magnitude of the residual.

Original: Our technique for addressing this challenge is to learn such a relationship on the training set, byinjecting synthetic data corruptions.
Revised: Our technique for addressing this challenge is to learn such a relationship on the training set, byinjecting synthetic data corruptions.

Original: For simplicity we summarize our technique in Algorithm 1, andexplain it in details below.
Revised: For simplicity we summarize our technique in Algorithm 1, andexplain it in details below.

Original: We suppose that although the amount of noise in a test dataset T test isunknown, the type of the noise (e.g., Gaussian noise, shot noise, etc.) is known.
Revised: We suppose that although the amount of noise in a test dataset T test isunknown, the type of the noise (e.g., Gaussian noise, shot noise, etc.) is known.

Original: Given the noise type,we ﬁrst determine a set C which contains a set of values specifying the potential amount of noisein test data.
Revised: Given the noise type,we ﬁrst determine a set C which contains a set of values specifying the potential amount of noisein test data.

Original: For example, if we consider Gaussian noise, then C contains a set of values specifyingthe variance of the noise.
Revised: For example, if we consider Gaussian noise, then C contains a set of values specifyingthe variance of the noise.

Original: We also specify a set Λ of potential values for λ to be used for inference.
Revised: We also specify a set Λ of potential values for λ to be used for inference.

Original: After training the network as speciﬁed in Step 2, we use a procedure described in between Step 4 and
Revised: After training the network as speciﬁed in Step 2, we use a procedure described in between Step 4 and

Original: Step 12 to ﬁt a function that maps a residual value r to an optimal choice of λ .
Revised: Step 12 to ﬁt a function that maps a residual value r to an optimal choice of λ .

Original: The idea for ﬁttingsuch a relation is to generate synthetic noise of varying magnitudes in C on the training set, and foreach noise magnitude we sweep the parameter λ ∈ Λ to ﬁnd a λ that produces the best accuracy ontraining set.
Revised: The idea for ﬁttingsuch a relation is to generate synthetic noise of varying magnitudes in C on the training set, and foreach noise magnitude we sweep the parameter λ ∈ Λ to ﬁnd a λ that produces the best accuracy ontraining set.

Original: Once such a relationship is learned, we feed in the residual value computed on the testdataset T test to predict an optimal λ to be used for inference on test data, as described in Step 14.
Revised: Once such a relationship is learned, we feed in the residual value computed on the testdataset T test to predict an optimal λ to be used for inference on test data, as described in Step 14.

Original:  Experiments
Revised:  Experiments

Original: In this section, we provide experimental evidence for neural networks with CSC-layers as discussedin Sec. 3.
Revised: In this section, we provide experimental evidence for neural networks with CSC-layers as discussedin Sec. 3.

Original: Through experiments on CIFAR-10, CIFAR-100 3 , and ImageNet 4 , Sec. 4.1 shows that ournetworks have equally competitive classiﬁcation performance as mainstream architectures such as
Revised: Through experiments on CIFAR-10, CIFAR-100 2 , and ImageNet 3 , Sec. 4.1 shows that ournetworks have equally competitive classiﬁcation performance as mainstream architectures such as

Original: ResNet.
Revised: ResNet.

Original: Furthermore, we show in Sec.
Revised: Furthermore, we show in Sec.

Original: 4.2 that our network is able to handle input perturbations withthe robust inference technique.
Revised: 4.2 that our network is able to handle input perturbations withthe robust inference technique.

Original: Finally, we demonstrate in Sec.
Revised: Finally, we demonstrate in Sec.

Original: 4.3 that our network is able to handleadversarial perturbations as well.
Revised: 4.3 that our network is able to handleadversarial perturbations as well.

Original: More details about implementation are given in the Appendix.
Revised: More details about implementation are given in the Appendix.

Original: Datasets.
Revised: Datasets.

Original: We test the performance of our method using the CIFAR-10 and CIFAR-100 [43] datasets.
Revised: We test the performance of our method using the CIFAR-10 and CIFAR-100 [51] datasets.

Original: Each dataset contains 50,000 training images and 10,000 testing images where each image is of size32 × 32 with RGB channels.
Revised: Each dataset contains 50,000 training images and 10,000 testing images where each image is of size32 × 32 with RGB channels.

Original: We also use the ImageNet dataset [44] that contains 1,000 classes and atotal number of around 1 million images.
Revised: We also use the ImageNet dataset [52] that contains 1,000 classes and atotal number of around 1 million images.

Original: Network architecture.
Revised: Network architecture.

Original: We use the network architectures with the ﬁrst convolutional layers of
Revised: We use the network architectures with the ﬁrst convolutional layers of

Original: ResNet-18 and ResNet-34 [15] 5 replaced by CSC-layers, and refer to these networks as SDNet-18and SDNet-34, respectively.
Revised: ResNet-18 and ResNet-34 [21] 4 replaced by CSC-layers, and refer to these networks as SDNet-18and SDNet-34, respectively.

Original: We use λ = 0 .
Revised: We use λ = 0 .

Original: 1 as the trade-off parameter in (4) for all CSC-layersunless speciﬁed otherwise.
Revised: 1 as the trade-off parameter in (4) for all CSC-layersunless speciﬁed otherwise.

Original: Forward propagation through each CSC-layer is performed via unrollingtwo iterations of FISTA.
Revised: Forward propagation through each CSC-layer is performed via unrollingtwo iterations of FISTA.

Original: Network training.
Revised: Network training.

Original: For CIFAR-10 and CIFAR-100, we use a cosine learning rate decay schedulewith an initial learning rate of 0.1, and train the model for 220 epochs.
Revised: For CIFAR-10 and CIFAR-100, we use a cosine learning rate decay schedulewith an initial learning rate of 0.1, and train the model for 220 epochs.

Original: We use the SGD optimizerwith 0.9 momentum and Nestrov.
Revised: We use the SGD optimizerwith 0.9 momentum and Nestrov.

Original: The weight decay is set to 5 × 10 − 4 , and batch size is set to 128.
Revised: The weight decay is set to 5 × 10 − 4 , and batch size is set to 128.

Original: All the experiments are conducted on a single NVIDIA GTX 2080Ti GPU.
Revised: All the experiments are conducted on a single NVIDIA GTX 2080Ti GPU.

Original: For ImageNet, we usemulti-step learning rate decay schedule with an initial learning rate of 0.1 decayed by a factor of 0 .
Revised: For ImageNet, we usemulti-step learning rate decay schedule with an initial learning rate of 0.1 decayed by a factor of 0 .

Original: 1at the 30th, 60th, and 90th epochs, and train the model for 100 epochs.
Revised: 1at the 30th, 60th, and 90th epochs, and train the model for 100 epochs.

Original: The batch size is 512, and theoptimizer is SGD with 0.9 momentum and Nestrov.
Revised: The batch size is 512, and theoptimizer is SGD with 0.9 momentum and Nestrov.

Original: All experiments on ImageNet are conducted on 

Revised: All experiments on ImageNet are conducted on

Original: NVIDIA RTX 3090 GPUs.
Revised: NVIDIA RTX 3090 GPUs.

Original: 4.1 Performance for Image Classiﬁcation
Revised: 4.1 Performance for Image Classiﬁcation

Original: We compare our method with standard network architectures ResNet-18 and ResNet-34 [15].
Revised: We compare our method with standard network architectures ResNet-18 and ResNet-34 [21].

Original: Inaddition, we compare with architectures with implicit layers (i.e., MDEQ [21]) and architectures withsparse modeling (i.e., SCN [9]).
Revised: Inaddition, we compare with architectures with implicit layers (i.e., MDEQ [27]) and architectures withsparse modeling (i.e., SCN [15]).

Original: For ResNet-18 and ResNet-34, we train the model using the same
Revised: For ResNet-18 and ResNet-34, we train the model using the samesetup as our SDNet models.

Revised: For MDEG and SCN, we train the models using the settings as stated intheir respective papers.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: setup as our SDNet models.

Original: For MDEG and SCN, we train the models using the settings as stated intheir respective papers.

Original: The results are reported in Table 1.
Revised: The results are reported in Table 1.

Original: We see that with a similar model size, SDNet-18/34 produces a
Revised: We see that with a similar model size, SDNet-18/34 produces a

Original: Top-1 accuracy that closely matches (for CIFAR-10 and ImageNet) or surpasses (for CIFAR-100)that of ResNet-18/34 while having a comparable Speed.
Revised: Top-1 accuracy that closely matches (for CIFAR-10 and ImageNet) or surpasses (for CIFAR-100)that of ResNet-18/34 while having a comparable Speed.

Original: This shows the potential of our network withmodeling based layers as a powerful alternative to existing data-driven models, since our model hasthe additional beneﬁt for handling corruptions.
Revised: This shows the potential of our network withmodeling based layers as a powerful alternative to existing data-driven models, since our model hasthe additional beneﬁt for handling corruptions.

Original: We also compare our SDNet-18 model with the MDEQ model, which has a similar model size, andsee that SDNet-18 is not only more accurate than MDEQ but is much ( > 7 times) faster.
Revised: We also compare our SDNet-18 model with the MDEQ model, which has a similar model size, andsee that SDNet-18 is not only more accurate than MDEQ but is much ( > 7 times) faster.

Original: Note that
Revised: Note that

Original: MDEQ cannot handle corrupted data as in our method as well.
Revised: MDEQ cannot handle corrupted data as in our method as well.

Original: The SCN network, which also uses sparse modeling, obtains an Top-1 accuracy that is highlycompetitive to all methods.
Revised: The SCN network, which also uses sparse modeling, obtains an Top-1 accuracy that is highlycompetitive to all methods.

Original: However, a signiﬁcant drawback of SCN is that it is very slow to train.
Revised: However, a signiﬁcant drawback of SCN is that it is very slow to train.

Original: The reason may be that SCN uses a patch-based sparse coding model for images, in contrast to aconvolutional sparse coding model as in our method, which requires solving many sparse codingproblems in each forward propagation that cannot beneﬁt from parallel computing.
Revised: The reason may be that SCN uses a patch-based sparse coding model for images, in contrast to aconvolutional sparse coding model as in our method, which requires solving many sparse codingproblems in each forward propagation that cannot beneﬁt from parallel computing.

Original: Replacing all convolution layer by the CSC-layer.
Revised: Replacing all convolution layer by the CSC-layer.

Original: We also train a version of SDNet obtainedfrom replacing all convolution layers of a ResNet with the CSC-layer (as opposed to only the ﬁrstconvolution layer), and refer to such a model as SDNet-18-All and SDNet-34-All.
Revised: We also train a version of SDNet obtainedfrom replacing all convolution layers of a ResNet with the CSC-layer (as opposed to only the ﬁrstconvolution layer), and refer to such a model as SDNet-18-All and SDNet-34-All.

Original: On ImageNet weobserve that SDNet-18-All and SDNet-34-All obtain 69.37% and 72.54% Top-1 accuracy, respectively.
Revised: On ImageNet weobserve that SDNet-18-All and SDNet-34-All obtain 69.37% and 72.54% Top-1 accuracy, respectively.

Original: Comparing such results with those of SDNet-18/34 reported in Table 1, we see that the performanceis not signiﬁcantly affected by replacing more convolution layers with CSC-layers.
Revised: Comparing such results with those of SDNet-18/34 reported in Table 1, we see that the performanceis not signiﬁcantly affected by replacing more convolution layers with CSC-layers.

Original: Moreover,
Revised: Moreover,

Original: SDNet-18/34-All enables us to develop a visualization technique as described in the Appendix.
Revised: SDNet-18/34-All enables us to develop a visualization technique as described in the Appendix.

Original: Effect of number of iterations.
Revised: Effect of number of iterations.

Original: Recall from Sec.
Revised: Recall from Sec.

Original: 3.1 that forward propagation through a CSC-layeris performed by running a few iterative steps of the FISTA algorithm.
Revised: 3.1 that forward propagation through a CSC-layeris performed by running a few iterative steps of the FISTA algorithm.

Original: Here we provide a study on howthe number of iterations affects model performance.
Revised: Here we provide a study on howthe number of iterations affects model performance.

Original: On ImageNet, we observe that SDNet-18 with 2,4, and 8 iterations of FISTA obtains 69.47%, 69.51%, and 69.79% Top-1 accuracy, respectively.
Revised: On ImageNet, we observe that SDNet-18 with 2,4, and 8 iterations of FISTA obtains 69.47%, 69.51%, and 69.79% Top-1 accuracy, respectively.

Original: Whileusing more iterations slightly increases the model performance, it comes at the cost of increasing thetraining time and memory requirement as a result of unrolling of the FISTA algorithm.
Revised: Whileusing more iterations slightly increases the model performance, it comes at the cost of increasing thetraining time and memory requirement as a result of unrolling of the FISTA algorithm.

Original: Hence, in allour experiments we use SDNet with 2 iterations.
Revised: Hence, in allour experiments we use SDNet with 2 iterations.

Original: 4.2 Handling Input Perturbations
Revised: 4.2 Handling Input Perturbations

Original: To test the robustness of our method to input perturbations, we use the CIFAR-10-C dataset [45]which contains a test set for CIFAR-10 that is corrupted with different types of synthetic noise and5 severity levels for each type.
Revised: To test the robustness of our method to input perturbations, we use the CIFAR-10-C dataset [53]which contains a test set for CIFAR-10 that is corrupted with different types of synthetic noise and5 severity levels for each type.

Original: Because the CSC model in (4) penalizes the entry-wise differencebetween input and reconstructed signals, it is more suited for handling additive noises.
Revised: Because the CSC model in (4) penalizes the entry-wise differencebetween input and reconstructed signals, it is more suited for handling additive noises.

Original: Hence, wefocus on four types of additive noises in CIFAR-10-C, namely, Gaussian noise, shot noise, specklenoise, and impulse noise.
Revised: Hence, wefocus on four types of additive noises in CIFAR-10-C, namely, Gaussian noise, shot noise, specklenoise, and impulse noise.

Original: We evaluate the accuracy of our SDNet-18 and compare its performancewith ResNet-18.
Revised: We evaluate the accuracy of our SDNet-18 and compare its performancewith ResNet-18.

Original: [Figure]
Revised: [Figure]

Original: Robustness as a function of λ .
Revised: Robustness as a function of λ .

Original: As discussed in Sec. 3.3, we may improve the performance of ourmodel to noisy test data by using values of λ that are different from the training phase.
Revised: As discussed in Sec. 3.3, we may improve the performance of ourmodel to noisy test data by using values of λ that are different from the training phase.

Original: Therefore, wereport the performance of our method with varying λ in the range of [0 . 1 , 1 . 5] in Figure 2 (recall thatλ = 0 .
Revised: Therefore, wereport the performance of our method with varying λ in the range of [0 . 1 , 1 . 5] in Figure 2 (recall thatλ = 0 .

Original: 1 is used for training).
Revised: 1 is used for training).

Original: It can be seen that for all types of noises and all severity levels (exceptfor impulse noise with level 0, 1, and 2), properly choosing a value of λ that is different from that usedduring training helps to improve the test performance.
Revised: It can be seen that for all types of noises and all severity levels (exceptfor impulse noise with level 0, 1, and 2), properly choosing a value of λ that is different from that usedduring training helps to improve the test performance.

Original: In particular, the accuracy curves as a functionof λ exhibit a unimodal shape where the performance ﬁrst increases then decreases.
Revised: In particular, the accuracy curves as a functionof λ exhibit a unimodal shape where the performance ﬁrst increases then decreases.

Original: Moreover, withineach corruption type the values of λ where a peak performance is achieved increases monotonicallywith the severity level of the corruption.
Revised: Moreover, withineach corruption type the values of λ where a peak performance is achieved increases monotonicallywith the severity level of the corruption.

Original: Such an observation is well-aligned with our discussion in
Revised: Such an observation is well-aligned with our discussion in

Original: Sec. 3.3.
Revised: Sec. 3.3.

Original: Choosing an optimal λ .
Revised: Choosing an optimal λ .

Original: While Figure 2 demonstrates that one may improve the performance oncorrupted data via a proper choice of λ , it does not show how to choose the best λ in practice.
Revised: While Figure 2 demonstrates that one may improve the performance oncorrupted data via a proper choice of λ , it does not show how to choose the best λ in practice.

Original: Herewe show that the technique presented in Algorithm 1 can be used to select λ for robust inference.
Revised: Herewe show that the technique presented in Algorithm 1 can be used to select λ for robust inference.

Original: Speciﬁcally, we apply Algorithm 1 with f ( · ; θ , λ 0 ) being SDNet-18 with λ 0 = 0 .
Revised: Speciﬁcally, we apply Algorithm 1 with f ( · ; θ , λ 0 ) being SDNet-18 with λ 0 = 0 .

Original: 1 , T train being the
Revised: 1 , T train being the

Original: CIFAR-10 training set, T test being CIFAR-10-C data with a particular type of corruption under aparticular severity level
Revised: CIFAR-10 training set, T test being CIFAR-10-C data with a particular type of corruption under aparticular severity level

Original: [Table]
Revised: [Table]

Original: and report the performance of the algorithm output in Table 2 and Table
Revised: and report the performance of the algorithm output in Table 2 and Table

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Table 2 shows the results with varying severity levels of Gaussian noise.
Revised: Table 2 shows the results with varying severity levels of Gaussian noise.

Original: We can see that if we usethe same λ = 0 .
Revised: We can see that if we usethe same λ = 0 .

Original: 1 as in training (i.e., SDNet-18 w/ λ
Revised: 1 as in training (i.e., SDNet-18 w/ λ

Original: = 0 . 1 ), then the performance of SDNet-18 is already signiﬁcantly better ResNet-18.
Revised: = 0 . 1 ), then the performance of SDNet-18is already signiﬁcantly better ResNet-18.

Original: This may be attributed to the fact that the CSC-layer isintrinsically more robust to input perturbations than convolution layers owning to its sparse modeling.
Revised: This may be attributed to the fact that the CSC-layer isintrinsically more robust to input perturbations than convolution layers owning to its sparse modeling.

Original: However, if we use an adaptive λ computed from Algorithm 1 (i.e., SDNet-18 w/ adaptive λ ), thenthe performance is further signiﬁcantly improved at all noise levels.
Revised: However, if we use an adaptive λ computed from Algorithm 1 (i.e., SDNet-18 w/ adaptive λ ), thenthe performance is further signiﬁcantly improved at all noise levels.

Original: To demonstrate that our methodworks beyond Gaussian noise, we report results for each of the four types of additive noises averagedover all severity levels in Table 3 for CIFAR-10-C as well as ImageNet-C.
Revised: To demonstrate that our methodworks beyond Gaussian noise, we report results for each of the four types of additive noises averagedover all severity levels in Table 3 for CIFAR-10-C as well as ImageNet-C.

Original: The results demonstratethat sparse modeling enables us to effectively handle various types of additive noises in test data veryeasily with the procedure in Algorithm 1.
Revised: The results demonstratethat sparse modeling enables us to effectively handle various types of additive noises in test data veryeasily with the procedure in Algorithm 1.

Original: [Figure]
Revised: [Figure]

Original: 314 Finally, in Figure 3 we plot the relationship between the optimal choice of λ (in y-axis) and themagnitude of residual in CSC-layers (in x-axis) learned from training data according to Algorithm 1.
Revised: 321 Finally, in Figure 3 we plot the relationship between the optimal choice of λ (in y-axis) and themagnitude of residual in CSC-layers (in x-axis) learned from training data according to Algorithm 1.

Original: We also plot the linear ﬁtting of such a relation, which can be seen to provide a good qualityapproximation for each of the four corruption types.
Revised: We also plot the linear ﬁtting of such a relation, which can be seen to provide a good qualityapproximation for each of the four corruption types.

Original: On the other hand, we note that the ﬁtted linearrelations differ across different types of corruptions.
Revised: On the other hand, we note that the ﬁtted linearrelations differ across different types of corruptions.

Original: Hence, it is important that the relationship isestimated for each corruption types separately.
Revised: Hence, it is important that the relationship isestimated for each corruption types separately.

Original: 4.3 Handling Adversarial Perturbations
Revised: 4.3 Handling Adversarial Perturbations

Original: We show that our method also exhibits robustness to adversarial perturbations.
Revised: We show that our method also exhibits robustness to adversarial perturbations.

Original: In this experiment, wegenerate adversarial perturbations on the CIFAR-10 test dataset using PGD attack on our SDNet (withλ = 0 . 1 ), with L ∞ norm of the perturbation being (cid:15) = 8 / 255 and L 2 norm of the perturbation being(cid:15)
Revised: In this experiment, wegenerate adversarial perturbations on the CIFAR-10 test dataset using PGD attack on our SDNet (withλ = 0 . 1 ), with L ∞ norm of the perturbation being (cid:15) = 8 / 255 and L 2 norm of the perturbation being(cid:15)

Original: = 0 .
Revised: = 0 .

Original: 5 , respectively.
Revised: 5 , respectively.

Original: The robust accuracy of our method is reported in Table 4 and is compared withthat of ResNet-18.
Revised: The robust accuracy of our method is reported in Table 4 and is compared withthat of ResNet-18.

Original: We can see that while SDNet does not perform much better than ResNet withλ = 0 .
Revised: We can see that while SDNet does not perform much better than ResNet withλ = 0 .

Original: 1 , we may tune the parameter λ to drastically improve the robust accuracy.
Revised: 1 , we may tune the parameter λ to drastically improve the robust accuracy.

Original: [Table]
Revised: [Table]

Original:  Conclusion and Discussion
Revised:  Conclusion and Discussion

Original: This paper revisits the classical sparse modeling and provides a simple way of using it to guidethe design of interpretable deep networks.
Revised: This paper revisits the classical sparse modeling and provides a simple way of using it to guidethe design of interpretable deep networks.

Original: Despite multiple prior attempts, our work is the ﬁrstto demonstrate that such a design can produce performance (in terms of accuracy, model size,and memory) that is on par with standard ConvNets on modern image datasets such as ImageNet.
Revised: Despite multiple prior attempts, our work is the ﬁrstto demonstrate that such a design can produce performance (in terms of accuracy, model size,and memory) that is on par with standard ConvNets on modern image datasets such as ImageNet.

Original: The success in combining sparse modeling with deep learning provides a means of borrowing andutilizing the rich results in the well-developed ﬁeld of sparse modeling [46, 47] for network designand analysis.
Revised: The success in combining sparse modeling with deep learning provides a means of borrowing andutilizing the rich results in the well-developed ﬁeld of sparse modeling [54, 55] for network designand analysis.

Original: While it is not the purpose of this work to fully explore all potentials, our experimentsalready demonstrate clear advantages of so designed networks in handling various forms of datacorruptions.
Revised: While it is not the purpose of this work to fully explore all potentials, our experimentsalready demonstrate clear advantages of so designed networks in handling various forms of datacorruptions.

Original: Looking forward, other fundamental principles, algorithm and techniques in sparsemodeling may be introduced to further enhance the capability of our presented framework.
Revised: Looking forward, other fundamental principles, algorithm and techniques in sparsemodeling may be introduced to further enhance the capability of our presented framework.

Original: Alongthis line, we provide some preliminary evidence on how sparse modeling enables interpretability inthe Appendix, and leave further study tofuture work.
Revised: Alongthis line, we provide some preliminary evidence on how sparse modeling enables interpretability inthe Appendix, and leave further study to

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] • Did you include the license to the code and datasets?
Revised: [Yes]• Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data areproprietary.•

Original: • Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes](b) Did you describe the limitations of your work?

Original: [Yes] Limitations are described in theConclusion as well as in a footnote.(c)
Revised: [Yes] Limitations are described in theConclusion as well as in a footnote.(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Discussed in Conclusion.
Revised: [Yes] Discussedin Conclusion.(d)

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] No theoretical results. (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] No theoreticalresults.(b) Did you include complete proofs of all theoretical results?

Original: [N/A] No theoretical results.
Revised: [N/A] No theoretical results.3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Please refer tosection 4 and appendix.
Revised: [Yes] Please refer tosection 4 and appendix.(b)

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] Please refer to Section 5 for training details
Revised: [Yes] Please refer to Section 5 for training details

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?

Original: [No]
Revised: [No]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] Pleas refer to Section 5 for details
Revised: [Yes] Pleas refer to Section 5 for details

Original: related to computation and resources
Revised: related to computation and resources

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] All datasets we used: CIFAR-
Revised: [Yes] All datasets we used: CIFAR-

Original: 10, CIFAR-100, and ImageNet are publicly available, and we included their urls in footnotes.
Revised: 10, CIFAR-100, and ImageNet are publicly available, and we included their urls infootnotes.

Original: We also included the license and copyright information of the baseline
Revised: We also included the license and copyright information of the baseline

Original: ResNet models in a footnote.
Revised: ResNet models in a footnote.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A] All datasets are publicly available.
Revised: [N/A] All datasets are publicly available.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A] All datasets are publicly available.[Table]
Revised: [N/A] All datasets are publicly available.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table] (a) Did you include the full text of instructions given to participants and screenshots, if[Table]
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]
Revised: [Table]

Original: Board (IRB) approvals, if applicable?

Original: [N/A] [Table]
Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?

Original: [N/A] [Table]
Revised: [Table]