Original: Revisiting Sparse Convolutional Model for Visual Recognition
Revised: Revisiting Sparse Convolutional Model for Visual Recognition

Original: Abstract
Revised: Abstract

Original: Despite strong empirical performance for image classiﬁcation, deep neural networks are often regarded as “black boxes” and they are difﬁcult to interpret.
Revised: Despite strong empirical performance for image classiﬁcation, deep neural networks are often regarded as “black boxes” and they are difﬁcult to interpret.

Original: On the other hand, sparse convolutional models, which assume that a signal can be expressed by a linear combination of a few elements from a convolutional dictionary, are powerful tools for analyzing natural images with good theoretical interpretability and biological plausibility.
Revised: On the other hand, sparse convolutional models, which assume that a signal can be expressed by a linear combination of a few elements from a convolutional dictionary, are powerful tools for analyzing natural images with good theoretical interpretability and biological plausibility.

Original: However, such principled models have not demonstrated competitive performance when compared with empirically designed deep networks.
Revised: However, such principled models have not demonstrated competitive performance when compared with empirically designed deep networks.

Original: This paper revisits the sparse convolutional modeling for image classiﬁcation and bridges the gap between good empirical performance (of deep learning) and good interpretability (of sparse convolutional models).
Revised: This paper revisits the sparse convolutional modeling for image classiﬁcation and bridges the gap between good empirical performance (of deep learning) and good interpretability (of sparse convolutional models).

Original: Our method uses differentiable optimization layers that are deﬁned from convolutional sparse coding as drop-in replacements of standard convolutional layers in conventional deep neural networks.
Revised: Our method uses differentiable optimization layers that are deﬁned from convolutional sparse coding as drop-in replacements of standard convolutional layers in conventional deep neural networks.

Original: We show that such models have equally strong empirical performance on CIFAR-10, CIFAR-100 and ImageNet datasets when compared to conventional neural networks.
Revised: We show that such models have equally strong empirical performance on CIFAR-10, CIFAR-100 and ImageNet datasets when compared to conventional neural networks.

Original: By leveraging stable recovery property of sparse modeling, we further show that such models can be much more robust to input corruptions as well as adversarial perturbations in testing through a simple proper trade-off between sparse regularization and data reconstruction terms.
Revised: By leveraging stable recovery property of sparse modeling, we further show that such models can be much more robust to input corruptions as well as adversarial perturbations in testing through a simple proper trade-off between sparse regularization and data reconstruction terms.

Original: Source code can be found at https://github.com/Delay-Xili/SDNet .
Revised: Source code can be found at https://github.com/Delay-Xili/SDNet .

Original:  Introduction
Revised:  Introduction

Original: In recent years, deep learning has been a dominant approach for image classiﬁcation and has signiﬁcantly advanced the performance over previous shallow models.
Revised: In recent years, deep learning has been a dominant approach for image classiﬁcation and has signiﬁcantly advanced the performance over previous shallow models.

Original: Despite the phenomenal empirical success, it has been increasingly realized as well as criticized that deep convolutional networks (ConvNets) are “black boxes” for which we are yet to develop clear understanding [1].
Revised: Despite the phenomenal empirical success, it has been increasingly realized as well as criticized that deep convolutional networks (ConvNets) are “black boxes” for which we are yet to develop clear understanding [1].

Original: The layer operations such as convolution, nonlinearity and normalization are geared towards minimizing an end-to-end training loss and do not have much data-speciﬁc meaning .
Revised: The layer operations such as convolution, nonlinearity and normalization are geared towards minimizing an end-to-end training loss and do not have much data-speciﬁc meaning .

Original: As such, the functionality of each intermediate layer in a trained ConvNets is mostly unclear and the feature maps that they produce are hard to interpret.
Revised: As such, the functionality of each intermediate layer in a trained ConvNets is mostly unclear and the feature maps that they produce are hard to interpret.

Original: The lack of interpretability also contributes to the notorious difﬁculty in enhancing such learning systems for practical data which are usually corrupted by various forms of perturbation.
Revised: The lack of interpretability also contributes to the notorious difﬁculty in enhancing such learning systems for practical data which are usually corrupted by various forms of perturbation.

Original: This paper presents a visual recognition framework by introducing layers that have explicit data modeling to tackle shortcomings of current deep learning systems.
Revised: This paper presents a visual recognition framework by introducing layers that have explicit data modeling to tackle shortcomings of current deep learning systems.

Original: We work under the assumption that the layer input can be represented by a few atoms from a dictionary shared by all data points.
Revised: We work under the assumption that the layer input can be represented by a few atoms from a dictionary shared by all data points.

Original: This is the classical sparse data modeling that, as shown in a pioneering work of [2], can easily discover meaningful structures from natural image patches.
Revised: This is the classical sparse data modeling that, as shown in a pioneering work of [2], can easily discover meaningful structures from natural image patches.

Original: Backed by its ability in learning interpretable representations and strong theoretical guarantees [3, 4, 5, 6, 7, 8] (e.g. for handling corrupted data), sparse modeling has been used broadly in many signal and image processing applications [9].
Revised: Backed by its ability in learning interpretable representations and strong theoretical guarantees [3, 4, 5, 6, 7, 8] (e.g. for handling corrupted data), sparse modeling has been used broadly in many signal and image processing applications [9].

Original: However, the empirical performance of sparse methods have been surpassed by deep learning methods for classiﬁcation of modern image datasets.
Revised: However, the empirical performance of sparse methods have been surpassed by deep learning methods for classiﬁcation of modern image datasets.

Original: Because of the complementary beneﬁts of sparse modeling and deep learning, there exist many efforts that leverage sparse modeling to gain theoretical insights into ConvNets and/or to develop computational methods that further improve upon existing ConvNets.
Revised: Because of the complementary beneﬁts of sparse modeling and deep learning, there exist many efforts that leverage sparse modeling to gain theoretical insights into ConvNets and/or to develop computational methods that further improve upon existing ConvNets.

Original: One of the pioneering works is [10] which interpreted a ConvNet as approximately solving a multi-layer convolutional sparse coding model.
Revised: One of the pioneering works is [10] which interpreted a ConvNet as approximately solving a multi-layer convolutional sparse coding model.

Original: Based on this interpretation, the work [10] and its follow-ups [11, 12, 13, 14] presented alternative algorithms and models in order to further enhance the practical performance of such learning systems.
Revised: Based on this interpretation, the work [10] and its follow-ups [11, 12, 13, 14] presented alternative algorithms and models in order to further enhance the practical performance of such learning systems.

Original: However, there has been no empirical evidence that such systems can handle modern image datasets such as ImageNet and obtain comparable performance to deep learning.
Revised: However, there has been no empirical evidence that such systems can handle modern image datasets such as ImageNet and obtain comparable performance to deep learning.

Original: The only exception to the best of our knowledge is the work of [15, 16] which exhibited a performance on par to (on ImageNet) or better than (on CIFAR-10) ResNet.
Revised: The only exception to the best of our knowledge is the work of [15, 16] which exhibited a performance on par to (on ImageNet) or better than (on CIFAR-10) ResNet.

Original: However, the method in [15, 16] 1) requires a dedicated design of network architecture that may limit its applicability, 2) is computationally orders of magnitude slower to train, and 3) does not demonstrate beneﬁts in terms of interpretability and robustness.
Revised: However, the method in [15, 16] 1) requires a dedicated design of network architecture that may limit its applicability, 2) is computationally orders of magnitude slower to train, and 3) does not demonstrate beneﬁts in terms of interpretability and robustness.

Original: In a nutshell, sparse modeling is yet to demonstrate practicality that enables its broad applications.
Revised: In a nutshell, sparse modeling is yet to demonstrate practicality that enables its broad applications.

Original: Paper contributions.
Revised: Paper contributions.

Original: In this paper, we revisit sparse
Revised: In this paper, we revisit sparse

Original: modeling for image classiﬁcation and demonstrate through a simple design that sparse modeling can be combined with deep learning to obtain performance on par with standard ConvNets but with better layer-wise interpretability and stability.
Revised: modeling for image classiﬁcation and demonstrate through a simple design that sparse modeling can be combined with deep learning to obtain performance on par with standard ConvNets but with better layer-wise interpretability and stability.

Original: Our method encapsulates the sparse modeling into an implicit layer [17, 18, 19] and uses it as a drop-in replacement for any convolutional layer in standard ConvNets.
Revised: Our method encapsulates the sparse modeling into an implicit layer [17, 18, 19] and uses it as a drop-in replacement for any convolutional layer in standard ConvNets.

Original: The layer implements the convolutional sparse coding (CSC) model of [20], and is referred to as a CSC-layer , where the input signal is approximated by a sparse linear combination of atoms from a convolutional dictionary.
Revised: The layer implements the convolutional sparse coding (CSC) model of [20], and is referred to as a CSC-layer , where the input signal is approximated by a sparse linear combination of atoms from a convolutional dictionary.

Original: Such a convolutional dictionary is treated as the parameters of the CSC-layer that are amenable to training via back-propagation.
Revised: Such a convolutional dictionary is treated as the parameters of the CSC-layer that are amenable to training via back-propagation.

Original: Then, the overall network with the CSC-layers may be trained in an end-to-end fashion from labeled data by minimizing the cross-entropy loss as usual.
Revised: Then, the overall network with the CSC-layers may be trained in an end-to-end fashion from labeled data by minimizing the cross-entropy loss as usual.

Original: This paper demonstrates that such a learning framework has the following beneﬁts
Revised: This paper demonstrates that such a learning framework has the following beneﬁts

Original: : • Performance on standard datasets.
Revised: : • Performance on standard datasets.

Original: We demonstrate that our network obtains better (on CIFAR100) or on par (on CIFAR-10 and ImageNet) performance with similar training time compared with standard architectures such as ResNet [21].
Revised: We demonstrate that our network obtains better (on CIFAR100) or on par (on CIFAR-10 and ImageNet) performance with similar training time compared with standard architectures such as ResNet [21].

Original: This provides the ﬁrst evidence on the strong empirical performance of sparse modeling for deep learning to the best of our knowledge.
Revised: This provides the ﬁrst evidence on the strong empirical performance of sparse modeling for deep learning to the best of our knowledge.

Original: Compared to previous sparse methods [15] that obtained similar performance, our method is of orders of magnitude faster.
Revised: Compared to previous sparse methods [15] that obtained similar performance, our method is of orders of magnitude faster.

Original: • Robustness to input perturbations.
Revised: • Robustness to input perturbations.

Original: The stable recovery property of sparse convolution model equips the CSC-layers with the ability to remove perturbation in the layer input and to recover clean sparse code.
Revised: The stable recovery property of sparse convolution model equips the CSC-layers with the ability to remove perturbation in the layer input and to recover clean sparse code.

Original: As a result, our networks with CSC-layers are more robust to perturbations in the input images compared with classical neural networks.
Revised: As a result, our networks with CSC-layers are more robust to perturbations in the input images compared with classical neural networks.

Original: Unlike existing approaches for obtaining robustness that require heavy data augmentation [22] or additional training techniques [23], our method is light-weight and does not require modifying the training procedure at all.
Revised: Unlike existing approaches for obtaining robustness that require heavy data augmentation [22] or additional training techniques [23], our method is light-weight and does not require modifying the training procedure at all.

Original:  Related Work
Revised:  Related Work

Original: Implicit layers.
Revised: Implicit layers.

Original: The idea of trainable layers deﬁned from implicit functions can be traced back at least to the work of [24].
Revised: The idea of trainable layers deﬁned from implicit functions can be traced back at least to the work of [24].

Original: Recently, there is a revival of interests in implicit layers [17, 18, 19, 25, 26, 27, 28, 29] as an attractive alternative to explicit layers in existing neural networks.
Revised: Recently, there is a revival of interests in implicit layers [17, 18, 19, 25, 26, 27, 28, 29] as an attractive alternative to explicit layers in existing neural networks.

Original: However, a majority of the cited works above deﬁne an implicit layer by a ﬁxed point iteration, typically motivated from existing explicit layers such as residual layers, therefore they do not have clear interpretation in terms of modeling of the layer input.
Revised: However, a majority of the cited works above deﬁne an implicit layer by a ﬁxed point iteration, typically motivated from existing explicit layers such as residual layers, therefore they do not have clear interpretation in terms of modeling of the layer input.

Original: Consequently, such models do not have the ability to deal with input perturbations.
Revised: Consequently, such models do not have the ability to deal with input perturbations.

Original: The only exceptions are differentiable optimization layers [30, 17, 18, 31] that incorporate complex dependencies between hidden layers through the formulation of convex optimization.
Revised: The only exceptions are differentiable optimization layers [30, 17, 18, 31] that incorporate complex dependencies between hidden layers through the formulation of convex optimization.

Original: Nevertheless, most of the above works focus on differentiating through the convex optimization layers (such as disciplined parametrized programming [18]) without specializing in any particular signal models such as the sparse models considered in this paper nor demonstrating their performance when encapsulated in multi-layer neural networks.
Revised: Nevertheless, most of the above works focus on differentiating through the convex optimization layers (such as disciplined parametrized programming [18]) without specializing in any particular signal models such as the sparse models considered in this paper nor demonstrating their performance when encapsulated in multi-layer neural networks.

Original: Sparse prior in deep learning.
Revised: Sparse prior in deep learning.

Original: Aside from image classiﬁcation, sparse modeling has been introduced to deep learning for many image processing tasks such as super resolution [32], denoising [33] and so on [34, 35, 36, 37].
Revised: Aside from image classiﬁcation, sparse modeling has been introduced to deep learning for many image processing tasks such as super-resolution [32], denoising [33] and so on [34, 35, 36, 37].

Original: These works incorporate sparse modeling by using network architectures that are motivated by (but are not the same as) an unrolled sparse coding algorithmn LISTA [38].
Revised: These works incorporate sparse modeling by using network architectures that are motivated by (but are not the same as) an unrolled sparse coding algorithm LISTA [38].

Original: In sharp contrast to ours, there is no guarantee that such architectures perform a sparse encoding with respect to a particular (convolution) dictionary at all.
Revised: In sharp contrast to ours, there is no guarantee that such architectures perform a sparse encoding with respect to a particular (convolution) dictionary at all.

Original: As a result, they lack the capability of handling input perturbations as in our method.
Revised: As a result, they lack the capability of handling input perturbations as in our method.

Original: A notable exception is the work of [15] where each layer performs a precise sparse encoding and exhibits on par or better performance for image classiﬁcation over ResNet.
Revised: A notable exception is the work of [15] where each layer performs a precise sparse encoding and exhibits on par or better performance for image classiﬁcation over ResNet.

Original: However, the practical beneﬁt of the sparse modeling in terms of robustness is not demonstrated.
Revised: However, the practical beneﬁt of the sparse modeling in terms of robustness is not demonstrated.

Original: Moreover, [15] adopts a patch-based sparse coding model for images and has a large computational burden.
Revised: Moreover, [15] adopts a patch-based sparse coding model for images and has a large computational burden.

Original: Robustness.
Revised: Robustness.

Original: It is known that modern neural networks are extremely vulnerable to small perturbations in the input data.
Revised: It is known that modern neural networks are extremely vulnerable to small perturbations in the input data.

Original: A plethora of techniques have been proposed to address this instability issue, including stability training [23], adversarial training [39, 40, 41], data augmentation [42, 43, 22], etc.
Revised: A plethora of techniques have been proposed to address this instability issue, including stability training [23], adversarial training [39, 40, 41], data augmentation [42, 43, 22], etc.

Original: Nevertheless, these techniques either need a computational and memory overhead, or require a selection of appropriate augmentation strategies to cover all possible corruptions.
Revised: Nevertheless, these techniques either need a computational and memory overhead, or require a selection of appropriate augmentation strategies to cover all possible corruptions.

Original: With standard training only, our model can be made robust to input perturbations in test data by simply adapting sparse modeling to account for noise.
Revised: With standard training only, our model can be made robust to input perturbations in test data by simply adapting sparse modeling to account for noise.

Original: Closely related to our work are [44, 45, 46] which use sparse modeling to improve adversarial robustness.
Revised: Closely related to our work are [44, 45, 46] which use sparse modeling to improve adversarial robustness.

Original: However, they either only demonstrate performance on very simple networks [45, 46] or sacriﬁce natural accuracy for robustness [44].
Revised: However, they either only demonstrate performance on very simple networks [45, 46] or sacriﬁce natural accuracy for robustness [44].

Original: In contrast, our method is tested on realistic networks and does not affect natural accuracy.
Revised: In contrast, our method is tested on realistic networks and does not affect natural accuracy.

Original:  Neural Networks with Sparse Modeling
Revised:  Neural Networks with Sparse Modeling

Original: In this section, we show how sparse modeling is incorporated into a deep network via a speciﬁc type of network layer that we refer to as the convolutional sparse coding (CSC) layer.
Revised: In this section, we show how sparse modeling is incorporated into a deep network via a speciﬁc type of network layer that we refer to as the convolutional sparse coding (CSC) layer.

Original: We describe the
Revised: We describe the

Original: CSC-layer in Sec.
Revised: CSC-layer in Sec.

Original: 3.1 and explain how we use them for deep learning in Sec. 3.2.
Revised: 3.1 and explain how we use them for deep learning in Sec. 3.2.

Original: Finally, Sec. 3. explains how CSC enables robust inference with corrupted test data.
Revised: Finally, Sec. 3. explains how CSC enables robust inference with corrupted test data.

Original: Notations.
Revised: Notations.

Original: Given a single-channel image
Revised: Given a single-channel image

Original: ξ ∈ R H
Revised: ξ ∈ R H

Original: × W represented as a matrix, we may treat it as a 2D signal deﬁned on the discrete domain [1 , . . .
Revised: × W represented as a matrix, we may treat it as a 2D signal deﬁned on the discrete domain [1 , . . .

Original: , H ] × [1 , . . .
Revised: , H ] × [1 , . . .

Original: , W ] and extended to Z × Z by padding zeros.
Revised: , W ] and extended to Z × Z by padding zeros.

Original: Given a 2D kernel α ∈ R k × k , we may treat it as a 2D signal deﬁned on the discrete domain [ − k 0 · · · , k 0 ] × [ − k 0 , · · · , k 0 ] with k = 2 k 0 + 1 and extended to Z × Z by padding zeros.
Revised: Given a 2D kernel α ∈ R k × k , we may treat it as a 2D signal deﬁned on the discrete domain [ − k 0 · · · , k 0 ] × [ − k 0 , · · · , k 0 ] with k = 2 k 0 + 1 and extended to Z × Z by padding zeros.

Original: Then, for convenience, we use “ ∗ ” and “ (cid:63) ” to denote the convolution and correlation operators, respectively, between two 2D signals:
Revised: Then, for convenience, we use “ ∗ ” and “ (cid:63) ” to denote the convolution and correlation operators, respectively, between two 2D signals:

Original: [Equation]
Revised: [Equation]

Original: 3.1 Convolutional Sparse Coding (CSC) Layer
Revised: 3.1 Convolutional Sparse Coding (CSC) Layer

Original: Sparse modeling is introduced in the form of an implicit layer of a neural network.
Revised: Sparse modeling is introduced in the form of an implicit layer of a neural network.

Original: Unlike classical fully-connected or convolutional layers in which input-output relations are deﬁned by an explicit function, implicit layers are deﬁned from implicit functions.
Revised: Unlike classical fully-connected or convolutional layers in which input-output relations are deﬁned by an explicit function, implicit layers are deﬁned from implicit functions.

Original: For our case, in particular, we introduce an implicit layer that is deﬁned from an optimization problem involving the input to the layer as well as a weight parameter, where the output of the layer is the solution to the optimization problem.
Revised: For our case, in particular, we introduce an implicit layer that is deﬁned from an optimization problem involving the input to the layer as well as a weight parameter, where the output of the layer is the solution to the optimization problem.

Original: A generative model via sparse convolution.
Revised: A generative model via sparse convolution.

Original: Concretely, given a multi-dimensional input signal x ∈ R M × H × W to the layer where H, W are spatial dimensions and M is the number of channels for x .
Revised: Concretely, given a multi-dimensional input signal x ∈ R M × H × W to the layer where H, W are spatial dimensions and M is the number of channels for x .

Original: We assume the signal x is generated by a multi-channel sparse code z ∈ R C × H × W convoluting with a multi-dimensional kernel A ∈ R M × C × k × k , which is referred to as a convolution dictionary .
Revised: We assume the signal x is generated by a multi-channel sparse code z ∈ R C × H × W convoluting with a multi-dimensional kernel A ∈ R M × C × k × k , which is referred to as a convolution dictionary .

Original: Here C is the number of channels for z and the convolution kernel A .
Revised: Here C is the number of channels for z and the convolution kernel A .

Original: To be more precise, we denote z as z .
Revised: To be more precise, we denote z as z .

Original: = ( ζ 1 , . .
Revised: = ( ζ 1 , . .

Original: . , ζ C ) where each ζ c ∈ R H × W (presumably sparse), and denote the kernel A as
Revised: . , ζ C ) where each ζ c ∈ R H × W (presumably sparse), and denote the kernel A as

Original: A .
Revised: A .

Original: =  
Revised: =  

Original: ααα...
Revised: ααα...

Original: α 1 C ααα...
Revised: α 1 C ααα...

Original: α 2 C ..
Revised: α 2 C ..

Original: α M 1 α M 2 α M 3 ...
Revised: α M 1 α M 2 α M 3 ...

Original: α MC  
Revised: α MC  

Original: ∈ R M × C × k × k , (2) where each
Revised: ∈ R M × C × k × k , (2) where each

Original: α ij ∈ R k
Revised: α ij ∈ R k

Original: ×
Revised: ×

Original: k is a kernel of size k × k .
Revised: k is a kernel of size k × k .

Original: Then the signal x is generated via the following operator A ( · ) deﬁned by the kernel
Revised: Then the signal x is generated via the following operator A ( · ) deﬁned by the kernel

Original: A as:
Revised: A as:

Original: [Equation]
Revised: [Equation]

Original: A layer as convolutional sparse coding.
Revised: A layer as convolutional sparse coding.

Original: Given a multi-dimensional input signal x ∈ R M × H × W , we deﬁne that the function of “a layer” is to perform an (inverse) mapping to a preferably sparse output
Revised: Given a multi-dimensional input signal x ∈ R M × H × W , we deﬁne that the function of “a layer” is to perform an (inverse) mapping to a preferably sparse output

Original: z ∗ ∈ R C
Revised: z ∗ ∈ R C

Original: ×
Revised: ×

Original: H × W , where C is the number of output channels.
Revised: H × W , where C is the number of output channels.

Original: Under the above sparse generative model, we can seek the optimal sparse solution z by solving the following Lasso type optimization problem:
Revised: Under the above sparse generative model, we can seek the optimal sparse solution z by solving the following Lasso type optimization problem:

Original: [Equation]
Revised: [Equation]

Original: The optimization problem in (4) is based on the convolutional sparse coding (CSC) model [20] 2 .
Revised: The optimization problem in (4) is based on the convolutional sparse coding (CSC) model [20] 2 .

Original: Hence, we refer to the implicit layer deﬁned by (4) as a CSC-layer .
Revised: Hence, we refer to the implicit layer deﬁned by (4) as a CSC-layer .

Original: The goal of the CSC model is to reconstruct the input x via A ( z ) , where the feature map z speciﬁes the locations and magnitudes of the convolutional ﬁlters in A to be linearly combined (see Figure 1 for an illustration).
Revised: The goal of the CSC model is to reconstruct the input x via A ( z ) , where the feature map z speciﬁes the locations and magnitudes of the convolutional ﬁlters in A to be linearly combined (see Figure 1 for an illustration).

Original: The reconstruction is not required to be exact in order to tolerate modeling discrepancies, and the difference between x and A ( z ) is penalized by its entry-wise (cid:96) 2 -norm
Revised: The reconstruction is not required to be exact in order to tolerate modeling discrepancies, and the difference between x and A ( z ) is penalized by its entry-wise (cid:96) 2 -norm

Original: (i.e., the (cid:96) 2 norm of x −A ( z ) ﬂattened into a vector).
Revised: (i.e., the (cid:96) 2 norm of x −A ( z ) ﬂattened into a vector).

Original: Sparse modeling is introduced by the entry-wise (cid:96)
Revised: Sparse modeling is introduced by the entry-wise (cid:96)

Original: 1 -norm of z in the objective function, which enforces z to be sparse.
Revised: 1 -norm of z in the objective function, which enforces z to be sparse.

Original: The parameter λ > 0 controls the tradeoff between the sparsity of z and the magnitude of the residual x − A ( z ) , and is treated as a hyper-parameter that subjects to tuning via cross-validation.
Revised: The parameter λ > 0 controls the tradeoff between the sparsity of z and the magnitude of the residual x − A ( z ) , and is treated as a hyper-parameter that subjects to tuning via cross-validation.

Original: As we will show in Sec. 3.3, λ can be used to improve the performance of our model in the test phase when the input is corrupted.
Revised: As we will show in Sec. 3.3, λ can be used to improve the performance of our model in the test phase when the input is corrupted.

Original: Based on the input-output mapping of the CSC-layer given in (4), one may perform forward propagation by solving the associated optimization, and perform backward propagation by deriving the gradient of z ∗ with respect to the input x and parameter A .
Revised: Based on the input-output mapping of the CSC-layer given in (4), one may perform forward propagation by solving the associated optimization, and perform backward propagation by deriving the gradient of z ∗ with respect to the input x and parameter A .

Original: In this paper, we adopt the fast iterative shrinkage thresholding algorithm (FISTA) [47] for the forward propagation, which also produces an unrolled network architecture that can carry out automatic differentiation for backward propagation.
Revised: In this paper, we adopt the fast iterative shrinkage thresholding algorithm (FISTA) [47] for the forward propagation, which also produces an unrolled network architecture that can carry out automatic differentiation for backward propagation.

Original: We defer a discussion of the implementation details of the CSC-layer to the Appendix.
Revised: We defer a discussion of the implementation details of the CSC-layer to the Appendix.

Original: 3.2 Sparse Dictionary Learning Network Architecture and Training
Revised: 3.2 Sparse Dictionary Learning Network Architecture and Training

Original: Convolution layers are basic ingredients of ConvNets that appear in many common network architectures such as LeNet [48] and ResNet [49].
Revised: Convolution layers are basic ingredients of ConvNets that appear in many common network architectures such as LeNet [48] and ResNet [49].

Original: In this paper, we incorporate sparse modeling into a given existing/baseline network architecture by replacing certain / all convolution layers with the
Revised: In this paper, we incorporate sparse modeling into a given existing/baseline network architecture by replacing certain / all convolution layers with the

Original: CSC-layer.
Revised: CSC-layer.

Original: Meanwhile, all other layers such as normalization, nonlinear, and fully connected layers
Revised: Meanwhile, all other layers such as normalization, nonlinear, and fully connected layers

Original: [Table] [Table]
Revised: [Table] [Table]

Original: are retained.
Revised: are retained.

Original: This simple design choice allows the CSC-layers to be broadly applicable.
Revised: This simple design choice allows the CSC-layers to be broadly applicable.

Original: We will refer to our network with CSC-layers as Sparse Dictionary learning Network (SDNet).
Revised: We will refer to our network with CSC-layers as Sparse Dictionary learning Network (SDNet).

Original: Give a set of training data { x i , y i } Ni =1 where x i denotes an image and y i is the corresponding label, we train a network with CSC-layers by solving the following optimization problem:
Revised: Give a set of training data { x i , y i } Ni =1 where x i denotes an image and y i is the corresponding label, we train a network with CSC-layers by solving the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where (cid:96) CE denotes the cross-entropy loss.
Revised: where (cid:96) CE denotes the cross-entropy loss.

Original: In above, we use f ( · ; θ ) to denote the mapping that is performed by the neural network, where θ is a set of learnable parameters containing a subset of kernel parameters { A s } s ∈S associated with the set S of CSC-layers.
Revised: In above, we use f ( · ; θ ) to denote the mapping that is performed by the neural network, where θ is a set of learnable parameters containing a subset of kernel parameters { A s } s ∈S associated with the set S of CSC-layers.

Original: Following the convention in the sparse dictionary learning literature, we add the constraint A s ∈ N for the dictionaries in CSC-layers, where N denotes the set of normalized dictionaries:
Revised: Following the convention in the sparse dictionary learning literature, we add the constraint A s ∈ N for the dictionaries in CSC-layers, where N denotes the set of normalized dictionaries:

Original: [Equation]
Revised: [Equation]

Original: To handle such a constraint, we use the projected stochastic gradient descent (SGD) for solving the problem in (5).
Revised: To handle such a constraint, we use the projected stochastic gradient descent (SGD) for solving the problem in (5).

Original: That is, after each gradient update step for the parameters θ and { A s } s ∈S as in a regular SGD, an extra step is taken to project each A s onto the constraint set N .
Revised: That is, after each gradient update step for the parameters θ and { A s } s ∈S as in a regular SGD, an extra step is taken to project each A s onto the constraint set N .

Original: 3.3 Robust Inference
Revised: 3.3 Robust Inference

Original: The fundamental difference of the CSC-layer vis-a-vis a classical explicit layer (e.g., a convolutional layer) is that the CSC-layer imposes an assumption on the input feature map.
Revised: The fundamental difference of the CSC-layer vis-a-vis a classical explicit layer (e.g., a convolutional layer) is that the CSC-layer imposes an assumption on the input feature map.

Original: That is, it assumes that the input feature map (or image) can be approximated by a superposition of a few atoms of a dictionary, which is the layer parameter that is learnable and is shared across all data.
Revised: That is, it assumes that the input feature map (or image) can be approximated by a superposition of a few atoms of a dictionary, which is the layer parameter that is learnable and is shared across all data.

Original: In this section, we show that CSC-layers enables us to design a robust inference strategy to obtain robustness to corruptions in ways that cannot be achieved by classical explicit layers.
Revised: In this section, we show that CSC-layers enables us to design a robust inference strategy to obtain robustness to corruptions in ways that cannot be achieved by classical explicit layers.

Original: We leverage an attractive property of the CSC model is that it admits a stable recovery of the sparse signals with respect to input noise.
Revised: We leverage an attractive property of the CSC model is that it admits a stable recovery of the sparse signals with respect to input noise.

Original: In particular, Theorem 1 shows that any bounded perturbation to the input of a CSC-layer produces a bounded perturbation to its output and does not change the support of the output, if one uses a λ in (4) that is proportional to the norm of the perturbation in the layer input, provided that certain technical conditions are satisﬁed.
Revised: In particular, Theorem 1 shows that any bounded perturbation to the input of a CSC-layer produces a bounded perturbation to its output and does not change the support of the output, if one uses a λ in (4) that is proportional to the norm of the perturbation in the layer input, provided that certain technical conditions are satisﬁed.

Original: Theorem 1.
Revised: Theorem 1.

Original: (Informal version of [50, Theorem 19]) Suppose x (cid:92) has a representation A ( z (cid:92) ) as in (3) , and that it is contaminated by noise e to create the input x = x (cid:92) + e .
Revised: (Informal version of [50, Theorem 19]) Suppose x (cid:92) has a representation A ( z (cid:92) ) as in (3) , and that it is contaminated by noise e to create the input x = x (cid:92) + e .

Original: Then as long as z (cid:92) is sufﬁciently sparse, the solution z ∗ to (4) with λ =
Revised: Then as long as z (cid:92) is sufﬁciently sparse, the solution z ∗ to (4) with λ =

Original: O ( (cid:107) e (cid:107) 2 ) satisﬁes ( i ) the support of z ∗ is contained in that of z (cid:92) and ( ii ) (cid:107) z ∗ − z (cid:92) (cid:107) 

Revised: O ( (cid:107) e (cid:107) 2 ) satisﬁes ( i ) the support of z ∗ is contained in that of z (cid:92) and ( ii ) (cid:107) z ∗ − z (cid:92) (cid:107)

Original: = O ( (cid:107) e (cid:107) 2 ) .
Revised: = O ( (cid:107) e (cid:107) 2 ) .

Original: Intuitively, the parameter λ in (4) controls a balance between the sparsity regularization (cid:107) z (cid:107) 1 and the residual x − A ( z ) , the latter of which accounts for modeling discrepancies and increases when x is noisy.
Revised: Intuitively, the parameter λ in (4) controls a balance between the sparsity regularization (cid:107) z (cid:107) 1 and the residual x − A ( z ) , the latter of which accounts for modeling discrepancies and increases when x is noisy.

Original: Therefore, using a larger value of λ helps the model to handle a larger residual.
Revised: Therefore, using a larger value of λ helps the model to handle a larger residual.

Original: We thus present a very simple approach to obtain model robustness.
Revised: We thus present a very simple approach to obtain model robustness.

Original: We consider the setting where a model is trained on an uncorrupted dataset but is tested on data that is corrupted by random noise.
Revised: We consider the setting where a model is trained on an uncorrupted dataset but is tested on data that is corrupted by random noise.

Original: Corruptions in the input image may propagate into deeper feature maps (hence corrupting the input of all CSC-layers) during forward propagation.
Revised: Corruptions in the input image may propagate into deeper feature maps (hence corrupting the input of all CSC-layers) during forward propagation.

Original: Therefore, instead of directly using the CSC-layers that are obtained from the training phase, our method is to adjust the trade-off parameter λ in the
Revised: Therefore, instead of directly using the CSC-layers that are obtained from the training phase, our method is to adjust the trade-off parameter λ in the

Original: CSC-layers for the test data.
Revised: CSC-layers for the test data.

Original: As we show in the experiments (see Sec. 4.2), an optimal value of λ indeed increases with the variance of the noise in the input image, which is well-aligned with the result in Theorem 1.
Revised: As we show in the experiments (see Sec. 4.2), an optimal value of λ indeed increases with the variance of the noise in the input image, which is well-aligned with the result in Theorem 1.

Original: Choosing the optimal λ .
Revised: Choosing the optimal λ .

Original: In practical applications the amount of noise in a given test dataset is often unknown.
Revised: In practical applications, the amount of noise in a given test dataset is often unknown.

Original: Hence, the optimal choice of parameter λ becomes a nontrivial task.
Revised: Hence, the optimal choice of parameter λ becomes a nontrivial task.

Original: Here we present a practical technique for determining a proper choice of value λ , based on the simple observation that the amount of noise in the test data correlates with the magnitude of the residual x − A ( z ) .
Revised: Here we present a practical technique for determining a proper choice of value λ , based on the simple observation that the amount of noise in the test data correlates with the magnitude of the residual x − A ( z ) .

Original: That is, for test data corrupted by a larger amount of noise, we expect the magnitude of the residual in
Revised: That is, for test data corrupted by a larger amount of noise, we expect the magnitude of the residual in

Original: CSC-layers to become larger with such data fed into the network.
Revised: CSC-layers to become larger with such data fed into the network.

Original: Since the residual for any test data can always be computed, the key question here is how we can ﬁnd a relationship between an optimal value
Revised: Since the residual for any test data can always be computed, the key question here is how we can ﬁnd a relationship between an optimal value

Original: λ and the magnitude of the residual.
Revised: λ and the magnitude of the residual.

Original: Our technique for addressing this challenge is to learn such a relationship on the training set, by injecting synthetic data corruptions.
Revised: Our technique for addressing this challenge is to learn such a relationship on the training set, by injecting synthetic data corruptions.

Original: For simplicity we summarize our technique in Algorithm 1, and explain it in details below.
Revised: For simplicity we summarize our technique in Algorithm 1, and explain it in details below.

Original: We assume that although the amount of noise in a test dataset T test is unknown, the type of the noise (e.g., Gaussian noise, shot noise, etc.) is known.
Revised: We assume that although the amount of noise in a test dataset T test is unknown, the type of the noise (e.g., Gaussian noise, shot noise, etc.) is known.

Original: Given the noise type, we ﬁrst determine a set C which contains a set of values specifying the potential amount of noise in test data.
Revised: Given the noise type, we ﬁrst determine a set C which contains a set of values specifying the potential amount of noise in test data.

Original: For example, if we consider Gaussian noise, then C contains a set of values specifying the variance of the noise.
Revised: For example, if we consider Gaussian noise, then C contains a set of values specifying the variance of the noise.

Original: We also specify a set Λ of potential values for λ to be used for inference.
Revised: We also specify a set Λ of potential values for λ to be used for inference.

Original: After training the network as speciﬁed in Step 2, we use a procedure described in between Step 4 and
Revised: After training the network as speciﬁed in Step 2, we use a procedure described in between Step 4 and

Original: Step 12 to ﬁt a function that maps a residual value r to an optimal choice of λ .
Revised: Step 12 to ﬁt a function that maps a residual value r to an optimal choice of λ .

Original: The idea for ﬁtting such a relation is to generate synthetic noise of varying magnitudes in C on the training set, and for each noise magnitude we sweep the parameter λ ∈ Λ to ﬁnd a λ that produces the best accuracy on training set.
Revised: The idea for ﬁtting such a relation is to generate synthetic noise of varying magnitudes in C on the training set, and for each noise magnitude we sweep the parameter λ ∈ Λ to ﬁnd a λ that produces the best accuracy on training set.

Original: Once such a relationship is learned, we feed in the residual value computed on the test dataset T test to predict an optimal λ to be used for inference on test data, as described in Step 14.
Revised: Once such a relationship is learned, we feed in the residual value computed on the test dataset T test to predict an optimal λ to be used for inference on test data, as described in Step 14.

Original:  Experiments
Revised:  Experiments

Original: In this section, we provide experimental evidence for neural networks with CSC-layers as discussed in Sec. 3.
Revised: In this section, we provide experimental evidence for neural networks with CSC-layers as discussed in Sec. 3.

Original: Through experiments on CIFAR-10, CIFAR-100 3 , and ImageNet 4 , Sec. 4.1 shows that our networks have equally competitive classiﬁcation performance as mainstream architectures such as
Revised: Through experiments on CIFAR-10, CIFAR-100 3 , and ImageNet 4 , Sec. 4.1 shows that our networks have equally competitive classiﬁcation performance as mainstream architectures such as

Original: ResNet.
Revised: ResNet.

Original: Furthermore, we show in Sec.
Revised: Furthermore, we show in Sec.

Original: 4.2 that our network is able to handle input perturbations with the robust inference technique.
Revised: 4.2 that our network is able to handle input perturbations with the robust inference technique.

Original: Finally, we demonstrate in Sec.
Revised: Finally, we demonstrate in Sec.

Original: 4.3 that our network is able to handle adversarial perturbations as well.
Revised: 4.3 that our network is able to handle adversarial perturbations as well.

Original: More details about implementation are given in the Appendix.
Revised: More details about implementation are given in the Appendix.

Original: Datasets.
Revised: Datasets.

Original: We test the performance of our method using the CIFAR-10 and CIFAR-100 [51] datasets.
Revised: We test the performance of our method using the CIFAR-10 and CIFAR-100 [51] datasets.

Original: Each dataset contains 50,000 training images and 10,000 testing images where each image is of size 32 × 32 with RGB channels.
Revised: Each dataset contains 50,000 training images and 10,000 testing images where each image is of size 32 × 32 with RGB channels.

Original: We also use the ImageNet dataset [52] that contains 1,000 classes and a total number of around 1 million images.
Revised: We also use the ImageNet dataset [52] that contains 1,000 classes and a total number of around 1 million images.

Original: Network architecture.
Revised: Network architecture.

Original: We use the network architectures with the ﬁrst convolutional layers of
Revised: We use the network architectures with the ﬁrst convolutional layers of

Original: ResNet-18 and ResNet-34 [21] 5 replaced by CSC-layers, and refer to these networks as SDNetand SDNet-34, respectively.
Revised: ResNet-18 and ResNet-34 [21] 5 replaced by CSC-layers, and refer to these networks as SDNetand SDNet-34, respectively.

Original: We use λ = 0 .
Revised: We use λ = 0 .

Original: 1 as the trade-off parameter in (4) for all CSC-layers
Revised: 1 as the trade-off parameter in (4) for all CSC-layers

Original: [Table]
Revised: [Table]

Original: unless speciﬁed otherwise.
Revised: unless speciﬁed otherwise.

Original: Forward propagation through each CSC-layer is performed via unrolling two iterations of FISTA.
Revised: Forward propagation through each CSC-layer is performed via unrolling two iterations of FISTA.

Original: Network training.
Revised: Network training.

Original: For CIFAR-10 and CIFAR-100, we use a cosine learning rate decay schedule with an initial learning rate of 0.1, and train the model for 220 epochs.
Revised: For CIFAR-10 and CIFAR-100, we use a cosine learning rate decay schedule with an initial learning rate of 0.1, and train the model for 220 epochs.

Original: We use the SGD optimizer with 0.9 momentum and Nestrov.
Revised: We use the SGD optimizer with 0.9 momentum and Nestrov.

Original: The weight decay is set to 5 × 10 − 4 , and batch size is set to 128.
Revised: The weight decay is set to 5 × 10 − 4 , and batch size is set to 128.

Original: All the experiments are conducted on a single NVIDIA GTX 2080Ti GPU.
Revised: All the experiments are conducted on a single NVIDIA GTX 2080Ti GPU.

Original: For ImageNet, we use multi-step learning rate decay schedule with an initial learning rate of 0.1 decayed by a factor of 0 .at
Revised: For ImageNet, we use multi-step learning rate decay schedule with an initial learning rate of 0.1 decayed by a factor of 0 .at

Original: the 30th, 60th, and 90th epochs, and train the model for 100 epochs.
Revised: the 30th, 60th, and 90th epochs, and train the model for 100 epochs.

Original: The batch size is 512, and the optimizer is SGD with 0.9 momentum and Nestrov.
Revised: The batch size is 512, and the optimizer is SGD with 0.9 momentum and Nestrov.

Original: All experiments on ImageNet are conducted on
Revised: All experiments on ImageNet are conducted on

Original: NVIDIA RTX 3090 GPUs.
Revised: NVIDIA RTX 3090 GPUs.

Original: 4.1 Performance for Image Classiﬁcation
Revised: 4.1 Performance for Image Classiﬁcation

Original: We compare our method with standard network architectures ResNet-18 and ResNet-34 [21].
Revised: We compare our method with standard network architectures ResNet-18 and ResNet-34 [21].

Original: In addition, we comparewith architectures with implicit layers (i.e., MDEQ [27]) and architectures with sparse modeling (i.e., SCN [15]).
Revised: In addition, we compare architectures with implicit layers (i.e., MDEQ [27]) and architectures with sparse modeling (i.e., SCN [15]).

Original: For ResNet-18 and ResNet-34, we train the model using the same setup as our SDNet models.
Revised: For ResNet-18 and ResNet-34, we train the model using the same setup as our SDNet models.

Original: For MDEG and SCN, we train the models using the settings as stated in their respective papers.
Revised: For MDEG and SCN, we train the models using the settings as stated in their respective papers.

Original: SCN has both a different sparse coding layer and a different network architecture compared to our SDNet.
Revised: SCN has both a different sparse coding layer and a different network architecture compared to our SDNet.

Original: Hence, we also include a baseline referred to as SCN-18, which is constructed by replacing the ﬁrst convolutional layer of ResNet-18 with the sparse coding layer from SCN (hence has the same architecture as SDNet-18 but a different sparse coding layer).
Revised: Hence, we also include a baseline referred to as SCN-18, which is constructed by replacing the ﬁrst convolutional layer of ResNet-18 with the sparse coding layer from SCN (hence has the same architecture as SDNet-18 but a different sparse coding layer).

Original: The results are reported in Table 1.
Revised: The results are reported in Table 1.

Original: We see that witha similar model size, SDNet-18/34 produces a
Revised: We see that with similar model size, SDNet-18/34 produces a

Original: Top-1 accuracy that closely matches (for CIFAR-10 and ImageNet) or surpasses (for CIFAR-100) that of ResNet-18/34 while having a comparable Speed.
Revised: Top-1 accuracy that closely matches (for CIFAR-10 and ImageNet) or surpasses (for CIFAR-100) that of ResNet-18/34 while having a comparable speed.

Original: This shows the potential of our network with modeling based layers as a powerful alternative to existing data-driven models, since our model has the additional beneﬁt for handling corruptions.
Revised: This shows the potential of our network with modeling based layers as a powerful alternative to existing data-driven models, since our model has the additional beneﬁt of handling corruptions.

Original: [Figure]
Revised: [Figure]

Original: We also compare our SDNet-18 model with the MDEQ model, which has a similar model size, and see that SDNet-18 is not only more accurate than MDEQ but is much ( > 7 times) faster.
Revised: We also compare our SDNet-18 model with the MDEQ model, which has a similar model size, and see that SDNet-18 is not only more accurate than MDEQ but is much ( > 7 times) faster.

Original: Note that
Revised: Note that

Original: MDEQ cannot handle corrupted data as in our method as well.
Revised: MDEQ cannot handle corrupted data as in our method as well.

Original: The SCN network, which also uses sparse modeling, obtains an Top-1 accuracy that is highly competitive to all methods.
Revised: The SCN network, which also uses sparse modeling, obtains a Top-1 accuracy that is highly competitive to all methods.

Original: However, a signiﬁcant drawback of SCN is that it is very slow to train.
Revised: However, a signiﬁcant drawback of SCN is that it is very slow to train.

Original: This is true even with SCN-18, where only one convolutional layer is replaced by the sparse coding layer.
Revised: This is true even with SCN-18, where only one convolutional layer is replaced by the sparse coding layer.

Original: The reason may be that SCN uses a patch-based sparse coding model for images, in contrast to a convolutional sparse coding model as in our method, which requires solving many sparse coding problems in each forward propagation that cannot beneﬁt from parallel computing.
Revised: The reason may be that SCN uses a patch-based sparse coding model for images, in contrast to a convolutional sparse coding model as in our method, which requires solving many sparse coding problems in each forward propagation that cannot beneﬁt from parallel computing.

Original: 4.2 Handling Input Perturbations
Revised: 4.2 Handling Input Perturbations

Original: To test the robustness of our method to input perturbations, we use the CIFAR-10-C dataset [53] which contains a test set for CIFAR-10 that is corrupted with different types of synthetic noise and 5 severity levels for each type.
Revised: To test the robustness of our method to input perturbations, we use the CIFAR-10-C dataset [53] which contains a test set for CIFAR-10 that is corrupted with different types of synthetic noise and 5 severity levels for each type.

Original: Because the CSC model in (4) penalizes the entry-wise difference between input and reconstructed signals, it is more suited for handling additive noises.
Revised: Because the CSC model in (4) penalizes the entry-wise difference between input and reconstructed signals, it is more suited for handling additive noises.

Original: Hence, we focus on four types of additive noises in CIFAR-10-C, namely, Gaussian noise, shot noise, speckle noise, and impulse noise.
Revised: Hence, we focus on four types of additive noises in CIFAR-10-C, namely, Gaussian noise, shot noise, speckle noise, and impulse noise.

Original: We evaluate the accuracy of our SDNet-18 and compare its performance with ResNet-18.
Revised: We evaluate the accuracy of our SDNet-18 and compare its performance with ResNet-18.

Original: Robustness as a function of λ .
Revised: Robustness as a function of λ .

Original: As discussed in Sec. 3.3, we may improve the performance of our model to noisy test data by using values of λ that are different from the training phase.
Revised: As discussed in Sec. 3.3, we may improve the performance of our model to noisy test data by using values of λ that are different from the training phase.

Original: Therefore, we report the performance of our method with varying λ in the range of [0 . 1 , 1 . 5] in Figure 2 (recall that λ = 0 .
Revised: Therefore, we report the performance of our method with varying λ in the range of [0 . 1 , 1 . 5] in Figure 2 (recall that λ = 0 .

Original: 1 is used for training).
Revised: 1 is used for training).

Original: It can be seen that for all types of noises and all severity levels (except for impulse noise with level 0, 1, and 2), properly choosing a value of λ that is different from that used during training helps to improve the test performance.
Revised: It can be seen that for all types of noises and all severity levels (except for impulse noise with levels 0, 1, and 2), properly choosing a value of λ that is different from that used during training helps to improve the test performance.

Original: In particular, the accuracy curves as a function of λ exhibit a unimodal shape where the performance ﬁrst increases  then decreases.
Revised: In particular, the accuracy curves as a function of λ exhibit a unimodal shape where the performance ﬁrst increases and then decreases.

Original: Moreover, within each corruption type the values of λ where a peak performance is achieved increases monotonically with the severity level of the corruption.
Revised: Moreover, within each corruption type the values of λ where a peak performance is achieved increase monotonically with the severity level of the corruption.

Original: Such an observation is well-aligned with our discussion in Sec. 3.3.
Revised: Such an observation is well-aligned with our discussion in Sec. 3.3.

Original: Choosing an optimal λ .
Revised: Choosing an optimal λ .

Original: While Figure 2 demonstrates that one may improve the performance on corrupted data via a proper choice of λ , it does not show how to choose the best λ in practice.
Revised: While Figure 2 demonstrates that one may improve the performance on corrupted data via a proper choice of λ , it does not show how to choose the best λ in practice.

Original: Here we show that the technique presented in Algorithm 1 can be used to select λ for robust inference.
Revised: Here we show that the technique presented in Algorithm 1 can be used to select λ for robust inference.

Original: Speciﬁcally, we apply Algorithm 1 with f ( · ; θ , λ 0 ) being SDNet-18 with λ 0 = 0 .
Revised: Speciﬁcally, we apply Algorithm 1 with f ( · ; θ , λ 0 ) being SDNet-18 with λ 0 = 0 .

Original: 1 , T train being the
Revised: 1 , T train being the

Original: CIFAR-10 training set, T test being CIFAR-10-C data with a particular type of corruption under a particular severity level, and report the performance of the algorithm output in Table 2 and Table 3.
Revised: CIFAR-10 training set, T test being CIFAR-10-C data with a particular type of corruption under a particular severity level, and report the performance of the algorithm output in Table 2 and Table 3.

Original: Table 2 shows the results with varying severity levels of Gaussian noise.
Revised: Table 2 shows the results with varying severity levels of Gaussian noise.

Original: We can see that if we use the same λ = 0 .
Revised: We can see that if we use the same λ = 0 .

Original: 1 as in training (i.e., SDNet-18 w/ λ
Revised: 1 as in training (i.e., SDNet-18 w/ λ

Original: = 0 . 1 ), then the performance of SDNetis already signiﬁcantly better ResNet-18.
Revised: = 0 . 1 ), then the performance of SDNetis already signiﬁcantly better ResNet-18.

Original: This may be attributed to the fact that the CSC-layer is intrinsically more robust to input perturbations than convolution layers owning to its sparse modeling.
Revised: This may be attributed to the fact that the CSC-layer is intrinsically more robust to input perturbations than convolution layers owning to its sparse modeling.

Original: However, if we use an adaptive λ computed from Algorithm 1 (i.e., SDNet-18 w/ adaptive λ ), then the performance is further signiﬁcantly improved at all noise levels.
Revised: However, if we use an adaptive λ computed from Algorithm 1 (i.e., SDNet-18 w/ adaptive λ ), then the performance is further signiﬁcantly improved at all noise levels.

Original: To demonstrate that our method works beyond Gaussian noise, we report results for each of the four types of additive noises averaged over all severity levels in Table 3 for CIFAR-10-C as well as ImageNet-C.
Revised: To demonstrate that our method works beyond Gaussian noise, we report results for each of the four types of additive noises averaged over all severity levels in Table 3 for CIFAR-10-C as well as ImageNet-C.

Original: The results demonstrate
Revised: The results demonstrate

Original: [Table] [Table]
Revised: [Table] [Table]

Original: that sparse modeling enables us to effectively handle various types of additive noises in test data very easily with the procedure in Algorithm 1.
Revised: that sparse modeling enables us to effectively handle various types of additive noises in test data very easily with the procedure in Algorithm 1.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Finally, in Figure 3 we plot the relationship between the optimal choice of λ (in y-axis) and the magnitude of residual in CSC-layers (in x-axis) learned from training data according to Algorithm 1.
Revised: Finally, in Figure 3 we plot the relationship between the optimal choice of λ (in y-axis) and the magnitude of residual in CSC-layers (in x-axis) learned from training data according to Algorithm 1.

Original: We also plot the linear ﬁtting of such a relation, which can be seen to provide a good quality approximation for each of the four corruption types.
Revised: We also plot the linear ﬁtting of such a relation, which can be seen to provide a good quality approximation for each of the four corruption types.

Original: On the other hand, we note that the ﬁtted linear relations differ across different types of corruptions.
Revised: On the other hand, we note that the ﬁtted linear relations differ across different types of corruption.

Original: Hence, it is important that the relationship is estimated for each corruption types separately.
Revised: Hence, it is important that the relationship is estimated for each corruption type separately.

Original: 4.3 Handling Adversarial Perturbations
Revised: 4.3 Handling Adversarial Perturbations

Original: We show that our method also exhibits robustness to adversarial perturbations.
Revised: We show that our method also exhibits robustness to adversarial perturbations.

Original: In this experiment, we generate adversarial perturbations on the CIFAR-10 test dataset using PGD attack on our SDNet (with λ = 0 . 1 ), with L ∞ norm of the perturbation being (cid:15) = 8 / 255 and L 2 norm of the perturbation being (cid:15)
Revised: In this experiment, we generate adversarial perturbations on the CIFAR-10 test dataset using PGD attack on our SDNet (with λ = 0 . 1 ), with L ∞ norm of the perturbation being (cid:15) = 8 / 255 and L 2 norm of the perturbation being (cid:15)

Original: = 0 .
Revised: = 0 .

Original: 5 , respectively.
Revised: 5 , respectively.

Original: The robust accuracy of our method is reported in Table 4 and is compared with that of ResNet-18.
Revised: The robust accuracy of our method is reported in Table 4 and is compared with that of ResNet-18.

Original: We can see that while SDNet does not perform much better than ResNet with λ = 0 .
Revised: We can see that while SDNet does not perform much better than ResNet with λ = 0 .

Original: 1 , we may tune the parameter λ to drastically improve the robust accuracy.
Revised: 1 , we may tune the parameter λ to drastically improve the robust accuracy.

Original: 4.4 Analysis
Revised: 4.4 Analysis

Original: Effect of number of iterations.
Revised: Effect of number of iterations.

Original: Recall from Sec.
Revised: Recall from Sec.

Original: 3.1 that forward propagation through a CSC-layer is performed by running a few iterative steps of the FISTA algorithm.
Revised: 3.1 that forward propagation through a CSC-layer is performed by running a few iterative steps of the FISTA algorithm.

Original: Here we provide a study on how the number of iterations affects model performance on ImageNet and ImageNet-C using
Revised: Here we provide a study on how the number of iterations affects model performance on ImageNet and ImageNet-C using

Original: SDNet-18.
Revised: SDNet-18.

Original: The results are shown in Table 5.
Revised: The results are shown in Table 5.

Original: With  increasing number of FISTA iterations, the model performance on both natural accuracy and robust accuracy improves.
Revised: With the increasing number of FISTA iterations, the model performance on both natural accuracy and robust accuracy improves.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Replacing all convolution layer by the CSC-layer.
Revised: Replacing all convolution layer by the CSC-layer.

Original: We also train a version of SDNet obtained from replacing all convolution layers of a ResNet with the CSC-layer (as opposed to only the ﬁrst convolution layer)
Revised: We also train a version of SDNet obtained from replacing all convolution layers of a ResNet with the CSC-layer (as opposed to only the ﬁrst convolution layer)

Original: [Table]
Revised: [Table]

Original: and refer to such a model as SDNet-18-All and SDNet-34-All.
Revised: and refer to such a model as SDNet-18-All and SDNet-34-All.

Original: On ImageNet we observe that SDNet-18-All and SDNet-34-All obtain 69.37% and 72.54% Top-1 accuracy, respectively.
Revised: On ImageNet we observe that SDNet-18-All and SDNet-34-All obtain 69.37% and 72.54% Top-1 accuracy, respectively.

Original: Comparing such results with those of SDNet-18/34 reported in Table 1, we see that the performance is not signiﬁcantly affected by replacing more convolution layers with CSC-layers (see Tablein Appendix for more results).
Revised: Comparing such results with those of SDNet-18/34 reported in Table [Table] we see that the performance is not signiﬁcantly affected by replacing more convolution layers with CSC-layers (see Table D. in Appendix for more results).

Original: Moreover, SDNet-18/34-All enables us to develop a visualization technique as described in the Appendix B.
Revised: Moreover, SDNet-18/34-All enables us to develop a visualization technique as described in the Appendix B.

Original:  Conclusion and Discussion
Revised:  Conclusion and Discussion

Original: This paper revisits the classical sparse modeling and provides a simple way of using it to guide the design of interpretable deep networks.
Revised: This paper revisits the classical sparse modeling and provides a simple way of using it to guide the design of interpretable deep networks.

Original: Despite multiple prior attempts, our work is the ﬁrst to demonstrate that such a design can produce performance (in terms of accuracy, model size, and memory) that is on par with standard ConvNets on modern image datasets such as ImageNet.
Revised: Despite multiple prior attempts, our work is the ﬁrst to demonstrate that such a design can produce performance (in terms of accuracy, model size, and memory) that is on par with standard ConvNets on modern image datasets such as ImageNet.

Original: The success in combining sparse modeling with deep learning provides a means of borrowing and utilizing the rich results in the well-developed ﬁeld of sparse modeling [54, 55] for network design and analysis.
Revised: The success in combining sparse modeling with deep learning provides a means of borrowing and utilizing the rich results in the well-developed ﬁeld of sparse modeling [54, 55] for network design and analysis.

Original: While it is not the purpose of this work to fully explore all potentials, our experiments already demonstrate clear advantages of so designed networks in handling various forms of data corruptions.
Revised: While it is not the purpose of this work to fully explore all potentials, our experiments already demonstrate clear advantages of so-designed networks in handling various forms of data corruption.

Original: Looking forward, other fundamental principles, algorithm and techniques in sparse modeling may be introduced to further enhance the capability of our presented framework.
Revised: Looking forward, other fundamental principles, algorithms, and techniques in sparse modeling may be introduced to further enhance the capability of our presented framework.

Original: Along this line, we provide some preliminary evidence on how sparse modeling enables interpretability in the Appendix, and leave further study to future work.
Revised: Along this line, we provide some preliminary evidence on how sparse modeling enables interpretability in the Appendix, and leave further study to future work.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: Zhihui Zhu acknowledges support from NSF grants CCF-2008460.
Revised: Zhihui Zhu acknowledges support from NSF grants CCF-2240708.

Original: Shao-Lun Huang acknowledges support from Shenzhen Science and Technology Program under Grant KQTD20170810150821146,
Revised: Shao-Lun Huang acknowledges support from Shenzhen Science and Technology Program under Grant KQTD20170810150821146,

Original: National Key R&D Program of China under Grant 2021YFA0715202 and High-end Foreign Expert
Revised: National Key R&D Program of China under Grant 2021YFA0715202 and High-end Foreign Expert

Original: Talent Introduction Plan under Grant G2021032013L. Yi Ma acknowledges support from ONR grants N00014-20-1-2002 and N00014-22-1-2102, the joint Simons Foundation-NSF DMS grant #2031899, as well as partial support from Berkeley FHL Vive Center for Enhanced Reality and
Revised: Talent Introduction Plan under Grant G2021032013L. Yi Ma acknowledges support from ONR grants N00014-20-1-2002 and N00014-22-1-2102, the joint Simons Foundation-NSF DMS grant #2031899, as well as partial support from Berkeley FHL Vive Center for Enhanced Reality and

Original: Berkeley Center for Augmented Cognition, Tsinghua-Berkeley Shenzhen Institute (
Revised: Berkeley Center for Augmented Cognition, Tsinghua-Berkeley Shenzhen Institute (

Original: TBSI) Research
Revised: TBSI) Research

Original: Fund, and Berkeley AI Research (BAIR).Checklist
Revised: Fund, and Berkeley AI Research (BAIR).

Original: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] or [N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:

Original: • Did you include the license to the code and datasets?

Original: [Yes] • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?

Original: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the

Original: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] Limitations are described in the Conclusion as well as in a footnote.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Discussed in Conclusion.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] No theoretical results.

Original: (b) Did you include complete proofs of all theoretical results?

Original: [N/A] No theoretical results.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Please refer to section 4 and appendix.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Please refer to Section 5 for training details (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Pleas refer to Section 5 for details related to computation and resources

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] All datasets we used: CIFAR10, CIFAR-100, and ImageNet are publicly available, and we included their urls in footnotes.

Original: We also included the license and copyright information of the baseline ResNet models in a footnote.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] All datasets are publicly available.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] All datasets are publicly available.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]

Original: Appendices

Original: We provide the implementation details for the CSC-layers in Appendix A, followed by a discussion on how sparse modeling enables us to easily visualize the feature maps in the intermediate layers of a convolutional neural network in Appendix B.

Original: A Implementation Details for CSC-Layers

Original: Forward propagation.

Original: Forward propagation of the sparse coding layer is carried out by solving the optimization problem in (4).

Original: In this paper, we adopt the fast iterative shrinkage thresholding algorithm (FISTA) [47].

Original: Starting with an arbitrarily initialized z (we used z [0] = 0 ), FISTA operates by introducing an auxiliary variable y initialized as y [1] =

Original: z [0] , as well as a scalar m initialized as m 1 = 1 , and carrying out iteratively the following steps for (cid:96) ≥ 1 :

Original: [Equation]

Original: In above, A ∗ ( · ) is the adjoint operator of A ( · ) and it is deﬁned as

Original: [Equation]

Original: for any x = ( ξ 1 , . . .

Original: , ξ M ) ∈ R M × H × W , where each ξ c ∈ R H × W .

Original: The FISTA iteration in (6) automatically gives rise to a nonlinear operator T λt , which denotes a shrinkage thresholding operator that is applied entry-wise to the input variable.

Original: In particular, shrinkage thresholding for a scalar η ∈ R is deﬁned as

Original: [Equation]

Original: Finally, the parameter t is the step size for FISTA.

Original: The iterations (6) converge to a solution to (4) as long as the step size t is smaller than the inverse of the dominant eigenvalue of the operator A (cid:62)

Original: ( A ( · )) .

Original: In our implementation, we estimate this dominant eigenvalue by the power iteration, which iteratively carries out the following calculation to estimate the dominant eigenvector:

Original: [Equation]

Original: The iteration is terminated when the the update v [ k +1] − v [ k ] has an entry-wise (cid:96) 2 -norm smaller than a predeﬁned threshold (we use √ 10 − 5 ) or when the maximum number of iterations (we use 50 ) is reached.

Original: Assuming that the iteration carries out for K times, the estimated dominant eigenvalue is given by

Original: [Equation]

Original: [-We may then set t to be a value that is smaller than 1 . 
-]
Revised: [-We may then set t to be a value that is smaller than 1 . 
-]

Original: /λ K to guarantee the convergence of ISTA.

Original: In our experiment we set t = 0 .

Original: 9 /λ K .

Original: Note that the dominant eigenvalue for A (cid:62) ( A ( · )) changes after each update on the layer parameters A .

Original: Hence, the calculation of λ K is performed after each parameter update during training.

Original: Once training is completed, λ K can be ﬁxed during testing and therefore does not incur additional inference time.

Original: Backward propagation.

Original: A beneﬁt of adopting the FISTA iteration for forward propagation in the sparse coding layer is that it naturally leads to an optimization-driven network [36], a network architecture that is constructed from an unrolled optimization algorithm, for which backward propagation can be carried out by auto-differentiation.

Original: We note that there are many other algorithms for solving the sparse coding and convolutional sparse coding [56] problems, which may lead to more efﬁcient implementation of the forward propagation.

Original: Moreover, there are also other algorithms for performing the backward propagation [24, 18] by leveraging the fact that the sparse coding problem (4) is convex.

Original: We choose FISTA because it has a very simple implementation for both forward and backward propagation.

Original: We leave the acceleration of our approach with other solvers as future work.

Original: Implementation.

Original: Implementation of the sparse coding layer requires carrying out the computation of the operators A ( · ) and A ∗ ( · ) .

Original: Both of these two operators can be easily implemented by many modern deep learning packages, as noted in [57].

Original: In PyTorch, for example, A ( z ) can be implemented by the convolutional function as follows:

Original: [Equation]

Original: where input is set to z (after inserting a batch dimension) and weight is set to A .

Original: Similarly, A ∗ x can be easily implemented by replacing the conv2d function in (11) with the conv_transposed2d function.

Original: B Visualization of Feature Maps with CSC-layers

Original: Visualization of the feature maps in the intermediate layers of a deep neural network is an important aspect toward establishing the interpretability of deep learning models.

Original: Here we illustrate how sparse modeling with CSC-layers enables us to very easily obtain such a visualization in ways that cannot be accomplished with standard convolutional layers.

Original: Before we introduce our visualization method, we mention that existing convolutional neural network visualization methods primarily include those based on 1) synthesizing an input image pattern that maximally activates one [58, 59] or multiple [60] CNN neurons and 2) mapping the intermediate features into visually perceptible signals by training reversed convolutional ﬁlters [61] or by using the transposed versions of the (forward-pass) convolutional ﬁlters along with approximated unpooling operations [62].

Original: These methods have the shortcomings such as requiring extra training steps [58, 59, 60, 61], visualizing only in the neuron level and lacking layer-level perspectives [58, 59, 62], and relying on numerically approximated inverse operators [62, 61].

Original: In contrast to previous methods, our method enables an analytical reconstruction of the layer input from the layer output (feature map) by simply applying the learned convolutional dictionary on the feature map without additional training, visualization level restrictions, or any approximated inverse operators.

Original: As we explain next, this is made possible by sparse modeling with CSC

Original: B.1 Method

Original: Recall that each CSC-layer solves an optimization problem (4) where the layer input is reconstructed by the layer output with layer parameter being the dictionary.

Original: Speciﬁcally, let A l be the (learned) convolutional dictionary and z l − 1 , z l be the input and output feature maps respectively, at layer (cid:96) .

Original: In

Original: CSC-layers, z l is the optimal solution to (4), replicated here for convenience:

Original: [Equation]

Original: Given a feature map z l at the output of a layer (cid:96) , one may obtain a reconstructed layer input ˜

Original: z l −from the convolution operation, i.e., ˜ z l − 1 .

Original: =

Original: A l ( z l ) .

Original: In a deep model with multiple CSC-layers, one may apply this reconstruction step recursively from the (cid:96) -th layer to the ﬁrst layer, to obtain an reconstructed input image ˜ x :

Original: [Equation]

Original: To express this more compactly, we have

Original: [Equation]

Original: The output ˜ x , which is in the input image space, can then be visualized as an image.

Original: Note that the procedure in (14) for obtaining ˜ x is analytical and thus does not require numerical approximation or additional training, as opposed to [62, 61] and [58, 59, 60, 61].

Original: B.2 Results

Original: We apply our visualization method described above to the SDNet-18 trained on ImageNet (see Sec. 4) to visualize the feature maps associated with ﬁve different images.

Original: The results are provided in

Original: Figure 5.

Original: It can be observed that the shallow layers (e.g., layer 1 - 5) capture rich details of the input image.

Original: On the other hand, feature maps at deeper layers become more blurry and only capture a rough contour of the contents in the corresponding input image.

Original: This is suggesting that the layers of

Original: SDNet-18 progressively remove some of the unrelated details from the network input.

Original: C Visualizing the Learned Dictionary of SDNet-

Original: In Figure 4, we provide a visualization of the learned dictionary in the ﬁrst layer of SDNet-18 trained on ImageNet.

Original: The dimension of the dictionary in ﬁrst layer of SDNet-18 is 64 × 3 × 7 × 7 , which is treated as 64 small patches of shape 7 × 7 with RGB channels that are arranged into a 8 × 8 grid.

Original: [Figure]

Original: D More Experimental Results D.1 Sparsity of CSC-layer Feature Map

Original: To evaluate the sparsity of our CSC-layer output feature map

Original: [Figure]

Original: we generate a histogram of the values at the output of CSC-layer in SDNet-18 for 10,000 CIFAR-10 test images and report the results in

Original: Figure 6.

Original: The result shows our CSC-layer feature maps are highly sparse.

Original: The zero value accounts for 52% of whole CSC-layer feature maps while common convolution layer outputs dense feature maps

Original: [Figure]

Original: D.2 Comparison of SDNet-18 and SDNet-18-All

Original: Table 6 shows the comparison of SDNet-18/34 and SDNet-18/34-All on CIFAR-10, CIFARand ImageNet datasets, including accuracy, model complexity and speed.

Original: SDNet-18-All means all convolution layers are replaced with CSC-layer.

Original: And the number of FISTA iteration is set to 2 for all

Original: CSC-layers.

Original: Both models have high accuracy performance while SDNet-18 is signiﬁcantly faster.

Original: [Table]