Original: A J OINT S UBSPACE V IEW TO C ONVOLUTIONAL N EURAL N ETWORKS
Revised: A J OINT S UBSPACE V IEW TO C ONVOLUTIONAL N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Motivated by the intuition that important image regions remain important across different layers and scales in a CNN, we propose in this paper a joint subspace view to convolutional ﬁlters across network layers.
Revised: Motivated by the intuition that important image regions remain important across different layers and scales in a CNN, we propose in this paper a joint subspace view to convolutional ﬁlters across network layers.

Original: When we construct for each layer a ﬁlter subspace by decomposing convolutional ﬁlters over a small set of layer-speciﬁc ﬁlter atoms, we observe a low-rank structure within subspace coefﬁcients across layers.
Revised: When we construct for each layer a ﬁlter subspace by decomposing convolutional ﬁlters over a small set of layer-speciﬁc ﬁlter atoms, we observe a low-rank structure within subspace coefﬁcients across layers.

Original: The above observation matches widely-known cross-layer ﬁlter correlation and redundancy.
Revised: The above observation matches widely-known cross-layer ﬁlter correlation and redundancy.

Original: Thus, we propose to jointly model ﬁlter subspace across different layers by enforcing cross-layer shared subspace coefﬁcients.
Revised: Thus, we propose to jointly model ﬁlter subspace across different layers by enforcing cross-layer shared subspace coefﬁcients.

Original: In other words, a CNN is now reduced to layers of ﬁlter atoms, typically a few hundred of parameters per layer, with a common block of subspace coefﬁcients shared across layers.
Revised: In other words, a CNN is now reduced to layers of ﬁlter atoms, typically a few hundred of parameters per layer, with a common block of subspace coefﬁcients shared across layers.

Original: We further show that such subspace coefﬁcient sharing can be easily extended to other network sub-structures, from sharing across the entire network to sharing within ﬁlter groups in a layer.
Revised: We further show that such subspace coefﬁcient sharing can be easily extended to other network sub-structures, from sharing across the entire network to sharing within ﬁlter groups in a layer.

Original: While signiﬁcantly reducing the parameter redundancy of a wide range of network architectures, the proposed joint subspace view also preserves the expressiveness of CNNs, and brings many additional advantages, such as easy model adaptation and better interpretation.
Revised: While signiﬁcantly reducing the parameter redundancy of a wide range of network architectures, the proposed joint subspace view also preserves the expressiveness of CNNs, and brings many additional advantages, such as easy model adaptation and better interpretation.

Original: We support our ﬁndings with extensive empirical evidence.
Revised: We support our ﬁndings with extensive empirical evidence.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Modern developments of deep network components (Ioffe & Szegedy, 2015; Xu et al., 2015), structures (He et al., 2016; Huang et al., 2017), and training techniques (Kingma & Ba, 2014; Zoph &
Revised: Modern developments of deep network components (Ioffe & Szegedy, 2015; Xu et al., 2015), structures (He et al., 2016; Huang et al., 2017), and training techniques (Kingma & Ba, 2014; Zoph &

Original: Le, 2017) have enabled deep convolutional neural networks to consistently grow deeper and wider, with hundreds of millions of parameters.
Revised: Le, 2017) have enabled deep convolutional neural networks to consistently grow deeper and wider, with hundreds of millions of parameters.

Original: The trend of growing network scales leads to both severe challenges and urgent needs to fast adaptations (Finn et al., 2017; Lopez-Paz & Ranzato, 2017; Shin et al., 2017), parameter efﬁciency (Cheng et al., 2017; Han et al., 2015a;
Revised: The trend of growing network scales leads to both severe challenges and urgent needs to fast adaptations (Finn et al., 2017; Lopez-Paz & Ranzato, 2017; Shin et al., 2017), parameter efﬁciency (Cheng et al., 2017; Han et al., 2015a;

Original: Luo et al., 2017; Savarese & Maire, 2019;
Revised: Luo et al., 2017; Savarese & Maire, 2019;

Original: Yang et al., 2019; Wang et al., 2018b), and interpretability (Selvaraju et al., 2017;
Revised: Yang et al., 2019; Wang et al., 2018b), and interpretability (Selvaraju et al., 2017;

Original: Zhou et al., 2016).
Revised: Zhou et al., 2016).

Original: While recent works on efﬁcient convolution operations (Chollet, 2017; Howard et al., 2017; Ma et al., 2018; Zhang et al., 2018) alleviate the long recognized over-parametrization problem of deep CNNs, and practical methods on modeling the shared components across CNN ﬁlters have been proposed (Ha et al., 2016; Savarese & Maire, 2019), there is still a lack of a principled view to exploiting the common structure of the ﬁlters within a CNN.
Revised: While recent works on efﬁcient convolution operations (Chollet, 2017; Howard et al., 2017; Ma et al., 2018; Zhang et al., 2018) alleviate the long recognized over-parametrization problem of deep CNNs, and practical methods on modeling the shared components across CNN ﬁlters have been proposed (Ha et al., 2016; Savarese & Maire, 2019), there is still a lack of a principled view to exploiting the common structure of the ﬁlters within a CNN.

Original: Motivated by thewidely observed redundancies in deep network parameters (Michel et al., 2019; Raghu et al., 2017), and the fact thatunder certain linear transforms, deep features across layers are highly correlated  (Kornblith et al., 2019; Morcos et al., 2018; Raghu et al., 2017), we introduce a joint subspace view to ﬁlters across a CNN by exploring the intrinsic ﬁlter correlations.
Revised: While preliminary studies have revealed the redundancies in deep network parameters (Michel et al., 2019; Raghu et al., 2017), and the fact that deep features across layers are highly correlated under certain linear transforms (Kornblith et al., 2019; Morcos et al., 2018; Raghu et al., 2017), exploiting this potential redundancy remains challenging. We suspect that the spatial patterns of convolutional ﬁlters, which correspond to different semantics levels, are the main reason why we cannot observe obvious cross-layer ﬁlter low-rankness.

Original: We ﬁrst decompose each convolutional ﬁlter  over layer-speciﬁc 2D ﬁlter atoms, linearly combined by atom coefﬁcients.
Revised: To validate our hypothesis, we decouple the spatial patterns and channel mixing of a convolutional ﬁlter by decomposing it over layer-speciﬁc 2D ﬁlter atoms, linearly combined using atom coefﬁcients.

Original: Then with proper alignments to the layer outputs, we  observe a low-rank structure among those atom coefﬁcients across different layers.
Revised: Then with proper alignments to the layer outputs, we ﬁnally observe a low-rank structure among those atom coefﬁcients across different layers.

Original: This low-rank structure hints the feasibility of enforcing a common block of atom coefﬁcients across layers, and motivates us to introduce atom-coefﬁcient decomposed convolution (ACDC), a new plug-and-play replacement to standard convolutions that allows coefﬁcients to be shared across different network sub-structures.
Revised: This highly non-trivial observation, which to our knowledge has never been reported before, hints the feasibility of enforcing a common block of atom coefﬁcients across layers and suggests a joint subspace view to CNNs.

Original: By applying the joint subspace view to a wide range of network architectures, each layer is now left with only ﬁlter atoms, which typically include only a few hundreds of parameters, together with a common block of atom coefﬁcients shared across layers as shown in
Revised: Applying to a wide range of network architectures, this joint subspace view leads subsequently to a novel CNN architecture in which the majority of parameters are shared across layers as a common

Original: [Figure]
Revised: [Figure]

Original: ⨂ [Figure] Channels that describe important region features will highly likely be consistently assigned withlarge weights across layers, in order tocarry on those critical information.
Revised: will highly likely be consistently assigned with weights layers order to

Original: Thus, atom coefﬁcients, i.e., weights to linearly combine channels, now become shareable across layers to permit a joint subspace view.
Revised: Thus, atom coefﬁcients, i e., weights to linearly combine channels, now become shareable across layers to permit a joint subspace view.

Revised: Such joint subspace view leads naturally to a novel CNN architecture in which the majority of parameters are shared across layers as a common block of atom coefﬁcients, with only a few hundred parameters remaining speciﬁc to each layer as block of atom coefﬁcients, with only a few hundred parameters remaining speciﬁc to each layer as ﬁlter atoms as shown in Figure 1.

Original:  intuition visualized in Figure 1 that important image regions, e.g., the wings, beak, and tail of a bird, remain important across different layers and scales to output an informative deep representation.
Revised: This matches the intuition visualized in Figure 1 that important image regions, e.g., the wings, beak, and tail of a bird, remain important across different layers and scales to output an informative deep representation.

Original: Atom-coefﬁcient decomposition disentangles spatial convolution and channel mixing.
Revised: Atom-coefﬁcient decomposition disentangles spatial convolution and channel mixing.

Original: Thus, channels that describe important region features will highly likely be consistently assigned with large weights across layers, in order to carry on those critical information.
Revised: Thus, channels that describe important region features will highly likely be consistently assigned with large weights across layers, in order to carry on those critical information.

Original: Thus, atom coefﬁcients, i.e., weights to linearly combine channels, now become shareable across layers to permit joint subspace CNN modeling.
Revised: Thus, atom coefﬁcients, i.e., weights to linearly combine channels, now become shareable across layers to permit joint subspace CNN modeling.

Original: Our approach accepts many easily constructed variants, e.g., using different ﬁlter atom numbers and sharing substructures, to allow highly ﬂexible trade-offs between parameter reduction and model expressiveness.
Revised: Our approach accepts many easily constructed variants, e.g., using different ﬁlter atom numbers and sharing substructures, to allow highly ﬂexible trade-offs between parameter reduction and model expressiveness.

Original: This simple and plug-and-play joint subspace view opens the door to better interpreting, training, adapting, and compacting deep models.
Revised: This simple and plug-and-play joint subspace view opens the door to better interpreting, training, adapting, and compacting deep models.

Original: We support our ﬁndings with extensive empirical evidence in this paper.
Revised: We support our ﬁndings with extensive empirical evidence in this paper.

Original: By using variants of ACDC as plug-and-play replacements to the standard convolutions, we observe comparable and even better performance on challenging image classiﬁcation datasets with orders of magnitude smaller models.
Revised: By using variants of ACDC as plug-and-play replacements to the standard convolutions, we observe comparable and even better performance on challenging image classiﬁcation datasets with orders of magnitude smaller models.

Original: We further demonstrate with few shot learning experiments that ACDC improves the adaptation of deep networks on novel tasks with limited supervisions.
Revised: We further demonstrate with few shot learning experiments that ACDC improves the adaptation of deep networks on novel tasks with limited supervisions.

Original: We end our paper with a model interpretability discussion.
Revised: We end our paper with a model interpretability discussion.

Original: Our main contributions are summarized as follows:
Revised: Our main contributions are summarized as follows:

Original: • We introduce a joint subspace view to CNN, which is compatible to a wide range of network architectures, to provide a structural regularization over CNN ﬁlters for better model interpretability, training, adaptation and compaction.
Revised: • We introduce a joint subspace view to CNN, which is compatible to a wide range of network architectures, to provide a structural regularization over CNN ﬁlters for better model interpretability, training, adaptation and compaction.

Original: • Highlighting the remarkable ﬂexibility and compatibility, we introduce various model variants constructed easily by sharing coefﬁcient within different network sub-structures.
Revised: • Highlighting the remarkable ﬂexibility and compatibility, we introduce various model variants constructed easily by sharing coefﬁcient within different network sub-structures.

Original: • We validate the effectiveness of our approach by plug-and-playing them into modern CNN architectures for various real-world tasks.  
Revised: • We validate the effectiveness of our approach by plug-and-playing them into modern CNN architectures for various real-world tasks.  R ELATED W ORK

Original: J OINT S UBSPACE CNN M ODELING
Revised: Before presenting the connections of our method with the literature, let us comment on the complementary efforts on network compression. Contrary to recent efforts on network compression and efﬁcient architectures, we instead explore and exploit the underlying correspondence and correlations among convolutional layers within a given CNN. We decompose the convolutional kernels among layers, and enforce shared composition coefﬁcients within sub-structures of a CNN. This plug-and-play reparametrization of convolutional kernels signiﬁcantly reduces the network redundancy while maintaining performance, improving adaptation, providing additional interpretability, reducing computational cost, and adding compression (as a by-product) comparable to many modern heavily tailored light-weight networks.

Revised: We consider directions such as neural architecture search (Howard et al., 2019), post processing methods to deep networks (Phan et al., 2020; Stock et al., 2020; Li et al., 2019a; Wu et al., 2018; Son et al., 2018), and heavily tailored efﬁcient networks and training (Tan & Le, 2019) complementary to our efforts.

Revised: CNN architectures.

Revised: The tremendous success of applying convolutional neural networks (CNNs) on numerous tasks has stimulated rapid developments for more effective and efﬁcient network architectures in both hand-crafted (Chen et al., 2017;

Revised: He et al., 2016; Howard et al., 2017; Iandola et al., 2016; Sandler et al., 2018) and automatically discovered (Elsken et al., 2018; Liu et al., 2018;

Revised: Pham et al., 2018; Zoph & Le, 2017) manners.

Revised: We consider our work orthogonal to such topologybased methods, as the plug-and-play property of the proposed ACDC allows it to be added to all the aforementioned methods as a replacement to the standard convolution.

Revised: Besides efforts on studying efﬁcient network architectures, methods for network compression and pruning (Han et al., 2015a; 2016; 2015b; He et al., 2017; Li et al., 2016; Luo et al., 2017) have been extensively studied for decreasing the model size by pruning the inconsequential connections and weights of a network.

Revised: Methods (Ha et al., 2016; Savarese & Maire, 2019) align with our direction as they are also insensitive to network topologies.

Revised: And as shown in the experiments, ACDC can achieve higher performance in terms of parameter reduction and classiﬁcation accuracy with greater ﬂexibility.

Revised: Kernel decomposition in CNNs Convolutional kernel decomposition has been studied for various objectives.

Revised: (Sosnovik et al., 2019) utilizes kernel decomposition as a tool for constructing same kernel with multiple receptive ﬁelds.

Revised: DCFNet (Qiu et al., 2018) is proposed as a principle way of regularizing the convolutional ﬁlter structures by decomposing convolutional ﬁlters in CNN as a truncated expansion with pre-ﬁxed bases.

Revised: Split-wise decomposition over basis elements is introduced in (Li et al., 2019a) to retrain a CNN with reduced parameters.

Revised: Weight sharing in networks Exploiting sharing weight within CNNs is discussed in (Son et al., 2018; Wu et al., 2018) as a post-processing step for model compression.

Revised: Hypernetworks (Ha et al., 2016) adopt weight generation from a shared network (denoted as Hypernet) to exploit correlations among weights empirically.

Revised:  J OINT S UBSPACE CNN M ODELING

Original: In this section, we start with a brief introduction to atom-coefﬁcient ﬁlter decomposition, which builds the foundation of our joint subspace CNN modeling (Section 2.1).
Revised: In this section, we start with a brief introduction to atom-coefﬁcient ﬁlter decomposition, which builds the foundation of our joint subspace CNN modeling (Section 3.1).

Original: We then present the empirical support for the joint subspace view to CNN ﬁlters through a motivating experiment (Section 2.2).
Revised: We then present the empirical support for the joint subspace view to CNN ﬁlters through a motivating experiment (Section 3.2).

Original: We show in this motivating experiment that, with atom-coefﬁcient decomposition, a jointly low-rank structure can be clearly observed among coefﬁcients trained independently in different layers of a
Revised: We show in this motivating experiment that, with atom-coefﬁcient decomposition, a jointly low-rank structure can be clearly observed among coefﬁcients trained independently in different layers of a

Original: CNN under certain linear transformations.
Revised: CNN under certain linear transformations.

Original: This observation, together with the well recognized overparametrization problem of CNNs, leads to the idea of subspace coefﬁcients sharing enforced across network sub-structures, which can then be constructed as plug-and-play replacements to the standard convolutions (Section 2.3 and Section 2.4).
Revised: This observation, together with the well recognized overparametrization problem of CNNs, leads to the idea of subspace coefﬁcients sharing enforced across network sub-structures, which can then be constructed as plug-and-play replacements to the standard convolutions (Section 3.3 and Section 3.4).

Original: 2. C ONVOLUTIONAL F ILTER D ECOMPOSITION
Revised: 3. C ONVOLUTIONAL F ILTER D ECOMPOSITION

Original: Previous works have shown that a convolutional ﬁlter in a CNN can be decomposed as a linear combination of pre-ﬁxed basis (Qiu et al., 2018).
Revised: Previous works have shown that a convolutional ﬁlter in a CNN can be decomposed as a linear combination of pre-ﬁxed basis (Qiu et al., 2018).

Original: In our approach, we adopt a similar decomposition as shown in Figure 2, in which a convolutional ﬁlter is represented as a linear combination of trainable 2D ﬁlter atoms.
Revised: In our approach, we adopt a similar decomposition as shown in Figure 2, in which a convolutional ﬁlter is represented as a linear combination of trainable 2D ﬁlter atoms.

Original: After decomposition, a convolution layer with c -channel output Y and c (cid:48) -channel input X becomes
Revised: After decomposition, a convolution layer with c -channel output Y and c (cid:48) -channel input X becomes

Original: [Equation]
Revised: [Equation]

Original: where ∗ denotes the convolution operation.
Revised: where ∗ denotes the convolution operation.

Original: As illustrated in Figure 2, in (1), a convolutional ﬁlter
Revised: As illustrated in Figure 2, in (1), a convolutional ﬁlter

Original: K ∈ R c × c (cid:48)
Revised: K ∈ R c × c (cid:48)

Original: × l × l , which can be seen as a stack of c × c (cid:48)
Revised: × l × l , which can be seen as a stack of c × c (cid:48)

Original: 2D convolutional ﬁlters with the size of l × l , is reconstructed by multiplying m 2D ﬁlter atoms, denoted collectively as D ∈ R m × l × l , with the corresponding atom coefﬁcients A ∈ R c × c (cid:48)  m .
Revised: 2D convolutional ﬁlters with the size of l × l , is reconstructed by multiplying m 2D ﬁlter atoms, denoted collectively as D ∈ R m × l × l , with the corresponding atom coefﬁcients A ∈ R c × c (cid:48) × m .

Original: Note that square ﬁlters are assumed here for simplicity, while all ﬁlter shapes are supported.
Revised: Note that square ﬁlters are assumed here for simplicity, while all ﬁlter shapes are supported.

Original: Since both convolution and tensor multiplication are commutative, a convolutional layer can now be decomposed into two:
Revised: Since both convolution and tensor multiplication are commutative, a convolutional layer can now be decomposed into two:

Original: • A atom sub-layer where each atom involves spatial-only convolution with the ﬁlter atoms, i.e., Z ∈ R c (cid:48) m × h × w = D ∗ X ; • A coefﬁcient sub-layer that linearly combines feature channels from the atom sub-layer : Y ∈ R c × h × w = AZ .
Revised: • A atom sub-layer where each atom involves spatial-only convolution with the ﬁlter atoms, i.e., Z ∈ R c (cid:48) m × h × w = D ∗ X ; • A coefﬁcient sub-layer that linearly combines feature channels from the atom sub-layer : Y ∈ R c × h × w = AZ .

Original: Note that Z here denotes atom sub-layer outputs, and stride 1 and same padding are assumed for the sake of discussion.
Revised: Note that Z here denotes atom sub-layer outputs, and stride 1 and same padding are assumed for the sake of discussion.

Original: 2. T HE M OTIVATION B EHIND
Revised: 3. T HE M OTIVATION B EHIND

Original: l l c′ c × = …
Revised: l l c′ c × = …

Original: K D l l m m c c′
Revised: K D l l m m c c′

Original: A
Revised: A

Original: Deep CNNs are long recognized to be over-parametrized.
Revised: Deep CNNs are long recognized to be over-parametrized.

Original: The very deep layers in modern CNN structures (He et al., 2016; Huang et al., 2017; Zagoruyko & Komodakis, 2016) and the high-dimensional ﬁlters with little structural regularizations lead to hundreds of millions of parameters.
Revised: The very deep layers in modern CNN structures (He et al., 2016; Huang et al., 2017; Zagoruyko & Komodakis, 2016) and the high-dimensional ﬁlters with little structural regularizations lead to hundreds of millions of parameters.

Original: Such over-parametrization problem is also observed in the studies of deep representations (Raghu et al., 2017), and empirically alleviated by new network structures (Chollet, 2017; Howard et al., 2017), network compression, and parameter reduction methods (Ha et al., 2016; Savarese & Maire, 2019).
Revised: Such over-parametrization problem is also observed in the studies of deep representations (Raghu et al., 2017), and empirically alleviated by new network structures (Chollet, 2017; Howard et al., 2017), network compression, and parameter reduction methods (Ha et al., 2016; Savarese & Maire, 2019).

Original: Meanwhile, recent studies on deep representations (Kornblith et al., 2019; Morcos et al., 2018;
Revised: Meanwhile, recent studies on deep representations (Kornblith et al., 2019; Morcos et al., 2018;

Original: Raghu et al., 2017) have shown that there exists obvious correlations in features across layers within a CNN after proper linear transformations.
Revised: Raghu et al., 2017) have shown that there exists obvious correlations in features across layers within a CNN after proper linear transformations.

Original: The correlation of features across network layers motivates us to explore and exploit correlations across ﬁlters for structural regularizations.
Revised: The correlation of features across network layers motivates us to explore and exploit correlations across ﬁlters for structural regularizations.

Original: We present here a motivating experiment on MNIST by applying CCA alignments as in (Raghu et al., 2017) to the atom sub-layer outputs and the coefﬁcient sub-layer outputs of layer i and layer j .
Revised: We present here a motivating experiment on MNIST by applying CCA alignments as in (Raghu et al., 2017) to the atom sub-layer outputs and the coefﬁcient sub-layer outputs of layer i and layer j .

Original: Note that no atom coefﬁcient sharing is yet imposed here, and the network reports the same testing accuracy before and after ﬁlter decomposition.
Revised: Note that no atom coefﬁcient sharing is yet imposed here, and the network reports the same testing accuracy before and after ﬁlter decomposition.

Original: Formally, c , m , d , and hw denote the number of channels, number of ﬁlter atoms, test set size, and the 2D feature dimensions, respectively.
Revised: Formally, c , m , d , and hw denote the number of channels, number of ﬁlter atoms, test set size, and the 2D feature dimensions, respectively.

Original: The atom sub-layer outputs of the i -th and j -th layer, Z i and Z j ∈ R cm × dhw , are ﬁrstly aligned by linear transformations P i and P j ∈ R cm × cm that maximize the correlation ρ z = max P i , P j corr ( P i Z i , P j Z j ) .
Revised: The atom sub-layer outputs of the i -th and j -th layer, Z i and Z j ∈ R cm × dhw , are ﬁrstly aligned by linear transformations P i and P j ∈ R cm × cm that maximize the correlation ρ z = max P i , P j corr ( P i Z i , P j Z j ) .

Original: And similarly, the coefﬁcient sub-layer outputs of both layers, Y i and Y j ∈ R c × dhw , are aligned by Q i and Q j ∈ R c × c that maximize the correlation ρ = max Q i , Q j corr ( Q i Y i , Q j Y j ) .
Revised: And similarly, the coefﬁcient sub-layer outputs of both layers, Y i and Y j ∈ R c × dhw , are aligned by Q i and Q j ∈ R c × c that maximize the correlation ρ = max Q i , Q j corr ( Q i Y i , Q j Y j ) .

Original: Omitting the layer indexes, the feed forwards of both layers can be rewritten as
Revised: Omitting the layer indexes, the feed forwards of both layers can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: By merging the transform into the coefﬁcients A by (cid:101)
Revised: By merging the transform into the coefﬁcients A by (cid:101)

Original: A =
Revised: A =

Original: QAP − 1 , we obtain ‘aligned coefﬁcients’ (cid:101)
Revised: QAP − 1 , we obtain ‘aligned coefﬁcients’ (cid:101)

Original: A i and (cid:101)
Revised: A i and (cid:101)

Original: A j , that reside in a low rank structure reﬂected by the very similar effective ranks of (cid:101)
Revised: A j , that reside in a low rank structure reﬂected by the very similar effective ranks of (cid:101)

Original: A i[Figure] [Figure] and [ (cid:101)
Revised: A i and [ (cid:101)

Original: A i , (cid:101)
Revised: A i , (cid:101)

Original: A j ] .
Revised: A j ] .

Original: For example, in our MNIST experiment using a 4-layer 32-channel CNN, out of thepossible ( i, j ) pairs [Figure] the average effective rank of (cid:101)
Revised: For example, in our MNIST experiment using a 4-layer 32-channel CNN, out of thepossible ( i, j ) pairs, the average effective rank of (cid:101)

Original: A i and [ (cid:101)
Revised: A i and [ (cid:101)

Original: A i , (cid:101)
Revised: A i , (cid:101)

Original: A j ] are 31 .
Revised: A j ] are 31 .

Original: 98 and 38 . 56 , respectively.
Revised: 98 and 38 . 56 , respectively.

Original: Our observations agree with and further support recent studies on cross-layer feature correlations (Kornblith et al., 2019; Morcos et al., 2018; Raghu et al., 2017).
Revised: Our observations agree with and further support recent studies on cross-layer feature correlations (Kornblith et al., 2019; Morcos et al., 2018; Raghu et al., 2017).

Original: This observation hints that, instead of treating ﬁlters in each layers independently, it is feasible to exploit such intrinsic low-rank structure of coefﬁcients across layers, and impose a joint subspace view to ﬁlters in a CNNs.
Revised: This observation hints that, instead of treating ﬁlters in each layers independently, it is feasible to exploit such intrinsic low-rank structure of coefﬁcients across layers, and impose a joint subspace view to ﬁlters in a CNNs.

Original: And practically, this observation supports the feasibility of directly reducing the parameter redundancy of a CNN by enforcing shared atom coefﬁcients across layers.
Revised: And practically, this observation supports the feasibility of directly reducing the parameter redundancy of a CNN by enforcing shared atom coefﬁcients across layers.

Original: 2. C OEFFICIENTS S HARING A CROSS L AYERS
Revised: 3. C OEFFICIENTS S HARING A CROSS L AYERS

Original: Based on the observations above and (1), a joint subspace CNN modeling is constructed by enforcing common atom coefﬁcients A across layers in a CNN, as illustrated in Figure 1.
Revised: Based on the observations above and (1), a joint subspace CNN modeling is constructed by enforcing common atom coefﬁcients A across layers in a CNN, as illustrated in Figure 1.

Original: Formally, given a
Revised: Formally, given a

Original: N -layers CNN, the n -th convolutional ﬁlter is constructed by
Revised: N -layers CNN, the n -th convolutional ﬁlter is constructed by

Original: [Equation]
Revised: [Equation]

Original: Assuming for now all layers have identical channel number with c (cid:48)
Revised: Assuming for now all layers have identical channel number with c (cid:48)

Original: =
Revised: =

Original: c , the amount of parameters is reduced from c 2 c 2 N to c 2 m + Nkl 2 . 
Revised: c , the amount of parameters is reduced from c 2 c 2 N to c 2 m + Nkl 2 . [Figure] [Figure]

Original: In practice, convolution layers within a network can have different numbers of channels.
Revised: In practice [Figure] convolution layers within a network can have different numbers of channels.

Original: When sharing coefﬁcients across layers with different channels numbers, we initialize the dimensions of the shared coefﬁcients to be the largest dimensions needed by the corresponding layers.
Revised: When sharing coefﬁcients across layers with different channels numbers, we initialize the dimensions of the shared coefﬁcients to be the largest dimensions needed by the corresponding layers.

Original: For example, given a N -layer CNN with convolutional kernels { K n ∈ R c n × c (cid:48) n × l × l ; n = 1 , . . .
Revised: For example, given a N -layer CNN with convolutional kernels { K n ∈ R c n × c (cid:48) n × l × l ; n = 1 , . . .

Original: , N } , the network is constructed by initializing the shared coefﬁcient as A ∈ R c max × c (cid:48) max × m , where c max = max { c n ; n = 1 , . . .
Revised: , N } , the network is constructed by initializing the shared coefﬁcient as A ∈ R c max × c (cid:48) max × m , where c max = max { c n ; n = 1 , . . .

Original: , N } and c (cid:48) max = max { c (cid:48) n ; n = 1 , . .
Revised: , N } and c (cid:48) max = max { c (cid:48) n ; n = 1 , . .

Original: . , N } .
Revised: . , N } .

Original: The ﬁlters with fewer channels are reconstructed by multiplying the ﬁlter atoms with a subset of the shared coefﬁcients
Revised: The ﬁlters with fewer channels are reconstructed by multiplying the ﬁlter atoms with a subset of the shared coefﬁcients

Original: K n = A [1 : c n , 1 : c (cid:48) n , 1 : m ] D n .
Revised: K n = A [1 : c n , 1 : c (cid:48) n , 1 : m ] D n .

Original: A 3-layer illustration with progressively increased channels is shown in Figure 3.
Revised: A 3-layer illustration with progressively increased channels is shown in Figure 3.

Original: Such a design choice is motivated by multi-scale decomposition (Mallat, 1999), and proves to be highly effective with our extensive experimental validation.
Revised: Such a design choice is motivated by multi-scale decomposition (Mallat, 1999), and proves to be highly effective with our extensive experimental validation.

Original: Note that a relaxed version can be formulated as follows: Instead of enforcing common coefﬁcients across all layers, we allow the sharing to happen among a few consecutive layers in a network.
Revised: Note that a relaxed version can be formulated as follows: Instead of enforcing common coefﬁcients across all layers, we allow the sharing to happen among a few consecutive layers in a network.

Original: We refer a group of consecutive layers with identical number of output channels and identical feature sizes as a block in a deep network, and enforce coefﬁcient sharing in each block.
Revised: We refer a group of consecutive layers with identical number of output channels and identical feature sizes as a block in a deep network, and enforce coefﬁcient sharing in each block.

Original: For example, a
Revised: For example, a

Original: VGG16 (Simonyan & Zisserman, 2014) can be implemented by sharing coefﬁcients within 5 groups, each of which consists of conv layers with 64, 128, 256, 512, 512 channels, respectively.
Revised: VGG16 (Simonyan & Zisserman, 2014) can be implemented by sharing coefﬁcients within 5 groups, each of which consists of conv layers with 64, 128, 256, 512, 512 channels, respectively.

Original: 2. C OEFFICIENTS S HARING A CROSS F ILTER G ROUPS
Revised: 3. C OEFFICIENTS S HARING A CROSS F ILTER G ROUPS

Original: Motivated by the observation in (Raghu et al., 2017) that the representation learned at a layer is not fully determined by the number of neurons in the layer, and the existence of parameter redundancy within a single layer as studied in (Michel et al., 2019), we suggest to further extend our 
Revised: Motivated by the observation in (Raghu et al., 2017) that the representation learned at a layer is not fully determined by the number of neurons in the layer, and the existence of parameter redundancy within a single layer as studied in (Michel et al., 2019), we suggest to further extend our joint subspace view from coefﬁcient sharing across layers to sharing across groups of ﬁlters in a convolutional layer.

Original: Practically, we further break down the smallest sharing unit from a layer to part of a layer.
Revised: Practically, we further break down the smallest sharing unit from a layer to part of a layer.

Original: A high-dimensional convolutional layer can now be separated into several groups with identical sizes, and sharing coefﬁcients are imposed across groups.
Revised: A high-dimensional convolutional layer can now be separated into several groups with identical sizes, and sharing coefﬁcients are imposed across groups.

Original: Formally, given a convolutional layer with c (cid:48) input channels and c output channels, respectively, we divide input channels into g identical-sized groups, and each group is convolved with a convolution kernel K j ∈ R cg × c (cid:48)
Revised: Formally, given a convolutional layer with c (cid:48) input channels and c output channels, respectively, we divide input channels into g identical-sized groups, and each group is convolved with a convolution kernel K j ∈ R cg × c (cid:48)

Original: g × l × l , j = 1 , . . .
Revised: g × l × l , j = 1 , . . .

Original: , g .
Revised: , g .

Original: After grouping, we decompose { K j ; j = 1 , . . .
Revised: After grouping, we decompose { K j ; j = 1 , . . .

Original: , g } into shared coefﬁcients A ∈ R cg × c (cid:48)
Revised: , g } into shared coefﬁcients A ∈ R cg × c (cid:48)

Original: g × m , and g independent sets of ﬁlter atoms { D j ∈ R m × l × l ; j = 1 , . .
Revised: g × m , and g independent sets of ﬁlter atoms { D j ∈ R m × l × l ; j = 1 , . .

Original: . , g } .
Revised: . , g } .

Original: In this way, the number of shared coefﬁcients is reduced by g 2 times, and the number of ﬁlter atoms is increased by g times.
Revised: In this way, the number of shared coefﬁcients is reduced by g 2 times, and the number of ﬁlter atoms is increased by g times.

Original: Since ﬁlter atoms have orders of magnitude smaller dimension comparing to the coefﬁcients, we achieve further parameter reduction.
Revised: Since ﬁlter atoms have orders of magnitude smaller dimension comparing to the coefﬁcients, we achieve further parameter reduction.

Original: Since each K j only convolves with a subset of the input feature, this method reduces the overall computations.
Revised: Since each K j only convolves with a subset of the input feature, this method reduces the overall computations.

Original: However, directly deploying this sharing mechanism breaks the network into several paralleled subnetworks with no feature passing and gradient propagation among them.
Revised: However, directly deploying this sharing mechanism breaks the network into several paralleled subnetworks with no feature passing and gradient propagation among them.

Original: To remedy this without adding any additional parametric components, we utilize channel shufﬂe (Zhang et al., 2018) that enables information to be efﬁciently propagated among groups in a non-parametric way.
Revised: To remedy this without adding any additional parametric components, we utilize channel shufﬂe (Zhang et al., 2018) that enables information to be efﬁciently propagated among groups in a non-parametric way.

Original: An illustration is presented in Figure 4.
Revised: An illustration is presented in Figure 4.

Original: Since the size of the shared coefﬁcient now does not depend on the largest feature dimension of a network but the size of the groups, we obtain a more compact model.
Revised: Since the size of the shared coefﬁcient now does not depend on the largest feature dimension of a network but the size of the groups, we obtain a more compact model.

Original: Those with higher than baseline accuracies but fewer parameters are marked
Revised: In practice, to handle the expanding channels numbers in typical CNNs, we can easily allow each group in the shallow layer to feed the features to multiple groups in the deep layer. Therefore the proposed coefﬁcient sharing across ﬁlter groups permits remarkable parameter sharing with little modiﬁcation to the original network structures such as the numbers of channels and layers.

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Revised: In this section

Original: In this section, we support our ﬁndings with extensive empirical evidence, and evaluate variants of our joint subspace CNN modeling, referred to as ACDC, as plug-and-play replacements to the standard convolution, with several different levels of atom coefﬁcient sharing:
Revised: [Table] we support our ﬁndings with extensive empirical evidence [Table] and evaluate variants of our joint subspace CNN modeling, referred to as ACDC, as plug-and-play replacements to the standard convolution, with several different levels of atom coefﬁcient sharing:

Original: • ACDC-net : Sharing across all layers.
Revised: • ACDC-net : Sharing across all layers.

Original: • ACDC-block : Sharing across blocks of layers.
Revised: • ACDC-block : Sharing across blocks of layers.

Original: • ACDC-g : Sharing across ﬁlter groups.
Revised: • ACDC-g : Sharing across ﬁlter groups.

Original: ACDC-g naturally further leads to three variants by allowing sharing within the entire network, blocks within a network, and layers within a network, and are thus named as ACDC-g-net , ACDCg-block , and ACDC-g-layer , respectively.
Revised: ACDC-g naturally further leads to three variants by allowing sharing within the entire network, blocks within a network, and layers within a network, and are thus named as ACDC-g-net , ACDCg-block , and ACDC-g-layer , respectively.

Original: We use m and s to denote the number of ﬁlter atoms and grouping size, respectively.
Revised: We use m and s to denote the number of ﬁlter atoms and grouping size, respectively.

Original: 3. I MAGE C LASSIFICATION
Revised: 4. I MAGE C LASSIFICATION

Original: In this section [Table] we perform standard image classiﬁcation experiments, and validate our joint subspace CNN modeling with extensive experiments on CIFAR-10, CIFAR-100, Tiny ImageNet, and
Revised: In this section, we perform standard image classiﬁcation experiments, and validate our joint subspace CNN modeling with extensive experiments on CIFAR-10, CIFAR-100, Tiny ImageNet, and

Original: ImageNet.
Revised: ImageNet.

Original: Evaluating variants of ACDC.
Revised: Evaluating variants of ACDC.

Original: We ﬁrst report on CIFAR-10 self-comparisons on variants of
Revised: We ﬁrst report on CIFAR-10 self-comparisons on variants of

Original: ACDC constructed with different numbers of ﬁlter atoms as well as grouping sizes.
Revised: ACDC constructed with different numbers of ﬁlter atoms as well as grouping sizes.

Original: We present performance in terms of both parameter size and classiﬁcation error in Table 1.
Revised: We present performance in terms of both parameter size and classiﬁcation error in Table 1.

Original: VGG16 (Simonyan & Zisserman, 2014), ResNet18 (He et al., 2016) [Table] and Wide ResNet (WRN) (Zagoruyko & Komodakis, 2016) are adopted as the underlying network architectures in order to show the remarkable compatibility of the proposed approach.
Revised: VGG16 (Simonyan & Zisserman, 2014), ResNet18 (He et al., 2016), and Wide ResNet (WRN) (Zagoruyko & Komodakis, 2016) are adopted as the underlying network architectures in order to show the remarkable compatibility of the proposed approach.

Original: This set of experiments show our joint subspace modeling maintains or improves CNN performance with a signiﬁcantly more compact model, e [Table] g., 98% reduction with comparable performance, and 70% reduction with even higher accuracy.[Table]
Revised: This set of experiments show our joint subspace modeling maintains or improves CNN performance with a signiﬁcantly more compact model, e.g., 98% reduction with comparable performance, and 70% reduction with even higher accuracy.

Original: CIFAR-10, CIFAR-100 and Tiny ImageNet.
Revised: CIFAR-10, CIFAR-100 and Tiny ImageNet.

Original: We further present experiment results on CIFAR-10,
Revised: We further present experiment results on CIFAR-10,

Original: CIFAR- [Table] and Tiny ImageNet.
Revised: CIFAR-100, and Tiny ImageNet in Table 2.

Original: We compare exampled variants of ACDC against HyperNetworks (Ha et al., 2016) and Soft Parameter Sharing (Savarese & Maire, 2019)
Revised: We compare exampled variants of ACDC against HyperNetworks (Ha et al., 2016) and Soft Parameter Sharing (Savarese & Maire, 2019), both of which serve as plug-and-play replacements to standard convolutions as well. Though HyperNetworks (Ha et al., 2016) achieves remarkable parameter reduction, ACDC is able to achieve higher accuracies

Original: [Table]
Revised: [Table]

Original: both of which serve as plugand-play replacements to standard convolutions as well.
Revised: We use ResNet34 and ResNet50 as baselines

Original: Though HyperNetworks (Ha et al., 2016) achieves remarkable parameter reduction, ACDC is able to achieve higher accuracies with even fewer parameters.
Revised: [Table] [Figure] [Table]

Original: The parameter reductions in Soft Parameter Sharing (Savarese & Maire, 2019) are highly restricted by the large scale elements in the ﬁlter bank.
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: with even fewer parameters.

Revised: The parameter reductions in Soft Parameter Sharing (Savarese & Maire, 2019) are highly restricted by the large scale elements in the ﬁlter bank.

Original: For example, SWRN 28-10-1, as the smallest variant of Soft Parameter Sharing on WRN, adopts a single template per sharing group, and can only achieve 66% of parameter reduction.
Revised: For example, SWRN 28-101, as the smallest variant of Soft Parameter Sharing on WRN [Figure] adopts a single template per sharing group [Figure]

Revised: and can only achieve 66% of parameter reduction.

Original: By adopting ACDC-net and ACDC-block to WRN, we are able to achieve both higher parameter reductions and accuracy.
Revised: By adopting ACDC-net and ACDC-block to WRN [Figure] we are able to achieve both higher parameter reductions and accuracy.

Original: Though model compaction is not our mere objective, we include here as references some state-of-the-art light CNN architectures (Iandola et al., 2016; Zhang et al., 2018;
Revised: Though model compaction is not our mere objective, we include here as references some state-of-the-art light CNN architectures (Iandola et al., 2016; Zhang et al., 2018;

Original: Ma et al., 2018; Sandler et al., 2018) and architecture based on neural architecture search (Zoph et al., 2018).
Revised: Ma et al., 2018; Sandler et al., 2018) and architecture based on neural architecture search (Zoph et al., 2018).

Original: Additional comparisons against network compression and pruning methods are in supplementary material Section C.
Revised: Additional comparisons against network compression and pruning methods are in supplementary material Section C.

Original: ImageNet.
Revised: ImageNet.

Original: To fully validate our joint subspace CNN modeling, we perform further experiments on the large-scale ImageNet dataset (Deng et al., 2009).
Revised: To fully validate our joint subspace CNN modeling, we perform further experiments on the large-scale ImageNet dataset (Deng et al., 2009).

Original: We use ResNet34 and ResNet50 as baseline models, and compare ACDC against LegoNet (Yang et al., 2019), Versatile (Wang et al., 2018b),  network pruning method ChPrune (He et al., 2017), and recently proposed Structured Conv (Bhalgat et al., 2020).
Revised: We use ResNet34 and ResNet50 as baseline models, and compare ACDC against LegoNet (Yang et al., 2019), Versatile (Wang et al., 2018b), [Figure] [Figure] network pruning method ChPrune (He et al., 2017), and recently proposed Structured Conv (Bhalgat et al., 2020).

Original: Results are presented in Table 3.
Revised: Results are presented in Table 3.

Original: For ResNet34, one variant of ACDC with grouping[Table] [Figure] in layers uses only 1 . 66 million parameters with acceptable performance decrease.
Revised: For ResNet34, one variant of ACDC with grouping in layers uses only 1 . 66 million parameters with acceptable performance decrease.

Original: For ResNet50, since a large proportion of the parameters are in the 1 × 1 conv layers, ACDC achieves fair parameter reductions with maintained performance.
Revised: For ResNet50, since a large proportion of the parameters are in the 1 × 1 conv layers, ACDC achieves fair parameter reductions with maintained performance.

Original: [Equation] E XPERIMENTS
Revised: 4. F EW - SHOT E XPERIMENTS

Original: We further demonstrate that our joint subspace modeling improves deep model adaptation on novel tasks with limited supervisions, which is validated by few-shot classiﬁcation using commonly adopted experimental settings.
Revised: We further demonstrate that our joint subspace modeling improves deep model adaptation on novel tasks with limited supervisions, which is validated by few-shot classiﬁcation using commonly adopted experimental settings.

Original: Speciﬁcally, we adopt ACDC-net on the model-agnostic metalearning (MAML)
Revised: Speciﬁcally, we adopt ACDC-net on the model-agnostic metalearning (MAML)

Original: (Finn et al., 2017) algorithm, which is a method that adapts to a novel task by tuning the entire network from a learned initialization.
Revised: (Finn et al., 2017) algorithm, which is a method that adapts to a novel task by tuning the entire network from a learned initialization.

Original: Although MAML is designed to be modelagnostic, we consistently observe that it struggles for further performance improvements when using deeper and wider networks.
Revised: Although MAML is designed to be modelagnostic, we consistently observe that it struggles for further performance improvements when using deeper and wider networks.

Original: Same observations are reported in (Chen et al., 2019).
Revised: Same observations are reported in (Chen et al., 2019).

Original: We show that such limitation can be alleviated by structural regularizations with ACDC.
Revised: We show that such limitation can be alleviated by structural regularizations with ACDC.

Original: We follow the same experimental settings as (Chen et al., 2019) and perform both 5-way 1-shot and 5-way 5-shot image classiﬁcations on mini ImageNet dataset.
Revised: We follow the same experimental settings as (Chen et al., 2019) and perform both 5-way 1-shot and 5-way 5-shot image classiﬁcations on mini ImageNet dataset.

Original: The comparisons are shown in Figure 5.
Revised: The comparisons are shown in Figure 5.

Original: Though adopting ResNet10 with MAML achieves improvements over simple network with few stacked layers, the performance drops with more residual layers as shown by the results on ResNet18.
Revised: Though adopting ResNet10 with MAML achieves improvements over simple network with few stacked layers, the performance drops with more residual layers as shown by the results on ResNet18.

Original: By using
Revised: By using

Original: ACDC-net with deeper ResNets, performance is not only maintained but also improved when more layers are used, which thus permits larger models in MAML. 
Revised: ACDC-net with deeper ResNets, performance is not only maintained but also improved when more layers are used, which thus permits larger models in MAML. [Table]

Original: [Table]
Revised: [Table]

Original: C OMPUTATIONAL E FFICIENCY
Revised: C OMPUTATIONAL E FFICIENCY

Original: ACDC enjoys another merit of being computationally efﬁcient when using sharing with grouping .
Revised: ACDC enjoys another merit of being computationally efﬁcient when using sharing with grouping .

Original: Since after grouping, each group of convolutional ﬁlters only convolves with a subset of input features, ACDC-g-block and ACDC-g-net substantially reduce the number of FLOPs and accelerate the speed.
Revised: Since after grouping, each group of convolutional ﬁlters only convolves with a subset of input features, ACDC-g-block and ACDC-g-net substantially reduce the number of FLOPs and accelerate the speed.

Original: We report comparisons with ResNet18 and VGG16 in Table 4.
Revised: We report comparisons with ResNet18 and VGG16 in Table 4.

Original: All numbers are obtained by feeding the network a typical batch with 100 64 × 64 images.
Revised: All numbers are obtained by feeding the network a typical batch with 100 64 × 64 images.

Original: It is clearly shown that by using small groups, the computation can be reduced dramatically, and larger number of ﬁlter atoms only effects the speed the total FLOPs slightly.
Revised: It is clearly shown that by using small groups, the computation can be reduced dramatically, and larger number of ﬁlter atoms only effects the speed the total FLOPs slightly.

Original: We present further speed comparisons against recent efﬁcient network structures in Appendix Table F.[Figure] [Figure]
Revised: We present further speed comparisons against recent efﬁcient network structures in Appendix Table F.

Original: Number of ﬁlter atoms One additional hyperparameter introduced is the number of ﬁlter atoms m per sub-structure.
Revised: Number of ﬁlter atoms One additional hyperparameter introduced is the number of ﬁlter atoms m per sub-structure.

Original: As shown in Figure 6, having more ﬁlter atoms in each sub-structure leads to performance improvements that saturate at m = 32 .
Revised: As shown in Figure 6, having more ﬁlter atoms in each sub-structure leads to performance improvements that saturate at m = 32 .

Original: More ﬁlter atoms also result in larger parameter sizes, which are unfavourable.
Revised: More ﬁlter atoms also result in larger parameter sizes, which are unfavourable.

Original: Regularization by atom-drop.
Revised: Regularization by atom-drop.

Original: To improve the robustness of the ﬁlter atoms and the corresponding reconstructed kernels, we further propose a structural regularization to ﬁlter atoms named atomdrop inspired by widely used dropout.
Revised: To improve the robustness of the ﬁlter atoms and the corresponding reconstructed kernels, we further propose a structural regularization to ﬁlter atoms named atomdrop inspired by widely used dropout.

Original: Speciﬁcally, when training the network, we randomly drop a ﬁlter atom with probability p , which is referred as atom drop rate , by temporarily setting values of the dropped ﬁlter atoms to 0 , and the values of all other remained ﬁlter atoms are multiplied by 11 − p in order to maintain consistent scales of the reconstructed convolutional kernels.
Revised: Speciﬁcally, when training the network, we randomly drop a ﬁlter atom with probability p , which is referred as atom drop rate , by temporarily setting values of the dropped ﬁlter atoms to 0 , and the values of all other remained ﬁlter atoms are multiplied by 11 − p in order to maintain consistent scales of the reconstructed convolutional kernels.

Original: At test time, all ﬁlter atoms are presented with no dropping.
Revised: At test time, all ﬁlter atoms are presented with no dropping.

Original: As shown in Figure 7, atom-drop improves generalization when p ≤ 0 .
Revised: As shown in Figure 7, atom-drop improves generalization when p ≤ 0 .

Original: Higher values of p potentially degrade the performance as the training becomes unstable.
Revised: Higher values of p potentially degrade the performance as the training becomes unstable.

Original: Thus we use p = 0 .
Revised: Thus we use p = 0 .

Original: 1 as the default setting.
Revised: 1 as the default setting.

Original: We show in Appendix Section B further comparison against typical dropout methods.
Revised: We show in Appendix Section B further comparison against typical dropout methods.

Original: [Figure]
Revised: [Figure]

Original: sharing allows CAM to better features regions

Original: R ELATED W ORK

Original: Before presenting the connections of our method with the literature, let us comment on the complementary efforts on network compression.

Original: Contrary to recent efforts on network compression and efﬁcient architectures, we instead explore and exploit the underlying correspondence and correlations among convolutional layers within a given CNN.

Original: We decompose the convolutional kernels among layers, and enforce shared composition coefﬁcients within sub-structures of a CNN.

Original: This plug-and-play reparametrization of convolutional kernels signiﬁcantly reduces the network redundancy while maintaining performance, improving adaptation, providing additional interpretability, reducing computational cost, and adding compression (as a by-product) comparable to many modern heavily tailored light-weight networks.

Original: We consider directions such as neural architecture search (Howard et al., 2019), post processing methods to deep networks (Phan et al., 2020;

Original: Stock et al., 2020; Li et al., 2019; Wu et al., 2018; Son et al., 2018), and heavily tailored efﬁcient networks and training (Tan & Le, 2019) complementary to our efforts.

Original: CNN architectures.

Original: The tremendous success of applying convolutional neural networks (CNNs) on numerous tasks has stimulated rapid developments for more effective and efﬁcient network architectures in both hand-crafted (Chen et al., 2017;

Original: He et al., 2016; Howard et al., 2017; Iandola et al., 2016; Sandler et al., 2018) and automatically discovered (Elsken et al., 2018; Liu et al., 2018;

Original: Pham et al., 2018; Zoph & Le, 2017) manners.

Original: We consider our work orthogonal to such topologybased methods, as the plug-and-play property of the proposed ACDC allows it to be added to all the aforementioned methods as a replacement to the standard convolution.

Original: Besides efforts on studying efﬁcient network architectures, methods for network compression and pruning (Han et al., 2015a; 2016; 2015b; He et al., 2017; Li et al., 2016; Luo et al., 2017) have been extensively studied for decreasing the model size by pruning the inconsequential connections and weights of a network.

Original: Methods (Ha et al., 2016; Savarese & Maire, 2019) align with our direction as they are also insensitive to network topologies.

Original: And as shown in the experiments, ACDC can achieve higher performance in terms of parameter reduction and classiﬁcation accuracy with greater ﬂexibility.

Original: Kernel decomposition in CNNs Convolutional kernel decomposition has been studied for various objectives.

Original: (Sosnovik et al., 2019) utilizes kernel decomposition as a tool for constructing same kernel with multiple receptive ﬁelds.

Original: DCFNet (Qiu et al., 2018) is proposed as a principle way of regularizing the convolutional ﬁlter structures by decomposing convolutional ﬁlters in CNN as a truncated expansion with pre-ﬁxed bases.

Original: Split-wise decomposition over basis elements is introduced in (Li et al., 2019) to retrain a CNN with reduced parameters.

Original: Weight sharing in networks Exploiting sharing weight within CNNs is discussed in (Son et al., 2018; Wu et al., 2018) as a post-processing step for model compression.

Original: Hypernetworks (Ha et al., 2016) adopt weight generation from a shared network (denoted as Hypernet) to exploit correlations among weights empirically.

Original:  C ONCLUSION AND I NTERPRETABILITY D ISCUSSION
Revised:  C ONCLUSION AND I NTERPRETABILITY D ISCUSSION

Original: In this paper, we introduced a joint subspace approach to CNNs.
Revised: In this paper, we introduced a joint subspace approach to CNNs.

Original: We presented observations that, due to the underlying cross-layer correlations, atom coefﬁcients in the decomposed convolution layers across a CNN reside in a low-rank structure.
Revised: We presented observations that, due to the underlying cross-layer correlations, atom coefﬁcients in the decomposed convolution layers across a CNN reside in a low-rank structure.

Original: We explicitly exploited such observations by enforcing atom coefﬁcients to be shared within sub-structures of CNNs.
Revised: We explicitly exploited such observations by enforcing atom coefﬁcients to be shared within sub-structures of CNNs.

Original: Variants of the proposed method can be constructed with different sharing structures, number of atoms, and grouping sizes.
Revised: Variants of the proposed method can be constructed with different sharing structures, number of atoms, and grouping sizes.

Original: We reported extensive experiment results that show the effectiveness of the proposed method on standard image classiﬁcation and adaptations.
Revised: We reported extensive experiment results that show the effectiveness of the proposed method on standard image classiﬁcation and adaptations.

Original: This joint subspace modeling has the potential for better interpretability of CNNs, due to the crosslayer shared coefﬁcients.
Revised: This joint subspace modeling has the potential for better interpretability of CNNs, due to the crosslayer shared coefﬁcients.

Original: Class activation mapping (CAM) (Zhou et al., 2016), is a method that is originally proposed to explain the importance of image regions only at the ﬁnal convolution layer , and is calculated by weighted averaging the features of the ﬁnal convolution layer by the weight vector of a particular class.
Revised: Class activation mapping (CAM) (Zhou et al., 2016), is a method that is originally proposed to explain the importance of image regions only at the ﬁnal convolution layer , and is calculated by weighted averaging the features of the ﬁnal convolution layer by the weight vector of a particular class.

Original: We close our paper with an illustration in Figure 8 that extends class activation mapping (CAM) to all layers of a ResNet-18 and a ResNet-18 with ACDC-net .
Revised: We close our paper with an illustration in Figure 8 that extends class activation mapping (CAM) to all layers of a ResNet-18 and a ResNet-18 with ACDC-net .

Original: The networks are trained from scratch on CUB-200 (Wah et al., 2011) high-resolution ( 448 × 448 ) ﬁne-grained bird classiﬁcation dataset.
Revised: The networks are trained from scratch on CUB-200 (Wah et al., 2011) high-resolution ( 448 × 448 ) ﬁne-grained bird classiﬁcation dataset.

Original: It is clearly shown that, in ACDC-net , while the class activation maps for shallow layers are inevitably noisy due to the limited receptive ﬁelds, features in deeper layers are progressively reﬁned to the discriminative regions.
Revised: It is clearly shown that, in ACDC-net , while the class activation maps for shallow layers are inevitably noisy due to the limited receptive ﬁelds, features in deeper layers are progressively reﬁned to the discriminative regions.

Original: However, in standard CNNs, due to no explicit correspondence among ﬁlters across layers, CAM can only explain the feature of the last Conv layer.
Revised: However, in standard CNNs, due to no explicit correspondence among ﬁlters across layers, CAM can only explain the feature of the last Conv layer.

Original: This shows the great potential for better interpretability with ACDC, and we will keep this as a direction of future effort.
Revised: This shows the great potential for better interpretability with ACDC, and we will keep this as a direction of future effort.

Original: A PPENDIX A I MPLEMENTATION D ETAILS
Revised: A PPENDIX A I MPLEMENTATION D ETAILS

Original: All experiments are conducted on a server with 8 Nvidia RTX 3090 graphic cards, and each has 24GB memory.
Revised: All experiments are conducted on a server with 8 Nvidia RTX 3090 graphic cards, and each has 24GB memory.

Original: Experiments with few-shot image classiﬁcation and CIFAR trained and tested on a single card.
Revised: Experiments with few-shot image classiﬁcation and CIFAR trained and tested on a single card.

Original: Experiments with ImageNet are trained with 8 cards in parallel.
Revised: Experiments with ImageNet are trained with 8 cards in parallel.

Original: The machine is also equipped with 512GB memory and two AMD EPYC 7502 CPUs.
Revised: The machine is also equipped with 512GB memory and two AMD EPYC 7502 CPUs.

Original: All experiments are performed using PyTorch (Paszke et al., 2019).
Revised: All experiments are performed using PyTorch (Paszke et al., 2019).

Original: Initialization.
Revised: Initialization.

Original: We train every ACDC network from scratch.
Revised: We train every ACDC network from scratch.

Original: We use orthogonal initialization for all ﬁlter atoms, and Kaiming normal initialization for all coefﬁcients.
Revised: We use orthogonal initialization for all ﬁlter atoms, and Kaiming normal initialization for all coefﬁcients.

Original: Training details.
Revised: Training details.

Original: All networks for CIFAR10 and CIFAR100 are trained for 350 epochs.
Revised: All networks for CIFAR10 and CIFAR100 are trained for 350 epochs.

Original: The initial learning rate is set to be 0 .
Revised: The initial learning rate is set to be 0 .

Original: The learning rate decays by a factor of 10 at the 150 -th and the 250 -th epoch.
Revised: The learning rate decays by a factor of 10 at the 150 -th and the 250 -th epoch.

Original: All ACDC networks are trained with a weight decay of 10 − 4 .
Revised: All ACDC networks are trained with a weight decay of 10 − 4 .

Original: The reported numbers are averaged over 5 runs.
Revised: The reported numbers are averaged over 5 runs.

Original: Networks for ImageNet are trained for 90 epochs.
Revised: Networks for ImageNet are trained for 90 epochs.

Original: The initial learning rate is 0 . 1 and decays by a factor of 10 every 30 epochs.
Revised: The initial learning rate is 0 . 1 and decays by a factor of 10 every 30 epochs.

Original: The weight decay for ImageNet experiments are set to be 2 .
Revised: The weight decay for ImageNet experiments are set to be 2 .

Original: 5 × 10 − 5 .
Revised: 5 × 10 − 5 .

Original: We consistently observe that lower values for weight decay yield better results on ACDC networks.
Revised: We consistently observe that lower values for weight decay yield better results on ACDC networks.

Original: We believe the reason is that the structural regularization of ACDC already reduces the risk of overﬁtting.
Revised: We believe the reason is that the structural regularization of ACDC already reduces the risk of overﬁtting.

Original: B C OMPARISONS OF A TOM - DROP
Revised: B C OMPARISONS OF A TOM - DROP

Original: The proposed atom-drop shares the similar motivation with dropout, which is introduced to improve the robustness of networks weights by randomly closing activations in the training of the the deep network.
Revised: The proposed atom-drop shares the similar motivation with dropout, which is introduced to improve the robustness of networks weights by randomly closing activations in the training of the the deep network.

Original: Atom-drop is introduced in ACDC as a method to improve the robustness of the learned atoms at each layer.
Revised: Atom-drop is introduced in ACDC as a method to improve the robustness of the learned atoms at each layer.

Original: Most importantly, since atom-drop is performed on atoms, which are network parameters, and dropout is performed on feature activations, they can be applied simultaneously.
Revised: Most importantly, since atom-drop is performed on atoms, which are network parameters, and dropout is performed on feature activations, they can be applied simultaneously.

Original: We present in Table E the performance of ACDC-net m16 with WRN-40-4 and different dropping regularizations.
Revised: We present in Table E the performance of ACDC-net m16 with WRN-40-4 and different dropping regularizations.

Original: Dropout-1d and dropout-2d correspond to randomly dropping pixels and feature channels, respectively.
Revised: Dropout-1d and dropout-2d correspond to randomly dropping pixels and feature channels, respectively.

Original: Using drop-atom alone delivers better performance comparing to both dropout-1d and dropout-2d, and drop-atom is compatible with dropout-1d and dropout-2d for better performance.
Revised: Using drop-atom alone delivers better performance comparing to both dropout-1d and dropout-2d, and drop-atom is compatible with dropout-1d and dropout-2d for better performance.

Original: [Table]
Revised: [Table]

Original: C C OMPARISONS A GAINST N ETWORK C OMPRESSION AND P RUNING M ETHODS
Revised: C C OMPARISONS A GAINST N ETWORK C OMPRESSION AND P RUNING M ETHODS

Original: We present here further comparisons against network compression and pruning methods.
Revised: We present here further comparisons against network compression and pruning methods.

Original: Performance is measured with error rates and proportion of remained parameters (both lower the better).
Revised: Performance is measured with error rates and proportion of remained parameters (both lower the better).

Original: Note that different from post-processing using compression and pruning, ACDC is primarily proposed as a structural regularization and a plug-and-play replacement to standard convolutional layers, so the networks with ACDC remain trained end-to-end.
Revised: Note that different from post-processing using compression and pruning, ACDC is primarily proposed as a structural regularization and a plug-and-play replacement to standard convolutional layers, so the networks with ACDC remain trained end-to-end.

Original: We include SNIP (Lee et al., 2018), TR (Wang et al., 2018a), and recently proposed T-Basis (Obukhov et al., 2020) and GraSP (Wang et al., 2020) into the comparisons.
Revised: We include SNIP (Lee et al., 2018), TR (Wang et al., 2018a), and recently proposed T-Basis (Obukhov et al., 2020) and GraSP (Wang et al., 2020) into the comparisons.

Original: Results shown in Figure I clearly demonstrate that ACDC can achieve even smaller parameter size with negligent scariﬁcation to accuracy.[Figure]
Revised: Results shown in Figure I clearly demonstrate that ACDC can achieve even smaller parameter size with negligent scariﬁcation to accuracy.

Original: Performance is and parameters compression terms of parameter and prediction

Original: D C OMPARISONS ON S PEED
Revised: D C OMPARISONS ON S PEED

Original: We present the comparisons on speed against some recent efﬁcient network structures in Table F.
Revised: We present the comparisons on speed against some recent efﬁcient network structures in Table F.

Original: All numbers are obtained by an average of 10,000 runs.
Revised: All numbers are obtained by an average of 10,000 runs.

Original: Each run contains a batch of 100 samples  with a resolution of 224 × 224 , which saturate the GPU cores.
Revised: Each run contains a batch of 100 samples [Figure] Performance is measured with error rates and proportion of remained parameters (both lower the better).As a plug-and-play method, ACDC can outperform network compression and pruning methods in terms of both parameter reduction rate and prediction accuracy. with a resolution of 224 × 224 , which saturate the GPU cores.

Original: While fast speed is not the primary goal of the paper, our joint subspace modeling allows comparably fast speed with the state-of-the-art efﬁcient network structures. 
Revised: While fast speed is not the primary goal of the paper [Figure] our joint subspace modeling allows comparably fast speed with the state-of-the-art efﬁcient network structures. [Figure]

Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: E A RCHITECTURE A DAPTIVITY

Revised: ACDC offers a variety of variants with varying degrees of sharing and performance.

Revised: The selection of the variants can be determined based on the factors like the size and difﬁculty of the datasets at hand.

Revised: Furthermore, the shared atom-coefﬁcients decomposition for all ACDC variants allows for easy transfer and, as a result, an adaptive model adaptation scheme.

Revised: For example, one can start training the network with coefﬁcients sharing across all layers ( ACDC-net ) with a relatively small number of atoms.

Revised: And the network capacity can be gradually increased by relating the coefﬁcient sharing to blocks only and adding more atoms.

Revised: Meanwhile, the majority of the parameters can be inherited from the earlier trained model to guarantee training efﬁciency.

Revised: To show this, we use

Revised: WideResNet (WRN-28-10) as the backbone and train an expanding model on the Tiny ImageNet dataset.

Revised: As shown in Table G, in stage 1, we start with training ACDC-net m=8 .

Revised: When further performance improvement is desired, we can relax the sharing by using the parameters obtained with ACDC-net m=8 to initialize ACDC-stage m=8 and continue training the network at stage 2.

Revised: Similarly, we can further expand the network to ACDC-stage m=12 with most of the parameter inherited from ACDC-stage m=8 .