Original: Learning unfolded networks with a cyclic group structure
Revised: Learning unfolded networks with a cyclic group structure

Revised: Editors: Sophia Sanborn, Christian Shewmake, Simone Azeglio, Arianna Di Bernardo, Nina Miolane

Original: Abstract
Revised: Abstract

Original: Deep neural networks lack straightforward ways to incorporate domain knowledge and are notoriously considered black boxes.
Revised: Deep neural networks lack straightforward ways to incorporate domain knowledge and are notoriously considered black boxes.

Original: Prior works attempted to inject domain knowledge into architectures implicitly through data augmentation.
Revised: Prior works attempted to inject domain knowledge into architectures implicitly through data augmentation.

Original: Building on recent advances on equivariant neural networks, we propose networks that explicitly encode domain knowledge, speciﬁcally equivariance with respect to rotations.
Revised: Building on recent advances on equivariant neural networks, we propose networks that explicitly encode domain knowledge, speciﬁcally equivariance with respect to rotations.

Original: By using unfolded architectures, a rich framework that originated from sparse coding and has theoretical guarantees, we present interpretable networks with sparse activations.
Revised: By using unfolded architectures, a rich framework that originated from sparse coding and has theoretical guarantees, we present interpretable networks with sparse activations.

Original: The equivariant unfolded networks compete favorably with baselines, with only a fraction of their parameters, as showcased on (rotated)
Revised: The equivariant unfolded networks compete favorably with baselines, with only a fraction of their parameters, as showcased on (rotated)

Original: MNIST and CIFAR-10.
Revised: MNIST and CIFAR-10.

Original: 1. Introduction
Revised: 1. Introduction

Original: While advances in deep neural networks have yielded groundbreaking results in various ﬁelds such as computer vision (Redmon and Farhadi, 2017; Pavlakos et al., 2017; Mildenhall et al., 2020), natural language processing (Devlin et al., 2019; Brown et al., 2020), and their intersection (Radford et al., 2021), interpreting their structure and explaining their performance is not straightforward.
Revised: While advances in deep neural networks have yielded groundbreaking results in various ﬁelds such as computer vision (Redmon and Farhadi, 2017; Pavlakos et al., 2017; Mildenhall et al., 2020), natural language processing (Devlin et al., 2019; Brown et al., 2020), and their intersection (Radford et al., 2021), interpreting their structure and explaining their performance is not straightforward.

Original: At the same time, applying deep learning techniques to novel ﬁelds comes with challenges, as it is not clear how to integrate domain knowledge into existing architectures.
Revised: At the same time, applying deep learning techniques to novel ﬁelds comes with challenges, as it is not clear how to integrate domain knowledge into existing architectures.

Original: In this work, we propose a novel architecture to address both of these shortcomings at the same time, leading to an interpretable architecture that respects expert knowledge.
Revised: In this work, we propose a novel architecture to address both of these shortcomings at the same time, leading to an interpretable architecture that respects expert knowledge.

Original: Convolutional Neural Networks (CNNs) are equivariant in their representations with respect to translation; however, there are other operators that is natural for image models to be equivariant to, such as rotations.
Revised: Convolutional Neural Networks (CNNs) are equivariant in their representations with respect to translation; however, there are other operators that is natural for image models to be equivariant to, such as rotations.

Original: While data augmentation techniques have been used to model equivariances they require large amounts of data and increase the computational demands for training, frequently by an order of magnitude.
Revised: While data augmentation techniques have been used to model equivariances they require large amounts of data and increase the computational demands for training, frequently by an order of magnitude.

Original: At the same time, if we know the desired equivariances for a speciﬁc application, investing computational resources to re-learn these equivariances is wasteful.
Revised: At the same time, if we know the desired equivariances for a speciﬁc application, investing computational resources to re-learn these equivariances is wasteful.

Original: This was also acknowledged by Dieleman et al.
Revised: This was also acknowledged by Dieleman et al.

Original: (2016) and Cohen and Welling (2016) who concurently introduced CNN frameworks that incorporate rotated ﬁlters in order to create equivariant representations with respect to rotations; however both works were limited to elementary rotations and the reﬂections of
Revised: (2016) and Cohen and Welling (2016) who concurently introduced CNN frameworks that incorporate rotated ﬁlters in order to create equivariant representations with respect to rotations; however both works were limited to elementary rotations and the reﬂections of

Original: D 4 .
Revised: D 4 .

Original: Follow up works by diﬀerent authors extended the ideas to vector ﬁelds (Marcos et al., 2017), applied rotations directly on the sphere to avoid interpolation artifacts (Esteves et al., 2020), and incorporated harmonic functions to model equivariance with respect to arbitrary rotations (Worrall et al., 2017).
Revised: Follow up works by diﬀerent authors extended the ideas to vector ﬁelds (Marcos et al., 2017), applied rotations directly on the sphere to avoid interpolation artifacts (Esteves et al., 2020), and incorporated harmonic functions to model equivariance with respect to arbitrary rotations (Worrall et al., 2017).

Original: There have been several attempts to tackle interpretability, ranging from prototype learning approaches (Chen et al., 2019; Arik and Pﬁster, 2020) that learn prototypical parts for each class in a classiﬁcation task, to post-hoc methods (Ribeiro et al., 2016) that reverse engineer predictions from arbitrary classiﬁers.
Revised: There have been several attempts to tackle interpretability, ranging from prototype learning approaches (Chen et al., 2019; Arik and Pﬁster, 2020) that learn prototypical parts for each class in a classiﬁcation task, to post-hoc methods (Ribeiro et al., 2016) that reverse engineer predictions from arbitrary classiﬁers.

Original: In this work, we focus on model-based networks (Shlezinger et al., 2020): in these approaches, interpretability is directly encoded into the model by constructing a neural network to mimic the steps of an optimization algorithm.
Revised: In this work, we focus on model-based networks (Shlezinger et al., 2020): in these approaches, interpretability is directly encoded into the model by constructing a neural network to mimic the steps of an optimization algorithm.

Original: First introduced by Gregor and LeCun (2010), unrolled neural networks have inspired a vast array of works, ranging from theoretical contributions (Nguyen et al., 2019;
Revised: First introduced by Gregor and LeCun (2010), unrolled neural networks have inspired a vast array of works, ranging from theoretical contributions (Nguyen et al., 2019;

Original: Arora et al., 2015) to state-of-the-art results (Tolooshams et al., 2020).
Revised: Arora et al., 2015) to state-of-the-art results (Tolooshams et al., 2020).

Original: In this work, we propose an unrolled architecture, inspired from algorithms for sparse coding, whose layer weights employ a cyclic group structure to achieve rotational equivariance.
Revised: In this work, we propose an unrolled architecture, inspired from algorithms for sparse coding, whose layer weights employ a cyclic group structure to achieve rotational equivariance.

Original: This allows us to signiﬁcantly reduce the number of trainable ﬁlters compared to baseline architectures.
Revised: This allows us to signiﬁcantly reduce the number of trainable ﬁlters compared to baseline architectures.

Original: Concretely, our contributions can be summarized as follows:
Revised: Concretely, our contributions can be summarized as follows:

Original: We propose an unrolled architecture, modeled after sparse coding, that is by construction interpretable and equivariant with respect to rotations, 2.
Revised: We propose an unrolled architecture, modeled after sparse coding, that is by construction interpretable and equivariant with respect to rotations, 2.

Original: we showcase its eﬃcacy in learning ﬁlters that are governed by a cyclic group structure with a fraction of the learnable parameters, and 3.
Revised: we showcase its eﬃcacy in learning ﬁlters that are governed by a cyclic group structure with a fraction of the learnable parameters, and 3.

Original: we evaluate the proposed architecture on MNIST, rotated MNIST, and CIFAR-10, standard benchmarks for rotationally equivariant architectures and demonstrate its performance.
Revised: we evaluate the proposed architecture on MNIST, rotated MNIST, and CIFAR-10, standard benchmarks for rotationally equivariant architectures and demonstrate its performance.

Original: We make the code for our experiments and architectures available online on GitHub athttps://github.com/manosth/cyclical_groups .
Revised: We make the code for our experiments and architectures available online on GitHub at

Original: 2. Background
Revised: 2. Background

Original: Equivariance.
Revised: Equivariance.

Original: In lay terms, an operator is equivariant with respect to some actions if it behaves in a predictable manner under them.
Revised: In lay terms, an operator is equivariant with respect to some actions if it behaves in a predictable manner under them.

Original: Formally, we say that an operator f is equivariant with respect to a family of actions T if, for any T ∈ T it holds that
Revised: Formally, we say that an operator f is equivariant with respect to a family of actions T if, for any T ∈ T it holds that

Original: [Equation]
Revised: [Equation]

Original: for some other transform T (cid:48) .
Revised: for some other transform T (cid:48) .

Original: Note that, in general, T and T (cid:48) belong in diﬀerent families as T acts on the input space (Dom( f )) and T (cid:48) on the encoding space (Im( f )).
Revised: Note that, in general, T and T (cid:48) belong in diﬀerent families as T acts on the input space (Dom( f )) and T (cid:48) on the encoding space (Im( f )).

Original: Constant functions are trivially equivariant, and a special case, invariance , arises when T (cid:48) is the identity map.
Revised: Constant functions are trivially equivariant, and a special case, invariance , arises when T (cid:48) is the identity map.

Original: Note that convolution is not equivariant to rotation (Cohen and Welling, 2016; Dieleman et al., 2016); instead, the two are related by
Revised: Note that convolution is not equivariant to rotation (Cohen and Welling, 2016; Dieleman et al., 2016); instead, the two are related by

Original: [Equation]
Revised: [Equation]

Original: where x denotes an input image, R is a rotation, and h is the convolving ﬁlter.
Revised: where x denotes an input image, R is a rotation, and h is the convolving ﬁlter.

Original: Cyclic groups.
Revised: Cyclic groups.

Original: We call a ﬁnite group G a cyclic group if there exists a generating element g such that
Revised: We call a ﬁnite group G a cyclic group if there exists a generating element g such that

Original: [Equation]
Revised: [Equation]

Original: where e denotes the identity element.
Revised: where e denotes the identity element.

Original: We denote the family of cyclic groups as G ; several groups belong to this family, with most notable being D 4 , the symmetry group of the square.
Revised: We denote the family of cyclic groups as G ; several groups belong to this family, with most notable being D 4 , the symmetry group of the square.

Original: Cyclic groups are of interest for our model since all elements can be identiﬁed by the generator g .
Revised: Cyclic groups are of interest for our model since all elements can be identiﬁed by the generator g .

Original: This will enable us, in Section 3 to signiﬁcantly reduce the trainable parameters of our networks, while retaining (and even improving) performance and interpretability.
Revised: This will enable us, in Section 3 to signiﬁcantly reduce the trainable parameters of our networks, while retaining (and even improving) performance and interpretability.

Original: Unrolled sparse autoencoders.
Revised: Unrolled sparse autoencoders.

Original: In their most general form, unrolled networks temporally unroll the steps of optimization algorithms, mapping algorithm iterations to network layers.
Revised: In their most general form, unrolled networks temporally unroll the steps of optimization algorithms, mapping algorithm iterations to network layers.

Original: In that way, the output of the neural network can be interpreted as the output of the optimization algorithm, with theoretical guarantees under certain assumptions.
Revised: In that way, the output of the neural network can be interpreted as the output of the optimization algorithm, with theoretical guarantees under certain assumptions.

Original: The
Revised: The

Original: Iterative Soft Thresholding Algorithn (ISTA), an algorithm for sparse coding, has inspired several architectures (Simon and Elad, 2019; Sulam et al., 2020; Tolooshams et al., 2021), due to the desirability of sparse representations.
Revised: Iterative Soft Thresholding Algorithn (ISTA), an algorithm for sparse coding, has inspired several architectures (Simon and Elad, 2019; Sulam et al., 2020; Tolooshams et al., 2021), due to the desirability of sparse representations.

Original: Within that framework, the representation at layer l + 1 is given by
Revised: Within that framework, the representation at layer l + 1 is given by

Original: [Equation]
Revised: [Equation]

Original: where x is the original input, z ( l ) is the representation at the previous layer, W l are the weights of layer l , α is a constant such that 1 α ≥ σ max ( W Tl W l ), and S λ is the soft thresholding operator deﬁned as
Revised: where x is the original input, z ( l ) is the representation at the previous layer, W l are the weights of layer l , α is a constant such that 1 α ≥ σ max ( W Tl W l ), and S λ is the soft thresholding operator deﬁned as

Original: [Equation]
Revised: [Equation]

Original: If the weights of all the L layers are equal, i.e. W 1 = . . .
Revised: If the weights of all the L layers are equal, i.e. W 1 = . . .

Original: =
Revised: =

Original: W L , we call the network tied .
Revised: W L , we call the network tied .

Original: As a ﬁnal remark, Equation (3) can be rewriten as
Revised: As a ﬁnal remark, Equation (3) can be rewriten as

Original: [Equation]
Revised: [Equation]

Original: where we let W z
Revised: where we let W z

Original: = ( I − α W Tl W l ) and W x = α W Tl , which can be interpreted as a residual network (He et al., 2016), with a residual connection to the input.
Revised: = ( I − α W Tl W l ) and W x = α W Tl , which can be interpreted as a residual network (He et al., 2016), with a residual connection to the input.

Original: 3. Equivariant autoencoders
Revised: 3. Equivariant autoencoders

Original: We will combine the ideas of cyclic groups and unrolled autoencoders from Section 2 to create an equivariant unrolled architecture, where the bulk of the weights in each layer are cyclic rotations of some “basis” weights .
Revised: We will combine the ideas of cyclic groups and unrolled autoencoders from Section 2 to create an equivariant unrolled architecture, where the bulk of the weights in each layer are cyclic rotations of some “basis” weights .

Original: Let R θ denote a two-dimensional rotation by θ degrees.
Revised: Let R θ denote a two-dimensional rotation by θ degrees.

Original: If 360 mod θ = 

Revised: If 360 mod θ =

Original: and we let k = 360 ÷ θ , then the group deﬁned as
Revised: and we let k = 360 ÷ θ , then the group deﬁned as

Original: [Equation]
Revised: [Equation]

Original: R θ

Original: [Equation]

Original: [Equation]

Original: is a cyclic group generated by the generator g
Revised: is a cyclic group generated by the generator g

Original: = R θ .
Revised: = R θ .

Original: This construction allows us to extend this framework, in future work, to arbitrary linear operators and the possibility to learn the generator g , leading to data-driven approaches for the cyclic group structure.
Revised: This construction allows us to extend this framework, in future work, to arbitrary linear operators and the possibility to learn the generator g , leading to data-driven approaches for the cyclic group structure.

Original: Regardless, the weights of layer l satisfy
Revised: Regardless, the weights of layer l satisfy

Original: [Equation]
Revised: [Equation]

Original: where k is the number of rotations we are considering, per layer.
Revised: where k is the number of rotations we are considering, per layer.

Original: Note that Equation (7) shows a single basis weight per layer; this can be readily extended to having m basis weights w l 1 , . . .
Revised: Note that Equation (7) shows a single basis weight per layer; this can be readily extended to having m basis weights w l 1 , . . .

Original: , w l m along with the k rotations for each of them, as is done in Section 4.
Revised: , w l m along with the k rotations for each of them, as is done in Section 4.

Original: The networks are constructed analogously to Sulam et al.
Revised: The networks are constructed analogously to Sulam et al.

Original: ; Tolooshams et al.
Revised: ; Tolooshams et al.

Original: but with one (or m ) learnable ﬁlter(s) per layer.
Revised: but with one (or m ) learnable ﬁlter(s) per layer.

Original: During training, the remaining ﬁlters are constructed and errors are backpropagated through all ﬁlters.
Revised: During training, the remaining ﬁlters are constructed and errors are backpropagated through all ﬁlters.

Original: The experiments of Sectionuse untied networks for improved performance.
Revised: The experiments of Sectionuse untied networks for improved performance.

Original: 4. Experiments
Revised: 4. Experiments

Original: We present experiments on MNIST, rotated MNIST, and CIFAR-10, standard benchmarks for equivariant architectures, to evaluate the architecture introduced in Section 3.
Revised: We present experiments on MNIST, rotated MNIST, and CIFAR-10, standard benchmarks for equivariant architectures, to evaluate the architecture introduced in Section 3.

Original: We used batch normalization (Ioﬀe and Szegedy, 2015) in all of our architectures, following best practices.
Revised: We used batch normalization (Ioﬀe and Szegedy, 2015) in all of our architectures.

Original: The normalization was applied at the output of every layer, except the last.
Revised: The normalization was applied at the output of every layer, except the last.

Original: FISTA (Beck and Teboulle, 2009), a faster version of ISTA, is used for faster convergence of the sparse coding.
Revised: FISTA (Beck and Teboulle, 2009), a faster version of ISTA, is used for faster convergence of the sparse coding.

Original: All of our networks use L = 4 layers, λ (the parameter of S λ ) is set to 0 .
Revised: All of our networks use L = 4 layers, λ (the parameter of S λ ) is set to 0 .

Original: 5, and the stepsize of FISTA in (3) is set to α = 0 .
Revised: 5, and the stepsize of FISTA in (3) is set to α = 0 .

Original: A summary of our main results is given in Table 1.
Revised: A summary of our main results is given in Table 1.

Original: We test three models: an unrolled sparse networkthat is considered as a baseline; R 90CNN, an equivariant unfolded network with the elementary rotations; and R 60 -CNN, with 60 ° rotations.
Revised: We test three models: an unrolled sparse network as a baseline; R 90 -CNN, an equivariant unfolded network with the elementary rotations; and R 60 -CNN, with 60 ° rotations.

Original: Every architecture has 60 ﬁlters per layer; in the case of the baseline, all 60 are learnable, R 90 -CNN has 15 learnable ﬁlters per layer the rest are constructed as elementary rotations,and R 60 -CNN has only 10 learnable ﬁlters per layer with the rest being their 60 ° rotations. The learned ﬁlters are 7 × 7 in the case of MNIST and 8 × 8 in the
Revised: Every architecture has 60 ﬁlters per layer; in the case of the baseline, all 60 are learnable, R 90CNN has 15 learnable ﬁlters per layer the rest are constructed as elementary rotations,

Original: Method
Revised: Method

Revised: MNIST rot-MNIST

Revised: CIFAR

Original: [Table]
Revised: [Table]

Original:  case of CIFAR-10.
Revised: and R 60 -CNN has only 10 learnable ﬁlters per layer with the rest being their 60 ° rotations. The learned ﬁlters are 7 × 7 in the case of MNIST and 8 × 8 in the case of CIFAR-10.

Original: A visualization of R 60 -CNN’s learned ﬁlters when trained on CIFARis show in Figure 1.
Revised: A visualization of R 60 -CNN’s learned ﬁlters when trained on CIFAR-10 is show in Figure 1.

Original: MNIST.
Revised: MNIST.

Original: We ﬁrst evaluate all three models on MNIST, a relatively easy dataset, and ﬁnd that all of them performed similarly.
Revised: We ﬁrst evaluate all three models on MNIST, a relatively easy dataset, and ﬁnd that all of them performed similarly.

Original: However, we note that R 90 -CNN has only 14 the trainable ﬁlters of the baseline, as the rest are generated as rotations of the basis weights;
Revised: However, we note that R 90 -CNN has only 14 the trainable ﬁlters of the baseline, as the rest are generated as rotations of the basis weights;

Original: R 60 -CNNH has only 16 .
Revised: R 60 -CNNH has only 16 .

Original: When evaluating the architectures on the rotated MNIST, a harder dataset, we observe that the R 90 -CNN, with a fraction of the parameters of the baseline model leads to the best performance.
Revised: When evaluating the architectures on the rotated MNIST, a harder dataset, we observe that the R 90 -CNN, with a fraction of the parameters of the baseline model leads to the best performance.

Original: This showcases that the encoded equivariance in the representation is actually beneﬁcial for the classiﬁcation of the inputs.
Revised: This showcases that the encoded equivariance in the representation is actually beneﬁcial for the classiﬁcation of the inputs.

Original: Evaluating the architectures on the rotated MNIST produces similar results and R 90CNN outperforms the baseline.
Revised: Evaluating the architectures on the rotated MNIST produces similar results and R 90CNN outperforms the baseline.

Original: However, R 90 -CNN displays reduced performance on this dataset; we speculate that this is because of interpolation artifacts, which was one of the motivating reasons for Esteves et al.
Revised: However, R 90 -CNN displays reduced performance on this dataset; we speculate that this is because of interpolation artifacts, which was one of the motivating reasons for Esteves et al.

Original: (2020) to consider rotations on the sphere.
Revised: (2020) to consider rotations on the sphere.

Original: To further demonstrate the beneﬁt of the equivariant unfolded networks, we trained dense variants (where the ﬁlters are now 28 × 28 images) of the three models on MNIST, and
Revised: To further demonstrate the beneﬁt of the equivariant unfolded networks, we trained dense variants (where the ﬁlters are now 28 × 28 images) of the three models on MNIST, and

Original: [Table]
Revised: [Table]

Original: evaluate their performance on the rotated dataset.
Revised: evaluate their performance on the rotated dataset.

Original: The results are presented in Table 2.
Revised: The results are presented in Table 2.

Original: This experiment showcases the generalization capabilities of the equivariant unrolled networks.
Revised: This experiment showcases the generalization capabilities of the equivariant unrolled networks.

Original: While we see similar performance across all models on the trained dataset, we see that both the equivariant models are able to generalize better than the baseline.
Revised: While we see similar performance across all models on the trained dataset, we see that both the equivariant models are able to generalize better than the baseline.

Original: Dense architectures were chosen for this experiment to highlight the distribution shift when evaluating on rot-MNIST.
Revised: Dense architectures were chosen for this experiment to highlight the distribution shift when evaluating on rot-MNIST.

Original: CIFAR-10.
Revised: CIFAR-10.

Original: When training on CIFAR-10, a comparetively harder dataset, we found that both the equivariant models outperform the baseline, even if marginally, with only a fraction of the learnable ﬁlters per layer.
Revised: When training on CIFAR-10, a comparetively harder dataset, we found that both the equivariant models outperform the baseline, even if marginally, with only a fraction of the learnable ﬁlters per layer.

Original: Moreover, upon further examination, the ﬁlters of R 90 -CNN and R 60 -CNN exhibit a topographic structure by construction as ﬁlters are generated as rotations of one another.
Revised: Moreover, upon further examination, the ﬁlters of R 90 -CNN and R 60 -CNN exhibit a topographic structure by construction as ﬁlters are generated as rotations of one another.

Original: That topographic structure is not present in the ﬁlters of the baseline model which doesn’t incorporate domain knowledge regarding rotations.
Revised: That topographic structure is not present in the ﬁlters of the baseline model which doesn’t incorporate domain knowledge regarding rotations.

Original: The learned ﬁlters of the diﬀerent models, when trained on CIFAR-10, are presented in
Revised: The learned ﬁlters of the diﬀerent models, when trained on CIFAR-10, are presented in

Original: Appendix A.
Revised: Appendix A.

Original: 5. Conclusions and future work
Revised: 5. Conclusions and future work

Original: In this work we introduced equivariant unrolled networks, where the ﬁlters of each layer are constructed as discrete rotations of one another.
Revised: In this work we introduced equivariant unrolled networks, where the ﬁlters of each layer are constructed as discrete rotations of one another.

Original: By exploiting this cyclical group structure, we are able to facilitate training and maintain performance while drastically reducing the number of trainable parameters of the model.
Revised: By exploiting this cyclical group structure, we are able to facilitate training and maintain performance while drastically reducing the number of trainable parameters of the model.

Original: We experimentally validated the equivariant unrolled networks against a baseline network and showed comparable, and even favorable, results, with only a fraction of the learnable parameters on multiple datasets that are used as benchmarks for equivariant architectures.
Revised: We experimentally validated the equivariant unrolled networks against a baseline network and showed comparable, and even favorable, results, with only a fraction of the learnable parameters on multiple datasets that are used as benchmarks for equivariant architectures.

Original: Finally, we identify learning the generator g from data, as hinted in Section 2, an exciting avenue for future work.
Revised: Finally, we identify learning the generator g from data, as hinted in Section 2, an exciting avenue for future work.

Original: Acknowledgements
Revised: Acknowledgements

Original: Demba Ba was supported by the National Science Foundation under Grant number #S5206,
Revised: Demba Ba was supported by the National Science Foundation under Grant number #S5206,

Original: PO 
Revised: PO 560825.

Original: Appendix A. Learned ﬁlters on CIFAR- 
Revised: Appendix A. Learned ﬁlters on CIFAR-

Original: We present ﬁlters learned on CIFAR-10 (without whitening) by the three architectures.
Revised: We present ﬁlters learned on CIFAR-10 (without whitening) by the three architectures.

Original: We choose to present ﬁlters from the ﬁrst layer, as those resemble edge detectors the most and thus are more straightforward to reason about.
Revised: We choose to present ﬁlters from the ﬁrst layer, as those resemble edge detectors the most and thus are more straightforward to reason about.

Original: While all models seem to be learning similarly-looking ﬁlters, the equivariant models do not need to “waste” computation on learning diﬀerent orientations.
Revised: While all models seem to be learning similarly-looking ﬁlters, the equivariant models do not need to “waste” computation on learning diﬀerent orientations.

Original: Indeed, if we observe Figure 2, the ﬁrst ﬁlter from the top and the third from the bottom of the ﬁrst column seem to be rotated versions of one another.
Revised: Indeed, if we observe Figure 2, the ﬁrst ﬁlter from the top and the third from the bottom of the ﬁrst column seem to be rotated versions of one another.

Original: In stark contrast, the third column of Figure 3 seems to learning the elementary rotations of that same ﬁlter, without investing resources on learning that information from the data.
Revised: In stark contrast, the third column of Figure 3 seems to learning the elementary rotations of that same ﬁlter, without investing resources on learning that information from the data.

Original: That is also observed in the ﬁrst column of Figure 4, which has even more rotated versions of that same ﬁlter.
Revised: That is also observed in the ﬁrst column of Figure 4, which has even more rotated versions of that same ﬁlter.