Original: B UILDING D EEP E QUIVARIANT C APSULE N ETWORKS
Revised: B UILDING D EEP E QUIVARIANT C APSULE N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Capsule networks are constrained by the parameter-expensive nature of their layers, and the general lack of provable equivariance guarantees.
Revised: Capsule networks are constrained by the parameter-expensive nature of their layers, and the general lack of provable equivariance guarantees.

Original: We present a variation of capsule networks that aims to remedy this.
Revised: We present a variation of capsule networks that aims to remedy this.

Original: We identify that learning all pairwise part-whole relationships between capsules of successive layers is inefﬁcient.
Revised: We identify that learning all pairwise part-whole relationships between capsules of successive layers is inefﬁcient.

Original: Further, we also realise that the choice of prediction networks and the routing mechanism are both key to equivariance.
Revised: Further, we also realise that the choice of prediction networks and the routing mechanism are both key to equivariance.

Original: Based on these, we propose an alternative framework for capsule networks that learns to projectively encode the manifold of pose-variations, termed the space-of-variation (SOV), for every capsule-type of each layer.
Revised: Based on these, we propose an alternative framework for capsule networks that learns to projectively encode the manifold of pose-variations, termed the space-of-variation (SOV), for every capsule-type of each layer.

Original: This is done using a trainable, equivariant function deﬁned over a grid of group-transformations.
Revised: This is done using a trainable, equivariant function deﬁned over a grid of group-transformations.

Original: Thus, the prediction-phase of routing involves projection into the SOV of a deeper capsule using the corresponding function.
Revised: Thus, the prediction-phase of routing involves projection into the SOV of a deeper capsule using the corresponding function.

Original: As a speciﬁc instantiation of this idea, and also in order to reap the beneﬁts of increased parameter-sharing, we use type-homegenous group-equivariant convolutions of shallower capsules in this phase.
Revised: As a speciﬁc instantiation of this idea, and also in order to reap the beneﬁts of increased parameter-sharing, we use type-homogeneous group-equivariant convolutions of shallower capsules in this phase.

Original: We also introduce an equivariant routing mechanism based on degree-centrality.
Revised: We also introduce an equivariant routing mechanism based on degree-centrality.

Original: We show that this particular instance of our general model is equivariant, and hence preserves the compositional representation of an input under transformations.
Revised: We show that this particular instance of our general model is equivariant, and hence preserves the compositional representation of an input under transformations.

Original: We conduct several experiments on standard object-classiﬁcation datasets that showcase the increased transformation-robustness, as well as general performance, of our model to several capsule baselines.
Revised: We conduct several experiments on standard object-classiﬁcation datasets that showcase the increased transformation-robustness, as well as general performance, of our model to several capsule baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The hierarchical component-structure of visual objects motivates their description as instances of class-dependent spatial grammars.
Revised: The hierarchical component-structure of visual objects motivates their description as instances of class-dependent spatial grammars.

Original: The production-rules of such grammars specify this structure by laying out valid type-combinations for components of an object, their inter-geometry, as well as the behaviour of these with respect to transformations on the input.
Revised: The production-rules of such grammars specify this structure by laying out valid type-combinations for components of an object, their inter-geometry, as well as the behaviour of these with respect to transformations on the input.

Original: A system that aims to truly understand a visual scene must accurately learn such grammars for all constituent objects - in effect, learning their aggregational structures.
Revised: A system that aims to truly understand a visual scene must accurately learn such grammars for all constituent objects - in effect, learning their aggregational structures.

Original: One means of doing so is to have the internal representation of a model serve as a component-parsing of an input across several semantic resolutions.
Revised: One means of doing so is to have the internal representation of a model serve as a component-parsing of an input across several semantic resolutions.

Original: Further, in order to mimic latent compositionalities in objects, such a representation must be reﬂective of detected strengths of possible spatial relationships.
Revised: Further, in order to mimic latent compositionalities in objects, such a representation must be reﬂective of detected strengths of possible spatial relationships.

Original: A natural structure for such a representation is a parse-tree whose nodes denote components, and whose weighted parent-child edges denote the strengths of detected aggregational relationships.
Revised: A natural structure for such a representation is a parse-tree whose nodes denote components, and whose weighted parent-child edges denote the strengths of detected aggregational relationships.

Original: Capsule networks (Hinton et al., 2011), (Sabour et al., 2017) are a family of deep neural networks that aim to build such distributed, spatially-aware representations in a multi-class setting.
Revised: Capsule networks (Hinton et al., 2011), (Sabour et al., 2017) are a family of deep neural networks that aim to build such distributed, spatially-aware representations in a multi-class setting.

Original: Each layer of a capsule network represents and detects instances of a set of components (of a visual scene) at a particular semantic resolution.
Revised: Each layer of a capsule network represents and detects instances of a set of components (of a visual scene) at a particular semantic resolution.

Original: It does this by using vector-valued activations, termed ’capsules’.
Revised: It does this by using vector-valued activations, termed ’capsules’.

Original: Each capsule is meant to be interpreted as being representative of a set of generalised pose-coordinates for a visual object.
Revised: Each capsule is meant to be interpreted as being representative of a set of generalised pose-coordinates for a visual object.

Original: Each layer consists of capsules of several types that may be instantiated at all spatial locations depending on the nature of the image.
Revised: Each layer consists of capsules of several types that may be instantiated at all spatial locations depending on the nature of the image.

Original: Thus, given an image, a capsule network provides a description of its components at various ’levels’ of semantics.
Revised: Thus, given an image, a capsule network provides a description of its components at various ’levels’ of semantics.

Original: In order that this distributed representation across layers be an accurate component-parsing of a visual scene, and capture meaningful and inherent spatial relationships, deeper capsules are constructed from shallower capsules using a mechanism that combines backpropagation-based learning, and consensus-based heuristics.
Revised: In order that this distributed representation across layers be an accurate component-parsing of a visual scene, and capture meaningful and inherent spatial relationships, deeper capsules are constructed from shallower capsules using a mechanism that combines backpropagation-based learning, and consensus-based heuristics.

Original: Brieﬂy, the mechanism of creating deeper capsules from a set of shallower capsules is as follows.
Revised: Brieﬂy, the mechanism of creating deeper capsules from a set of shallower capsules is as follows.

Original: Each deeper capsule of a particular type receives a set of predictions for its pose from a local pool of shallower capsules.
Revised: Each deeper capsule of a particular type receives a set of predictions for its pose from a local pool of shallower capsules.

Original: This happens by using a set of trainable neural networks that the shallower capsules are given as input into.
Revised: This happens by using a set of trainable neural networks that the shallower capsules are given as input into.

Original: These networks can be interpreted as aiming to capture possible part-whole relationships between the corresponding deeper and shallower capsules.
Revised: These networks can be interpreted as aiming to capture possible part-whole relationships between the corresponding deeper and shallower capsules.

Original: The predictions thus obtained are then combined in a manner that ensures that the result reﬂects agreement among them.
Revised: The predictions thus obtained are then combined in a manner that ensures that the result reﬂects agreement among them.

Original: This is so that capsules are activated only when their component-capsules are in the right spatial relationship to form an instance of the object-type it represents.
Revised: This is so that capsules are activated only when their component-capsules are in the right spatial relationship to form an instance of the object-type it represents.

Original: The agreement-based aggregation described just now is termed ’routing’.
Revised: The agreement-based aggregation described just now is termed ’routing’.

Original: Multiple routing algorithms exist, for example
Revised: Multiple routing algorithms exist, for example

Original: dynamic routing (Sabour et al., 2017), EM-routing (Hinton et al., 2018), SVD-based routing (Bahadori, 2018), and routing based on a clustering-like objective function (Wang & Liu, 2018).
Revised: dynamic routing (Sabour et al., 2017), EM-routing (Hinton et al., 2018), SVD-based routing (Bahadori, 2018), and routing based on a clustering-like objective function (Wang & Liu, 2018).

Original: These are based on differing notions of consensus, and consequently affect the capsule-decomposition of an input differently.
Revised: These are based on differing notions of consensus, and consequently affect the capsule-decomposition of an input differently.

Original: Based on their explicit learning of compositional structures, capsule networks can be seen as an alternative (to CNNs) for better learning of compositional representations.
Revised: Based on their explicit learning of compositional structures, capsule networks can be seen as an alternative (to CNNs) for better learning of compositional representations.

Original: Indeed, CNN-based models do not have an inherent mechanism to explicitly learn or use spatial relationships in a visual scene.
Revised: Indeed, CNN-based models do not have an inherent mechanism to explicitly learn or use spatial relationships in a visual scene.

Original: Further, the common use of layers that enforce local transformation-invariance, such as pooling, further limit their ability to accurately detect compositional structures by allowing for relaxations in otherwise strict spatial relations (Hinton et al., 2011).
Revised: Further, the common use of layers that enforce local transformation-invariance, such as pooling, further limit their ability to accurately detect compositional structures by allowing for relaxations in otherwise strict spatial relations (Hinton et al., 2011).

Original: Thus, despite some manner of hierarchical learning - as seen in their layers capturing simpler to more complex features as a function of depth - CNNs do not form the ideal representational model we seek.
Revised: Thus, despite some manner of hierarchical learning - as seen in their layers capturing simpler to more complex features as a function of depth - CNNs do not form the ideal representational model we seek.

Original: It is our belief that capsule-based models may serve us better in this regard.
Revised: It is our belief that capsule-based models may serve us better in this regard.

Original: This much said, research in capsule networks is still in its infancy, and several issues have to be overcome before capsule networks can become universally applicable like CNNs.
Revised: This much said, research in capsule networks is still in its infancy, and several issues have to be overcome before capsule networks can become universally applicable like CNNs.

Original: We focus on two of these that we consider as fundamental to building better capsule network models.
Revised: We focus on two of these that we consider as fundamental to building better capsule network models.

Original: First, most capsule-network models, in their current form, do not scale well to deep architectures.
Revised: First, most capsule-network models, in their current form, do not scale well to deep architectures.

Original: A signiﬁcant factor is the fact that all pair-wise relationships between capsules of two layers (upto a local pool) are explicitly modelled by a unique neural network.
Revised: A signiﬁcant factor is the fact that all pair-wise relationships between capsules of two layers (upto a local pool) are explicitly modelled by a unique neural network.

Original: Thus, for a ’convolutional capsule’ layer - the number of trainable neural networks depends on the product of the spatial extent of the windowing and the product of the number of capsule-types of each the two layers.
Revised: Thus, for a ’convolutional capsule’ layer - the number of trainable neural networks depends on the product of the spatial extent of the windowing and the product of the number of capsule-types of each the two layers.

Original: We argue that this design is not only expensive, but also inefﬁcient.
Revised: We argue that this design is not only expensive, but also inefﬁcient.

Original: Given two successive capsule-layers, not all pairs of capsule-types have signiﬁcant relationships.
Revised: Given two successive capsule-layers, not all pairs of capsule-types have signiﬁcant relationships.

Original: This is due to them either representing object-components that are part of different classes, or being just incompatible in compositional structures.
Revised: This is due to them either representing object-components that are part of different classes, or being just incompatible in compositional structures.

Original: The consequences of this inefﬁciency go beyond poor scalability.
Revised: The consequences of this inefﬁciency go beyond poor scalability.

Original: For example, due to the large number of prediction-networks in this design, only simple functions - often just matrices - are used to model part-whole relationships.
Revised: For example, due to the large number of prediction-networks in this design, only simple functions - often just matrices - are used to model part-whole relationships.

Original: While building deep capsule networks, such a linear inductive bias can be inaccurate in layers where complex objects are represented.
Revised: While building deep capsule networks, such a linear inductive bias can be inaccurate in layers where complex objects are represented.

Original: Thus, for the purpose of building deeper architectures, as well as more expressive layers, this inefﬁciency in the prediction phase must be handled.
Revised: Thus, for the purpose of building deeper architectures, as well as more expressive layers, this inefﬁciency in the prediction phase must be handled.

Original: The second issue with capsule networks is more theoretical, but nonetheless has implications in practice.
Revised: The second issue with capsule networks is more theoretical, but nonetheless has implications in practice.

Original: This is the lack, in general, of theoretical guarantees on equivariance.
Revised: This is the lack, in general, of theoretical guarantees on equivariance.

Original: Most capsule networks only use intuitive heuristics to learn transformation-robust spatial relations among components.
Revised: Most capsule networks only use intuitive heuristics to learn transformation-robust spatial relations among components.

Original: This is acceptable, but not ideal.
Revised: This is acceptable, but not ideal.

Original: A capsule network model that can detect compositionalities in a provablyinvariant manner are more useful, and more in line with the basic motivations for capsules.
Revised: A capsule network model that can detect compositionalities in a provablyinvariant manner are more useful, and more in line with the basic motivations for capsules.

Original: Both of the above issues are remedied in the following description of our model.
Revised: Both of the above issues are remedied in the following description of our model.

Original: First, instead of learning pair-wise relationships among capsules, we learn to projectively encode a description of each capsule-type for every layer.
Revised: First, instead of learning pair-wise relationships among capsules, we learn to projectively encode a description of each capsule-type for every layer.

Original: This we do by associating each capsule-type with a vector-valued function, given by a trainable neural network.
Revised: This we do by associating each capsule-type with a vector-valued function, given by a trainable neural network.

Original: This network assumes the role of the prediction mechanism in capsule networks.
Revised: This network assumes the role of the prediction mechanism in capsule networks.

Original: We interpret the role of this network as a means of encoding the manifold of legal pose-variations for its associated capsule-type.
Revised: We interpret the role of this network as a means of encoding the manifold of legal pose-variations for its associated capsule-type.

Original: It is expected that, given proper training, shallower capsules that have no relationship with a particular capsule-type will project themselves to a vector of low activation (for example, two-norm), when input to the corresponding network.
Revised: It is expected that, given proper training, shallower capsules that have no relationship with a particular capsule-type will project themselves to a vector of low activation (for example, 2-norm), when input to the corresponding network.

Original: As an aside, it is this mechanism that gives the name to our model.
Revised: As an aside, it is this mechanism that gives the name to our model.

Original: We term this manifold the ’space-of-variation’ of a capsule-type.
Revised: We term this manifold the ’space-of-variation’ of a capsule-type.

Original: Since, we attempt to learn such spaces at each layer, we name our model ’space-of-variation’ networks (SOVNET).
Revised: Since, we attempt to learn such spaces at each layer, we name our model ’space-of-variation’ networks (SOVNET).

Original: In this design, the number of trainable networks for a given layer depend on the number of capsule-types of that layer.
Revised: In this design, the number of trainable networks for a given layer depend on the number of capsule-types of that layer.

Original: As mentioned earlier, the choice of prediction networks and routing algorithm is important to having guarantees on learning transformation-invariant compositional relationships.
Revised: As mentioned earlier, the choice of prediction networks and routing algorithm is important to having guarantees on learning transformation-invariant compositional relationships.

Original: Thus, in order to ensure equivariance, which we show is sufﬁcient for the above, we use group-equivariant convolutions (GCNN)
Revised: Thus, in order to ensure equivariance, which we show is sufﬁcient for the above, we use group-equivariant convolutions (GCNN)

Original: (Cohen & Welling, 2016) in the prediction phase.
Revised: (Cohen & Welling, 2016) in the prediction phase.

Original: Thus, shallower capsules of a ﬁxed type are input to a GCNN associated with a deeper capsule-type to obtain predictions for it.
Revised: Thus, shallower capsules of a ﬁxed type are input to a GCNN associated with a deeper capsule-type to obtain predictions for it.

Original: Apart from ensuring equivariance to transformations, GCNNs also allow for greater parameter-sharing (across a set of transformations), resulting in greater awareness of local object-structures.
Revised: Apart from ensuring equivariance to transformations, GCNNs also allow for greater parameter-sharing (across a set of transformations), resulting in greater awareness of local object-structures.

Original: We argue that this could potentially improve the quality of predictions when compared to isolated predictions made by convolutional capsule layers, such as those of (Hinton et al., 2018).
Revised: We argue that this could potentially improve the quality of predictions when compared to isolated predictions made by convolutional capsule layers, such as those of (Hinton et al., 2018).

Original: The last contribution of this paper is an equivariant degree-centrality based routing algorithm.
Revised: The last contribution of this paper is an equivariant degree-centrality based routing algorithm.

Original: The main idea of this method is to treat each prediction for a capsule as a vertex of a graph, whose weighted edges are given by a similarity measure on the predictions themselves.
Revised: The main idea of this method is to treat each prediction for a capsule as a vertex of a graph, whose weighted edges are given by a similarity measure on the predictions themselves.

Original: Our method uses the softmaxed values of the degree scores of the afﬁnity matrix of this graph as a set of weights for aggregating predictions.
Revised: Our method uses the softmaxed values of the degree scores of the afﬁnity matrix of this graph as a set of weights for aggregating predictions.

Original: The key idea being that predictions that agree with a majority of other predictions for the same capsule get a larger weight - following the principle of routing-by-agreement.
Revised: The key idea being that predictions that agree with a majority of other predictions for the same capsule get a larger weight - following the principle of routing-by-agreement.

Original: While this method is only heuristic in the sense of optimality, it is provably equivariant and preserves the capsule-decomposition of an input.
Revised: While this method is only heuristic in the sense of optimality, it is provably equivariant and preserves the capsule-decomposition of an input.

Original: We summarise the contributions of this paper in the following:
Revised: We summarise the contributions of this paper in the following:

Original: A general framework for a scalable capsule-network model.
Revised: A general framework for a scalable capsule-network model.

Original: A particular instantiation of this model that uses equivariant convolutions, and an equivariant, degree-centrality-based routing algorithm.
Revised: A particular instantiation of this model that uses equivariant convolutions, and an equivariant, degree-centrality-based routing algorithm.

Original: A graph-based framework for studying the representation of a capsule network, and the proof of
Revised: A graph-based framework for studying the representation of a capsule network, and the proof of

Original: the
Revised: the

Original: sufﬁciency of equivariance for the (qualiﬁed) preservation of this representation under transformations of the input.
Revised: sufﬁciency of equivariance for the (qualiﬁed) preservation of this representation under transformations of the input.

Original: A set of proof-of-concept, evaluative experiments on afﬁnely transformed variations of MNIST, FASHIONMNIST, and CIFAR10, as well as separate experiments on KMNIST and SVHN that showcase the superior adapatability of SOVNET architectures to train and test-time geometric perturbations of the data, as well as their general performance.
Revised: A set of proof-of-concept, evaluative experiments on afﬁnely transformed variations of MNIST, FASHIONMNIST, and CIFAR10, as well as separate experiments on KMNIST and SVHN that showcase the superior adapatability of SOVNET architectures to train and test-time geometric perturbations of the data, as well as their general performance.

Original:  S OVNET , EQUIVARIANCE , AND COMPOSITIONALITY
Revised:  S OVNET , EQUIVARIANCE , AND COMPOSITIONALITY

Original: We begin with essential deﬁnitions for a layer of SOVNET, and the properties we wish to guarantee.
Revised: We begin with essential deﬁnitions for a layer of SOVNET, and the properties we wish to guarantee.

Original: Given a group ( G, ◦ ) , we formally describe the l th layer of a SOVNET architecture as the set of function-tuples { ( f li , a li ) : 0 ≤ i ≤ N l − 1; f li : G → R d l ; a li : G → [0 , 1] } .
Revised: Given a group ( G, ◦ ) , we formally describe the l th layer of a SOVNET architecture as the set of function-tuples { ( f li , a li ) : 0 ≤ i ≤ N l − 1; f li : G → R d l ; a li : G → [0 , 1] } .

Original: Here, N l denotes the number of capsule-types at the l th layer, f li is a functional description of the d l -dimensional posevectors of instances of the i th capsule-type, and a li is a functional description of the corresponding activations.
Revised: Here, N l denotes the number of capsule-types at the l th layer, f li is a functional description of the d l -dimensional posevectors of instances of the i th capsule-type, and a li is a functional description of the corresponding activations.

Original: We model each capsule-type as a function over a group of transformations so as to allow for formal guarantees on transformation-equivariance.
Revised: We model each capsule-type as a function over a group of transformations so as to allow for formal guarantees on transformation-equivariance.

Original: Thus, we also model images as function from a group to a representation-space.
Revised: Thus, we also model images as function from a group to a representation-space.

Original: The main assumption being that the translation-group is a subgroup of the group in question.
Revised: The main assumption being that the translation-group is a subgroup of the group in question.

Original: This is similar in approach to (Cohen & Welling, 2016).
Revised: This is similar in approach to (Cohen & Welling, 2016).

Original: We wish for each capsule-type, both pose and activation-wise, to display equivariance.
Revised: We wish for each capsule-type, both pose and activation-wise, to display equivariance.

Original: We present a formal deﬁnition of this notion.
Revised: We present a formal deﬁnition of this notion.

Original: Consider a group ( G, ◦ ) and vector spaces V , W .
Revised: Consider a group ( G, ◦ ) and vector spaces V , W .

Original: Let T and T (cid:48) be two group-representations for elements of G over V and W , respectively.
Revised: Let T and T (cid:48) be two group-representations for elements of G over V and W , respectively.

Original: Φ : V → W is said to be equivariant with respect to T and T (cid:48)
Revised: Φ : V → W is said to be equivariant with respect to T and T (cid:48)

Original: if ∀ g ∈ G , ∀ x ∈ V , Φ( T g x ) = T (cid:48)
Revised: if ∀ g ∈ G , ∀ x ∈ V , Φ( T g x ) = T (cid:48)

Original: g Φ( x ) .
Revised: g Φ( x ) .

Original: This deﬁnition translates to a preservation on transformations in the input-space to the output-space - something that allows no loss of information in compositional structures.
Revised: This deﬁnition translates to a preservation on transformations in the input-space to the output-space - something that allows no loss of information in compositional structures.

Original: As in (Cohen & Welling, 2016), we restrict the notion of equivariance in our model by using the operator L g in place of the group-representation.
Revised: As in (Cohen & Welling, 2016), we restrict the notion of equivariance in our model by using the operator L g in place of the group-representation.

Original: L g is given by [ L g f ]( x ) =
Revised: L g is given by [ L g f ]( x ) =

Original: f ( g − 1 x ) .
Revised: f ( g − 1 x ) .

Original: Thus, if ⊗ denotes an operation between two functions, we require ([ L g f ] ⊗ Ψ)( x ) =
Revised: Thus, if ⊗ denotes an operation between two functions, we require ([ L g f ] ⊗ Ψ)( x ) =

Original: [ L g ( f ⊗ Ψ)]( x ) .
Revised: [ L g ( f ⊗ Ψ)]( x ) .

Original: The operator ⊗ describes the change in representation space, and is dependent on the nature of the deep learning model.
Revised: The operator ⊗ describes the change in representation space, and is dependent on the nature of the deep learning model.

Original: In our case, ⊗ is given by routing among capsules.
Revised: In the case of capsule networks (and SOVNET), this change is given by routing among capsules as described in subsection 2.1.

Original: 2. SOVNET LAYER
Revised: 2. SOVNET LAYER

Original: We deﬁne the capsule-types of a particular layer as an output of an agreement-based aggregation of predictions made by the preceding layer.
Revised: We deﬁne the capsule-types of a particular layer as an output of an agreement-based aggregation of predictions made by the preceding layer.

Original: A recursive application of this deﬁnition is enough to deﬁne a SOVNET architecture, given an initial set of capsules.
Revised: A recursive application of this deﬁnition is enough to deﬁne a SOVNET architecture, given an initial set of capsules.

Original: A means of obtaining this initial set is given in section 3.
Revised: A means of obtaining this initial set is given in section 3.

Original: We provide a general framework for the summation-based family of routing procedures in Algorithm 1. Algorithm 
Revised: We provide a general framework for the summation-based family of routing procedures in Algorithm 1. Algorithm 1 A general routing algorithm for SOVNET.

Original: Input : { ( f li , a li ) | i ∈ { 0 , ..., N l − 1 } , f li : G → R d l , a li : G → [0 , 1] }
Revised: Input : { ( f li , a li ) | i ∈ { 0 , ..., N l − 1 } , f li : G → R d l , a li : G → [0 , 1] }

Original: Output : { ( f l +1 j , a l +1 j ) | j ∈ { 0 , ..., N l +1 − 1 } , f l +1 j : G → R d l +1 , a l +1 j : G → [0 , 1] }
Revised: Output : { ( f l +1 j , a l +1 j ) | j ∈ { 0 , ..., N l +1 − 1 } , f l +1 j : G → R d l +1 , a l +1 j : G → [0 , 1] }

Original: Trainable Functions : (Ψ l +1 j , · ) - projection networks that use operator ·
Revised: Trainable Functions : (Ψ l +1 j , · ) - projection networks that use operator ·

Original: S l +1 ij ( g ) =
Revised: S l +1 ij ( g ) =

Original: (( f li , a li ) · Ψ l +1 j )( g ) ∀
Revised: (( f li , a li ) · Ψ l +1 j )( g ) ∀

Original: i, j , ∀ g ∈ G ( c l +10 j ( g ) , ..., c l +1 N l − 1 j ( g )) =
Revised: i, j , ∀ g ∈ G ( c l +10 j ( g ) , ..., c l +1 N l − 1 j ( g )) =

Original: GetWeights ( S l +10 j ( g ) , ..., S l +1 N l − 1 j ( g ))
Revised: GetWeights ( S l +10 j ( g ) , ..., S l +1 N l − 1 j ( g ))

Original: ∀
Revised: ∀

Original: j , ∀ g ∈ G
Revised: j , ∀ g ∈ G

Original: f l +1 j ( g )
Revised: f l +1 j ( g )

Original: = (cid:80) N l −[-1 i =
-]
Revised: = (cid:80) N l −[-1 i =
-]

Original: [-c l +
 i,j ( g ) S l +1 ij [Equation]-] [Table]
Revised: [-c l +
 i,j ( g ) S l +1 ij [Equation]-] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: The steps in Algorithm 1 are exactly the same as in any other capsule network: shallower capsules make a prediction for deeper capsules, theimportance of these predictions are scored, and ﬁnally these are used to build the deeper capsules.
Revised: The weighted-sum family of routing algorithms builds deeper capsules using a weighted sum of predictions made for them by shallower capsules. To ensure that the predictions are combined in a meaningful manner, different methods can be used to obtain the weights. The role of the function GetWeights is to represent any such mechanism.

Original: The extent ofagreement among the predictions for a particular capsule of a type deﬁnes its activation.
Revised: The activation of a capsule, representative of the probability of existence of the object it represents, is determined by the extent of the consensus among its predictions.

Original: We must note that several other families of routing algorithms exist, especially those that do not use weighted-summation.
Revised: This is based on the routing-by-agreement principle of capsule networks.

Original: An example of such an algorithm is spectral-routing (Bahadori, 2018).
Revised: The

Original: We do not, however, pursue a description of  such methods as our aims are more limited in scope.
Revised: Agreement function represents any means of evaluating such consensus.

Original: We instantiate the above algorithm to a speciﬁc model, as given in Algorithm 2.
Revised: We instantiate the above algorithm to a speciﬁc model, as given in Algorithm 2.

Original: In this model, the Ψ lj are group-equivariant convolutional ﬁlters, and the operator · is the corresponding groupequivariant correlation operator (cid:63) .
Revised: In this model, the Ψ lj are group-equivariant convolutional ﬁlters, and the operator · is the corresponding groupequivariant correlation operator (cid:63) .

Original: The weights c l +1 ij ( g ) are, in this routing method, the softmaxed degree-scores of the afﬁnities among predictions for the same deeper capsule.
Revised: The weights c l +1 ij ( g ) are, in this routing method, the softmaxed degree-scores of the afﬁnities among predictions for the same deeper capsule.

Original: Further, like in dynamic routing (Sabour et al., 2017), we also assume that the activation of a capsule is given by its two-norm.
Revised: Further, like in dynamic routing (Sabour et al., 2017), we also assume that the activation of a capsule is given by its 2-norm.

Original: To ensure that this value is in [0 , 1] , we use the ’squash’ function of dynamic routing.
Revised: To ensure that this value is in [0 , 1] , we use the ’squash’ function of dynamic routing.

Original: Thus, we do not mention it explicitly.
Revised: Thus, we do not mention it explicitly.

Original: Note that we have used the subscript notation to also denote that a variable is part of a vector, for example S l +1 ijp ( g ) denotes the p th element of the d l +1 -dimensional vector S l +1 ij ( g ) .
Revised: Note that we have used the subscript notation to also denote that a variable is part of a vector, for example S l +1 ijp ( g ) denotes the p th element of the d l +1 -dimensional vector S l +1 ij ( g ) .

Original: This new routing algorithm is meant to serve as an alternative to existing iterative routing strategies such as dynamic routing.
Revised: This new routing algorithm is meant to serve as an alternative to existing iterative routing strategies such as dynamic routing.

Original: An important strength of our method being that there is no hyperparameter, like that of the number of iterations in dynamic routing or EM routing.
Revised: An important strength of our method being that there is no hyperparameter, like that of the number of iterations in dynamic routing or EM routing.

Original: 2. E QUIVARIANCE , COMPOSITIONALITY AND SOVNET
Revised: 2. E QUIVARIANCE , COMPOSITIONALITY AND SOVNET

Original: The SOVNET layer we introduced in Algorithm 2 is group-equivariant with respect to the group action L g , where g ∈ G - the set of transformations over which the group-convolution is deﬁned.
Revised: The SOVNET layer we introduced in Algorithm 2 is group-equivariant with respect to the group action L g , where g ∈ G - the set of transformations over which the group-convolution is deﬁned.

Original: For notational convenience, we deﬁne ⊗ to be an operator that encapsulates the degree-routing procedurefor with prediction networks Ψ l +1 j .
Revised: For notational convenience, we deﬁne ⊗ to be an operator that encapsulates the degree-routing procedure with prediction networks Ψ l +1 j .

Original: Thus, the j th capsule-type of the l + 1 th layer is functionally depicted as f l +1 j
Revised: Thus, the j th capsule-type of the l +1 th layer is functionally depicted as f l +1 j

Original: = ( F l ⊗ Ψ l +1 j ) , where F l = ( f l 0 , ..., f lN l − 1 ) .
Revised: = ( F l ⊗ Ψ l +1 j ) , where F l = ( f l 0 , ..., f lN l − 1 ) .

Original: The formal statement of this result is given below; the proof is presented in the appendix.
Revised: The formal statement of this result is given below; the proof is presented in the appendix.

Original: Theorem 2.1.
Revised: Theorem 2.1.

Original: The SOVNET layer deﬁned in Algorithm 2, and denoted by the operator ⊗ as given above, satisﬁes ([ L g F l ] ⊗ Ψ l +1 j ) =
Revised: The SOVNET layer deﬁned in Algorithm 2, and denoted by the operator ⊗ as given above, satisﬁes ([ L g F l ] ⊗ Ψ l +1 j ) =

Original: ( L g [ F l ⊗ Ψ l +1 j ]) , where g belongs to the underlying group of the equivariant convolution. 
Revised: ( L g [ F l ⊗ Ψ l +1 j ]) , where g belongs to the underlying group of the equivariant convolution. Input : { f li | i [Equation]

Revised: [Equation]

Original: Input : { f li | i ∈ { 0 , ..., N l − 1 } , f li: G → R d l }
Revised: [Equation] N l − [Equation] f li

Original:  Output : { f l +1 j | j ∈ {0 , ..., N l +1 − 1 } , f l +1 j : G → R d l +1 }
Revised: [Equation] G → [Equation] Output : { f l +1 j | j ∈ {

Original: Trainable Functions : (Ψ l +1 j , (cid:63) ) , 0 ≤ j ≤ N l +1 − 1 , - a set of d l +1 group-equivariant convolutional ﬁlters (per capsule-type) that use the group-equivariant correlation operator (cid:63)
Revised: [Equation] [Equation]

Original: S l +1 ijp ( g ) =
Revised: N l + [Equation] −

Original: ( fli (cid:63) Ψ l 
Revised: [Equation] f l [Equation]

Original: +1 ,p j )( g ) =
Revised: 1 j : G → R d l +1 }

Original: (cid:80) h ∈ G (cid:80) d l  − 1 k =0 f lik ( h )Ψ
Revised: Trainable Functions : (Ψ l +1 j , (cid:63) ) , 0 ≤ j ≤ N l +1 − 1 , - a set of d l +1 group-equivariant convolutional ﬁlters (per capsule-type) that use the group-equivariant correlation operator (cid:63) [Equation] [Equation]

Original: l +1 ,p k
Revised: c [Equation]

Original: ( g − 1 ◦ h ) ; p ∈ { 0 , ...,
Revised: g [Equation] = DegreeScore ( S

Original: d l +1 − 1 } ( c l +10 j ( g ) , ..., c l +1 N l − 1 j ( g )) =
Revised: [Equation] ( g ) , ... [Equation] S l +

Original: DegreeScore ( S l +10 j ( g ) , ..., S l +1 N l − 1 j ( g )); ∀ 0 ≤ j ≤ N l
Revised: [Equation] N [Equation]

Original: +1 − 1 ,
Revised: − [Equation]

Original: ∀ g f l +1 j ( g ) 
Revised: ( g ) [Equation] ∀

Original: [-= (cid:80) N l − 1 i =
 c l +1 ij ( g ) S-]
Revised: [-= (cid:80) N l − 1 i =
 c l +1 ij ( g ) S-][Equation]

Original: l +1 ij ( g ) ∀ 0 ≤ j ≤ N l 
Revised: ≤ [Equation] ≤ N l + [Equation]

Original: +1 −1 , ∀ g ∈ G f l +1 j ( g ) =
Revised: −

Original: Squash ( f l +1 j ( g ))
Revised: [Equation] ∀ [Equation]

Original: = (cid:107) f 
Revised: f [Equation]

Original: [-l +1 j ( g ) (cid:107) 
-]
Revised: [-l +1 j ( g ) (cid:107) 
-][Table]

Original: [-+ (cid:107) f l +1 j ( g ) (cid:107) 
 f l +1 j ( g ) ; ∀ 0 ≤ j ≤ N l − 1 , ∀ g ∈ G-]
Revised: [-+ (cid:107) f l +1 j ( g ) (cid:107) 
 f l +1 j ( g ) ; ∀ 0 ≤ j ≤ N l − 1 , ∀ g ∈ G-][Table]

Original: +10 j ( g ) , ..., S l +1 N l − 1 j ( g ) )
Revised: ., S l +1 N l − 1 j ( g ) )

Original: A jik ( g ) =
Revised: A jik ( g ) =

Original: S l +1 ij ( g ) .S
Revised: S l +1 ij ( g ) .S

Original: l +1 kj ( g ) (cid:107)
Revised: l +1 kj ( g ) (cid:107)

Original: S l +1 ij ( g ) (cid:107)  .
Revised: S l +1 ij ( g ) (cid:107) 2 .

Original: (cid:107)
Revised: (cid:107)

Original: S
Revised: S

Original: l +1 kj ( g ) (cid:107)  ; 0 ≤ i, k ≤ N l −
Revised: l +1 kj ( g ) (cid:107) 2 ; 0 ≤ i, k ≤ N l −

Original: Degree ji ( g ) =
Revised: Degree ji ( g ) =

Original: (cid:80) N l − 1 k =

Revised: (cid:80) N l − 1 k =

Original: ( A jik ( g )) ; 0 ≤ i ≤ N l −c ij ( g ) =
Revised: ( A jik ( g )) ; 0 ≤ i ≤ N l −c ij ( g ) =

Original: exp( Degree ji ( g ))
Revised: exp( Degree ji ( g ))

Original: (cid:80)
Revised: (cid:80)

Original: Nl −k =0 exp( Degree jk ( g )) ; 0 ≤ i ≤ N l −return c ij ( g ) ∀ 0 ≤ i ≤ N l −
Revised: Nl −k =0 exp( Degree jk ( g )) ; 0 ≤ i ≤ N l −return c ij ( g ) ∀ 0 ≤ i ≤ N l −

Original: Proof.
Revised: Proof.

Original: The proof is given in the appendix.
Revised: The proof is given in the appendix.

Original: Equivariance is widely considered a desirable inductive bias for a variety of reasons.
Revised: Equivariance is widely considered a desirable inductive bias for a variety of reasons.

Original: First, equivariance mirrors natural label-invariance under transformations.
Revised: First, equivariance mirrors natural label-invariance under transformations.

Original: Second, it lends predictability to the output of a network under (ﬁxed) transformations of the input.
Revised: Second, it lends predictability to the output of a network under (ﬁxed) transformations of the input.

Original: These, of course, lead to a greater robustness in handling transformations of the data.
Revised: These, of course, lead to a greater robustness in handling transformations of the data.

Original: We aim at adding to this list by showing that equivariance guarantees the preservation of detected compositionalities in a SOVNET architecture.
Revised: We aim at adding to this list by showing that equivariance guarantees the preservation of detected compositionalities in a SOVNET architecture.

Original: This is of course quite unsurprising, and has been a signiﬁcant undercurrent of the capsule-network idea.
Revised: This is of course quite unsurprising, and has been a signiﬁcant undercurrent of the capsule-network idea.

Original: Our work completes this intuition with a formal result.
Revised: Our work completes this intuition with a formal result.

Original: We begin by ﬁrst deﬁning the notion of a capsule-decomposition graph.
Revised: We begin by ﬁrst deﬁning the notion of a capsule-decomposition graph.

Original: This graph is formed from the activations and the routing weights of a SOVNET.
Revised: This graph is formed from the activations and the routing weights of a SOVNET.

Original: Speciﬁcally, given an input to a SOVNET model, each capsule of every type is a vertex in this graph.
Revised: Speciﬁcally, given an input to a SOVNET model, each capsule of every type is a vertex in this graph.

Original: We construct an edge between capsules that are connected by routing, with the direction from the shallower capsule to the deeper capsule.
Revised: We construct an edge between capsules that are connected by routing, with the direction from the shallower capsule to the deeper capsule.

Original: Each of these edges are weighted by the corresponding routing coefﬁcient.
Revised: Each of these edges are weighted by the corresponding routing coefﬁcient.

Original: Capsules not related to each other by routing are not connected by an edge.
Revised: Capsules not related to each other by routing are not connected by an edge.

Original: This graph is a direct formalisation of the various detected compositionalities with their strengths.
Revised: This graph is a direct formalisation of the various detected compositionalities with their strengths.

Original: What should the ideal behaviour of this graph be under the change-of-viewpoint of an input?
Revised: What should the ideal behaviour of this graph be under the change-of-viewpoint of an input?

Original: The answer to this lies in the expected behaviour of natural compositionalities.
Revised: The answer to this lies in the expected behaviour of natural compositionalities.

Original: Thus, while the pose of objects, and their components, is changed under transformations of the input, the relative geometry is constant.
Revised: Thus, while the pose of objects, and their components, is changed under transformations of the input, the relative geometry is constant.

Original: Thus, it is desirable that the capsule-decomposition graphs of a particular input (and its transformed variations) be isomorphic to each other.
Revised: Thus, it is desirable that the capsule-decomposition graphs of a particular input (and its transformed variations) be isomorphic to each other.

Original: We show that a SOVNET model that is equivariant with respect to a set of transformations satisﬁes the above property for that set.
Revised: We show that a SOVNET model that is equivariant with respect to a set of transformations satisﬁes the above property for that set.

Original: A more formal description of the capsule-decomposition graph, and the statement for the above theorem are given below.
Revised: A more formal description of the capsule-decomposition graph, and the statement for the above theorem are given below.

Original: Consider a L -layer SOVNET model, whose routing procedure belongs to the family of methods given by Algorithm 1.
Revised: Consider an L -layer SOVNET model, whose routing procedure belongs to the family of methods given by Algorithm 1.

Original: Let us consider a ﬁxed input x : G → R c .
Revised: Let us consider a ﬁxed input x : G → R c .

Original: We deﬁne the capsule-decomposition graph of such a model, for this input x , as G ( x ) =
Revised: We deﬁne the capsule-decomposition graph of such a model, for this input x , as G ( x ) =

Original: ( V ( x ) , E ( x )) .
Revised: ( V ( x ) , E ( x )) .

Original: Here, V ( x ) and E ( x ) denote the vertex-set and the edge-set, respectively.
Revised: Here, V ( x ) and E ( x ) denote the vertex-set and the edge-set, respectively.

Original: V ( x ) = { ˜ f li ( g ) : 0 ≤ i ≤ N l − 1 , 0 ≤ l ≤ L − 1 , g ∈ G } , where ˜ f li ( g ) =
Revised: V ( x ) = { ˜ f li ( g ) : 0 ≤ i ≤ N l − 1 , 0 ≤ l ≤ L − 1 , g ∈ G } , where ˜ f li ( g ) =

Original: ( g, i, f li ( g ) , a li ( g )) , g ∈ G , 0 ≤ i N l − 1 .
Revised: ( g, i, f li ( g ) , a li ( g )) , g ∈ G , 0 ≤ i N l − 1 .

Original: E ( x ) = { ( ˜ f li ( g 1 ) , ˜ f
Revised: E ( x ) = { ( ˜ f li ( g 1 ) , ˜ f

Original: l +1 i ( g 2 ) , 
Revised: l +1 j ( g 2 ) , c

Original: c l +1 ij ( g 2 )) :
Revised: l +1 ij ( g 2 )) :

Original: g 1 ∈ Pool  ( g 2 ) } . {+Pool l +
+}
Revised: g 1 ∈ Pool l +1 j ( g 2 ) } . {+Pool l +
+}

Original: Pool ( g 2 ) denotes the pool of grid-positions  that route to  g 2 .
Revised: j ( g 2 ) denotes the pool of grid-positions at layer l that route to the deeper capsule of type j of layer l + 1 at g 2 .

Original: This deﬁnition uses the positional, as well as type information of each capsule to uniquely identify it.
Revised: A more formal deﬁnition is given the appendix.

Original: Moreover, we also use the notation L h ˜ f
Revised: We also use the notation L h ˜ f

Original: l i ( g ) to denote ( h − 1 ◦ g, i, f l i ( h − 1 ◦ g ) , a l i ( h − 1 ◦ g )) .
Revised: l i ( g ) to denote ( h − 1 ◦ g, i, f l i ( h − 1 ◦ g ) , a l i ( h − 1 ◦ g )) .

Original: Theorem 2.2.
Revised: Theorem 2.2.

Original: Consider an L -layer SOVNET whose activations are routed according to a procedure belonging to the family given by Algorithm 1.
Revised: Consider an L -layer SOVNET whose activations are routed according to a procedure belonging to the family given by Algorithm 1.

Original: Further, assume that this routing procedure is equivariant with respect to the group G .
Revised: Further, assume that this routing procedure is equivariant with respect to the group G .

Original: Then, given an input x and ∀ g ∈ G , G ( x ) and G ([ L g x ]) are isomorphic.
Revised: Then, given an input x and ∀ g ∈ G , G ( x ) and G ([ L g x ]) are isomorphic.

Original: Proof.
Revised: Proof.

Original: The proof is given in the appendix.
Revised: The proof is given in the appendix.

Original: Based on above theorem, and the fact that degree-centrality based routing is equivariant, the above result applies to SOVNET models that use Algorithm 2 .
Revised: Based on above theorem, and the fact that degree-centrality based routing is equivariant, the above result applies to SOVNET models that use Algorithm 2 .

Original:  E XPERIMENTS AND RESULTS
Revised:  E XPERIMENTS AND RESULTS

Original: This section presents a description of the experiments we performed.
Revised: This section presents a description of the experiments we performed.

Original: We conducted two sets of experiments; the ﬁrst to compare SOVNET architectures to other capsule network baselines with respect to transformation robustness on classiﬁcation, and the second to compare SOVNET to certain capsule as well as convolutional baselines based on classiﬁcation performance.
Revised: We conducted two sets of experiments; the ﬁrst to compare SOVNET architectures to other capsule network baselines with respect to transformation robustness on classiﬁcation, and the second to compare SOVNET to certain capsule as well as convolutional baselines based on classiﬁcation performance.

Original: Before we present the details of these experiments, we brieﬂy describe some details of the SOVNET architecture we used.
Revised: Before we present the details of these experiments, we brieﬂy describe some details of the SOVNET architecture we used.

Original: We only present an outline - the complete details, both architecture-wise and about the training, can be found in the anonymised github repository https://github.com/
Revised: We only present an outline - the complete details, both architecture-wise and about the training, can be found in the anonymised github repository https://github.com/

Original: AnonymousCapsuleSOVNET/SOVNET .
Revised: AnonymousCapsuleSOVNET/SOVNET .

Original: The ﬁrst detail of the architecture pertains to the construction of the ﬁrst layer of capsules.
Revised: The ﬁrst detail of the architecture pertains to the construction of the ﬁrst layer of capsules.

Original: While many approaches are possible, we used the following methodology that is similar in spirit to other capsule network models.
Revised: While many approaches are possible, we used the following methodology that is similar in spirit to other capsule network models.

Original: The ﬁrst layer of the SOVNET architectures we constructed use a modiﬁed residual block that uses the SELU activation, along with group-equivariant convolutions.
Revised: The ﬁrst layer of the SOVNET architectures we constructed use a modiﬁed residual block that uses the SELU activation, along with group-equivariant convolutions.

Original: This is so as to allow a meaningful set of equivariant feature maps to be used for the creation of the ﬁrst set of capsules.
Revised: This is so as to allow a meaningful set of equivariant feature maps to be used for the creation of the ﬁrst set of capsules.

Original: Intuition and some literature, for example Rosario et al.
Revised: Intuition and some literature, for example Rosario et al.

Original: (2019), suggest that the construction of primary capsules plays a signiﬁcant role in the performance of the capsule network.
Revised: (2019), suggest that the construction of primary capsules plays a signiﬁcant role in the performance of the capsule network.

Original: Thus, it is necessary to build a sufﬁciently expressive layer that yields the ﬁrst set of meaningful capsule-activations.
Revised: Thus, it is necessary to build a sufﬁciently expressive layer that yields the ﬁrst set of meaningful capsule-activations.

Original: To this end, each capsule-type in the primary capsule layer is associated with a group-convolution layer followed by a modiﬁed residual block.
Revised: To this end, each capsule-type in the primary capsule layer is associated with a group-convolution layer followed by a modiﬁed residual block.

Original: The convolutional feature-maps from the preceding layer passes through each of these sub-networks to yield the primary capsules.
Revised: The convolutional feature-maps from the preceding layer passes through each of these sub-networks to yield the primary capsules.

Original: No routing is performed in this layer.
Revised: No routing is performed in this layer.

Original: We now describe the SOVNET blocks.
Revised: We now describe the SOVNET blocks.

Original: Since the design of SOVNET signiﬁcantly reduces the number of prediction networks, and thereby the number of trainable parameters, we are able to build architectures whose each layer uses more expressive prediction mechanisms than a simple matrix.
Revised: Since the design of SOVNET signiﬁcantly reduces the number of prediction networks, and thereby the number of trainable parameters, we are able to build architectures whose each layer uses more expressive prediction mechanisms than a simple matrix.

Original: Speciﬁcally, each hidden layer of the SOVNET architectures we consider uses a (group-equivariant) modiﬁed residual block as the prediction mechanism.
Revised: Speciﬁcally, each hidden layer of the SOVNET architectures we consider uses a (group-equivariant) modiﬁed residual block as the prediction mechanism.

Original: We use a SOVNET architecture that useshidden layers for MNIST, FashionMNIST, KMNIST, and SVHN, and a model that uses 6 hidden layers for CIFAR-10.
Revised: We use a SOVNET architecture that useshidden layers for MNIST, FashionMNIST, KMNIST, and SVHN, and a model that uses 6 hidden layers for CIFAR-10.

Original: Unlike DeepCaps - another capsule network whose predictions use (regular) convolution, each of the hidden layers of out SOVNET models use degree-routing.
Revised: Unlike DeepCaps - another capsule network whose predictions use (regular) convolution, each of the hidden layers of our SOVNET models use degree-routing.

Original: The hidden layers of DeepCaps (excepting the last), in contrast, are not strictly capsule-based - being just convolutions whose outputs are reshaped to a capsule-form.
Revised: The hidden layers of DeepCaps (excepting the last), in contrast, are not strictly capsule-based - being just convolutions whose outputs are reshaped to a capsule-form.

Original: The output capsule-layer of SOVNET is designed similar to the hidden capsule-layers, with the difference that the prediction-mechanism is a group-convolutional implementation of a fullyconnected layer.
Revised: The output capsule-layer of SOVNET is designed similar to the hidden capsule-layers, with the difference that the prediction-mechanism is a group-convolutional implementation of a fullyconnected layer.

Original: In order to make a prediction for the class of an input, the maximum across the rotational (and reﬂectional) positions of the two-norm of the capsule-activations of this layer are taken for each class-type.
Revised: In order to make a prediction for the class of an input, the maximum across the rotational (and reﬂectional) positions of the two-norm of the capsule-activations of this layer are taken for each class-type.

Original: This is an equivariant operation, as it corresponds to the subgroup-pooling of Cohen & Welling (2016).
Revised: This is an equivariant operation, as it corresponds to the subgroup-pooling of Cohen & Welling (2016).

Original: The predictions that this layer yields is the type of the capsule with the maximum two-norm.
Revised: The predictions that this layer yields is the type of the capsule with the maximum 2-norm.

Original: In order to guarantee the robustness to translations and rotations, we used the p4-convolutions (Cohen & Welling, 2016) for the prediction mechanism in all the networks used in the ﬁrst set of experiments.
Revised: In order to guarantee the robustness to translations and rotations, we used the p4-convolutions (Cohen & Welling, 2016) for the prediction mechanism in all the networks used in the ﬁrst set of experiments.

Original: For the second set, we used the p4m-convolution (Cohen & Welling, 2016), that is equivariant to rotations, translations and reﬂections - for greater ability to learn from augmentations.
Revised: For the second set, we used the p4m-convolution (Cohen & Welling, 2016), that is equivariant to rotations, translations and reﬂections - for greater ability to learn from augmentations.

Original: The architectures, however are identical but for this difference.
Revised: The architectures, however are identical but for this difference.

Original: As in (Sabour et al., 2017), we used a margin loss and a regularising reconstruction loss to train the networks.
Revised: As in (Sabour et al., 2017), we used a margin loss and a regularising reconstruction loss to train the networks.

Original: The positive and negative margins for half of the training epochs were set to 0.9 and 0.1, respectively.
Revised: The positive and negative margins for half of the training epochs were set to 0.9 and 0.1, respectively.

Original: Further, the negative margin-loss was weighted by 0.5, as in (Sabour et al., 2017).
Revised: Further, the negative margin-loss was weighted by 0.5, as in (Sabour et al., 2017).

Original: These values were used for the ﬁrst half of the training epochs.
Revised: These values were used for the ﬁrst half of the training epochs.

Original: In order to facilitate better predictions, these values were changed to 0.95, 0.05, and 0.8, respectively for the second half of the training.
Revised: In order to facilitate better predictions, these values were changed to 0.95, 0.05, and 0.8, respectively for the second half of the training.

Original: We adopt this from (Rajasegaran et al., 2019).
Revised: We adopt this from (Rajasegaran et al., 2019).

Original: The reconstruction loss was computed by masking the incorrect classes, and by feeding the ’true’ class-capsule to a series of transposed convolutions to reconstruct the image.
Revised: The reconstruction loss was computed by masking the incorrect classes, and by feeding the ’true’ class-capsule to a series of transposed convolutions to reconstruct the image.

Original: The mean square loss was computed for the reconstruction and original image.
Revised: The mean square loss was computed for the reconstruction and original image.

Original: The main idea being that this loss guides the capsule network to build meaningful capsules.
Revised: The main idea being that this loss guides the capsule network to build meaningful capsules.

Original: This loss was weighed by 0.0005 as in (Sabour et al., 2017).
Revised: This loss was weighed by 0.0005 as in (Sabour et al., 2017).

Original: We used the Adam optimiser and an exponential learning rate scheduler that reduced the learning rate by a factor of 0.9 each epoch.
Revised: We used the Adam optimiser and an exponential learning rate scheduler that reduced the learning rate by a factor of 0.9 each epoch.

Original: With this outline of the architecture and details of the training, we now describe the ﬁrst set of experiments we conducted on SOVNET.
Revised: With this outline of the architecture and details of the training, we now describe the ﬁrst set of experiments we conducted on SOVNET.

Original: The preservation of detected compositionalities under transformations in SOVNET leads us to the expectation that SOVNET models, when properly trained, will display greater robustness to changes in viewpoint of the input.
Revised: The preservation of detected compositionalities under transformations in SOVNET leads us to the expectation that SOVNET models, when properly trained, will display greater robustness to changes in viewpoint of the input.

Original: Apart from handling test-time transformations, as is the commonly held notion of transformation robustness, a robust model must also effectively learn from train-time perturbations of the data.
Revised: Apart from handling test-time transformations, as is the commonly held notion of transformation robustness, a robust model must also effectively learn from train-time perturbations of the data.

Original: Based on these ideas, we designed a set of experiments that compare SOVNET architectures to other capsule networks on their ability to handle train and test-time afﬁne transformations of the data.
Revised: Based on these ideas, we designed a set of experiments that compare SOVNET architectures to other capsule networks on their ability to handle train and test-time afﬁne transformations of the data.

Original: Speciﬁcally, we perform experiments on MNIST (LeCun & Cortes, 2010), FashionMNIST (Xiao et al., 2017), and CIFAR-10 (Krizhevsky & Hinton, 2009).
Revised: Speciﬁcally, we perform experiments on MNIST (LeCun & Cortes, 2010), FashionMNIST (Xiao et al., 2017), and CIFAR-10 (Krizhevsky & Hinton, 2009).

Original: For each of these datasets, we created 5 variations of the train and test-splits by randomly transforming data according to the extents of the transformations given in Table 1.
Revised: For each of these datasets, we created 5 variations of the train and test-splits by randomly transforming data according to the extents of the transformations given in Table 1.

Original: We train a given model on each transformed version of the training-split, and test each model on each of the versions of the test-split.
Revised: We train a given model on each transformed version of the training-split, and test each model on each of the versions of the test-split.

Original: Thus we obtain, for a single model, 25 accuracies per dataset - each corresponding to a pair of train and test-splits.
Revised: Thus we obtain, for a single model, 25 accuracies per dataset - each corresponding to a pair of train and test-splits.

Original: There is a single modiﬁcation to these transformations for the case of CIFAR-10.
Revised: There is a single modiﬁcation to these transformations for the case of CIFAR-10.

Original: In order to compare
Revised: In order to compare

Original: SOVNET against the closest competitor DeepCaps, we use their strategy of ﬁrst resizing CIFARimages to 64 × 64, followed by translations and rotations.
Revised: SOVNET against the closest competitor DeepCaps, we use their strategy of ﬁrst resizing CIFARimages to 64 × 64, followed by translations and rotations.

Original: We tested SOVNET against four capsule network baselines, namely Capsnet (Sabour et al., 2017),
Revised: We tested SOVNET against four capsule network baselines, namely Capsnet (Sabour et al., 2017),

Original: EMcaps (Hinton et al., 2018), DeepCaps Rajasegaran et al.
Revised: EMcaps (Hinton et al., 2018), DeepCaps Rajasegaran et al.

Original: (2019), and GCaps (Lenssen et al., 2018).
Revised: (2019), and GCaps (Lenssen et al., 2018).

Original: The results of these experiments are given in Tables 2 to 4.
Revised: The results of these experiments are given in Tables 2 to 4.

Original: In the majority of the cases, SOVNET obtains the highest accuracy - showing that it is more robust to transformations of the data.
Revised: In the majority of the cases, SOVNET obtains the highest accuracy - showing that it is more robust to transformations of the data.

Original: Note that we had to conduct these experiments as such a robustness study was not done in the original papers for the baselines.
Revised: Note that we had to conduct these experiments as such a robustness study was not done in the original papers for the baselines.

Original: We used, and modiﬁed, code from the following github sources for the implementation of the baselines: (Li, 2019) for CAPSNET; (Yang, 2019) for EMCAPS; (Rajasegaran, 2019) and (HopefulRational, 2019) for DeepCaps, and (Lenssen, 2019) for GCaps. 
Revised: We used, and modiﬁed, code from the following github sources for the implementation of the baselines: (Li, 2019) for CAPSNET; (Yang, 2019) for EMCAPS; (Rajasegaran, 2019) and (HopefulRational, 2019) for DeepCaps, and (Lenssen, 2019) for GCaps. We also tested against a group-equivariant convolution network (GCNN).

Original: The second set of experiments we conducted, tested SOVNET against several capsule as well as convolutional baselines.
Revised: The second set of experiments we conducted, tested SOVNET against several capsule as well as convolutional baselines.

Original: We trained and tested SOVNET on KMNIST (Clanuwat et al., 2018) and
Revised: We trained and tested SOVNET on KMNIST (Clanuwat et al., 2018) and

Original: SVHN (Netzer et al., 2011).
Revised: SVHN (Netzer et al., 2011).

Original: With fairly standard augmentation - mild translations (and resizing for
Revised: With fairly standard augmentation - mild translations (and resizing for

Original: SVHN to 64 × 64) - the SOVNET architecture with p4m-convolutions was able to achieve on-par, or above, comparative performance.
Revised: SVHN to 64 × 64) - the SOVNET architecture with p4m-convolutions was able to achieve on-par, or above, comparative performance.

Original: The results of this experiment are in Table 5.
Revised: The results of this experiment are in Table 5.

Original: In order to compare the performance of SOVNET architectures against more sophisticated CNN-baselines, we also trained
Revised: In order to compare the performance of SOVNET architectures against more sophisticated CNN-baselines, we also trained

Original: ResNet-18, ResNet-34 on the most extreme of the transformations of the list - translation by  2 pixels, and rotation by  180°.
Revised: ResNet-18, ResNet-34 on the most extreme transformation - translation by up to ± 2 pixels, and rotation by up to ± 180°.

Original: The results of these experiments are presented in the appendix.[Table] [Table] [Table]
Revised: The results of these experiments are presented in the appendix.

Original:  D ISCUSSION AND RELATED WORK
Revised:  D ISCUSSION AND RELATED WORK

Original: A number of insights can be drawn from an observation of the accuracies obtained from the experiments.
Revised: A number of insights can be drawn from an observation of the accuracies obtained from the experiments.

Original: First, the most obvious, is that SOVNET is signiﬁcantly more robust to train and test-time geometric transformations of the input.
Revised: First, the most obvious, is that SOVNET is signiﬁcantly more robust to train and test-time geometric transformations of the input.

Original: Indeed, SOVNET learns to use even extreme transformations of the training data and generalises better to test-time transformationsbetter in a majority of the cases.
Revised: Indeed, SOVNET learns to use even extreme transformations of the training data and generalises better to test-time transformations in a majority of the cases.

Original: However, in certain splits, some baselines perform better than SOVNET.
Revised: However, in certain splits, some baselines perform better than SOVNET.

Original: These cases are brieﬂy discussed below.
Revised: These cases are brieﬂy discussed below.

Original:  On the CIFAR-10 experiments, DeepCaps performs signiﬁcantly better than SOVNET on the untransformed case - generalising to test-time transformations better.
Revised: [Table] [Table] On the CIFAR-10 experiments, DeepCaps performs signiﬁcantly better than SOVNET on the untransformed case - generalising to test-time transformations better.

Original: However, SOVNET learns from train-time transformations better than DeepCaps - outperforming it in a large majority of the other cases.
Revised: However, SOVNET learns from train-time transformations better than DeepCaps - outperforming it in a large majority of the other cases.

Original: We hypothesize that the ﬁrst observation is due to the increased (almost double) number of parameters of DeepCaps that allows it to learn features that generalise better to transformations.
Revised: We hypothesize that the ﬁrst observation is due to the increased (almost double) number of parameters of DeepCaps that allows it to learn features that generalise better to transformations.

Original: Further, as p4-convolutions (the prediction-mechanisms used) are equivariant only to rotations in multiples of 90°, its performance is signiﬁcantly lower for test-time transformations of 30°and 60°for the untransformed case.
Revised: Further, as p4-convolutions (the prediction-mechanisms used) are equivariant only to rotations in multiples of 90°, its performance is signiﬁcantly lower for test-time transformations of 30°and 60°for the untransformed case.

Original: However, the equivariance of SOVNET allows it to learn better from train-time geometric transforms than DeepCaps, explaining the second observation.[Table] [Table] [Table]
Revised: However, the equivariance of SOVNET allows it to learn better from train-time geometric transforms than DeepCaps, explaining the second observation.

Original: [Table]

Original: Sabour et al.

Original: [Table]

Original: [Table]

Original: Deliège et al.

Original: [Table]

Original: et al.

Original: [Table]

Original: Phaye et al.

Original: [Table]

Original: [Table]

Original: Clanuwat et al.

Original: [Table]

Original: Tissera et al.

Original: [Table]

Original: SOVNET [Table] [Table] The second case is that GCaps outperforms SOVNET on generalising to extreme transformations on (mainly) MNIST, and once on FashionMNIST, under mild train-time conditions.
Revised: The second case is that GCaps outperforms SOVNET on generalising to extreme transformations on (mainly) MNIST, and once on FashionMNIST, under mild train-time conditions.

Original: However, it is unable to sustain this under more extreme train-time perturbations.
Revised: However, it is unable to sustain this under more extreme train-time perturbations.

Original: We infer that this is caused largely by the explicit geometric parameterisation of capsules in G-Caps.
Revised: We infer that this is caused largely by the explicit geometric parameterisation of capsules in G-Caps.

Original: While under mild-tomoderate train-time conditions, and on simple datasets, this approach could yield better results, this parameterisation, especially with very simple prediction-mechanisms, can prove detrimental.
Revised: While under mild-tomoderate train-time conditions, and on simple datasets, this approach could yield better results, this parameterisation, especially with very simple prediction-mechanisms, can prove detrimental.

Original: Thus, the convolutional nature of the prediction-mechanisms, which can capture more complex features, and also the greater depth of SOVNET allows it to learn better from more complex training scenarios.
Revised: Thus, the convolutional nature of the prediction-mechanisms, which can capture more complex features, and also the greater depth of SOVNET allows it to learn better from more complex training scenarios.

Original: This makes the case for deeper models with more expressive and equivariant prediction-mechanisms.
Revised: This makes the case for deeper models with more expressive and equivariant prediction-mechanisms.

Original: A related point of interest is that G-Caps performs very poorly on the CIFAR-10 dataset - achieving the least accuracy on most cases on this dataset - despite provable guarantees on equivariance.
Revised: A related point of interest is that G-Caps performs very poorly on the CIFAR-10 dataset - achieving the least accuracy on most cases on this dataset - despite provable guarantees on equivariance.

Original: We argue that this is signiﬁcantly due to the nature of the capsules of this model itself.
Revised: We argue that this is signiﬁcantly due to the nature of the capsules of this model itself.

Original: In GCaps, each capsule is explicitly modelled as an element of a Lie group.
Revised: In GCaps, each capsule is explicitly modelled as an element of a Lie group.

Original: Thus, capsules capture exclusively geometric information, and use only this information for routing.
Revised: Thus, capsules capture exclusively geometric information, and use only this information for routing.

Original: In contrast, other capsule models have no such parameterisation.
Revised: In contrast, other capsule models have no such parameterisation.

Original: In the case of CIFAR-10, where non-geometric features such as  texture are important, we see that purely spatio-geometric based routing is not effective.
Revised: In the case of CIFAR-10, where non-geometric features such as [Table] [Table] texture are important, we see that purely spatio-geometric based routing is not effective.

Original: This observation allows us to make a more general hypothesis that could deal with the fundamentals of capsule networks.
Revised: This observation allows us to make a more general hypothesis that could deal with the fundamentals of capsule networks.

Original: We propose a trade-off in capsule networks, based on the notion of equivariance.
Revised: We propose a trade-off in capsule networks, based on the notion of equivariance.

Original: To appreciate this, some background is necessary on both equivariance and capsule networks.
Revised: To appreciate this, some background is necessary on both equivariance and capsule networks

Original:  As the body of literature concerning equivariance is quite vast, we only mention a relevant selection of papers.
Revised: [Table] As the body of literature concerning equivariance is quite vast [Table] we only mention a relevant selection of papers.

Original: Equivariance can be seen as a desirable, if not fundamental, inductive bias for neural networks used in computer vision.
Revised: Equivariance can be seen as a desirable, if not fundamental, inductive bias for neural networks used in computer vision.

Original: Indeed, the fact that AlexNet (Krizhevsky et al., 2012) automatically learns representation that are equivariant to ﬂips, rotation and scaling(Lenc & Vedaldi, 2015) shows the importance of equivariance as well as its natural necessity.
Revised: Indeed, the fact that AlexNet (Krizhevsky et al., 2012) automatically learns representation that are equivariant to ﬂips, rotation and scaling shows the importance of equivariance as well as its natural necessity (Lenc & Vedaldi, 2015).

Original: Thus, a neural network model that can formally guarantee this property is essential.
Revised: Thus, a neural network model that can formally guarantee this property is essential.

Original: An early work in this regard is thework on group-equivariant convolutions proposed in (Cohen & Welling, 2016).
Revised: An early work in this regard is the group-equivariant convolution proposed in (Cohen & Welling, 2016).

Original: There, the authors proposed a generalisation of the 2-D spatial convolution operation to act on a general group of symmetry transforms - increasing the parameter-sharing and, thereby, improving performance.
Revised: There, the authors proposed a generalisation of the 2-D spatial convolution operation to act on a general group of symmetry transforms - increasing the parameter-sharing and, thereby, improving performance.

Original: Since then, several other models exhibiting equivariance to certain groups of transformations have been proposed, for example (Cohen et al., 2018), where a spherical correlation operator that exhibits rotationequivariance was introduced; (Carlos Esteves & Daniilidis, 2017), where a network equivariant to  rotation and scale, but invariant to translations was presented, and Worrall & Brostow (2018), where a model equivariant to translations and 3D right-angled rotations was developed. 
Revised: Since then, several other models exhibiting equivariance to certain groups of transformations have been proposed, for example (Cohen et al., 2018b), where a spherical correlation operator that exhibits rotationequivariance was introduced; (Carlos Esteves & Daniilidis, 2017), where a network equivariant to [Table] rotation and scale, but invariant to translations was presented, and Worrall & Brostow (2018), where a model equivariant to translations and 3D right-angled rotations was developed. A general theory of equivariant CNNs was developed in (Cohen et al., 2018a).

Revised: In their paper, they show that convolutions with equivariant kernels are the most general class of equivariant maps between feature spaces.

Original: A fundamental issue witha general group-equivariant convolutional model is the fact that the grid the convolution works with increases exponentially with the type of the transformations considered.
Revised: A fundamental issue with group-equivariant convolutional networks is the fact that the grid the convolution works with increases exponentially with the type of the transformations considered.

Original: This was pointed out in (Sabour et al., 2017); capsules were proposed as an efﬁcient alternative.
Revised: This was pointed out in (Sabour et al., 2017); capsules were proposed as an efﬁcient alternative.

Original: In a general capsule network model, each capsule is supposed to represent the pose-coordinates of an object-component.
Revised: In a general capsule network model, each capsule is supposed to represent the pose-coordinates of an object-component.

Original: Thus, to increase the scope of equivariance, only a linear increase in the dimension of each capsule is necessary.
Revised: Thus, to increase the scope of equivariance, only a linear increase in the dimension of each capsule is necessary.

Original: This was however not formalised in most capsule architectures, which focused on other aspects such as routing (Hinton et al., 2018), (Bahadori, 2018), (Wang & Liu, 2018); general architecture (Rajasegaran et al., 2019), (Deliège et al., 2018), (Rawlinson et al., 2018), Jeong et al.
Revised: This was however not formalised in most capsule architectures, which focused on other aspects such as routing (Hinton et al., 2018), (Bahadori, 2018), (Wang & Liu, 2018); general architecture (Rajasegaran et al., 2019), (Deliège et al., 2018), (Rawlinson et al., 2018), Jeong et al.

Original: (2019), (Phaye et al., 2018), Rosario et al.
Revised: (2019), (Phaye et al., 2018), Rosario et al.

Original: (2019); or application Afshar et al.
Revised: (2019); or application Afshar et al.

Original: It was only in group-equivariant capsules (Lenssen et al., 2018) that this idea of efﬁcient equivariance was formalised.
Revised: It was only in group-equivariant capsules (Lenssen et al., 2018) that this idea of efﬁcient equivariance was formalised.

Original: Indeed, in that paper equivariance changed fromthe preserving the action of a group on a vector space to preserving the group-transformation on an element.
Revised: Indeed, in that paper, equivariance changed from preserving the action of a group on a vector space to preserving the group-transformation on an element.

Original: While such models scale well to larger transformation groups in the sense of preserving equivariance guarantees, we argue that they cannot efﬁciently handle compositionalities that involve more than spatial geometry.
Revised: While such models scale well to larger transformation groups in the sense of preserving equivariance guarantees, we argue that they cannot efﬁciently handle compositionalities that involve more than spatial geometry.

Original: The direct use of capsules as geometric pose-coordinates could lead to exponential representational inefﬁciencies in the number of capsules.
Revised: The direct use of capsules as geometric pose-coordinates could lead to exponential representational inefﬁciencies in the number of capsules.

Original: This is the tradeoff we referred to.
Revised: This is the tradeoff we referred to.

Original: We do not attempt a formalisation of this, and instead make the observation given next.
Revised: We do not attempt a formalisation of this, and instead make the observation given next.

Original: While SOVNET (using GCNNs) lacks in transformational efﬁciency, the use of convolutions allows it better capture non-geometric structures well.
Revised: While SOVNET (using GCNNs) lacks in transformational efﬁciency, the use of convolutions allows it to capture non-geometric structures well.

Original: Further, SOVNET still retains the advantage of learning compositional structures better than CNN models due to the use of routing, placing it in a favourable position between two extremes.
Revised: Further, SOVNET still retains the advantage of learning compositional structures better than CNN models due to the use of routing, placing it in a favourable position between two extremes.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Revised: We presented a general model for capsule networks that is scalable to deep architectures.

Revised: We introduced a prediction mechanism based on group-equivariant convolutions, and a routing procedure based on degree-centrality.

Revised: We showed that this model is equivariant, depending on the nature of the equivariant convolutions.

Original:  A A PPENDIX
Revised: We proved that this results in a preservation of detected compositionalities under transformations. We presented the results of experiments on afﬁne variations of various classiﬁcation datasets, and showed that our model performs better than several capsule network baselines. A second set of experiments showed that our model performs comparably to convolutional baselines on two other datasets. We also discussed a possible tradeoff between efﬁciency in the transformational sense and efﬁciency in the representation of non-geometric compositional relations.

Revised: As future work, we aim at understanding the role of the routing algorithm in the optimality of the capsule-decomposition graph, and various other properties of interest based on it.

Revised: We also note that

Revised: SOVNET allows other equivariant prediction mechanisms - each of which could result in a wider application of SOVNET to different domains.

Revised: A A PPENDIX

Original: We present proofs for the theorems mentioned in the main body.
Revised: We present formal deﬁnitions for the following concepts. A. G ROUP

Revised: A tuple ( G, ◦ ) , where G is a non-empty set and ◦ deﬁnes a binary operation on G is said to form a group if the following properties are satisﬁed:

Revised: Closure: ∀ g 1 , g 2 ∈ G , g 1 ◦ g 2 ∈ G .

Original: {+Associativity: ∀ g 1 , g 2 , g 3 ∈ G , ( g 1 ◦ g 2 ) ◦ g 
+}
Revised: {+Associativity: ∀ g 1 , g 2 , g 3 ∈ G , ( g 1 ◦ g 2 ) ◦ g 
+}

Revised: = g 1 ◦

Revised: ( g 2 ◦ g 3 ) .

Revised: Existence of the identity element: ∃ e ∈ G (cid:51) ∀ g ∈ G , e ◦ g =

Revised: g ◦ e = g .

Revised: Existence of an inverse: ∀ g ∈ G , ∃ g − 1 ∈ G (cid:51)

Revised: [Equation]

Revised: A. G ROUP ACTION AND G ROUP REPRESENTATION

Revised: Given a group ( G, ◦ ) and a vector space V , a group action is a function f : G × V → V satisfying the following properties.

Revised: 1: ∀ a ∈ V , f ( e, a ) = a . 2: ∀ g, h ∈ G and ∀ a ∈ V , f ( h, f ( g, a ))

Revised: =

Revised: f ( h ◦ g, a ) .

Revised: A group representation is a group action by invertible linear maps.

Revised: More formally, a group representation of a group ( G, ◦ ) with respect to a vector space V is a homomorphism from G to

Revised: GL ( V ) - the set of linear, invertible maps from V to V .

Revised: A. P OOL

Revised: Consider a one-layer GCNN-convolutional prediction network Ψ l +1 j for a SOVNET layer l + 1 , and for the d l +1 - dimensional j th capsule-type.

Revised: Intuitively, Pool l +1 j ( g ) is deﬁned by the extent of the support of the g-transformed ﬁlter Ψ l +1 j .

Revised: More formally,

Revised: [Equation]

Revised: For a general L -layer GCNN prediction-

Revised: network, Pool l +1 j ( g ) is deﬁned by recursively applying the above deﬁnition through all the layers of the prediction network.

Revised: A. 2- NORM

Revised: The 2-norm of a vector x = ( x 0 , ..., x n − 1 ) T ∈ R n , and denoted by (cid:107) x (cid:107) 2 , is deﬁned as (cid:107) x (cid:107) 2 =

Revised: ( (cid:80) n − 1 i

Revised: =0 x 2 i ) 12 .

Revised: B P ROOFS OF THEOREMS

Revised: We present proofs for the theorems mentioned in the main body.

Original: Theorem A.1.
Revised: Theorem B.1.

Original: The SOVNET layer deﬁned in Algorithm 2, and denoted by the operator ⊗ as given above, satisﬁes ([ L g F l ] ⊗ Ψ l +1 j ) =
Revised: The SOVNET layer deﬁned in Algorithm 2, and denoted by the operator ⊗ as given above, satisﬁes ([ L g F l ] ⊗ Ψ l +1 j ) =

Original: ( L g [ F l ⊗ Ψ l +1 j ]) , where g belongs to the underlying group of the equivariant convolution.
Revised: ( L g [ F l ⊗ Ψ l +1 j ]) , where g belongs to the underlying group of the equivariant convolution.

Original: Proof.
Revised: Proof.

Original: For the theorem to be true, we must show that each step of Algorithm 2 is equivariant.
Revised: For the theorem to be true, we must show that each step of Algorithm 2 is equivariant.

Original: We do this step-wise.
Revised: We do this step-wise.

Original: The predictions S l +1 ij made in the ﬁrst step are group-equivariant.
Revised: The predictions S l +1 ij made in the ﬁrst step are group-equivariant.

Original: This follows from the fact that
Revised: This follows from the fact that

Original: S l +1 ijp ( g ) =
Revised: S l +1 ijp ( g ) =

Original: ( f li (cid:63) Ψ l
Revised: ( f li (cid:63) Ψ l

Original: +1 ,p j )( g ) , and that ([ L h f li ] (cid:63) Ψ l
Revised: +1 ,p j )( g ) , and that ([ L h f li ] (cid:63) Ψ l

Original: +1 ,p j )
Revised: +1 ,p j )

Original: =
Revised: =

Original: L h ( f li (cid:63) Ψ l
Revised: L h ( f li (cid:63) Ψ l

Original: +1 ,p j ) - proved in (Cohen &
Revised: +1 ,p j ) - proved in (Cohen &

Original: Welling, 2016).
Revised: Welling, 2016).

Original: We now show that the DegreeScore procedure is equivariant.
Revised: We now show that the DegreeScore procedure is equivariant.

Original: We see that Degree ji ( g )
Revised: We see that Degree ji ( g )

Original: [Equation]{+= (cid:80) N l − 1 k =
+}
Revised: {+= (cid:80) N l − 1 k =
+}

Original: Each S l 
Revised: S l +1 ij ( g ) .S l +1 kj ( g ) (cid:107) S l +1 ij ( g ) (cid:107) 2 .

Original: [Equation]
Revised: (cid:107)

Original: g
Revised: S

Original: [Equation]
Revised: l +1 kj ( g ) (cid:107) 2 ) ; 0 ≤ i ≤ N l − 1 . Each S l +1 ij ( g ) .S l +1 kj ( g ) =

Original:  ( f li (cid:63) Ψ l +1 ,p j )( g )( f lk (cid:63)
Revised: (cid:80) d l +1 − 1 p = ( f li (cid:63) Ψ l +1 ,p j )( g )( f lk (cid:63)

Original: Ψ
Revised: Ψ

Original: l +1 ,p j )( g ) .
Revised: l +1 ,p j )( g ) .

Original: From the equivariance of (cid:63) , (cid:80) d l +1 − 1 p =
Revised: From the equivariance of (cid:63) , (cid:80) d l +1 − 1 p =

Original: ([ L h f li ] (cid:63) Ψ l +1 ,p j )( g )( 
Revised: ([ L h f li ] (cid:63) Ψ l +1 ,p j )( g )( [ L h f lk ] (cid:63) Ψ l +1 ,p j )( g ) = (cid:80) d l +1 − 1 p =

Original: [ L h  f lk ] (cid:63) Ψ l +1 ,p j )( g ) =
Revised: L h ( f li (cid:63) Ψ l +1 ,p j )( g ) L h

Original: [Equation]
Revised: [Equation]

Original: l
Revised: Ψ

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: Ψ

Original: [Equation]
Revised: [Equation]

Original: Ψ l +1 ,p j )]( g ) .
Revised: j )]( g ) .

Original: Moreover, the two-norm of an equivariant map is also equivariant - from the equivariance of the post-composition of non-linearities over equivariant maps (Cohen & Welling, 2016).
Revised: Moreover, the 2-norm of an equivariant map is also equivariant - from the equivariance of the post-composition of non-linearities over equivariant maps (Cohen & Welling, 2016).

Original: Also, the division of two (non-zero) equivariant maps is also equivariant.
Revised: Also, the division of two (non-zero) equivariant maps is also equivariant.

Original: Thus, obtaining the degree-scores is equivariant.
Revised: Thus, obtaining the degree-scores is equivariant.

Original: Again, the softmax function preserves the equivariance as it is a point-wise non-linearity.
Revised: Again, the softmax function preserves the equivariance as it is a point-wise non-linearity.

Original: The proof is concluded by pointing out that the product and sum of equivariant maps is also equivariant.
Revised: The proof is concluded by pointing out that the product and sum of equivariant maps is also equivariant.

Original: Theorem A.2.
Revised: Theorem B.2.

Original: Consider an L -layer SOVNET whose activations are routed according to a procedure belonging to the family given by Algorithm 1.
Revised: Consider an L -layer SOVNET whose activations are routed according to a procedure belonging to the family given by Algorithm 1.

Original: Further, assume that this routing procedure is equivariant with respect to the group G .
Revised: Further, assume that this routing procedure is equivariant with respect to the group G .

Original: Then, given an input x and ∀ g ∈ G , G ( x ) and G ([ L g x ]) are isomorphic.
Revised: Then, given an input x and ∀ g ∈ G , G ( x ) and G ([ L g x ]) are isomorphic.

Original: Proof.
Revised: Proof.

Original: Consider a ﬁxed L -layer SOVNET that is equivariant to transformations from a group G , and an input x : G → R c .
Revised: Consider a ﬁxed L -layer SOVNET that is equivariant to transformations from a group G , and an input x : G → R c .

Original: Let G ( x ) be the capsule-decomposition graph corresponding to x .
Revised: Let G ( x ) be the capsule-decomposition graph corresponding to x .

Original: Then G ( L h x ) denotes the the capsule-decomposition graph of the transformed input L h x .
Revised: Then G ( L h x ) denotes the the capsule-decomposition graph of the transformed input L h x .

Original: We show that the map ˜ f li ( g ) → ˜ f li ( h − 1 ◦ g ) is an isomorphism from G ( x ) to G ( L h x ) .
Revised: We show that the map ˜ f li ( g ) → ˜ f li ( h − 1 ◦ g ) is an isomorphism from G ( x ) to G ( L h x ) .

Original: First, we note that ˜ f li ( g ) → ˜ f li ( h − 1 ◦ g ) is a bijection from V ( x ) to V ( L h x ) .
Revised: First, we note that ˜ f li ( g ) → ˜ f li ( h − 1 ◦ g ) is a bijection from V ( x ) to V ( L h x ) .

Original: This is from the deﬁnition of the vertex set of a capsule-decomposition graph and the fact that the map g → h − 1 ◦ g is a bijection.
Revised: This is from the deﬁnition of the vertex set of a capsule-decomposition graph and the fact that the map g → h − 1 ◦ g is a bijection.

Original: We now show that ( ˜ f li ( g 1 ) , ˜ f
Revised: We now show that ( ˜ f li ( g 1 ) , ˜ f

Original: l +1 j ( g 2 ) , c ij ( g 2 ))
Revised: l +1 j ( g 2 ) , c ij ( g 2 ))

Original: ∈ E ( x ) if and only if ( ˜ f l +1 i ( h − 1 ◦ g 1 ) , ˜ f
Revised: ∈ E ( x ) if and only if ( ˜ f l +1 i ( h − 1 ◦ g 1 ) , ˜ f

Original: l +1 j ( h − 1 ◦ g 2 ) , c ij ( h − 1 ◦ g 2 ))
Revised: l +1 j ( h − 1 ◦ g 2 ) , c ij ( h − 1 ◦ g 2 ))

Original: ∈ E ( L h x ) .
Revised: ∈ E ( L h x ) .

Original: First, let us assume ( ˜ f li ( g 1 ) , ˜ f
Revised: First, let us assume ( ˜ f li ( g 1 ) , ˜ f

Original: l +1 j ( g 2 ) , c ij ( g 2 ))
Revised: l +1 j ( g 2 ) , c ij ( g 2 ))

Original: ∈ E ( x ) .
Revised: ∈ E ( x ) .

Original: Thus, ˜ f li ( g 1 ) is routed to ˜ f lj ( g 2 ) with routing-coefﬁcient c ij ( g 2 ) .
Revised: Thus, ˜ f li ( g 1 ) is routed to ˜ f lj ( g 2 ) with routing-coefﬁcient c ij ( g 2 ) .

Original: However, due to the assumed equivariance of the model, ˜ f
Revised: However, due to the assumed equivariance of the model, ˜ f

Original: l +1 i (
Revised: l +1 i (

Original: h − 1 ◦ g 1 ) is routed to ˜ f l +1 j ( h − 1 ◦ g 2 ) with routing-coefﬁcient c ij ( h − 1 ◦ g 2 ) .
Revised: h − 1 ◦ g 1 ) is routed to ˜ f l +1 j ( h − 1 ◦ g 2 ) with routing-coefﬁcient c ij ( h − 1 ◦ g 2 ) .

Original: This, of course, implies ( ˜ f
Revised: This, of course, implies ( ˜ f

Original: l +1 i (
Revised: l +1 i (

Original: h − 1 ◦ g 1 ) , ˜ f
Revised: h − 1 ◦ g 1 ) , ˜ f

Original: l +1 j ( h − 1 ◦ g 2 ) , c ij ( h − 1 ◦ g 2 ))
Revised: l +1 j ( h − 1 ◦ g 2 ) , c ij ( h − 1 ◦ g 2 ))

Original: ∈ E ( L h x ) .
Revised: ∈ E ( L h x ) .

Original: The converse of this result is proved in the same way by considering E ( L h x ) , noting that
Revised: The converse of this result is proved in the same way by considering E ( L h x ) , noting that

Original: E
Revised: E

Original: ( L h − 1 L h x ) =
Revised: ( L h − 1 L h x ) =

Original: E ( x ) , and applying the above result to E ( L h x ) and E ( L h − 1 L h x ) .
Revised: E ( x ) , and applying the above result to E ( L h x ) and E ( L h − 1 L h x ) .

Revised: C F URTHER E XPERIMENTS C. E MPIRICAL VALIDATION OF CAPSULE DECOMPOSITION ISOMORPHISM

Original: In order to compare SOVNET with more sophisticated CNN models, we performed a limited set of experiments on  MNIST and FashionMNIST. 
Revised: We performed two experiments to verify that the capsule decomposition-graphs of the transformed and untransformed images are isomorphic. For the ﬁrst of these, we trained a p4-convolution based SOVNET architecture on untransformed images of MNIST and FashionMNIST. We then considered four variations of the two test-datasetsuntransformed, and three versions rotated exactly by multiples of 90 degrees: 90, 180, and 270. Our experiment veriﬁes that the mapping deﬁned in the proof of Theorem 2.2 is indeed an isomorphism.

Revised: To this end, we considered the capsule-activations as well as the degree-scores, obtained across all the capsule-layers, for each image of all the variations of the test split of the corresponding dataset.

Revised: We then mapped the activations and the degree-scores for the untransformed images by the aforesaid mapping for each of the transformations.

Revised: This corresponds to ’rotating’ the activations and degree-scores by each transformation.

Revised: We then computed the squared error of these with each of the activations and degree-scores obtained from the correspondingly transformed image, respectively.

Revised: A successful veriﬁcation would result in zero error (up to machine precision).

Revised: The results in Tableshow that this happens.

Revised: The second of our experiments is an empirical veriﬁcation that the test-accuracies remain unchanged under transformations for which SOVNET exhibits equivariance.

Revised: We use the same trained architecture as above, and verify that the accuracy remains unchanged under exact transformations of the images.

Revised: The results are presented in Table 7.

Revised: [Table]

Revised: [Table]

Revised: C. R ESULTS ON TESTING ON UNSEEN TRANSFORMS : A FFNIST

Revised: We trained a SOVNET architecture on MNIST images that are padded to size 40x40 - the size of

Revised: AFFNIST images.

Revised: We augment these images by translation, as is the standard approach.

Revised: Note that the changed size of the images necessitates a different architecture.

Revised: The result of this experiment is given in Table.

Revised: We see that our SOVNET architecture obtains the highest accuracy when compared to other recent capsule network models.

Revised: The results of this experiment are presented in Table

Revised: [Table]

Revised: We also trained the above SOVNET architecture on MNIST with translations in the range of [-6,6] pixels and rotations from [-30 ,30] degrees.

Revised: While this increases the extent of train-time augmentation, there are several test-time transformations that are unseen.

Revised: With this scheme, we achieve state-of-the-art accuracy of 98.80%.

Revised: This improves over the best, to our knowledge, accuracy of 98.3% obtained by (Tai et al., 2019).

Revised: [Table]

Revised: C. R ESULTS OF SOVNET ON CIFAR-

Revised: We have trained a SOVNET architecture on CIFAR100.

Revised: Our model has achieved an accuracy of 71.55%, an almost 4 percentage improvement over a recent capsule network model - STARCAPS (Karim Ahmed, 2019) which achieved 67.66%.

Revised: C. C OMPARISON AGAINST R ES N ET ARCHITECTURES

Revised: In order to compare SOVNET with more sophisticated CNN models

Revised: [Table]

Revised: we performed a limited set of experiments on MNIST and FashionMNIST.

Original: We trained ResNet18 and ResNet34 on the train split of these transformed by random translations of up to ± 2 pixels, and random rotations of up to ± 180°.
Revised: We trained ResNet18 and ResNet34 on the train split of MNIST and FashionMNIST transformed by random translations of up to ± 2 pixels [Table] and random rotations of up to ± 180°.

Original: The models were tested on various transformed versions of the test-splits.
Revised: The models were tested on various transformed versions of the test-splits.

Original: The results of these experiments are given in Table 6.
Revised: The results of these experiments are given in Table 9.

Original: As can be seen in the table, SOVNET compares with the two much deeper CNN models.
Revised: As can be seen in the table, SOVNET compares with the two much deeper CNN models.

Original: More testing on more complex datasets, as well as deeper SOVNET models must be done, however, to obtain a better understanding of the relative performance of these two kinds of models.
Revised: More testing on more complex datasets, as well as deeper SOVNET models must be done, however, to obtain a better understanding of the relative performance of these two kinds of models

Original: We also performed this experiment on FashionMNIST for a simple 8-layer deep GCNN-based on p4-convolutions.
Revised: [Table]

Original: [Table]
Revised: [Table]