Original: M ATRIX CAPSULES WITH EM ROUTING
Revised: M ATRIX CAPSULES WITH EM ROUTING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A capsule is a group of neurons whose outputs represent different properties of the same entity.
Revised: A capsule is a group of neurons whose outputs represent different properties of the same entity.

Original: We describe a version of capsules in which each capsule has a logistic unit to represent the presence of an entity and a 4x4 pose matrix which could learn to represent the relationship between that entity and the viewer.
Revised: We describe a version of capsules in which each capsule has a logistic unit to represent the presence of an entity and a 4x4 pose matrix which could learn to represent the relationship between that entity and the viewer.

Original: A capsule in one layer votes for the pose matrix of many different capsules in the layer above by multiplying its own pose matrix by viewpoint-invariant transformation matrices that could learn to represent part-whole relationships.
Revised: A capsule in one layer votes for the pose matrix of many different capsules in the layer above by multiplying its own pose matrix by viewpoint-invariant transformation matrices that could learn to represent part-whole relationships.

Original: Each of these votes is weighted by an assignment coefﬁcient.
Revised: Each of these votes is weighted by an assignment coefﬁcient.

Original: These coefﬁcients are iteratively updated using the EM algorithm such that the output of each capsule is routed to a capsule in the layer above that receives a cluster of similar votes.
Revised: These coefﬁcients are iteratively updated using the EM algorithm such that the output of each capsule is routed to a capsule in the layer above that receives a cluster of similar votes.

Original: The whole system is trained discriminatively by unrolling 3 iterations of EM between each pair of adjacent layers.
Revised: The whole system is trained discriminatively by unrolling 3 iterations of EM between each pair of adjacent layers.

Original: On the smallNORB benchmark, capsules reduce the number of test errors by 45% compared to the state-of-the-art.
Revised: On the smallNORB benchmark, capsules reduce the number of test errors by 45% compared to the state-of-the-art.

Original: Capsules also show far more resistance to white box adversarial attack than our baseline convolutional neural network.
Revised: Capsules also show far more resistance to white box adversarial attack than our baseline convolutional neural network.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Convolutional neural nets are based on a simple fact and a simple mechanism: a vision system needs to use the same knowledge at all locations in the image; this can be achieved by tying the weights of features detectors so that features learned at one location are available at other locations.
Revised: Convolutional neural nets are based on a simple fact and a simple mechanism: a vision system needs to use the same knowledge at all locations in the image; this can be achieved by tying the weights of features detectors so that features learned at one location are available at other locations.

Original: Convolutional capsules extend the sharing of knowledge across locations to include knowledge about the part-whole relationships that characterize a familiar shape.
Revised: Convolutional capsules extend the sharing of knowledge across locations to include knowledge about the part-whole relationships that characterize a familiar shape.

Original: Viewpoint changes have complicated effects on pixel intensities but simple, linear effects on the pose matrix that represents the relationship between an object or object-part and the viewer.
Revised: Viewpoint changes have complicated effects on pixel intensities but simple, linear effects on the pose matrix that represents the relationship between an object or object-part and the viewer.

Original: The aim of capsules is to make good use of this underlying linearity, both for dealing with viewpoint variation and improving segmentation decisions.
Revised: The aim of capsules is to make good use of this underlying linearity, both for dealing with viewpoint variation and improving segmentation decisions.

Original: Capsules use high-dimensional coincidence ﬁltering: a familiar object can be detected by looking for agreement between votes for its pose matrix.
Revised: Capsules use high-dimensional coincidence ﬁltering: a familiar object can be detected by looking for agreement between votes for its pose matrix.

Original: These votes come from parts that have already been detected.
Revised: These votes come from parts that have already been detected.

Original: A part produces a vote by multiplying its own pose matrix by a transformation matrix that represents the viewpoint invariant relationship between the part and the whole.
Revised: A part produces a vote by multiplying its own pose matrix by a transformation matrix that represents the viewpoint invariant relationship between the part and the whole.

Original: As the viewpoint changes, the pose matrices of the parts and the whole will change in a coordinated way so that any agreement between votes from different parts will persist.
Revised: As the viewpoint changes, the pose matrices of the parts and the whole will change in a coordinated way so that any agreement between votes from different parts will persist.

Original: Finding tight clusters of high-dimensional votes that agree in a mist of irrelevant votes is one way of solving the problem of assigning parts to wholes.
Revised: Finding tight clusters of high-dimensional votes that agree in a mist of irrelevant votes is one way of solving the problem of assigning parts to wholes.

Original: It is non-trivial because we cannot grid the high-dimensional space.
Revised: It is non-trivial because we cannot grid the high-dimensional space.

Original: To solve this challenge, we use a fast iterative process called ”routing-by-agreement” that updates the probability with which a part is assigned to a whole based on the proximity of the vote coming from that part to the votes coming from other parts that are assigned to the whole.
Revised: To solve this challenge, we use a fast iterative process called ”routing-by-agreement” that updates the probability with which a part is assigned to a whole based on the proximity of the vote coming from that part to the votes coming from other parts that are assigned to the whole.

Original: This is a powerful segmentation principle that allows knowledge of familiar shapes to drive segmentation, rather than just using low-level cues such as proximity or agreement in color or velocity.
Revised: This is a powerful segmentation principle that allows knowledge of familiar shapes to drive segmentation, rather than just using low-level cues such as proximity or agreement in color or velocity.

Original: 1. P REVIOUS WORK ON CAPSULES
Revised: 1. P REVIOUS WORK ON CAPSULES

Original: Hinton et al.
Revised: Hinton et al.

Original: (2011) used a transformation matrix in a transforming autoencoder that learned to transform a stereo pair of images into a stereo pair from a slightly different viewpoint.
Revised: (2011) used a transformation matrix in a transforming autoencoder that learned to transform a stereo pair of images into a stereo pair from a slightly different viewpoint.

Original: However, that system required the transformation matrix to be supplied externally.
Revised: However, that system required the transformation matrix to be supplied externally.

Original: More recently, routing-byagreement was shown to be very effective for segmenting highly overlapping digits (Sabour et al.
Revised: More recently, routing-byagreement was shown to be very effective for segmenting highly overlapping digits (Sabour et al.

Original: but that system has several deﬁciencies that we have defeated in our work:
Revised: but that system has several deﬁciencies that we have defeated in our work:

Original: It uses the length of the pose vector to represent the probability that the entity represented by a capsule is present.
Revised: It uses the length of the pose vector to represent the probability that the entity represented by a capsule is present.

Original: To keep the length less than 1 requires an unprincipled non-linearity that prevents there from being any sensible objective function that is minimized by the iterative routing procedure.
Revised: To keep the length less than 1 requires an unprincipled non-linearity that prevents there from being any sensible objective function that is minimized by the iterative routing procedure.

Original: It uses the cosine of the angle between two pose vectors to measure their agreement.
Revised: It uses the cosine of the angle between two pose vectors to measure their agreement.

Original: Unlike the log variance of a Gaussian cluster, the cosine is not good at distinguishing between quite good agreement and very good agreement.
Revised: Unlike the log variance of a Gaussian cluster, the cosine is not good at distinguishing between quite good agreement and very good agreement.

Original: It uses a vector of length n rather than a matrix with n elements to represent a pose, so its transformation matrices have n 2 parameters rather than just n .
Revised: It uses a vector of length n rather than a matrix with n elements to represent a pose, so its transformation matrices have n 2 parameters rather than just n .

Original: 1. R ELATED WORK
Revised: 1. R ELATED WORK

Original: Among the multiple recent attempts on bringing underlying afﬁne manifold of images to CNNs, the two main streams are attempts on gaining viewpoint invarient vs equivarient.
Revised: Among the multiple recent attempts on bringing underlying afﬁne manifold of images to CNNs, the two main streams are attempts on gaining viewpoint invarient vs equivarient.

Original: Spatial transformer networks (Jaderberg et al.
Revised: Spatial transformer networks (Jaderberg et al.

Original: (2015) seeks viewpoint invarient by changing the sampling of CNNs according to a selection of afﬁne transformations.
Revised: (2015) seeks viewpoint invarient by changing the sampling of CNNs according to a selection of afﬁne transformations.

Original: De Brabandere et al.
Revised: De Brabandere et al.

Original: (2016) extends spatial transformer networks where ﬁlters are adapted during inference depending on the input.
Revised: (2016) extends spatial transformer networks where ﬁlters are adapted during inference depending on the input.

Original: They generate different ﬁlters for each locality in the feature map rather than applying the same transformation to all ﬁlters.
Revised: They generate different ﬁlters for each locality in the feature map rather than applying the same transformation to all ﬁlters.

Original: Their approach is a step toward input covariance detection from traditional pattern matching frameworks like standard convolutional neural networks (LeCun et al.
Revised: Their approach is a step toward input covariance detection from traditional pattern matching frameworks like standard convolutional neural networks (LeCun et al.

Original: Dai et al.
Revised: Dai et al.

Original: (2017) also improves upon STNs by generalizing the sampling method of ﬁlters.
Revised: (2017) also improves upon STNs by generalizing the sampling method of ﬁlters.

Original: Our work differs substantially in that a unit is not activated based on the matching score with a ﬁlter (either ﬁxed or dynamically changing during inference).
Revised: Our work differs substantially in that a unit is not activated based on the matching score with a ﬁlter (either ﬁxed or dynamically changing during inference).

Original: In our case, a capsule is activated only if the transformed poses coming from the layer below match each other.
Revised: In our case, a capsule is activated only if the transformed poses coming from the layer below match each other.

Original: This is a more effective way to capture covariance and leads to models with many fewer parameters that generalize better.
Revised: This is a more effective way to capture covariance and leads to models with many fewer parameters that generalize better.

Original: The success of CNNs has motivated many researchers to extend the translational equivariance built in to CNNs to include rotational equivariance (Cohen & Welling (2016), Dieleman et al.
Revised: The success of CNNs has motivated many researchers to extend the translational equivariance built in to CNNs to include rotational equivariance (Cohen & Welling (2016), Dieleman et al.

Original: Oyallon & Mallat (2015)).
Revised: Oyallon & Mallat (2015)).

Original: The recent approach in Harmonic Networks (Worrall et al.
Revised: The recent approach in Harmonic Networks (Worrall et al.

Original: (2017)) achieves rotation equivariant feature maps by using circular harmonic ﬁlters and returning both the maximal response and orientation using complex numbers.
Revised: (2017)) achieves rotation equivariant feature maps by using circular harmonic ﬁlters and returning both the maximal response and orientation using complex numbers.

Original: This shares the basic representational idea of capsules: By assuming that there is only one instance of the entity at a location, we can use several different numbers to represent its properties.
Revised: This shares the basic representational idea of capsules: By assuming that there is only one instance of the entity at a location, we can use several different numbers to represent its properties.

Original: They used a ﬁxed number of streams of rotation orders.
Revised: They used a ﬁxed number of streams of rotation orders.

Original: By enforcing the equality of the sum of rotation orders along any path they achieve patch-wise rotation equivariance.
Revised: By enforcing the equality of the sum of rotation orders along any path they achieve patch-wise rotation equivariance.

Original: This approach is more parameter-efﬁcient than data augmentation approaches, duplicating feature maps, or duplicating ﬁlters (Fasel & Gatica-Perez (2006), Laptev et al.
Revised: This approach is more parameter-efﬁcient than data augmentation approaches, duplicating feature maps, or duplicating ﬁlters (Fasel & Gatica-Perez (2006), Laptev et al.

Original: Our approach encodes general viewpoint equivarience rather than only Afﬁne 2D rotations.
Revised: Our approach encodes general viewpoint equivarience rather than only Afﬁne 2D rotations.

Original: Symmetry networks (Gens & Domingos (2014)) uses iterative Lucas-Kanade optimization to ﬁnd poses that are supported by the most low-level features.
Revised: Symmetry networks (Gens & Domingos (2014)) uses iterative Lucas-Kanade optimization to ﬁnd poses that are supported by the most low-level features.

Original: Their key weakness is that the iterative algorithm always starts at the same poses, rather than the mean of the bottom-up votes.
Revised: Their key weakness is that the iterative algorithm always starts at the same poses, rather than the mean of the bottom-up votes.

Original: Lenc & Vedaldi (2016) proposes a feature detection mechanism (DetNet) that is equivariant to afﬁne transformations.
Revised: Lenc & Vedaldi (2016) proposes a feature detection mechanism (DetNet) that is equivariant to afﬁne transformations.

Original: DetNet is designed to detect the same points in the image under different viewpoint variations.
Revised: DetNet is designed to detect the same points in the image under different viewpoint variations.

Original: This effort is orthogonal to our work but DetNet might be a good way to implement the de-rendering ﬁrst-stage that activates the layer of primary capsules.
Revised: This effort is orthogonal to our work but DetNet might be a good way to implement the de-rendering ﬁrst-stage that activates the layer of primary capsules.

Original: Our routing algorithm can be seen as an attention mechanism.
Revised: Our routing algorithm can be seen as an attention mechanism.

Original: In this view it is related to work of
Revised: In this view it is related to work of

Original: Gregor et al.
Revised: Gregor et al.

Original: (2015), where they improved the decoder performance in a generative model by using
Revised: (2015), where they improved the decoder performance in a generative model by using

Original: Gaussian kernels to attend to different parts of the feature map generated by the encoder.
Revised: Gaussian kernels to attend to different parts of the feature map generated by the encoder.

Original: Vaswani et al.
Revised: Vaswani et al.

Original: (2017) uses a softmax attention mechanism to match parts of the query sequence to parts of the input sequence for the translation task and when generating an encoding for the query.
Revised: (2017) uses a softmax attention mechanism to match parts of the query sequence to parts of the input sequence for the translation task and when generating an encoding for the query.

Original: They show improvement upon previous translation efforts using recurrent architectures.
Revised: They show improvement upon previous translation efforts using recurrent architectures.

Original: Our algorithm has attention in the opposite direction.
Revised: Our algorithm has attention in the opposite direction.

Original: The competition is not between the lower-level capsules that a higher-level capsule might attend to.
Revised: The competition is not between the lower-level capsules that a higher-level capsule might attend to.

Original: It is between the higher-level capsules that a lower-level capsule might send its vote to.
Revised: It is between the higher-level capsules that a lower-level capsule might send its vote to.

Original:  U SING EM FOR ROUTING - BY - AGREEMENT
Revised:  U SING EM FOR ROUTING - BY - AGREEMENT

Original: Let us suppose that we have already decided on the poses and activation probabilities of all the capsules in a layer and we now want to decide which capsules to activate in the layer above and how to assign each active lower-level capsule to one active higher-level capsule.
Revised: Let us suppose that we have already decided on the poses and activation probabilities of all the capsules in a layer and we now want to decide which capsules to activate in the layer above and how to assign each active lower-level capsule to one active higher-level capsule.

Original: For explanatory purposes, we start by assuming that all of the transformation matrices are the identity matrix because this makes it easy to map the assignment task onto the task of ﬁtting a mixture of Gaussians.
Revised: For explanatory purposes, we start by assuming that all of the transformation matrices are the identity matrix because this makes it easy to map the assignment task onto the task of ﬁtting a mixture of Gaussians.

Original: Each capsule in the higher layer corresponds to a Gaussian and each active capsule in the lower layer corresponds to a data-point (or a fraction of a data-point if the capsule is partially active).
Revised: Each capsule in the higher layer corresponds to a Gaussian and each active capsule in the lower layer corresponds to a data-point (or a fraction of a data-point if the capsule is partially active).

Original: The only difference from a standard mixture of Gaussians is that we have far too many Gaussians so we need to introduce a penalty to stop us from assigning every data-point to a different Gaussian.
Revised: The only difference from a standard mixture of Gaussians is that we have far too many Gaussians so we need to introduce a penalty to stop us from assigning every data-point to a different Gaussian.

Original: Using the minimum description length principle we have a choice when deciding whether or not to activate a higher-level capsule.
Revised: Using the minimum description length principle we have a choice when deciding whether or not to activate a higher-level capsule.

Original: Choice 0 : if we do not activate it, we must pay a ﬁxed cost per data-point for describing the means of all the lower-level capsules that are assigned to a higherlevel capsule (assuming an improper uniform prior).
Revised: Choice 0 : if we do not activate it, we must pay a ﬁxed cost per data-point for describing the means of all the lower-level capsules that are assigned to a higherlevel capsule (assuming an improper uniform prior).

Original: For fractional assignments we pay that fraction of the ﬁxed cost.
Revised: For fractional assignments we pay that fraction of the ﬁxed cost.

Original: Choice 1 : if we do activate the higher-level capsule we must pay a ﬁxed cost for coding its mean and the fact that it is active and then pay additional costs, pro-rated by the assignment probabilities, for describing the discrepancies between the lower-level means and the values predicted for them when the mean of the higher-level capsule is used to predict them via the inverse of the transformation matrix.
Revised: Choice 1 : if we do activate the higher-level capsule we must pay a ﬁxed cost for coding its mean and the fact that it is active and then pay additional costs, pro-rated by the assignment probabilities, for describing the discrepancies between the lower-level means and the values predicted for them when the mean of the higher-level capsule is used to predict them via the inverse of the transformation matrix.

Original: A much simpler way to compute the cost of describing the discrepancies is to use the squared distance between the mean of the higher-level capsule and the vote from the lower-level capsule, scaled by the variance of the higher-level capsule.
Revised: A much simpler way to compute the cost of describing the discrepancies is to use the squared distance between the mean of the higher-level capsule and the vote from the lower-level capsule, scaled by the variance of the higher-level capsule.

Original: This is incorrect because the transformation matrix may compress or expand probability density
Revised: This is incorrect because the transformation matrix may compress or expand probability density

Original: but we use it anyway because it saves us from having to perform any additional transformations during the iterative routing process.
Revised: but we use it anyway because it saves us from having to perform any additional transformations during the iterative routing process.

Original: The difference in cost between these two choices, is put through the logistic function on each iteration to determine the higher-level capsule’s activation probability.
Revised: The difference in cost between these two choices, is put through the logistic function on each iteration to determine the higher-level capsule’s activation probability.

Original: If the transformation matrices are not the identity matrix, the iterative routing process differs from ﬁtting a standard mixture of Gaussians and the difference makes it converge much faster.
Revised: If the transformation matrices are not the identity matrix, the iterative routing process differs from ﬁtting a standard mixture of Gaussians and the difference makes it converge much faster.

Original: The higherlevel capsules all get different views of the same set of data-points because they see the data-points transformed by different part-whole relationships.
Revised: The higherlevel capsules all get different views of the same set of data-points because they see the data-points transformed by different part-whole relationships.

Original: This means that data-points that form a tight cluster from the perspective of one capsule may be widely scattered from the perspective of another capsule.
Revised: This means that data-points that form a tight cluster from the perspective of one capsule may be widely scattered from the perspective of another capsule.

Original: These different perspectives break the symmetry of the Gaussians much more strongly than simply starting with different means and variances.
Revised: These different perspectives break the symmetry of the Gaussians much more strongly than simply starting with different means and variances.

Original: The cost of explaining a whole data-point i by using capsule c that has an axis-aligned covariance matrix is simply the sum over all dimensions of the cost of explaining each dimension, h , of i .
Revised: The cost of explaining a whole data-point i by using capsule c that has an axis-aligned covariance matrix is simply the sum over all dimensions of the cost of explaining each dimension, h , of i .

Original: This is simply − ln
Revised: This is simply − ln

Original: ( P ih ) where P ih is the probability density of the h th component of the vote from i under the capsule’s Gaussian model for dimension h which has variance σ 2 h :
Revised: ( P ih ) where P ih is the probability density of the h th component of the vote from i under the capsule’s Gaussian model for dimension h which has variance σ 2 h :

Original: [Equation]
Revised: [Equation]

Original: Summing over all lower-level capsules for a single dimension of one higher-level capsule we get: cost h = (cid:88)
Revised: Summing over all lower-level capsules for a single dimension of one higher-level capsule we get: cost h = (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Where (cid:80) i r i is the amount of data assigned to c ( i.e. the sum of its assignment probabilities, r i ), k is a constant and V ih
Revised: Where (cid:80) i r i is the amount of data assigned to c ( i.e. the sum of its assignment probabilities, r i ), k is a constant and V ih

Original: is the value on dimension h of the vote from capsule i to capsule c .
Revised: is the value on dimension h of the vote from capsule i to capsule c .

Original: V ih is the product of the the transformation matrix W ic that is learned discriminatively.
Revised: V ih is the product of the the transformation matrix W ic that is learned discriminatively.

Original: Turning on c increases the description length for the means of the lower-level capsules assigned to c by the sum of the cost over all dimensions, so we deﬁne the activation function of capsules c to be:
Revised: Turning on c increases the description length for the means of the lower-level capsules assigned to c by the sum of the cost over all dimensions, so we deﬁne the activation function of capsules c to be:

Original: [Equation]
Revised: [Equation]

Original: where − b represents the cost of describing the mean of capsule c and λ is an inverse temperature parameter.
Revised: where − b represents the cost of describing the mean of capsule c and λ is an inverse temperature parameter.

Original: We learn b discriminatively and set a ﬁxed schedule for λ as a hyper-parameter.
Revised: We learn b discriminatively and set a ﬁxed schedule for λ as a hyper-parameter.

Original: For ﬁnalizing the pose parameters and activations of the capsules in layer L + 1 we run the EM algorithm for few iterations (normally 3 ) after the pose parameters and activations have already been ﬁnalized in layer L .
Revised: For ﬁnalizing the pose parameters and activations of the capsules in layer L + 1 we run the EM algorithm for few iterations (normally 3 ) after the pose parameters and activations have already been ﬁnalized in layer L .

Original: The non-linearity implemented by a whole capsule layer is a form of cluster ﬁnding using the EM algorithm, so we call it EM Routing .
Revised: The non-linearity implemented by a whole capsule layer is a form of cluster ﬁnding using the EM algorithm, so we call it EM Routing .

Original: Procedure 1 Routing algorithm 1 returns activation and pose of the capsules in layer L + 1 given the activations and votes of capsules in layer L .
Revised: Procedure 1 Routing algorithm 1 returns activation and pose of the capsules in layer L + 1 given the activations and votes of capsules in layer L .

Original: V ich is an H dimensional vote from capsule i with activation a i in layer L to capsule c in layer L + 1 .
Revised: V ich is an H dimensional vote from capsule i with activation a i in layer L to capsule c in layer L + 1 .

Original: β a , β v are learned discriminatively and the inverse temperature λ increases at each iteration with a ﬁxed schedule.
Revised: β a , β v are learned discriminatively and the inverse temperature λ increases at each iteration with a ﬁxed schedule.

Original: 1: procedure EM R OUTING ( a , V ) 2: ∀ i, c : R ic ← 1 /size
Revised: 1: procedure EM R OUTING ( a , V ) 2: ∀ i, c : R ic ← 1 /size

Original: ( L + 1) 3: for t iterations do 4: ∀ c : M c : , S c : , a (cid:48)
Revised: ( L + 1) 3: for t iterations do 4: ∀ c : M c : , S c : , a (cid:48)

Original: c
Revised: c

Original: ← M-
Revised: ← M-

Original: STEP ( R : c , a , V : c : ) 5: ∀ i : R i : ← E- STEP ( M , S , a (cid:48) , V i :: ) return a (cid:48) , M 1: procedure
Revised: STEP ( R : c , a , V : c : ) 5: ∀ i : R i : ← E- STEP ( M , S , a (cid:48) , V i :: ) return a (cid:48) , M 1: procedure

Original: M-
Revised: M-

Original: STEP ( r , a , V (cid:48) )
Revised: STEP ( r , a , V (cid:48) )

Original: (cid:46) for one higher-level capsule 2: ∀
Revised: (cid:46) for one higher-level capsule 2: ∀

Original: i : r (cid:48)
Revised: i : r (cid:48)

Original: i ← r i ∗ a i 3: ∀ h : µ h ← (cid:80) i r (cid:48)
Revised: i ← r i ∗ a i 3: ∀ h : µ h ← (cid:80) i r (cid:48)

Original: i V (cid:48)
Revised: i V (cid:48)

Original: ih (cid:80) i r (cid:48)
Revised: ih (cid:80) i r (cid:48)

Original: i 4: ∀ h : σ 2 h ← (cid:80) i r (cid:48)
Revised: i 4: ∀ h : σ 2 h ← (cid:80) i r (cid:48)

Original: i ( V (cid:48) ih − µ h ) 2 (cid:80) i r (cid:48)
Revised: i ( V (cid:48) ih − µ h ) 2 (cid:80) i r (cid:48)

Original: i 5: cost h ← ( β v + log ( σ h ))
Revised: i 5: cost h ← ( β v + log ( σ h ))

Original: (cid:80) i r (cid:48)
Revised: (cid:80) i r (cid:48)

Original: i 6: a (cid:48) ← sigmoid ( λ ( β a − (cid:80) h cost h )) 7: return µ one level
Revised: i 6: a (cid:48) ← sigmoid ( λ ( β a − (cid:80) h cost h )) 7: return µ one level

Original: [Table]
Revised: [Table]

Original: a
Revised: a

Original: [Table]
Revised: [Table]

Original: 4: return r
Revised: 4: return r

Original: 2. T HE OBJECTIVE FUNCTION FOR EM ROUTING
Revised: 2. T HE OBJECTIVE FUNCTION FOR EM ROUTING

Original: There is a free energy function that is reduced by each step of the routing.
Revised: There is a free energy function that is reduced by each step of the routing.

Original: The recomputation of the means and the variances reduces an energy equal to the squared distances of the votes from the means, weighted by the assignment probabilities.
Revised: The recomputation of the means and the variances reduces an energy equal to the squared distances of the votes from the means, weighted by the assignment probabilities.

Original: The recomputation of the capsule activations reduces a free energy equal to the sum of the (scaled) costs used in the logistic function minus the entropy of the activation values.
Revised: The recomputation of the capsule activations reduces a free energy equal to the sum of the (scaled) costs used in the logistic function minus the entropy of the activation values.

Original: The recomputation of the assignment probabilities reduces the sum of the two energies above minus the entropies of the assignment probabilities.
Revised: The recomputation of the assignment probabilities reduces the sum of the two energies above minus the entropies of the assignment probabilities.

Original:  T HE C APSULES M ODEL
Revised:  T HE C APSULES M ODEL

Original: The general architecture of our model is shown in Fig.
Revised: The general architecture of our model is shown in Fig.

Original: The model starts with a 5x5 convolutional layer with 32 channels (A=32) and a stride of 2 with a ReLU non-linearity.
Revised: The model starts with a 5x5 convolutional layer with 32 channels (A=32) and a stride of 2 with a ReLU non-linearity.

Original: All the other layers are capsule layers starting with the primary capsule layer.
Revised: All the other layers are capsule layers starting with the primary capsule layer.

Original: Each capsule has a 4x4 pose matrix and one logistic activation unit.
Revised: Each capsule has a 4x4 pose matrix and one logistic activation unit.

Original: The 4x4 pose of each of the B=32 primary capsule types is a linear transformation of the output of all the lower layer ReLUs centered at that location.
Revised: The 4x4 pose of each of the B=32 primary capsule types is a linear transformation of the output of all the lower layer ReLUs centered at that location.

Original: The activations of the primary capsules are produced by applying the sigmoid function to weighted sums of the same set of loer layer ReLUs.
Revised: The activations of the primary capsules are produced by applying the sigmoid function to weighted sums of the same set of lower layer ReLUs.

Original: The primary capsules are followed by two 3x3 convolutional capsule layers (K=3), each withcapsule types (C=D=32) with strides of 2 and one, respectively.
Revised: The primary capsules are followed by two 3x3 convolutional capsule layers (K=3), each withcapsule types (C=D=32) with strides of 2 and one, respectively.

Original: The last layer of convolutional capsules is connected to the ﬁnal capsule layer which has one capsule per output class.
Revised: The last layer of convolutional capsules is connected to the ﬁnal capsule layer which has one capsule per output class.

Original: When connecting the last convolutional capsule layer to the ﬁnal layer we do not want to throw away information about the location of the convolutional capsules but we also want to make use of the fact that all capsules of the same type are extracting the same entity at different positions.
Revised: When connecting the last convolutional capsule layer to the ﬁnal layer we do not want to throw away information about the location of the convolutional capsules but we also want to make use of the fact that all capsules of the same type are extracting the same entity at different positions.

Original: We therefore share the transformation matrices between different positions of the same capsule type and add the scaled coordinate (row, column) of the center of the receptive ﬁeld of each capsule to the ﬁrst two elements of its vote.
Revised: We therefore share the transformation matrices between different positions of the same capsule type and add the scaled coordinate (row, column) of the center of the receptive ﬁeld of each capsule to the ﬁrst two elements of its vote.

Original: We refer to this technique as Coordinate Addition.
Revised: We refer to this technique as Coordinate Addition.

Original: This should encourage the shared ﬁnal transformations to produce values for those two elements that represent the ﬁne position of the entity relative to the center of the capsule’s receptive ﬁeld.
Revised: This should encourage the shared ﬁnal transformations to produce values for those two elements that represent the ﬁne position of the entity relative to the center of the capsule’s receptive ﬁeld.

Original: The routing procedure is used between each adjacent pair of capsule layers.
Revised: The routing procedure is used between each adjacent pair of capsule layers.

Original: For convolutional capsules, each capsule in layer L + 1 sends feedback only to capsules within its receptive ﬁeld in layer
Revised: For convolutional capsules, each capsule in layer L + 1 sends feedback only to capsules within its receptive ﬁeld in layer

Original: L .
Revised: L .

Original: Therefore each convolutional instance of a capsule in layer L receives at most kernel size X kernel size feedback from each capsule type in layer L + 1 .
Revised: Therefore each convolutional instance of a capsule in layer L receives at most kernel size X kernel size feedback from each capsule type in layer L + 1 .

Original: The instances closer to the border of the image receive fewer feedbacks with corner ones receiving only one feedback per capsule type in layer L + 1 .
Revised: The instances closer to the border of the image receive fewer feedbacks with corner ones receiving only one feedback per capsule type in layer L + 1 .

Original: [Table]
Revised: [Table]

Original: al.

Original: (2011) with

Original: [Table]
Revised: [Table]

Original: input
Revised: Our Best model (third row)

Original: [Table]
Revised: [Table]

Original: Our Best model (third row), with multiple crops during testing
Revised: with multiple crops during testing

Original: [Table]
Revised: [Table]

Original: 3. S PREAD L OSS
Revised: 3. S PREAD L OSS

Original: In order to make the training less sensitive to the initialization and hyper-parameters of the model, we use ”spread loss” to directly maximize the gap between the activation of the target class ( a t ) and the activation of the other classes.
Revised: In order to make the training less sensitive to the initialization and hyper-parameters of the model [Table] we use ”spread loss” to directly maximize the gap between the activation of the target class ( a t ) and the activation of the other classes.

Original: If the activation of a wrong class, a i , is closer than the margin, m , to a t then it is penalized by the squared distance to the margin:
Revised: If the activation of a wrong class, a i , is closer than the margin, m , to a t then it is penalized by the squared distance to the margin:

Original: [Equation]
Revised: [Equation]

Original: By starting with a small margin of 0 . 2 and linearly increasing it  to 0 .
Revised: By starting with a small margin of 0 . 2 and linearly increasing it during training to 0 .

Original: 9 , we avoid dead capsules.
Revised: 9 , we avoid dead capsules.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The smallNORB dataset (LeCun et al.
Revised: The smallNORB dataset (LeCun et al.

Original: (2004)) has gray-level stereo images of 5 classes of toy: airplanes, cars, trucks, humans and animals.
Revised: (2004)) has gray-level stereo images of 5 classes of toy: airplanes, cars, trucks, humans and animals.

Original: There are 10 physical instances of each class which are painted matte green.
Revised: There are 10 physical instances of each class which are painted matte green.

Original: 5 physical instances of a class are selected for the training data and the otherfor the test data.
Revised: 5 physical instances of a class are selected for the training data and the otherfor the test data.

Original: Every individual toy is pictured at 18 different azimuths (0-340), 9 elevations and 6 lighting conditions, so the training and test sets each contain 24,300 stereo pairs of 96x96 images.
Revised: Every individual toy is pictured at 18 different azimuths (0-340), 9 elevations and 6 lighting conditions, so the training and test sets each contain 24,300 stereo pairs of 96x96 images.

Original: We selected smallNORB as a benchmark for developing our capsules system because it is carefully designed to be a pure shape recognition task that is not confounded by context and color, but it is much closer to natural images than MNIST .
Revised: We selected smallNORB as a benchmark for developing our capsules system because it is carefully designed to be a pure shape recognition task that is not confounded by context and color, but it is much closer to natural images than MNIST .

Original: We downsample smallNORB to 48 × 48 pixels and normalize each image to have zero mean and unit variance.
Revised: We downsample smallNORB to 48 × 48 pixels and normalize each image to have zero mean and unit variance.

Original: During training we randomly crop 32 × 32 patches and, after add random brightness and contrast to the image.
Revised: During training we randomly crop 32 × 32 patches and, after add random brightness and contrast to the image.

Original: During test we crop a 32 × 32 patch from the center of the image.
Revised: During test we crop a 32 × 32 patch from the center of the image.

Original: We achieve 1 . 4 % test error on smallNORB when averaging the class activations over multiple patches and 1 . 8% when only using the cetral crop.
Revised: We achieve 1 . 4 % test error on smallNORB when averaging the class activations over multiple patches and 1 . 8% when only using the cetral crop.

Original: The best reported result on smallNORB is 2 .
Revised: The best reported result on smallNORB is 2 .

Original: 56% (Cires¸an et al.
Revised: 56% (Cires¸an et al.

Original: To achieve this they added two additional stereo pairs of input images that are created by using an on-center off-surround ﬁlter and an off-center on-surround ﬁlter.
Revised: To achieve this they added two additional stereo pairs of input images that are created by using an on-center off-surround ﬁlter and an off-center on-surround ﬁlter.

Original: They also applied afﬁne distortions to the images.
Revised: They also applied afﬁne distortions to the images.

Original: Our work also beats the Sabour et al.
Revised: Our work also beats the Sabour et al.

Original: (2017) capsule work which achieves 2.7% on smallNORB.
Revised: (2017) capsule work which achieves 2.7% on smallNORB.

Original: We also tested our model on NORB which is a jittered version of smallNORB with added background and we achieved a 2.6% error rate which is on par with the state-of-the-art of 2.7% (Ciresan et al.
Revised: We also tested our model on NORB which is a jittered version of smallNORB with added background and we achieved a 2.6% error rate which is on par with the state-of-the-art of 2.7% (Ciresan et al.

Original: As the baseline for our experiments on generalization to novel viewpoints we train a CNN which has two convolutional layers with 32 and 64 channels respectively.
Revised: As the baseline for our experiments on generalization to novel viewpoints we train a CNN which has two convolutional layers with 32 and 64 channels respectively.

Original: Both layers have kernel size 5 andstride of 1 with a 2 × 2 max pooling. The third layer is a 1024 unit fully connected layer with dropout
Revised: Both layers have kernel size 5 and

Original: [Table]
Revised: [Table]

Original:  and connects to the 5-way softmax output layer.
Revised: stride of 1 with a 2 × 2 max pooling. The third layer is a 1024 unit fully connected layer with dropout and connects to the 5-way softmax output layer.

Original: All hidden units use the ReLU non-linearity.
Revised: All hidden units use the ReLU non-linearity.

Original: We use the same image preparation for CNN baseline as described above for the capsule network.
Revised: We use the same image preparation for CNN baseline as described above for the capsule network.

Original: Our baseline CNN was the result of an extensive hyperparameter search over ﬁlter sizes, numbers of channels and learning rates.
Revised: Our baseline CNN was the result of an extensive hyperparameter search over ﬁlter sizes, numbers of channels and learning rates.

Original: The CNN baseline achieves 5 .
Revised: The CNN baseline achieves 5 .

Original: 2% test error rate on smallNORB and has 4.2M parameters.
Revised: 2% test error rate on smallNORB and has 4.2M parameters.

Original: We deduce that the Cires¸an et al.
Revised: We deduce that the Cires¸an et al.

Original: (2011) network has 2.7M parameters.
Revised: (2011) network has 2.7M parameters.

Original: By using small matrix multiplies we reduced the number of parameters by a factor of 15 to 310K compared with our baseline CNN (and 9 w.r.t Cires¸an et al.
Revised: By using small matrix multiplies we reduced the number of parameters by a factor of 15 to 310K compared with our baseline CNN (and 9 w.r.t Cires¸an et al.

Original: A smaller capsule network of A = 64 , B = 8 , C
Revised: A smaller capsule network of A = 64 , B = 8 , C

Original: = D
Revised: = D

Original: = 16 with only 68K trainable parameters achieves 2 .
Revised: = 16 with only 68K trainable parameters achieves 2 .

Original: 2% test error rate which also beats the prior state-of-theart.
Revised: 2% test error rate which also beats the prior state-of-theart.

Original: Replacing the MDL-derived capsule activation term with posterior probabilities that are proportional to the sum of the assignment probabilities increases the test error rate on smallNORB to 4 .
Revised: Replacing the MDL-derived capsule activation term with posterior probabilities that are proportional to the sum of the assignment probabilities increases the test error rate on smallNORB to 4 .

Original: Tab.summarizes the effects of the number of routing iterations, the type of loss and the use of matrices rather than vectors for the poses.
Revised: Tab.summarizes the effects of the number of routing iterations, the type of loss and the use of matrices rather than vectors for the poses.

Original: Exactly the same capsules model as Fig.
Revised: Exactly the same capsules model as Fig.

Original: 1 achieves 0.44% test error rate on MNIST.
Revised: 1 achieves 0.44% test error rate on MNIST.

Original: If the number of channels in the ﬁrst hidden layer is increased to 256, it achieves 11.9% test error rate on Cifar (Krizhevsky & Hinton (2009)).
Revised: If the number of channels in the ﬁrst hidden layer is increased to 256, it achieves 11.9% test error rate on Cifar (Krizhevsky & Hinton (2009)).

Original: 4. G ENERALIZATION TO NOVEL VIEWPOINTS
Revised: 4. G ENERALIZATION TO NOVEL VIEWPOINTS

Original: A more severe test of generalization is to use a limited range of viewpoints for training and to test on a much wider range.
Revised: A more severe test of generalization is to use a limited range of viewpoints for training and to test on a much wider range.

Original: We trained both our convolutional baseline and our capsule model on one third of the training data containing azimuths of (300, 320, 340, 0, 20, 40) and tested on the two thirds of the test data that contained azimuths from 60 to 280.
Revised: We trained both our convolutional baseline and our capsule model on one third of the training data containing azimuths of (300, 320, 340, 0, 20, 40) and tested on the two thirds of the test data that contained azimuths from 60 to 280.

Original: In a separate experiment we trained on thesmaller elevations and tested on the 6 larger elevations.
Revised: In a separate experiment we trained on thesmaller elevations and tested on the 6 larger elevations.

Original: It is hard to decide if the capsules model is better at generalizing to novel viewpoints because it is better at all viewpoints.
Revised: It is hard to decide if the capsules model is better at generalizing to novel viewpoints because it is better at all viewpoints.

Original: To eliminate this confounding factor we stopped training the capsules model when its performance matched the baseline CNN on the third of the test set that used the training viewpoints.
Revised: To eliminate this confounding factor we stopped training the capsules model when its performance matched the baseline CNN on the third of the test set that used the training viewpoints.

Original: Then we compared these matched models on the two thirds of the test set with novel viewpoints.
Revised: Then we compared these matched models on the two thirds of the test set with novel viewpoints.

Original: Results in Tab.
Revised: Results in Tab.

Original: 2 show that, compared with the baseline CNN, capsules with matched performance on familiar viewpoints reduce the test error rate on novel viewpoints by about 30% for both novel azimuths and novel elevations.
Revised: 2 show that, compared with the baseline CNN, capsules with matched performance on familiar viewpoints reduce the test error rate on novel viewpoints by about 30% for both novel azimuths and novel elevations.

Original:  A DVERSARIAL R OBUSTNESS
Revised:  A DVERSARIAL R OBUSTNESS

Original: There is growing interest in the vulnerability of neural networks to adversarial examples; inputs that have been slightly changed by an attacker so as to trick a neural net classiﬁer into making the wrong classiﬁcation.
Revised: There is growing interest in the vulnerability of neural networks to adversarial examples; inputs that have been slightly changed by an attacker so as to trick a neural net classiﬁer into making the wrong classiﬁcation.

Original: These inputs can be created in a variety of ways, but straightforward strategies such as
Revised: These inputs can be created in a variety of ways, but straightforward strategies such as

Original: FGSM (Goodfellow et al.
Revised: FGSM (Goodfellow et al.

Original: (2014)) have been shown to drastically decrease accuracy in convolutional neural networks on image classiﬁcation tasks.
Revised: (2014)) have been shown to drastically decrease accuracy in convolutional neural networks on image classiﬁcation tasks.

Original: We compared our capsule model and a traditional convolutional model on their ability to withstand such attack.
Revised: We compared our capsule model and a traditional convolutional model on their ability to withstand such attack.

Original: FGSM computes the gradient of the loss w.r.t.
Revised: FGSM computes the gradient of the loss w.r.t.

Original: each pixel intensity and then changes the pixel intensity by a ﬁxed amount (cid:15) in the direction that increases the loss.
Revised: each pixel intensity and then changes the pixel intensity by a ﬁxed amount (cid:15) in the direction that increases the loss.

Original: So the changes only depend on the sign of the gradient at each pixel.
Revised: So the changes only depend on the sign of the gradient at each pixel.

Original: This can be extended to a targeted attack by updating the input to maximize the classiﬁcation probability of a particular wrong class.
Revised: This can be extended to a targeted attack by updating the input to maximize the classiﬁcation probability of a particular wrong class.

Original: We generated adversarial attacks using FGSM because it has only one hyper-parameter and it is easy to compare models that have very different gradient magnitudes.
Revised: We generated adversarial attacks using FGSM because it has only one hyper-parameter and it is easy to compare models that have very different gradient magnitudes.

Original: To test the robustness of our model we generated adversarial images from the test set using a fully trained model.
Revised: To test the robustness of our model we generated adversarial images from the test set using a fully trained model.

Original: We then reported the accuracy of the model on these images.
Revised: We then reported the accuracy of the model on these images.

Original: We found that our model is signiﬁcantly less vulnerable to both general and targeted FGSM adversarial attacks; a small epsilon can be used to reduce a convolutional model’s accuracy much more than an equivalent epsilon can on the capsule model (Fig. 3).
Revised: We found that our model is signiﬁcantly less vulnerable to both general and targeted FGSM adversarial attacks; a small epsilon can be used to reduce a convolutional model’s accuracy much more than an equivalent epsilon can on the capsule model (Fig. 3).

Original: It should also be noted that the capsule model’s accuracy after the untargeted attack never drops below chance ( 20% ) whereas the convolutional model’s accuracy is reduced to signiﬁcantly below chance with an epsilon of as small as 0 . 2 .
Revised: It should also be noted that the capsule model’s accuracy after the untargeted attack never drops below chance ( 20% ) whereas the convolutional model’s accuracy is reduced to signiﬁcantly below chance with an epsilon of as small as 0 . 2 .

Original: We also tested our model on the slightly more sophisticated adversarial attack of a Basic Iterative
Revised: We also tested our model on the slightly more sophisticated adversarial attack of a Basic Iterative

Original: Method (Kurakin et al.
Revised: Method (Kurakin et al.

Original: (2016)) which is simply the aforementioned attack except we take multiple smaller steps when creating the adversarial image.
Revised: (2016)) which is simply the aforementioned attack except we take multiple smaller steps when creating the adversarial image.

Original: Here too we ﬁnd that our model is much more robust to the attack than the traditional convolutional model.
Revised: Here too we ﬁnd that our model is much more robust to the attack than the traditional convolutional model.

Original: It has been shown that some robustness to adversarial attacks in models can be due to simply numerical instability in the calculation of the gradient Brendel & Bethge (2017).
Revised: It has been shown that some robustness to adversarial attacks in models can be due to simply numerical instability in the calculation of the gradient Brendel & Bethge (2017).

Original: To insure that this was not the sole cause of our model’s robustness we calculated the percentage of zeros values in the gradient with respect to the image in the capsule model and found it to be smaller than that of the
Revised: To insure that this was not the sole cause of our model’s robustness we calculated the percentage of zeros values in the gradient with respect to the image in the capsule model and found it to be smaller than that of the

Original: CNN.
Revised: CNN.

Original: Furthermore the capsule gradients, although smaller that those of the CNN, are only smaller by 2 orders of magnitude, as opposed to 16 orders of magnitude in Brendel et al.
Revised: Furthermore the capsule gradients, although smaller that those of the CNN, are only smaller by 2 orders of magnitude, as opposed to 16 orders of magnitude in Brendel et al.

Original: ’s work.
Revised: ’s work.

Original: Finally we tested our model’s robustness to black box attacks by generating adversarial examples with a CNN and testing them on both our capsule model and a different CNN.
Revised: Finally we tested our model’s robustness to black box attacks by generating adversarial examples with a CNN and testing them on both our capsule model and a different CNN.

Original: We found that the capsule model did not perform noticeably better at this task than the CNN.
Revised: We found that the capsule model did not perform noticeably better at this task than the CNN.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Building on the work of Sabour et al.
Revised: Building on the work of Sabour et al.

Original: (2017), we have proposed a new capsule network architecture in which each capsule has a logistic unit to represent the presence of an entity and a 4x4 pose matrix to represent the relationship between that entity and the viewer.
Revised: (2017), we have proposed a new capsule network architecture in which each capsule has a logistic unit to represent the presence of an entity and a 4x4 pose matrix to represent the relationship between that entity and the viewer.

Original: We also introduced a new iterative routing procedure between capsule layers, making use of the EM algorithm, which allows the output of each lower-level capsule to be routed to a capsule in the layer above so that each higherlevel capsule receives a cluster of similar pose votes, if such a cluster exists.
Revised: We also introduced a new iterative routing procedure between capsule layers, making use of the EM algorithm, which allows the output of each lower-level capsule to be routed to a capsule in the layer above so that each higherlevel capsule receives a cluster of similar pose votes, if such a cluster exists.

Original: This new architecture achieves signiﬁcantly better accuracy on the smallNORB data set then the state-of-the-art CNN, reducing the number of errors by 45%.
Revised: This new architecture achieves signiﬁcantly better accuracy on the smallNORB data set then the state-of-the-art CNN, reducing the number of errors by 45%.

Original: We have also shown it to be signiﬁcantly more robust to white box adversarial attacks than a baseline CNN.
Revised: We have also shown it to be signiﬁcantly more robust to white box adversarial attacks than a baseline CNN.

Original: SmallNORB is an ideal data-set for developing new shape-recognition models precisely because it lacks many of the additional features of images in the wild.
Revised: SmallNORB is an ideal data-set for developing new shape-recognition models precisely because it lacks many of the additional features of images in the wild.

Original: Now that our capsules model works well on NORB, we plan to implement an efﬁcient version so that we can test much larger models on much larger data-sets such as ImageNet.
Revised: Now that our capsules model works well on NORB, we plan to implement an efﬁcient version so that we can test much larger models on much larger data-sets such as ImageNet.

Original: A S UPPLEMENTARY FIGURES
Revised: A S UPPLEMENTARY FIGURES

Original: [Figure]
Revised: [Figure]