Original: AlgebraNets
Revised: AlgebraNets

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: Neural networks have historically been built layerwise from the set of functions in f : R n → R m , i.e. with activations and weights/parameters represented by real numbers, R .
Revised: Neural networks have historically been built layerwise from the set of functions in f : R n → R m , i.e. with activations and weights/parameters represented by real numbers, R .

Original: Our work considers a richer set of objects for activations and weights, and undertakes a comprehensive study of alternative algebras as number representations by studying their performance on two challenging problems: largescale image classiﬁcation using the ImageNet dataset and language modeling using the enwiki8 and WikiText-103 datasets.
Revised: Our work considers a richer set of objects for activations and weights, and undertakes a comprehensive study of alternative algebras as number representations by studying their performance on two challenging problems: largescale image classiﬁcation using the ImageNet dataset and language modeling using the enwiki8 and WikiText-103 datasets.

Original: We denote this broader class of models as AlgebraNets.
Revised: We denote this broader class of models as AlgebraNets.

Original: Our ﬁndings indicate that the conclusions of prior work, which explored neural networks constructed from C (complex numbers) and H (quaternions) on smaller datasets, do not always transfer to these challenging settings.
Revised: Our ﬁndings indicate that the conclusions of prior work, which explored neural networks constructed from C (complex numbers) and H (quaternions) on smaller datasets, do not always transfer to these challenging settings.

Original: However, our results demonstrate that there are alternative algebras which deliver better parameter and computational eﬃciency compared with R .
Revised: However, our results demonstrate that there are alternative algebras which deliver better parameter and computational eﬃciency compared with R .

Original: We consider C ,
Revised: We consider C ,

Original: H , M 2 ( R ) (the set of 2 × 2 real-valued matrices), M 2 ( C ) , M 3 ( R ) , M 4 ( R ) , dual numbers and the R 3 cross product.
Revised: H , M 2 ( R ) (the set of 2 × 2 real-valued matrices), M 2 ( C ) , M 3 ( R ) , M 4 ( R ) , dual numbers and the R 3 cross product.

Original: Additionally, we note that multiplication in these algebras has higher compute density than real multiplication, a useful property in situations with inherently limited parameter reuse such as auto-regressive inference and sparse neural networks.
Revised: Additionally, we note that multiplication in these algebras has higher compute density than real multiplication, a useful property in situations with inherently limited parameter reuse such as auto-regressive inference and sparse neural networks.

Original: We therefore investigate how to induce sparsity within
Revised: We therefore investigate how to induce sparsity within

Original: AlgebraNets.
Revised: AlgebraNets.

Original: We hope that our strong results on large-scale, practical benchmarks will spur further exploration of these unconventional architectures which challenge the default choice of using real numbers for neural network weights and activations.
Revised: We hope that our strong results on large-scale, practical benchmarks will spur further exploration of these unconventional architectures which challenge the default choice of using real numbers for neural network weights and activations.

Original:  Introduction
Revised:  Introduction

Original: Nearly universally, the atomic building blocks of artiﬁcial neural networks are scalar real-valued weights and scalar real-valued neuron activations that interact using standard rules of multiplication and addition.
Revised: Nearly universally, the atomic building blocks of artiﬁcial neural networks are scalar real-valued weights and scalar real-valued neuron activations that interact using standard rules of multiplication and addition.

Original: We propose AlgebraNets, where we replace the commonly used real-valued algebra with other associative algebras.
Revised: We propose AlgebraNets, where we replace the commonly used real-valued algebra with other associative algebras.

Original: Brieﬂy, this amounts to replacing scalars by tuples and real multiplication by a tuple multiplication rule.
Revised: Brieﬂy, this amounts to replacing scalars by tuples and real multiplication by a tuple multiplication rule.

Original: For example, by replacing each scalar weight and activation with 2 ×matrices and standard real addition / multiplication with matrix addition / multiplication.
Revised: For example, by replacing each scalar weight and activation with 2 ×matrices and standard real addition / multiplication with matrix addition / multiplication.

Original: These alternative algebras provide three clear beneﬁts for deep learning at scale:
Revised: These alternative algebras provide three clear beneﬁts for deep learning at scale:

Original: Parameter eﬃciency.
Revised: Parameter eﬃciency.

Original: One sweeping beneﬁt of AlgebraNets is they are able to match baseline performance on a variety of tasks, spread over multiple domains, with fewer parameters than the competitive real-valued baselines.
Revised: One sweeping beneﬁt of AlgebraNets is they are able to match baseline performance on a variety of tasks, spread over multiple domains, with fewer parameters than the competitive real-valued baselines.

Original: This means that equivalently capable models can be trained on smaller hardware, and for a given amount of memory, a model with greater eﬀective capacity can be trained.
Revised: This means that equivalently capable models can be trained on smaller hardware, and for a given amount of memory, a model with greater eﬀective capacity can be trained.

Original: We ﬁnd some variants of AlgebraNets that are more parameter eﬃcient than the previously considered C and H algebras.
Revised: We ﬁnd some variants of AlgebraNets that are more parameter eﬃcient than the previously considered C and H algebras.

Original: Throughout the text, we count parameters as the total number of real values e.g. a complex number counts as two parameters.
Revised: Throughout the text, we count parameters as the total number of real values e.g. a complex number counts as two parameters.

Original: Computational eﬃciency.
Revised: Computational eﬃciency.

Original: For scaling large models, parameter eﬃciency is not the only bottleneck:
Revised: For scaling large models, parameter eﬃciency is not the only bottleneck:

Original: FLOP eﬃciency – reducing the relative number of ﬂoating-point operations to achieve an equivalent accuracy – is also important.
Revised: FLOP eﬃciency – reducing the relative number of ﬂoating-point operations to achieve an equivalent accuracy – is also important.

Original: We ﬁnd instantiations of AlgebraNets that are more FLOP eﬃcient than the previously considered C and H algebras and as FLOP eﬃcient as R .
Revised: We ﬁnd instantiations of AlgebraNets that are more FLOP eﬃcient than the previously considered C and H algebras and as FLOP eﬃcient as R .

Original: Additionally, all of the proposed algebras oﬀer parameter reuse greater than 1 (see Table 1).
Revised: Additionally, all of the proposed algebras oﬀer parameter reuse greater than 1 (see Table 1).

Original: That is, the ratio of multiplications performed to values consumed is greater than or equal to 1:1.
Revised: That is, the ratio of multiplications performed to values consumed is greater than or equal to 1:1.

Original: By contrast, for multiplication in R it is only 1:2.
Revised: By contrast, for multiplication in R it is only 1:2.

Original: Modern hardware requires a high ratio of ﬂoating point operations to bytes loaded (bandwidth) to become compute bound and saturate the arithmetic units.
Revised: Modern hardware requires a high ratio of ﬂoating point operations to bytes loaded (bandwidth) to become compute bound and saturate the arithmetic units.

Original: This is particularly problematic for auto-regressive inference (dominated by matrix-vector multiplies), sparse models, depthwise convolutions and other operations with low arithmetic density.
Revised: This is particularly problematic for auto-regressive inference (dominated by matrix-vector multiplies), sparse models, depthwise convolutions and other operations with low arithmetic density.

Original: Architectural exploration.
Revised: Architectural exploration.

Original: The choice of real numbers for weights and activations is usually taken for granted (with some exceptions, e.g. those discussed in Sec. 3).
Revised: The choice of real numbers for weights and activations is usually taken for granted (with some exceptions, e.g. those discussed in Sec. 3).

Original: With AlgebraNets, we challenge this established design choice and open up a vast new space for neural network architecture exploration by showing that real numbers can be easily replaced with a variety of algebraic structures.
Revised: With AlgebraNets, we challenge this established design choice and open up a vast new space for neural network architecture exploration by showing that real numbers can be easily replaced with a variety of algebraic structures.

Original: Leveraging these new building blocks, one can consider diﬀerent algebraic interactions, diﬀerent choices of non-linearities, and diﬀerent network architecture choices.
Revised: Leveraging these new building blocks, one can consider diﬀerent algebraic interactions, diﬀerent choices of non-linearities, and diﬀerent network architecture choices.

Original: Importantly, as we demonstrate in this work, AlgebraNets are not only scalable to large models and complex tasks, but they in fact oﬀer improvements in model eﬃciency, which makes them a viable practical choice.
Revised: Importantly, as we demonstrate in this work, AlgebraNets are not only scalable to large models and complex tasks, but they in fact oﬀer improvements in model eﬃciency, which makes them a viable practical choice.

Original: We believe we have only begun to scratch the surface of what these alternative building blocks can enable, and we hope that their broader adoption will usher in further progress across the ﬁeld.
Revised: We believe we have only begun to scratch the surface of what these alternative building blocks can enable, and we hope that their broader adoption will usher in further progress across the ﬁeld.

Original: In summary, our main contributions are as follows:
Revised: In summary, our main contributions are as follows:

Original: • We propose AlgebraNets — a novel class of neural networks, which replaces the nearly ubiquitously used real algebra with alternatives.
Revised: • We propose AlgebraNets — a novel class of neural networks, which replaces the nearly ubiquitously used real algebra with alternatives.

Original: We show that in contrast to previous work, algebra speciﬁc initializations and replacement of batch normalization by an expensive whitening procedure Trabelsi et al. ; Gaudet and Maida (2018); Wu et al. (2020); Pan et al. (2019) is not necessary, making them a near drop-in replacement to real-valued networks.
Revised: We show that in contrast to previous work, algebra speciﬁc initializations and replacement of batch normalization by an expensive whitening procedure (Trabelsi et al., 2018; Gaudet and Maida, 2018; Wu et al., 2020; Pan et al., 2019) is not necessary, making them a near drop-in replacement to real-valued networks.

Original: • We evaluate AlgebraNets based on a wide range of algebras on three challenging large scale benchmarks: ImageNet image classiﬁcation Russakovsky et al.{+al., 2015), Enwik
+}
Revised: • We evaluate AlgebraNets based on a wide range of algebras on three challenging large scale benchmarks: ImageNet image classiﬁcation (Russakovsky et {+al., 2015), Enwik
+}

Original: , Enwik8 LLC (2009), and WikiText language modelling Merity et al.
Revised: (LLC, 2009), and WikiText language modelling (Merity et al., 2016).

Original: • We explore sparse AlgebraNets to take advantage of their higher compute density.
Revised: • We explore sparse AlgebraNets to take advantage of their higher compute density.

Original: We ﬁnd that AlgebraNets oﬀer improved parameter eﬃciency and FLOP parity compared to the real-valued baselines, which establishes them as a viable choice for eﬃcient deep learning at scale.
Revised: We ﬁnd that AlgebraNets oﬀer improved parameter eﬃciency and FLOP parity compared to the real-valued baselines, which establishes them as a viable choice for eﬃcient deep learning at scale.

Original:  AlgebraNets 2.1 Why Algebras?
Revised:  AlgebraNets 2.1 Why Algebras?

Original: We consider algebras because they have the right properties to make them a drop-in replacement for real numbers in typical neural networks.
Revised: We consider algebras because they have the right properties to make them a drop-in replacement for real numbers in typical neural networks.

Original: This is not surprising as the real numbers are an algebra over themselves.
Revised: This is not surprising as the real numbers are an algebra over themselves.

Original: An algebra A over a ﬁeld K (which we take to always be the ﬁeld of real or complex numbers) satisﬁes the following properties 1 Wikipedia contributors (2020b;a):
Revised: An algebra A over a ﬁeld K (which we take to always be the ﬁeld of real or complex numbers) satisﬁes the following properties 1 (Wikipedia contributors, 2020b;a):

Original: It is a vector space over K .
Revised: It is a vector space over K .

Original: • It has an associative and commutative addition operator with an identity element ( x + 0 = x ) and inverse element ( x + ( − x ) = 0 ).
Revised: • It has an associative and commutative addition operator with an identity element ( x + 0 = x ) and inverse element ( x + ( − x ) = 0 ).

Original: • It is possible to multiply elements of ﬁeld K with vectors.2.
Revised: • It is possible to multiply elements of ﬁeld K with vectors.2.

Original: There is a right and left distributive multiplication operator • over vectors closed in A. 3.
Revised: There is a right and left distributive multiplication operator • over vectors closed in A. 3.

Original: Scalar multiplication combines with • in a compatible way: ( a x ) • ( b y ) =
Revised: Scalar multiplication combines with • in a compatible way: ( a x ) • ( b y ) =

Original: ( ab )( x • y ) .
Revised: ( ab )( x • y ) .

Original: We do not claim that these properties are all required as neural network building-blocks, merely that they are convenient.
Revised: We do not claim that these properties are all required as neural network building-blocks, merely that they are convenient.

Original: For example, one could imagine not having associative addition – this would require a careful implementation to get right but is possible.
Revised: For example, one could imagine not having associative addition – this would require a careful implementation to get right but is possible.

Original: One could eliminate the requirement that scalars from K multiply with vectors from A - this would make various normalizations (e.g. batch normalization) impossible, but they are not required.
Revised: One could eliminate the requirement that scalars from K multiply with vectors from A - this would make various normalizations (e.g. batch normalization) impossible, but they are not required.

Original: Most importantly, removing some of these requirements does not lead to an obviously useful class of mathematical objects to consider.
Revised: Most importantly, removing some of these requirements does not lead to an obviously useful class of mathematical objects to consider.

Original: In addition to the previously considered C and H algebras, we also consider the algebras of n × n matrices over R and C (i.e. M n ( R ) or M n ( C ) ) as they have higher compute density than R and map well to the matrix multiplication units that are becoming common in processors Oh (2019).
Revised: In addition to the previously considered C and H algebras, we also consider the algebras of n × n matrices over R and C (i.e. M n ( R ) or M n ( C ) ) as they have higher compute density than R and map well to the matrix multiplication units that are becoming common in processors (Oh, 2019).

Original: We note x+ b x
Revised: We note x

Original: [Figure] that M 2 ( R ) is isomorphic to the split-quaternion Cockle (1849) algebra and M 2 ( C ) is isomorphic to the biquaternion Hamilton (1844) algebra, but the matrix algebras are more familiar so we retain that terminology.
Revised: = a x + b x that M 2 ( R ) is isomorphic to the split-quaternion (Cockle, 1849) algebra and M 2 ( C ) is isomorphic to the biquaternion (Hamilton, 1844) algebra, but the matrix algebras are more familiar so we retain that terminology.

Original: Lastly, we consider the dual numbers and the cross product of length-3 tuples.
Revised: Lastly, we consider the dual numbers and the cross product of length-3 tuples.

Original: 2.2 Algebra Details
Revised: 2.2 Algebra Details

Original: [Table]
Revised: [Table]

Original: R ; Real Numbers All baseline networks are real-valued with scalar weights; standard multiplication rules apply.

Original: For two weight values, we load 2 scalars and perform 1 multiply.

Original: C a b a a b b b − a
Revised: C a b a a b b b − a

Original: We provide tables describing the multiplicative interaction between tuples.
Revised: We provide tables describing the multiplicative interaction between tuples.

Original: The interaction between two tuples ( o a , o b , ... )
Revised: The interaction between two tuples ( o a , o b , ... )

Original: =
Revised: =

Original: ( t a , t b , ... )
Revised: ( t a , t b , ... )

Original: • ( v a , v b , ... ) is described by a matrix where the indices of t are on the left, v are on the top and entries correspond to which component of o the interaction contributes to.
Revised: • ( v a , v b , ... ) is described by a matrix where the indices of t are on the left, v are on the top and entries correspond to which component of o the interaction contributes to.

Original: A 0 means there is no interaction and a negative index means the result is subtracted.
Revised: A 0 means there is no interaction and a negative index means the result is subtracted.

Original: C ; Complex Numbers Each weight, w , is a length 2 tuple ( t a , t b ) representing the complex number t a + t b i .
Revised: C ; Complex Numbers Each weight, w , is a length 2 tuple ( t a , t b ) representing the complex number t a + t b i .

Original: For two weight values we load 4 scalars and perform 4 multiplies.
Revised: For two weight values we load 4 scalars and perform 4 multiplies.

Original: M n ( R ); n × n Matrices
Revised: M n ( R ); n × n Matrices

Original: Each weight is a length n 2 tuple, representing an n × n matrix.
Revised: Each weight is a length n 2 tuple, representing an n × n matrix.

Original: Multiplication and addition proceed with standard rules for matrices.
Revised: Multiplication and addition proceed with standard rules for matrices.

Original: We consider up to M 4 ( R ) matrices.
Revised: We consider up to M 4 ( R ) matrices.

Original: For two weight values we load 2 n 2 scalars and perform n 3 multiplies.
Revised: For two weight values we load 2 n 2 scalars and perform n 3 multiplies.

Original: H a b c d a a b c d b b - a d - c c c - d - a b d d c - b - a
Revised: [Equation]

Original: M n ( C ); n × n Complex Matrices Weights are length 2 n 2 tuples representing n × n complex-valued matrices.
Revised: M n ( C ); n × n Complex Matrices Weights are length 2 n 2 tuples representing n × n complex-valued matrices.

Original: We consider only n = 2 .
Revised: We consider only n = 2 .

Original: For two weight values we load 4 n 2 scalars and perform 4 n 3 multiplies.
Revised: For two weight values we load 4 n 2 scalars and perform 4 n 3 multiplies.

Original: The multiplication table is in Appendix A.
Revised: The multiplication table is in Appendix A.

Original: H ; Quaternions Each weight, w i is replaced by a length 4 tuple, ( t a , t b , t c , t d ) .
Revised: H ; Quaternions Each weight, w i is replaced by a length 4 tuple, ( t a , t b , t c , t d ) .

Original: Multiplication is not commutative, with the product of two quaternions given by the Hamilton product Hamilton (1843).
Revised: Multiplication is not commutative, with the product of two quaternions given by the Hamilton product (Hamilton, 1843).

Original: For two weight values, we loadelements and perform 16 multiplies.
Revised: For two weight values, we loadelements and perform 16 multiplies.

Original: D a b c d a a 0 0b 0 b 0c 0 0 cd 0 0 0 d
Revised: D [Equation] c d [Equation]

Original: Diagonal Algebra
Revised: Diagonal Algebra

Original: The high FLOP cost of the whitening operation required by Trabelsi et al. ; Gaudet and Maida (2018); Wu et al. (2020); Pan et al. (2019) makes networks using it ineﬃcient at training and inference in terms of FLOPs.
Revised: The high FLOP cost of the whitening operation required by (Trabelsi et al., 2018; Gaudet and Maida, 2018; Wu et al., 2020; Pan et al., 2019) makes networks using it ineﬃcient at training and inference in terms of FLOPs.

Original: We attempt to design an algebra where using whitening would in fact be competitive by eliminating the interaction of terms through the algebra.
Revised: We attempt to design an algebra where using whitening would in fact be competitive by eliminating the interaction of terms through the algebra.

Original: Only when combining the ‘diagonal’ D algebra with whitening are there interactions between the diﬀerent tuple components. a b a a b b b
Revised: Only when combining the ‘diagonal’ D algebra with whitening are there interactions between the diﬀerent tuple components. [Equation]

Original: Dual Numbers Each weight is represented by a length 2 tuple representing the dual number ( t 0 + t 1 (cid:15) ) .
Revised: Dual Numbers Each weight is represented by a length 2 tuple representing the dual number ( t 0 + t 1 (cid:15) ) .

Original: For a multiplication, we load 4 values and perform 3 multiplies.
Revised: For a multiplication, we load 4 values and perform 3 multiplies.

Original: R 3 a b c a 0 c -b b -c 0 a c b -a
Revised: R 3 a b c a 0 c -b b -c 0 a c b -a

Original: R 3 Cross Product Each weight is represented by a length 3 tuple.

Original: We use the cross product between two tuples for the multiplication rule, resulting in 6 diﬀerent multiplies for 6 values loaded.

Original: 2.3 Initialization, Normalization, Non-Linearities, and Pruning
Revised: 2.3 Initialization, Normalization, Non-Linearities, and Pruning

Original: Prior work Trabelsi etal. ; Gaudet and Maida (2018) has advocated algebra-speciﬁc initializations and expensive whitening procedures to replace batch normalization.
Revised: Prior work (Trabelsi et al., 2018; Gaudet and Maida, 2018) has advocated algebra-speciﬁc initializations and expensive whitening procedures to replace batch normalization.

Original: We ﬁnd that this is not necessary to achieve good performance, and we are able to use the same initialization, normalization, and non-linearities across all algebras which facilitates exploring a wide variety of options.
Revised: We ﬁnd that this is not necessary to achieve good performance, and we are able to use the same initialization, normalization, and non-linearities across all algebras which facilitates exploring a wide variety of options.

Original: To initialize all the components of the algebra tuple at the beginning of a network we set the ﬁrst tuple component to the typical input.
Revised: To initialize all the components of the algebra tuple at the beginning of a network we set the ﬁrst tuple component to the typical input.

Original: For ResNet, MobileNet, and the RNN we initialize the other components of the tuple with a small one or two-layer MLP, i.e. t b,c,... =
Revised: For ResNet, MobileNet, and the RNN we initialize the other components of the tuple with a small one or two-layer MLP, i.e. t b,c,... =

Original: MLP ( t a ) .
Revised: MLP ( t a ) .

Original: For the transformer, we take advantage of the fact that the embedding is already a learned representation and simply reshape the output embedding appropriately.
Revised: For the transformer, we take advantage of the fact that the embedding is already a learned representation and simply reshape the output embedding appropriately.

Original: We ﬁnd that the speciﬁcs of the input initialization do not have a large eﬀect on performance, though allowing a learnable transformation outperformed initializing additional components to 0 or replicating the input.
Revised: We ﬁnd that the speciﬁcs of the input initialization do not have a large eﬀect on performance, though allowing a learnable transformation outperformed initializing additional components to 0 or replicating the input.

Original: We use standard Glorot Glorot and
Revised: We use standard Glorot (Glorot and

Original: Bengio (2010) weight initialization of each component independently.
Revised: Bengio, 2010) weight initialization of each component independently.

Original: Comparisons with the algebra speciﬁc initializations Trabelsi et al. (2017); Gaudet and Maida (2018) can be found in Appendix B.
Revised: Comparisons with the algebra speciﬁc initializations (Trabelsi et al., 2018; Gaudet and Maida, 2018) can be found in Appendix B.

Original: Existing activation functions can be applied component-wise ( t = ( f ( t a ) , · · · , f ( t d )) ) and we found that ReLU and swish Ramachandran et al. (2017) work well; tanh and sigmoid can also be applied component-wise as part of GRUs and LSTMs.
Revised: Existing activation functions can be applied component-wise ( t = ( f ( t a ) , · · · , f ( t d )) ) and we found that ReLU and swish (Ramachandran et al., 2017) work well; tanh and sigmoid can also be applied component-wise as part of GRUs and LSTMs.

Original: Applying the activation function to the entire tuple has possible computational advantages if it is ReLU-like as it would allow an entire tuple multiplication to be skipped.
Revised: Applying the activation function to the entire tuple has possible computational advantages if it is ReLU-like as it would allow an entire tuple multiplication to be skipped.

Original: For example, consider t = f
Revised: For example, consider t = f

Original: ( g ( t )) t .
Revised: ( g ( t )) t .

Original: If g ( • ) returns the mean of the tuple, and if f was H the Heaviside step function, then one can remove entire components.
Revised: If g ( • ) returns the mean of the tuple, and if f was H the Heaviside step function, then one can remove entire components.

Original: Appendix B examines diﬀerent choices for doing this, but we do not consider it further in the main text.
Revised: Appendix B examines diﬀerent choices for doing this, but we do not consider it further in the main text.

Original: The ﬁnal logits of an AlgebraNet must be real-valued.
Revised: The ﬁnal logits of an AlgebraNet must be real-valued.

Original: We use an Algebra-speciﬁc ﬁnal linear layer and convert the ﬁnal algebra tuple to a scalar with the tuple-wise L 2 norm before applying softmax.
Revised: We use an Algebra-speciﬁc ﬁnal linear layer and convert the ﬁnal algebra tuple to a scalar with the tuple-wise L 2 norm before applying softmax.

Original: More details are in Appendix B.
Revised: More details are in Appendix B.

Original: To apply magnitude pruning Zhu and Gupta (2017) ; Gale et al. (2019) to prune tuples we used the tuple L 2 norm as the criterion for pruning for all AlgebraNet variants.
Revised: To apply magnitude pruning (Zhu and Gupta, 2017; Gale et al., 2019) to prune tuples we used the tuple L 2 norm as the criterion for pruning for all AlgebraNet variants.

Original: For the M n ( R ) algebras we also experimented with criteria based on the eigenvalues, λ i , and singular values, σ i , of each n × n matrix.
Revised: For the M n ( R ) algebras we also experimented with criteria based on the eigenvalues, λ i , and singular values, σ i , of each n × n matrix.

Original: The Frobenius norm corresponds to ( (cid:80) i σ 2 i ) 1 / 2 and the determinant corresponds to ( (cid:81) i λ i ) .
Revised: The Frobenius norm corresponds to ( (cid:80) i σ 2 i ) 1 / 2 and the determinant corresponds to ( (cid:81) i λ i ) .

Original: We found pruning based on the Frobenius norm to be the most eﬀective, followed by pruning based on the largest eigenvalue.
Revised: We found pruning based on the Frobenius norm to be the most eﬀective, followed by pruning based on the largest eigenvalue.

Original: See Appendix C for a comparison between diﬀerent methods.
Revised: See Appendix C for a comparison between diﬀerent methods.

Original: Trabelsi et al. (2017), Gaudet and Maida (2018), and Wu et al. (2020) use whitening in place of batch normalization.
Revised: (Trabelsi et al., 2018), (Gaudet and Maida, 2018), and (Wu et al., 2020) use whitening in place of batch normalization.

Original: Whitening normalizes and de-correlates the diﬀerent tuple elements from one another.
Revised: Whitening normalizes and de-correlates the diﬀerent tuple elements from one another.

Original: However, this extension results in a substantial increase in both training and test time computational costs, as described in Pan et al.
Revised: However, this extension results in a substantial increase in both training and test time computational costs, as described in (Pan et al., 2019).

Original: The inclusion of the whitening cost to the FLOP count in
Revised: The inclusion of the whitening cost to the FLOP count in

Original: Fig.
Revised: Fig.

Original: 1 highlights the substantial cost inference cost.
Revised: 1 highlights the substantial cost inference cost.

Original: Cholesky decomposition Press et al. (2007) of the inverted covariance matrix is required during training and at inference it is not possible to fold the whitening transformation into adjacent convolutions.
Revised: Cholesky decomposition (Press et al., 2007) of the inverted covariance matrix is required during training and at inference it is not possible to fold the whitening transformation into adjacent convolutions.

Original: A contribution from each of the algebra elements contributes to each element in the whitened output.
Revised: A contribution from each of the algebra elements contributes to each element in the whitened output.

Original: We ﬁnd that batch normalization does not substantially decrease performance, trains 1 .
Revised: We ﬁnd that batch normalization does not substantially decrease performance, trains 1 .

Original: 9 × faster and has no inference cost, so we use it for all experiments, unless explicitly stated.
Revised: 9 × faster and has no inference cost, so we use it for all experiments, unless explicitly stated.

Original:  Related Work
Revised:  Related Work

Original: Trabelsi et al. (2017) applied complex-valued networks to convolutional neural networks trained on CIFAR-10, as well as to music transcription and Speech Spectrum Prediction.
Revised: (Trabelsi et al., 2018) applied complex-valued networks to convolutional neural networks trained on CIFAR-10, as well as to music transcription and Speech Spectrum Prediction.

Original: They ﬁnd that complex-valued networks with the same number of parameters and more FLOPs perform slightly better than real-valued networks.
Revised: They ﬁnd that complex-valued networks with the same number of parameters and more FLOPs perform slightly better than real-valued networks.

Original: Gaudet and Maida (2018) extend the procedure from Trabelsi et al. (2017) to quaternion valued weights, showing that they are able to reduce the parameter count by a factor of two over complex-valued networks and a factor of four over real-valued networks, while again slightly increasing the top-1 accuracy on CIFAR-10.
Revised: (Gaudet and Maida, 2018) extend the procedure from (Trabelsi et al., 2018) to quaternion valued weights, showing that they are able to reduce the parameter count by a factor of two over complex-valued networks and a factor of four over real-valued networks, while again slightly increasing the top-1 accuracy on CIFAR-10.

Original: Wu et al. (2020) further extend this approach to octonions (which are a non-associative algebra), demonstrating that they are able to further reduce the parameter count while increasing the accuracy of their models on CIFAR-10.
Revised: (Wu et al., 2020) further extend this approach to octonions (which are a non-associative algebra), demonstrating that they are able to further reduce the parameter count while increasing the accuracy of their models on CIFAR-10.

Original: These papers establish the eﬃcacy of some alternative algebras, though they focus purely on parameter eﬃciency, rather than FLOP eﬃciency which is equally important for image classiﬁcation tasks.
Revised: These papers establish the eﬃcacy of some alternative algebras, though they focus purely on parameter eﬃciency, rather than FLOP eﬃciency which is equally important for image classiﬁcation tasks.

Original: Additionally, the tested datasets are relatively small, and it is unclear how the results scale to larger datasets.
Revised: Additionally, the tested datasets are relatively small, and it is unclear how the results scale to larger datasets.

Original: Both the quaternion and octonion network papers do not test their models on language modeling tasks where parameter eﬃciency is often of greater importance.
Revised: Both the quaternion and octonion network papers do not test their models on language modeling tasks where parameter eﬃciency is often of greater importance.

Original: Parcollet et al. (2018) propose a quaternion recurrent neural network (QRNN) and quaternion LSTM (QLSTM).
Revised: (Parcollet et al., 2018) propose a quaternion recurrent neural network (QRNN) and quaternion LSTM (QLSTM).

Original: They show that quaternion based methods are able to reduce the parameter count while oﬀering better performance on the TIMIT and WSJ phoneme recognition tasks.
Revised: They show that quaternion based methods are able to reduce the parameter count while oﬀering better performance on the TIMIT and WSJ phoneme recognition tasks.

Original: Associative Long
Revised: Associative Long

Original: Short-Term Memory leverage complex-vectors to increase the memory capacity of LSTMs without a parameter increase Danihelka et al.
Revised: Short-Term Memory leverage complex-vectors to increase the memory capacity of LSTMs without a parameter increase (Danihelka et al., 2016).

Original: Recently, many methods to induce sparsity in neural networks have shown that it is possible to train models with an overwhelming fraction of the weights being 

Revised: Recently, many methods to induce sparsity in neural networks have shown that it is possible to train models with an overwhelming fraction of the weights being

Original: Molchanov et al. ; Gale et al. ; Frankle and Carbin (2018); Louizos et al.
Revised: (Molchanov et al., 2017; Gale et al., 2019; Frankle and Carbin, 2019; Louizos et al., 2018;

Original: ; Evci et al. ; Zhu and Gupta (2017).
Revised: Evci et al., 2019; Zhu and Gupta, 2017).

Original: Many of these methods gradually decrease the number of weights in the network through training by using some combination of each weight’s gradient and magnitude.
Revised: Many of these methods gradually decrease the number of weights in the network through training by using some combination of each weight’s gradient and magnitude.

Original: Fine grained sparsity is hard to accelerate on modern hardware, although there have been some recent results demonstrating that speedups are possible Elsen et al.
Revised: Fine grained sparsity is hard to accelerate on modern hardware, although there have been some recent results demonstrating that speedups are possible (Elsen et al., 2020).

Original: Vecchi et al. (2019) considered inducing sparsity in quaternion networks.
Revised: (Vecchi et al., 2019) considered inducing sparsity in quaternion networks.

Original: Primitives that increase computational density of fundamental interactions would increase the performance of sparse methods as demonstrated on the GPU by Mueller-Roemer et al. in scientiﬁc computing.
Revised: Primitives that increase computational density of fundamental interactions would increase the performance of sparse methods as demonstrated on the GPU by (Mueller-Roemer et al., 2019) in scientiﬁc computing.

Original: Jayakumar et al. (2020) emphasize the importance of multiplicative interaction layers providing a particularly useful inductive bias during the fusion of multiple information streams.
Revised: (Jayakumar et al., 2020) emphasize the importance of multiplicative interaction layers providing a particularly useful inductive bias during the fusion of multiple information streams.

Original: Speciﬁc
Revised: Speciﬁc

Original: AlgebraNets may provide strong, useful domain-speciﬁc inductive biases, for example as done by Worrall et al. (2017), leveraging the rotational invariance of complex numbers in convolutional networks and by Hinton et al. (2018) where they use a 4 × 4 pose matrix to represent orientations.[Figure]
Revised: AlgebraNets may provide strong, useful domain-speciﬁc inductive biases, for example as done by (Worrall et al., 2017), leveraging the rotational invariance of complex numbers in convolutional networks and by (Hinton et al., 2018) where they use a 4 × 4 pose matrix to represent orientations.

Original:  Experiments and Results 4.1 ImageNet
Revised:  Experiments and Results 4.1 ImageNet

Original: We examine the performance of AlgebraNet versions of ResNet-50 He et al. and MobileNetv1 Howard et al. (2017) on the ImageNet Russakovsky et al.
Revised: We examine the performance of AlgebraNet versions of ResNet-50 (He et al., 2016) and MobileNetv1 (Howard et al., 2017) on the ImageNet (Russakovsky et al., 2015) dataset.

Original: (2015) dataset.

Original: We use a width multiplier on the channels to adjust model capacity.
Revised: We use a width multiplier on the channels to adjust model capacity.

Original: For all experiments we use SGD with momentum of 0 . 9 .
Revised: For all experiments we use SGD with momentum of 0 . 9 .

Original: We increase the number of training epochs by a factor of two to 180, which we also use for the pruning experiments.
Revised: We increase the number of training epochs by a factor of two to 180, which we also use for the pruning experiments.

Original: This did not aﬀect the baseline, but it improves the pruning results.
Revised: This did not aﬀect the baseline, but it improves the pruning results.

Original: It also resulted in improved performance for H , so we used it throughout.
Revised: It also resulted in improved performance for H , so we used it throughout.

Original: For a batch size of 256, the initial learning rate for the ResNet experiments was set to 2 .
Revised: For a batch size of 256, the initial learning rate for the ResNet experiments was set to 2 .

Original: 5 and multiplied by 0.1 at epochs 60, 100, 140, and 160.
Revised: 5 and multiplied by 0.1 at epochs 60, 100, 140, and 160.

Original: We ﬁnd it is useful to reduce the amount of L 2 regularization that is used for AlgebraNets.
Revised: We ﬁnd it is useful to reduce the amount of L 2 regularization that is used for AlgebraNets.

Original: The baseline value 10 − 4 was reduced by a factor of 0 .
Revised: The baseline value 10 − 4 was reduced by a factor of 0 .

Original: 725 for ResNet-50 and 0 .
Revised: 725 for ResNet-50 and 0 .

Original: 625 for MobileNet-v1.
Revised: 625 for MobileNet-v1.

Original: We use the swish activation function for all experiments shown in Figures 1 and 2 including the baselines.
Revised: We use the swish activation function for all experiments shown in Figures 1 and 2 including the baselines.

Original: We found it improved performance across the board.
Revised: We found it improved performance across the board.

Original: Figures 1 and 2 compare the trade-oﬀs between accuracy, parameters, and FLOPs for diﬀerent ﬂavours of AlgebraNet.
Revised: Figures 1 and 2 compare the trade-oﬀs between accuracy, parameters, and FLOPs for diﬀerent ﬂavours of AlgebraNet.

Original: Notably, we do not ﬁnd that the parameter reduction without accuracy loss from Trabelsi et al. (2017) and Gaudet and Maida (2018) on CIFAR translates to ImageNet; we are unable to divide the number of parameters by a factor of two/four for C / H and match
Revised: Notably, we do not ﬁnd that the parameter reduction without accuracy loss from (Trabelsi et al., 2018) and (Gaudet and Maida, 2018) on CIFAR translates to ImageNet; we are unable to divide the number of parameters by a factor of two/four for C / H and match

Original: baseline performance.
Revised: baseline performance.

Original: We hypothesize that this is in part due to over-paramaterization of many networks trained on CIFAR and that AlgebraNets act as an additional regularizer, in part due to the greater weight reuse: each tuple component is now involved in multiple equations.
Revised: We hypothesize that this is in part due to over-paramaterization of many networks trained on CIFAR and that AlgebraNets act as an additional regularizer, in part due to the greater weight reuse: each tuple component is now involved in multiple equations.

Original: We feel that this highlights the need for testing methods on large-scale datasets.
Revised: We feel that this highlights the need for testing methods on large-scale datasets.

Original: We ﬁnd M 2 ( R ) AlgebraNets provide the best parameter eﬃciency of all considered algebras while requiring no more FLOPs than the real baseline on both ResNet-50 and MobileNet-v1.
Revised: We ﬁnd M 2 ( R ) AlgebraNets provide the best parameter eﬃciency of all considered algebras while requiring no more FLOPs than the real baseline on both ResNet-50 and MobileNet-v1.

Original: We also ﬁnd, for both ResNet-50 and MobileNet-v1, M 2 ( C ) AlgebraNets provide better FLOP eﬃciency than the previously studied H while having the same ratio of multiplies to values.
Revised: We also ﬁnd, for both ResNet-50 and MobileNet-v1, M 2 ( C ) AlgebraNets provide better FLOP eﬃciency than the previously studied H while having the same ratio of multiplies to values.

Original: The diagonal  algebra is extremely parameter eﬃcient and we do ﬁnd the interaction between diﬀerent components through whitening to be important as hypothesized – a network trained with whitening achieves 6% higher top-1 accuracy than the same network trained with batch normalization.
Revised: The diagonal [Figure] [Figure] algebra is extremely parameter eﬃcient and we do ﬁnd the interaction between diﬀerent components through whitening to be important as hypothesized – a network trained with whitening achieves 6% higher top-1 accuracy than the same network trained with batch normalization.

Original: Unfortunately, adding whitening increases the total number of inference FLOPs by a factor of 3 × .
Revised: Unfortunately, adding whitening increases the total number of inference FLOPs by a factor of 3 × .

Original: We are left to conclude that whitening is not currently competitive and recommend using batch normalization for all algebras.
Revised: We are left to conclude that whitening is not currently competitive and recommend using batch normalization for all algebras.

Original: Future work exploring the role of the interaction from whitening, and alternatives that are more computationally eﬃcient is an interesting direction.
Revised: Future work exploring the role of the interaction from whitening, and alternatives that are more computationally eﬃcient is an interesting direction.

Original: An additional beneﬁt is that AlgebraNets applied to MobileNet like architectures increase the computational density of the often bandwidth bound depthwise convolutions, while reducing the number of FLOPs in the more costly 1 × 1 convolution.
Revised: An additional beneﬁt is that AlgebraNets applied to MobileNet like architectures increase the computational density of the often bandwidth bound depthwise convolutions, while reducing the number of FLOPs in the more costly 1 × 1 convolution.

Original: [Figure]
Revised: [Figure]

Original: Pruning ResNet- 
Revised: Pruning ResNet-

Original: We use magnitude based pruning, with the schedule proposed in Zhu and Gupta (2017).
Revised: We use magnitude based pruning, with the schedule proposed in (Zhu and Gupta, 2017).

Original: We always begin pruning at 20% and end pruning at 80% of the total training iterations.
Revised: We always begin pruning at 20% and end pruning at 80% of the total training iterations.

Original: We prune everysteps.
Revised: We prune everysteps.

Original: At each pruning step, we set tuples with the lowest magnitude, given by (cid:112)(cid:80) t 2 i , to 0 .
Revised: At each pruning step, we set tuples with the lowest magnitude, given by (cid:112)(cid:80) t 2 i , to 0 .

Original: We do not prune terms in the ﬁnal linear layer, the tuple-initialization convolutions, or in the initial convolution of the ResNet.
Revised: We do not prune terms in the ﬁnal linear layer, the tuple-initialization convolutions, or in the initial convolution of the ResNet.

Original: To allow for comparisons with Gale et al. (2019), we use ReLU activations in pruning experiments, as opposed to swish as used in Fig.
Revised: To allow for comparisons with (Gale et al., 2019), we use ReLU activations in pruning experiments, as opposed to swish as used in Fig.

Original: Final top-1 accuracies of pruned networks are shown in Fig.
Revised: Final top-1 accuracies of pruned networks are shown in Fig.

Original: Despite pruning entire tuples, which allows skipping an entire tuple multiplication, we are still able to ﬁnd sparse networks that are similarly FLOP eﬃcient to those from Gale et al. (2019), while having higher compute density due to the algebra structure.
Revised: Despite pruning entire tuples, which allows skipping an entire tuple multiplication, we are still able to ﬁnd sparse networks that are similarly FLOP eﬃcient to those from (Gale et al., 2019), while having higher compute density due to the algebra structure.

Original: Pruning individual components, rather than setting entire tuples to 0 does improve performance, though does not provide the same computational beneﬁts oﬀered by AlgebraNets.
Revised: Pruning individual components, rather than setting entire tuples to 0 does improve performance, though does not provide the same computational beneﬁts oﬀered by AlgebraNets.

Original: We provide further results in
Revised: We provide further results in

Original: Appendix C.
Revised: Appendix C.

Original: 4.3 Transformer-XL on Enwik8 and RNNs on WikiText- 
Revised: 4.3 Transformer-XL on Enwik8 and RNNs on WikiText-

Original: We perform character level language modeling on the Enwik8 dataset from the Hutter Prize LLC (2009) with the Transformer-XL Dai et al. (2019) architecture.
Revised: We perform character level language modeling on the Enwik8 dataset from the Hutter Prize (LLC, 2009) with the Transformer-XL (Dai et al., 2019) architecture.

Original: We tuned the baseline model by halving the embedding size, number of heads and feedforward size, resulting in a more challenging ‘eﬃcient’ baseline with only 25% as many parameters as the 24 layer network from Dai et al.
Revised: We tuned the baseline model by halving the embedding size, number of heads and feedforward size, resulting in a more challenging ‘eﬃcient’ baseline with only 25% as many parameters as the 24 layer network from (Dai et al., 2019).

Original: We train with Adam Kingma and Ba (2014) (learning rate 2 × 10 − 4 ), dropout 0.25 (component-wise, not tuple-wise), and windows of 1536 at train and 4608 at test time.
Revised: We train with Adam (Kingma and Ba, 2014) (learning rate 2 × 10 − 4 ), dropout 0.25 (component-wise, not tuple-wise), and windows of 1536 at train and 4608 at test time.

Original: The M 2 ( R ) -Transformer uses a learning rate of . 0005 and dropout 0 .
Revised: The M 2 ( R ) -Transformer uses a learning rate of . 0005 and dropout 0 .

Original: Our ‘eﬃcient’ baseline 24 layer Transformer-XL model has an embedding size of 512, 4 heads of size 128 each, and a feed-forward hidden size of 1536 for a total
Revised: Our ‘eﬃcient’ baseline 24 layer Transformer-XL model has an embedding size of 512, 4 heads of size 128 each, and a feed-forward hidden size of 1536 for a total

Original: [Table]
Revised: [Table]

Revised: 18L (see (Dai et al., 2019))

Revised: R

Revised: [Table]

Revised: [Table]

Revised: Transformer

Revised: [Table]

Revised: (see (Dai et al.

Revised: [Table]

Revised: [Table]

Revised: R 277.

Revised: [Table]

Revised: [Table]

Revised: ours

Revised: [Table]

Revised: R

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: eﬃcient 42L

Revised: [Table]

Revised: ours

Revised: [Table]

Revised: M

Original:  character on par.
Revised: [Table] [Table] [Table] same time - again while keeping the validation bits per character on par.

Original: The reported inference FLOPs are representative of an incremental token employing a memory length 1538 or 4608, corresponding to the training and test regimes.
Revised: The reported inference FLOPs are representative of an incremental token employing a memory length 1538 or 4608, corresponding to the training and test regimes.

Original: parameter count of 69.4 million.
Revised: parameter count of 69.4 million.

Original: It achieves 0.99 bits per character (BPC), matching the results of Dai et al. (2019) while requiring 75% less parameters.
Revised: It achieves 0.99 bits per character (BPC), matching the results of (Dai et al., 2019) while requiring 75% less parameters.

Original: Using the M 2 ( R ) -algebra in all linear layers with ﬁxed activation size results in a further 75% reduction in parameter count.
Revised: Using the M 2 ( R ) -algebra in all linear layers with ﬁxed activation size results in a further 75% reduction in parameter count.

Original: We use these parameter savings to increase the depth of the model from 24 to 42 layers, resulting in a model with 45% as many parameters as the ‘eﬃcient’ baseline.
Revised: We use these parameter savings to increase the depth of the model from 24 to 42 layers, resulting in a model with 45% as many parameters as the ‘eﬃcient’ baseline.

Original: The resulting M 2 ( R ) AlgebraNet also achieves 0.
Revised: The resulting M 2 ( R ) AlgebraNet also achieves 0.

Original: BPC, but with only 31.2 million parameters.
Revised: BPC, but with only 31.2 million parameters.

Original: The character-embedding layers are computationally unchanged; they associate each character with a d 4 -sized M 2 ( R ) embedding which can be thought of as a reshaped R d embedding.
Revised: The character-embedding layers are computationally unchanged; they associate each character with a d 4 -sized M 2 ( R ) embedding which can be thought of as a reshaped R d embedding.

Original: Special consideration has to be paid to the R l × l attention matrix, which is often regarded as a practical memory and compute bottleneck Child et al. Roy et al.
Revised: Special consideration has to be paid to the R l × l attention matrix, which is often regarded as a practical memory and compute bottleneck (Child et al., 2019; Roy et al., 2020;

Original: Kitaev et al.
Revised: Kitaev et al., 2020).

Original: Using an l × l algebra valued attention matrix would increase the memory and compute requirements (e.g. by a factor 2 in the Complex-Transformer Yang et al. (2020) or 4 for
Revised: Using an l × l algebra valued attention matrix would increase the memory and compute requirements (e.g. by a factor 2 in the Complex-Transformer (Yang et al., 2020) or 4 for

Original: M 2 ( R ) ).
Revised: M 2 ( R ) ).

Original: Thus, we desire a real valued attention matrix from the sets of R k -valued key and query vectors ( k, q ) .
Revised: Thus, we desire a real valued attention matrix from the sets of R k -valued key and query vectors ( k, q ) .

Original: We do this by reshaping keys and queries from M 2 ( R ) to R .
Revised: We do this by reshaping keys and queries from M 2 ( R ) to R .

Original: Formally, we redeﬁne the attentions real-valued scalar product as (cid:104) k, q (cid:105) M 2 ( R ) :
Revised: Formally, we redeﬁne the attentions real-valued scalar product as (cid:104) k, q (cid:105) M 2 ( R ) :

Original: = (cid:104) F ( k ) , F ( q ) (cid:105) R where F ﬂattens the input into a real vector.
Revised: = (cid:104) F ( k ) , F ( q ) (cid:105) R where F ﬂattens the input into a real vector.

Original: Finally, we also consider a dataset approximately one order of magnitude larger by tokenizing
Revised: Finally, we also consider a dataset approximately one order of magnitude larger by tokenizing

Original: WikiText-103 Merity et al. (2016) into characters (instead of the more common words).
Revised: WikiText-103 (Merity et al., 2016) into characters (instead of the more common words).

Original: On this dataset we consider a single layer GRU architecture followed by 5 linear readout layers with the ReLU non-linearity, skip connections and layer normalization after each layer.
Revised: On this dataset we consider a single layer GRU architecture followed by 5 linear readout layers with the ReLU non-linearity, skip connections and layer normalization after each layer.

Original: We train using a batch size of 16, Adam (learning rate of 10 − 4 ), and L 2 regularization of 10 − 7 for 200,000 steps.
Revised: We train using a batch size of 16, Adam (learning rate of 10 − 4 ), and L 2 regularization of 10 − 7 for 200,000 steps.

Original: Training takes two days for the largest baseline variant on a single V100 GPU.
Revised: Training takes two days for the largest baseline variant on a single V100 GPU.

Original: We train using length 512 sequences and back propagation through time.
Revised: We train using length 512 sequences and back propagation through time.

Original: We initialize the diﬀerent components of each algebra with single linear layers from the input.
Revised: We initialize the diﬀerent components of each algebra with single linear layers from the input.

Original: We report results on the typical validation set.
Revised: We report results on the typical validation set.

Original: We replace a gated recurrent unit (GRU) Cho et al. (2014) with the AlgebraNet equivalent, as well as replacing the readout layers with AlgebraNet variants.
Revised: We replace a gated recurrent unit (GRU) (Cho et al., 2014) with the AlgebraNet equivalent, as well as replacing the readout layers with AlgebraNet variants.

Original: We consider M 2 ( R ) , C , H , and M 3 ( R ) .
Revised: We consider M 2 ( R ) , C , H , and M 3 ( R ) .

Original: Results are shown in Table 3.
Revised: Results are shown in Table 3.

Original: A C AlgebraNet with a hidden size of 1024 and 24.1 million parameters achieves a validation BPC of 1.26, comparable to a real-valued network with 1.45 times the parameter count.
Revised: A C AlgebraNet with a hidden size of 1024 and 24.1 million parameters achieves a validation BPC of 1.26, comparable to a real-valued network with 1.45 times the parameter count.

Original: We ﬁnd that M 2 ( R ) with a hidden size of 512 results in a validation BPC of 1.30, comparable to a model with twice as many parameters.
Revised: We ﬁnd that M 2 ( R ) with a hidden size of 512 results in a validation BPC of 1.30, comparable to a model with twice as many parameters.

Original: Again, demonstrating the parameter eﬃciency of M 2 ( R ) and the usefulness of AlgebraNets for problems such as language modeling where parameter eﬃciency is crucial.
Revised: Again, demonstrating the parameter eﬃciency of M 2 ( R ) and the usefulness of AlgebraNets for problems such as language modeling where parameter eﬃciency is crucial.

Original:  Conclusion
Revised:  Conclusion

Original: Conventional neural networks are composed of real-valued weights and activations along with real valued operators.
Revised: Conventional neural networks are composed of real-valued weights and activations along with real valued operators.

Original: In this work, we proposed AlgebraNets, a general paradigm of replacing real valued weights and operators with weights and operators from other associative algebras in a general fashion.
Revised: In this work, we proposed AlgebraNets, a general paradigm of replacing real valued weights and operators with weights and operators from other associative algebras in a general fashion.

Original: We show these methods to be more parameter eﬃcient than their real-valued counterparts while having higher compute density.
Revised: We show these methods to be more parameter eﬃcient than their real-valued counterparts while having higher compute density.

Original: We also ﬁnd that the M 2 ( R ) algebra is more FLOP eﬃcient than previously considered algebras – in fact it is as FLOP eﬃcient as the reals.
Revised: We also ﬁnd that the M 2 ( R ) algebra is more FLOP eﬃcient than previously considered algebras – in fact it is as FLOP eﬃcient as the reals.

Original: The increased compute density of the proposed algebras will prove particularly useful for sparse neural networks and auto-regressive inference, due to modern hardware favoring a relatively high compute density.
Revised: The increased compute density of the proposed algebras will prove particularly useful for sparse neural networks and auto-regressive inference, due to modern hardware favoring a relatively high compute density.

Original: We hope that our work enables further development of these methods and promotes broader research into the fundamental design choices upon which modern neural networks are based.
Revised: We hope that our work enables further development of these methods and promotes broader research into the fundamental design choices upon which modern neural networks are based.

Original: [Table]
Revised: [Table]

Original: R 18.
Revised: R 18.

Original: C 9.
Revised: C 9.

Original: R 27.
Revised: R 27.

Original: M
Revised: M

Original: [Table]
Revised: [Table]

Original: ( R ) 12.

Original: R 35.
Revised: R 35.

Original: [Table]
Revised: [Table]

Original: H 12.
Revised: H 12.

Original: R 47. [Table] 
Revised: [Table] [Table] R

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A Additional Algebra Information A.1 M 2 ( R ) Multiplication Table
Revised: A Additional Algebra Information A.1 M 2 ( R ) Multiplication Table

Original: [Table]
Revised: [Table]

Original: A.2 M 2 ( C ) Multiplication Table
Revised: A.2 M 2 ( C ) Multiplication Table

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: ) a b c d e f g h a a b c d    b b - a d - c    c    a b c d d    b - a d - c e e f g h    f f - e h - g
Revised: ) a b c d e f g h a a b c d    b b - a d - c    c    a b c d d    b - a d - c e e f g h    f f - e h - g

Original:    g    e f g h h
Revised:    g    e f g h h

Original: [Table]
Revised: [Table]

Original: A.3 Dual Number Multiplication Table
Revised: A.3 Dual Number Multiplication Table

Original: [Table]
Revised: [Table]

Original: Dual Number a b a a b b b
Revised: Dual Number a b a a b b b

Original: A.4 Cross Product Multiplication Table
Revised: A.4 Cross Product Multiplication Table

Original: Multiplicated uses the cross product between length-3 tuples ( t a , t b , t c ) .
Revised: Multiplicated uses the cross product between length-3 tuples ( t a , t b , t c ) .

Original: Cross Product a b c a c -b b -c a c b -a
Revised: Cross Product a b c a c -b b -c a c b -a

Original: A.5 Linear Layer Example
Revised: A.5 Linear Layer Example

Original: We give a concrete example of replacing a real linear layer with M 2 ( R ) -linear layer such that the activation memory is kept identical.
Revised: We give a concrete example of replacing a real linear layer with M 2 ( R ) -linear layer such that the activation memory is kept identical.

Original: Intuitively, this can be thought of as reshaping the R d input activations to have shape
Revised: Intuitively, this can be thought of as reshaping the R d input activations to have shape

Original: M 2 ( R ) d/ 4 that is processed by a f M : M 2 ( R ) d/ 4 → M 2 ( R ) d/ 4 linear layer resulting in output activations – when ﬂattened – with shape R d .
Revised: M 2 ( R ) d/ 4 that is processed by a f M : M 2 ( R ) d/ 4 → M 2 ( R ) d/ 4 linear layer resulting in output activations – when ﬂattened – with shape R d .

Original: Each such linear layer f M requires 14 of the parameters and 12 of the FLOPS compared to a real R d → R d linear layer counterpart.
Revised: Each such linear layer f M requires 14 of the parameters and 12 of the FLOPS compared to a real R d → R d linear layer counterpart.

Original: B AlgebraNet Choices: Activations, Initializations, etc B.1 Tuple-wise Nonlinearity
Revised: B AlgebraNet Choices: Activations, Initializations, etc B.1 Tuple-wise Nonlinearity

Original: We consider equations of the form:
Revised: We consider equations of the form:

Original: [Equation]
Revised: [Equation]

Original: We found that if g is the tuple mean, and f is H the Heaviside function, top-1 performance dropped on an M 2 ( R )
Revised: We found that if g is the tuple mean, and f is H the Heaviside function, top-1 performance dropped on an M 2 ( R )

Original: ResNet-

Revised: ResNet-

Original: AlgebraNet by 2.97%.
Revised: AlgebraNet by 2.97%.

Original: While this drop is signiﬁcant, the resulting activation sparsity might make it a desirable tradeoﬀ in some circumstances.
Revised: While this drop is signiﬁcant, the resulting activation sparsity might make it a desirable tradeoﬀ in some circumstances.

Original: Other methods, such as setting g to be the determinant resulted in greater than a 10% drop in performance.
Revised: Other methods, such as setting g to be the determinant resulted in greater than a 10% drop in performance.

Original: B.2 Initialization
Revised: B.2 Initialization

Original: For a ResNet-50 H -AlgebraNet with the standard number of channels divided by 4, we ﬁnd a top-1 performance of 74 . 0 ± 0 .
Revised: For a ResNet-50 H -AlgebraNet with the standard number of channels divided by 4, we ﬁnd a top-1 performance of 74 . 0 ± 0 .

Original: 14 using standard initialization and 74 .
Revised: 14 using standard initialization and 74 .

Original: 1 ± 0 .
Revised: 1 ± 0 .

Original: 15 using initialization from Gaudet and Maida (2018).
Revised: 15 using initialization from Gaudet and Maida (2018).

Original: These experiments are done using standard batch normalization instead of the more expensive whitening procedure.
Revised: These experiments are done using standard batch normalization instead of the more expensive whitening procedure.

Original: B.3 Conversion to Reals
Revised: B.3 Conversion to Reals

Original: For all considered algebras, the norm of the tuple is mathematically given by (cid:112)(cid:80) i t 2 i .
Revised: For all considered algebras, the norm of the tuple is mathematically given by (cid:112)(cid:80) i t 2 i .

Original: It is possible that the optimal choice for converting to the reals would be diﬀerent in models with very large ﬁnal layers, such as word based language modeling – which we do not consider.
Revised: It is possible that the optimal choice for converting to the reals would be diﬀerent in models with very large ﬁnal layers, such as word based language modeling – which we do not consider.

Original: C AlgebraNet Pruning C.1 Alternative tuple pruning of M 2 ( R )
Revised: C AlgebraNet Pruning C.1 Alternative tuple pruning of M 2 ( R )

Original: For M 2 ( R ) , we consider a variety of alternative pruning methods to remove entire tuples, based on the two eigenvalues, λ 

Revised: For M 2 ( R ) , we consider a variety of alternative pruning methods to remove entire tuples, based on the two eigenvalues, λ

Original: and λ 2 and singular values, σ 1 , σ 2 .
Revised: and λ 2 and singular values, σ 1 , σ 2 .

Original: Speciﬁcally, because our matrices are square but not symmetric, the Forbenius norm is deﬁned based on the singular values which correspond to the squared eigenvalues of AA T , if A is the matrix in question.
Revised: Speciﬁcally, because our matrices are square but not symmetric, the Forbenius norm is deﬁned based on the singular values which correspond to the squared eigenvalues of AA T , if A is the matrix in question.

Original: • Frobenius Norm: (cid:0) σ 21 + σ 22 (cid:1) 1 /•
Revised: • Frobenius Norm: (cid:0) σ 21 + σ 22 (cid:1) 1 /•

Original: Determinant: λ 1 λ• Smallest Eigenvalue min( | λ 1 | , | λ 2 | )
Revised: Determinant: λ 1 λ• Smallest Eigenvalue min( | λ 1 | , | λ 2 | )

Original: • Largest Eigenvalue max( | λ 1 | , | λ 2 | )
Revised: • Largest Eigenvalue max( | λ 1 | , | λ 2 | )

Original: In all cases, we remove tuples with the minimum magnitude of one of those options.
Revised: In all cases, we remove tuples with the minimum magnitude of one of those options.

Original: Sparsity det min max
Revised: Sparsity det min max

Original: [Table]
Revised: [Table]

Original: In Table 4, we show the resulting drop in top-1 accuracy relative to the Frobenius norm at three diﬀerent sparsities for three alternative pruning methods.
Revised: In Table 4, we show the resulting drop in top-1 accuracy relative to the Frobenius norm at three diﬀerent sparsities for three alternative pruning methods.

Original: In addition to always achieving the best performance, the Frobenius norm has the additional advantage that it is deﬁned for all Algebra variants that was consider, rather than an M n ( R ) speciﬁc variant, for example.
Revised: In addition to always achieving the best performance, the Frobenius norm has the additional advantage that it is deﬁned for all Algebra variants that was consider, rather than an M n ( R ) speciﬁc variant, for example.

Original: C.2 Pruning components of M 2 ( R ) and H
Revised: C.2 Pruning components of M 2 ( R ) and H

Original: For M 2 ( R ) and H , we also prune individual tuple elements based on element norms.
Revised: For M 2 ( R ) and H , we also prune individual tuple elements based on element norms.

Original: This equally reduces the number of non-zero weights in the network, though it does not result in entire matrix multiplies that can be skipped.
Revised: This equally reduces the number of non-zero weights in the network, though it does not result in entire matrix multiplies that can be skipped.

Original: [Table]
Revised: [Table]

Original: In Table 5, we show the resulting increase in top-1 accuracy that results from pruning individual tuple components, rather than entire tuples.
Revised: In Table 5, we show the resulting increase in top-1 accuracy that results from pruning individual tuple components, rather than entire tuples.

Original: However, due to the structure M n ( R ) and H multiplication, setting individual values to 0 does not result in 0 in the output.
Revised: However, due to the structure M n ( R ) and H multiplication, setting individual values to 0 does not result in 0 in the output.

Original: Therefore, pruning entire tuples provides more useful computational advantages.
Revised: Therefore, pruning entire tuples provides more useful computational advantages.

Original: D AlgebraNet Tests on CIFAR
Revised: D AlgebraNet Tests on CIFAR

Original: We use a network structure based on that described in Gaudet and Maida (2018).
Revised: We use a network structure based on that described in Gaudet and Maida (2018).

Original: We begin with the same ResNet structure, with 128, 256, and then 512 channels in each real block.
Revised: We begin with the same ResNet structure, with 128, 256, and then 512 channels in each real block.

Original: For the C networks, all channel counts are divided by two.
Revised: For the C networks, all channel counts are divided by two.

Original: For the M 2 ( R ) and H networks, we assign the initial convolution, before the residual blocks, to have half the original number of channels, all other channel counts are divided by four.
Revised: For the M 2 ( R ) and H networks, we assign the initial convolution, before the residual blocks, to have half the original number of channels, all other channel counts are divided by four.

Original: Thus, for H and M 2 ( R ) we have slightly more than 1/4 the parameters.
Revised: Thus, for H and M 2 ( R ) we have slightly more than 1/4 the parameters.

Original: We train with 24 × 24 random crops and evaluate on 32 × 32 images.
Revised: We train with 24 × 24 random crops and evaluate on 32 × 32 images.

Original: [Table]
Revised: [Table]

Original: CIFAR
Revised: CIFAR

Original: BN denotes Batch Normalization, denotes use of whitening.
Revised: BN denotes Batch Normalization, denotes use of whitening.

Original: We ﬁnd we are able to divide the channels in the ﬁlter by two and maintain the same performance using complex valued networks.
Revised: We ﬁnd we are able to divide the channels in the ﬁlter by two and maintain the same performance using complex valued networks.

Original: When reducing the parameter count by a factor of ∼ four, we ﬁnd we are able to again match baseline performance with quaternions and 2 × 2 matrices.
Revised: When reducing the parameter count by a factor of ∼ four, we ﬁnd we are able to again match baseline performance with quaternions and 2 × 2 matrices.

Original: Regularization has non-trivial eﬀect on performance, and by more ﬁnely adjusting the L 2 loss for the diﬀerent algebras may result in higher top-1 accuracy.
Revised: Regularization has non-trivial eﬀect on performance, and by more ﬁnely adjusting the L 2 loss for the diﬀerent algebras may result in higher top-1 accuracy.

Original: We note that the relative reduction in parameters on CIFAR-10 is not something we are able to replicate on ImageNet.
Revised: We note that the relative reduction in parameters on CIFAR-10 is not something we are able to replicate on ImageNet.

Original: The results from the main text also hold here – M 2 ( R ) is the only algebra that is able to maintain accuracy while having fewer FLOPs than the baseline real network.
Revised: The results from the main text also hold here – M 2 ( R ) is the only algebra that is able to maintain accuracy while having fewer FLOPs than the baseline real network.

Original: For these experiments, we used algebra speciﬁc weight initializations, though we again veriﬁed that this does not seem to have a substantial eﬀect.
Revised: For these experiments, we used algebra speciﬁc weight initializations, though we again veriﬁed that this does not seem to have a substantial eﬀect.

Original: E Example M 2 ( R ) Code
Revised: E Example M 2 ( R ) Code

Original: We write the update rule explicitly for readability.
Revised: We write the update rule explicitly for readability.

Original: Note that it is possible to concatenate the relevant terms on the channel axis to reduce the number of convolutions needed.
Revised: Note that it is possible to concatenate the relevant terms on the channel axis to reduce the number of convolutions needed.

Original: Convolution
Revised: Convolution

Original: ''' Simplified example code for M_2(R).
Revised: ''' Simplified example code for M_2(R).

Original: x: Input with an additional algebra axis.
Revised: x: Input with an additional algebra axis.

Original: In the case of a convolution, either (B, H, W, C, A) or (B, C, H, W, A) w: Corresponding weight matrix, with an additional
Revised: In the case of a convolution, either (B, H, W, C, A) or (B, C, H, W, A) w: Corresponding weight matrix, with an additional

Original: algebra axis
Revised: algebra axis.

Revised: # Rule that describes 2x

Original: [Equation]
Revised: [Equation]

Revised: multiplication

Original: [Equation]
Revised: [Equation]

Original: # Update each of the four algebra components.
Revised: # Update each of the four algebra components.

Original: x_new =
Revised: x_new =

Original: [Equation]
Revised: [0, 0, 0, 0] for i in range(4): for j in range(2): # w: weight with an extra algebra dimension. # x: Input with shape [B, ... , A] where A is the additional algebra dimension. x_new[i]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 0] for i in range(4): for j in range(2): # [Table]
Revised: [i][

Original: [Table]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: # Add bias if wanted.
Revised: # Add bias if wanted.

Original: Add (4,) to 
Revised: Add (4,) to shape.

Original: [Table]
Revised: Linear

Original:  # Update each of the four algebra components.
Revised: Layer # Update each of the four algebra components.

Original: x_new =
Revised: x_new =

Original: [0, 0, 0, 0] for i in range(4): for j in range(2): [Table]
Revised: [0, 0, 0, 0] for i in range(4): for j in range(2): # w: weight with an extra algebra dimension.

Original: [Table]
Revised: # x: Input with shape , A] where is the algebra dimension.

Original: [Equation]
Revised: [Equation]

Original: # Add bias if wanted.
Revised: # Add bias if wanted.

Original: Add (4,) to shape.
Revised: Add (4,) to shape.

Revised: F Increased Activation Memory

Revised: Due to the activations, there will be a slight increase in memory footprint from AlgebraNets in some cases.

Revised: For example, in a

Revised: M 2 ( R ) AlgebraNet for ResNet-50 with channels/4, there will be C/4 convolutions performed.

Revised: This would, in a naive implementation, result in twice the activation memory.

Revised: However, with a properly written kernel, this would not be the case.

Revised: There is, however, an additional factor: to reach comparable performance a slightly larger network than C/4 is needed.

Revised: In practice about a 1.