Original: P ARAMETERIZATION OF H YPERCOMPLEX M ULTIPLICATIONS
Revised: P ARAMETERIZATION OF H YPERCOMPLEX M ULTIPLICATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent works have demonstrated reasonable success of representation learning in hypercomplex space.
Revised: Recent works have demonstrated reasonable success of representation learning in hypercomplex space.

Original: Speciﬁcally, the Hamilton product (4D hypercomplex multiplication) enables learning effective representations while saving up to 75% parameters.
Revised: Speciﬁcally, the Hamilton product (4D hypercomplex multiplication) enables learning effective representations while saving up to 75% parameters.

Original: However, one key caveat is that hypercomplex space only exists at very few predeﬁned dimensions.
Revised: However, one key caveat is that hypercomplex space only exists at very few predeﬁned dimensions.

Original: This restricts the ﬂexibility of models that leverage hypercomplex multiplications.
Revised: This restricts the ﬂexibility of models that leverage hypercomplex multiplications.

Original: To this end, we propose parameterizing hypercomplex multiplications, allowing models to learn multiplication rules from data regardless of whether such rules are predeﬁned.
Revised: To this end, we propose parameterizing hypercomplex multiplications, allowing models to learn multiplication rules from data regardless of whether such rules are predeﬁned.

Original: As a result, our method not only subsumes the Hamilton product, but also learns to operate on any arbitrary n D hypercomplex space, providing more architectural ﬂexibility.
Revised: As a result, our method not only subsumes the Hamilton product, but also learns to operate on any arbitrary n D hypercomplex space, providing more architectural ﬂexibility.

Original: Experiments of applications to LSTM and Transformer on natural language inference, machine translation, text style transfer, and subject verb agreement demonstrate architectural ﬂexibility and effectiveness of the proposed approach.
Revised: Experiments of applications to LSTM and Transformer on natural language inference, machine translation, text style transfer, and subject verb agreement demonstrate architectural ﬂexibility and effectiveness of the proposed approach.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A Quaternion is a 4D hypercomplex number with one real component and three imaginary components.
Revised: A Quaternion is a 4D hypercomplex number with one real component and three imaginary components.

Original: The Hamilton product is the hypercomplex multiplication of two Quaternions.
Revised: The Hamilton product is the hypercomplex multiplication of two Quaternions.

Original: Recent works in Quaternion space and Hamilton products have demonstrated reasonable success (Parcollet et al., 2018b; 2019; Tay et al., 2019).
Revised: Recent works in Quaternion space and Hamilton products have demonstrated reasonable success (Parcollet et al., 2018b; 2019; Tay et al., 2019).

Original: The Hamilton product enjoys a parameter saving with 1 / 4 learnable parameters as compared with the real-valued matrix multiplication.
Revised: The Hamilton product enjoys a parameter saving with 1 / 4 learnable parameters as compared with the real-valued matrix multiplication.

Original: It also enables effective representation learning by modeling interactions between real and imaginary components.
Revised: It also enables effective representation learning by modeling interactions between real and imaginary components.

Original: One of the attractive properties of Quaternion models (Parcollet et al., 2018b; 2019; Tay et al., 2019) is its high applicability and universal usefulness to one of the most ubiquitous layers in deep learning, i.e., the fully-connected (or feed-forward) layer.
Revised: One of the attractive properties of Quaternion models (Parcollet et al., 2018b; 2019; Tay et al., 2019) is its high applicability and universal usefulness to one of the most ubiquitous layers in deep learning, i.e., the fully-connected (or feed-forward) layer.

Original: This transformation layer is one of the most dominant component in existing deep learning literature (Goodfellow et al., 2016; Zhang et al., 2020).
Revised: This transformation layer is one of the most dominant component in existing deep learning literature (Goodfellow et al., 2016; Zhang et al., 2020).

Original: Its pervasiveness cannot be understated, given its centrality to many core building blocks in neural network research.
Revised: Its pervasiveness cannot be understated, given its centrality to many core building blocks in neural network research.

Original: Given widespread adoptions of fully-connected layers, e.g., within
Revised: Given widespread adoptions of fully-connected layers, e.g., within

Original: LSTM networks (Hochreiter & Schmidhuber, 1997) and Transformer models (Vaswani et al., 2017), having ﬂexibility to balance between parameter savings and effectiveness could be extremely useful to many real-world applications.
Revised: LSTM networks (Hochreiter & Schmidhuber, 1997) and Transformer models (Vaswani et al., 2017), having ﬂexibility to balance between parameter savings and effectiveness could be extremely useful to many real-world applications.

Original: Unfortunately, hypercomplex space only exists at 4D (Quaternions), 8D (Octonions), and 16D (Sedonions), which generalizes the 2D complex space (Rishiyur, 2006).
Revised: Unfortunately, hypercomplex space only exists at 4D (Quaternions), 8D (Octonions), and 16D (Sedonions), which generalizes the 2D complex space (Rishiyur, 2006).

Original: Moreover, custom operators are required at each hypercomplex dimensionality.
Revised: Moreover, custom operators are required at each hypercomplex dimensionality.

Original: For instance, the Hamilton product is the hypercomplex multiplication in 4D hypercomplex space.
Revised: For instance, the Hamilton product is the hypercomplex multiplication in 4D hypercomplex space.

Original: Thus, no operator in such predeﬁned hypercomplex space is suitable for applications that prefer reducing parameters to 1 /n , where n (cid:54) = 4 , 8 , 16 .
Revised: Thus, no operator in such predeﬁned hypercomplex space is suitable for applications that prefer reducing parameters to 1 /n , where n (cid:54) = 4 , 8 , 16 .

Original: In view of the architectural limitation due to the very few choices of existing hypercomplex space, we propose parameterization of hypercomplex multiplications, i.e., learning the real and imaginary component interactions from data in a differentiable fashion.
Revised: In view of the architectural limitation due to the very few choices of existing hypercomplex space, we propose parameterization of hypercomplex multiplications, i.e., learning the real and imaginary component interactions from data in a differentiable fashion.

Original: Essentially, our method can operate on an arbitrary n D hypercomplex space, aside from subsuming those predeﬁned hypercomplex multiplication rules, facilitating using up to 1 /n learnable parameters while maintaining expressiveness.
Revised: Essentially, our method can operate on an arbitrary n D hypercomplex space, aside from subsuming those predeﬁned hypercomplex multiplication rules, facilitating using up to 1 /n learnable parameters while maintaining expressiveness.

Original: In practice, the hyperparameter n can be ﬂexibly speciﬁed or tuned by users based on applications.
Revised: In practice, the hyperparameter n can be ﬂexibly speciﬁed or tuned by users based on applications.

Original: Concretely, our prime contribution is a new module that parameterizes and generalizes the hypercomplex multiplication by learning the real and imaginary component interactions, i.e., multiplica- tion rules, from data.
Revised: Concretely, our prime contribution is a new module that parameterizes and generalizes the hypercomplex multiplication by learning the real and imaginary component interactions, i.e., multiplica- tion rules, from data.

Original: Our method, which we call the parameterized hypercomplex multiplication layer, is characterized by a sum of Kronecker products that generalize the vector outer products to higher dimensions in real space.
Revised: Our method, which we call the parameterized hypercomplex multiplication layer, is characterized by a sum of Kronecker products that generalize the vector outer products to higher dimensions in real space.

Original: To demonstrate applicability, we equip two well-established models (LSTM and Transformer) with our proposed method.
Revised: To demonstrate applicability, we equip two well-established models (LSTM and Transformer) with our proposed method.

Original: We conduct extensive experiments on different tasks, i.e., natural language inference for LSTM and machine translation for Transformer.
Revised: We conduct extensive experiments on different tasks, i.e., natural language inference for LSTM and machine translation for Transformer.

Original: Additionally, we perform further experiments on text style transfer and subject verb agreement tasks.
Revised: Additionally, we perform further experiments on text style transfer and subject verb agreement tasks.

Original: All in all, our method has demonstrated architectural ﬂexibility through different experimental settings, where it generally can use a fraction of the learnable parameters with minimal degradation or slight improvement in performance.
Revised: All in all, our method has demonstrated architectural ﬂexibility through different experimental settings, where it generally can use a fraction of the learnable parameters with minimal degradation or slight improvement in performance.

Original: The overall contributions of this work are summarized as follows:
Revised: The overall contributions of this work are summarized as follows:

Original: • We propose a new parameterization of hypercomplex multiplications: the parameterized hypercomplex multiplication (PHM) layers.
Revised: • We propose a new parameterization of hypercomplex multiplications: the parameterized hypercomplex multiplication (PHM) layers.

Original: The key idea behind PHM layers is to learn the interactions between real and imaginary components, i.e., multiplication rules, from data using a sum of Kronecker products.
Revised: The key idea behind PHM layers is to learn the interactions between real and imaginary components, i.e., multiplication rules, from data using a sum of Kronecker products.

Original: • We demonstrate the applicability of the PHM layers by leveraging them in two dominant neural architectures: LSTM and Transformer.
Revised: • We demonstrate the applicability of the PHM layers by leveraging them in two dominant neural architectures: LSTM and Transformer.

Original: • We empirically show architectural ﬂexibility and effectiveness of PHM layers by conducting extensive experiments on ﬁve natural language inference tasks, seven machine translation datasets, together with text style transfer and subject verb agreement tasks.
Revised: • We empirically show architectural ﬂexibility and effectiveness of PHM layers by conducting extensive experiments on ﬁve natural language inference tasks, seven machine translation datasets, together with text style transfer and subject verb agreement tasks.

Original:  B ACKGROUND ON Q UATERNIONS AND H AMILTON P RODUCTS
Revised:  B ACKGROUND ON Q UATERNIONS AND H AMILTON P RODUCTS

Original: We begin by introducing the background for the rest of the paper.
Revised: We begin by introducing the background for the rest of the paper.

Original: Concretely, we describe Quaternion algebra along with Hamilton products, which is at the heart of our proposed approach.
Revised: Concretely, we describe Quaternion algebra along with Hamilton products, which is at the heart of our proposed approach.

Original: Quaternion A Quaternion Q ∈ H is a hypercomplex number with one real component and three imaginary components as follows:
Revised: Quaternion A Quaternion Q ∈ H is a hypercomplex number with one real component and three imaginary components as follows:

Original: [Equation]
Revised: [Equation]

Original: whereby ijk = i 

Revised: whereby ijk = i

Original: =
Revised: =

Original: j 2 = k 2 = − 1 .
Revised: j 2 = k 2 = − 1 .

Original: In equation 2.1, noncommutative multiplication rules hold:
Revised: In equation 2.1, noncommutative multiplication rules hold:

Original: [Equation]
Revised: [Equation]

Original: kj = − i , ik = − j .
Revised: kj = − i , ik = − j .

Original: Here, Q r is the real component, Q x , Q y , Q z are real numbers that represent the imaginary components of the Quaternion Q .
Revised: Here, Q r is the real component, Q x , Q y , Q z are real numbers that represent the imaginary components of the Quaternion Q .

Original: Addition The addition of two Quaternions is deﬁned as
Revised: Addition The addition of two Quaternions is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where Q and P with subscripts denote the real and imaginary components of Quaternions Q and P .
Revised: where Q and P with subscripts denote the real and imaginary components of Quaternions Q and P .

Original: Scalar Multiplication
Revised: Scalar Multiplication

Original: Any scalar α multiplies across all the components:
Revised: Any scalar α multiplies across all the components:

Original: [Equation]
Revised: [Equation]

Original: Hamilton Product The Hamilton product, which represents the multiplication of two Quaternions
Revised: Hamilton Product The Hamilton product, which represents the multiplication of two Quaternions

Original: Q and P , is deﬁned as
Revised: Q and P , is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The multiplication rule in equation 2.2 forges interactions between real and imaginary components of Q and P .
Revised: The multiplication rule in equation 2.2 forges interactions between real and imaginary components of Q and P .

Original: The beneﬁts of Hamilton products have been demonstrated in recent works where the matrix multiplication in fully-connected layers is replaced with the Hamilton product: this reduces 75% parameters with comparable performance (Parcollet et al., 2018b; 2019; Tay et al., 2019).
Revised: The beneﬁts of Hamilton products have been demonstrated in recent works where the matrix multiplication in fully-connected layers is replaced with the Hamilton product: this reduces 75% parameters with comparable performance (Parcollet et al., 2018b; 2019; Tay et al., 2019).

Original:  P ARAMETERIZATION OF H YPERCOMPLEX M ULTIPLICATIONS
Revised:  P ARAMETERIZATION OF H YPERCOMPLEX M ULTIPLICATIONS

Original: The following introduces our proposed parameterized hypercomplex multiplication layer and elaborates on how it parameterizes and generalizes multiplications in hypercomplex space, such as subsuming the multiplication rules of Hamilton products in equation 2.2.
Revised: The following introduces our proposed parameterized hypercomplex multiplication layer and elaborates on how it parameterizes and generalizes multiplications in hypercomplex space, such as subsuming the multiplication rules of Hamilton products in equation 2.2.

Original: [Figure]
Revised: [Equation]

Original: Parameters for H :
Revised: Parameters for H :

Original: Size of H : ⌦ ⌦ +
Revised: Size of H : [Equation]

Original: Parameter size of H :
Revised: Parameter size of H :

Revised: [Figure]

Original: F ULLY -C ONNECTED (FC) L AYERS
Revised: F ULLY -C ONNECTED (FC) L AYERS

Original: Before we delve into our proposed method, recall the fully-connected (FC) layer that transforms an input x ∈ R d into an output y ∈ R k byy = FC ( x
Revised: Before we delve into our proposed method, recall the fully-connected (FC) layer that transforms an input x ∈ R d into an output y ∈ R k by

Original: [Equation]
Revised: [Equation]

Original: = Wx + b , (3.1) where the weight matrix of parameters W ∈ R k × d and the bias vector of parameters b ∈ R k .
Revised: (3.1) where the weight matrix of parameters W ∈ R k × d and the bias vector of parameters b ∈ R k .

Original: The
Revised: The

Original: FC layer in equation 3.1 is fundamental to many modern and traditional neural network architectures.
Revised: FC layer in equation 3.1 is fundamental to many modern and traditional neural network architectures.

Original: Note that the degree of freedom for the weight parameters W in equation 3.1 is kd .
Revised: Note that the degree of freedom for the weight parameters W in equation 3.1 is kd .

Original: Since W dominates parameterization, the parameter size of the FC layer in equation 3.1 is O ( kd ) .
Revised: Since W dominates parameterization, the parameter size of the FC layer in equation 3.1 is O ( kd ) .

Original: 3. P ARAMETERIZED H YPERCOMPLEX M ULTIPLICATION (PHM) L AYERS
Revised: 3. P ARAMETERIZED H YPERCOMPLEX M ULTIPLICATION (PHM) L AYERS

Original: We propose the parameterized hypercomplex multiplication (PHM) layer that transforms an input x into an output y by y =
Revised: We propose the parameterized hypercomplex multiplication (PHM) layer that transforms an input x into an output y by y =

Original: PHM ( x ) =
Revised: PHM ( x ) =

Original: Hx + b , (3.2) where the same notation from equation 3.1 is used but the replaced parameter H ∈ R
Revised: Hx + b , (3.2) where the same notation from equation 3.1 is used but the replaced parameter H ∈ R

Original: k × d is constructed by a sum of Kronecker products.
Revised: k × d is constructed by a sum of Kronecker products.

Original: For context, the Kronecker product is a generalization of the vector outer product to higher dimensions in real space.
Revised: For context, the Kronecker product is a generalization of the vector outer product to higher dimensions in real space.

Original: For any matrix X ∈ R m × n and
Revised: For any matrix X ∈ R m × n and

Original: Y ∈ R p × q , the Kronecker product X ⊗ Y is a block matrix:
Revised: Y ∈ R p × q , the Kronecker product X ⊗ Y is a block matrix:

Original: [Equation]
Revised: [Equation]

Original: where x ij is the element of X at its i th row and j th column.
Revised: where x ij is the element of X at its i th row and j th column.

Original: Note that the symbol ⊗ between two matrices is the Kronecker product while the same symbol between two Quaternions means the
Revised: Note that the symbol ⊗ between two matrices is the Kronecker product while the same symbol between two Quaternions means the

Original: Hamilton product.
Revised: Hamilton product.

Original: Now let us revisit equation 3.2 to explain H .
Revised: Now let us revisit equation 3.2 to explain H .

Original: Suppose that both k and d are divisible by a userdeﬁned hyperparameter n ∈ Z > 0 .
Revised: Suppose that both k and d are divisible by a userdeﬁned hyperparameter n ∈ Z > 0 .

Original: For i = 1 , . . . , n , denote by each parameter matrix A i ∈ R n × n and S i ∈ R kn × dn .
Revised: For i = 1 , . . . , n , denote by each parameter matrix A i ∈ R n × n and S i ∈ R kn × dn .

Original: The parameter H in equation 3.2 is a sum of n Kronecker products:
Revised: The parameter H in equation 3.2 is a sum of n Kronecker products:

Original: [Equation]
Revised: [Equation]

Original: As illustrated in Figure 1, it is the parameter matrices
Revised: As illustrated in Figure 1, it is the parameter matrices

Original: A i and S i ( i = 1 , . . . , n ) that determine the degree of freedom for H , which is kd/n + n 3 .
Revised: A i and S i ( i = 1 , . . . , n ) that determine the degree of freedom for H , which is kd/n + n 3 .

Original: Since H dominates parameterization, the parameter size of the PHM in equation 3.2 is O ( kd/n ) , where kd (cid:39) n 4 is assumed: this condition is mild for real-world problems, such as in our experiments (e.g., d = 512 , k = 2048 , n = 2 , 4 , 8 , 16 ).
Revised: Since H dominates parameterization, the parameter size of the PHM in equation 3.2 is O ( kd/n ) , where kd (cid:39) n 4 is assumed: this condition is mild for real-world problems, such as in our experiments (e.g., d = 512 , k = 2048 , n = 2 , 4 , 8 , 16 ).

Original: Thus, for the same input and output sizes, the parameter size of a PHM layer is approximately 1 /n of that of an FC layer under mild assumptions.
Revised: Thus, for the same input and output sizes, the parameter size of a PHM layer is approximately 1 /n of that of an FC layer under mild assumptions.

Original: The beneﬁt of parameterization reduction of PHM layers is due to reusing elements of both parameter matrices
Revised: The beneﬁt of parameterization reduction of PHM layers is due to reusing elements of both parameter matrices

Original: A i and S i in the Kronecker product.
Revised: A i and S i in the Kronecker product.

Original: As an alternative perspective, we can equivalently reconstruct H in equation 3.3 by reusing parameter matrices in real-valued matrix multiplications, followed by more operations.
Revised: As an alternative perspective, we can equivalently reconstruct H in equation 3.3 by reusing parameter matrices in real-valued matrix multiplications, followed by more operations.

Original: Due to limited space, this more complicated perspective is offered in the supplementary materials.
Revised: Due to limited space, this more complicated perspective is offered in the supplementary materials.

Original: As highlighted in our contributions, our goal is to parameterize hypercomplex multiplications to offer architectural ﬂexibility rather than to compress the matrix H in equation 3.2.

Original: Thus, though simply setting H =
Revised: Though simply setting H =

Original: A 1 ⊗ S 1 can further save parameters, it does not generalize hypercomplex multiplications hence is out of scope.
Revised: A 1 ⊗ S 1 can further save parameters, it does not generalize hypercomplex multiplications hence is out of scope.

Original: In the following, we show how the proposed PHM layer subsumes and generalizes both hypercomplex multiplications and real-valued matrix multiplications.
Revised: In the following, we show how the proposed PHM layer subsumes and generalizes both hypercomplex multiplications and real-valued matrix multiplications.

Original: 3. S UBSUMING H YPERCOMPLEX M ULTIPLICATIONS
Revised: 3. S UBSUMING H YPERCOMPLEX M ULTIPLICATIONS

Original: First, we explore how the PHM layer connects to the hypercomplex multiplication.
Revised: First, we explore how the PHM layer connects to the hypercomplex multiplication.

Original: For the sake of illustration, let us take the Hamilton product of two Quaternions Q and P in equation 2.2 as an example, which can be rewritten as
Revised: For the sake of illustration, let us take the Hamilton product of two Quaternions Q and P in equation 2.2 as an example, which can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: where the 4 output elements are the real values for the Quaternion unit basis [1 , i , j , k ] (cid:62) .
Revised: where the 4 output elements are the real values for the Quaternion unit basis [1 , i , j , k ] (cid:62) .

Original: Note that for models leveraging Hamilton products of Quaternions (Parcollet et al., 2018b; 2019; Tay et al., 2019), the components Q r , Q x , Q y , Q z of equation 3.4 are learnable parameters while the components P r , P x , P y , P z are the layer inputs.
Revised: Note that for models leveraging Hamilton products of Quaternions (Parcollet et al., 2018b; 2019; Tay et al., 2019), the components Q r , Q x , Q y , Q z of equation 3.4 are learnable parameters while the components P r , P x , P y , P z are the layer inputs.

Original: In practice, such a layer usually has more than 4 inputs ( d > 4 ).
Revised: In practice, such a layer usually has more than 4 inputs ( d > 4 ).

Original: To apply the Hamilton product, all the inputs are evenly split into 4 segments ( P r , P x , P y , P z ) of the right input vector of equation 3.4.
Revised: To apply the Hamilton product, all the inputs are evenly split into 4 segments ( P r , P x , P y , P z ) of the right input vector of equation 3.4.

Original: Then each component in the left matrix of equation 3.4 can be a block matrix (i) where all the elements take the same value; (ii) whose shape is aligned with the input length d and the output length k of the layer.
Revised: Then each component in the left matrix of equation 3.4 can be a block matrix (i) where all the elements take the same value; (ii) whose shape is aligned with the input length d and the output length k of the layer.

Original: It is noteworthy that the left 4 × 4 matrix of equation 3.4 can be rewritten as a sum of 4 Kronecker products:   1 0 00 1 00 0 10 0 0  (cid:124) (cid:123)(cid:122) (cid:125) A⊗
Revised: It is noteworthy that the left 4 × 4 matrix of equation 3.4 can be rewritten as a sum of 4 Kronecker products:   1 0 00 1 00 0 10 0 0  (cid:124) (cid:123)(cid:122) (cid:125) A⊗

Original: (cid:104) Q r (cid:105) (cid:124) (cid:123)(cid:122) (cid:125)
Revised: (cid:104) Q r (cid:105) (cid:124) (cid:123)(cid:122) (cid:125)

Original: S+   0 − 1       0 −    
Revised: S+   0 − 1       0 −    

Original:  (cid:124) (cid:123)(cid:122) (cid:125) A⊗
Revised:  (cid:124) (cid:123)(cid:122) (cid:125) A⊗

Original: (cid:104) Q x (cid:105) (cid:124) (cid:123)(cid:122) (cid:125)
Revised: (cid:104) Q x (cid:105) (cid:124) (cid:123)(cid:122) (cid:125)

Original: S+    − 1        0 −  
Revised: S+    − 1        0 −  

Original:  (cid:124) (cid:123)(cid:122) (cid:125) A⊗
Revised:  (cid:124) (cid:123)(cid:122) (cid:125) A⊗

Original: (cid:104) Q y (cid:105) (cid:124) (cid:123)(cid:122) (cid:125)
Revised: (cid:104) Q y (cid:105) (cid:124) (cid:123)(cid:122) (cid:125)

Original: S+   0 −0 0 − 0  1  
Revised: S+   0 −0 0 − 0  1  

Original:  (cid:124) (cid:123)(cid:122) (cid:125) A⊗
Revised:  (cid:124) (cid:123)(cid:122) (cid:125) A⊗

Original: (cid:104) Q z (cid:105) (cid:124) (cid:123)(cid:122) (cid:125)
Revised: (cid:104) Q z (cid:105) (cid:124) (cid:123)(cid:122) (cid:125)

Original: S.
Revised: S.

Original: According to equation 3.5, when n = 4 , the PHM layer can be learned to express the Hamilton product of Quaternions.
Revised: According to equation 3.5, when n = 4 , the PHM layer can be learned to express the Hamilton product of Quaternions.

Original: Speciﬁcally, matrices A 1 , . .
Revised: Specifically, matrices A 1 , . .

Original: A 4 in equation 3.3 parameterize the four matrices composed of − 1 , 0 , 1 in equation 3.5 that reﬂect interactions between real and imaginary components of Quaternions, which are the rule of Hamilton products.
Revised: A 4 in equation 3.3 parameterize the four matrices composed of − 1 , 0 , 1 in equation 3.5 that reﬂect interactions between real and imaginary components of Quaternions, which are the rule of Hamilton products.

Original: The single-element “matrices” S 1 , . . .
Revised: The single-element “matrices” S 1 , . . .

Original: , S 4 in equation 3.3 are equal to the learnable components Q r , Q x , Q y , Q z in equation 3.4.
Revised: , S 4 in equation 3.3 are equal to the learnable components Q r , Q x , Q y , Q z in equation 3.4.

Revised: Figure 2 shows that PHM layers can learn the rule of Hamilton products on artiﬁcial data.

Original: Likewise, hypercomplex multiplications of Octonions or Sedenions can also be learned by the PHM layer when n is set to 8 or 16.
Revised: Likewise, hypercomplex multiplications of Octonions or Sedenions can also be learned by the PHM layer when n is set to 8 or 16.

Original: 3. S UBSUMING R EAL -V ALUED M ATRIX M ULTIPLICATIONS
Revised: 3. S UBSUMING R EAL -V ALUED M ATRIX M ULTIPLICATIONS

Original: Next, we show how the PHM layer subsumes the matrix multiplication in real space.
Revised: Next, we show how the PHM layer subsumes the matrix multiplication in real space.

Original: In other words, the PHM layer is a generalization of the FC layer via the hyperparameter n .
Revised: In other words, the PHM layer is a generalization of the FC layer via the hyperparameter n .

Original: To explain, referring to equation 3.2, when n = 1 , H =
Revised: To explain, referring to equation 3.2, when n = 1 , H =

Original: A 1 ⊗ S 1 = a S 1 , where the scalar a is the single element of the 1 × 1 matrix A 1 and S 1 ∈ R k × d .
Revised: A 1 ⊗ S 1 = a S 1 , where the scalar a is the single element of the 1 × 1 matrix A 1 and S 1 ∈ R k × d .

Original: Since learning a and S 1 separately is equivalent to learning their multiplication jointly, scalar a can be dropped, which is learning the single weight matrix in an FC layer.
Revised: Since learning a and S 1 separately is equivalent to learning their multiplication jointly, scalar a can be dropped, which is learning the single weight matrix in an FC layer.

Original: Therefore, a PHM layer is degenerated to an FC layer when n = 1 .
Revised: Therefore, a PHM layer is degenerated to an FC layer when n = 1 .

Original: 3. G ENERALIZING H YPERCOMPLEX M ULTIPLICATIONS
Revised: 3. G ENERALIZING H YPERCOMPLEX M ULTIPLICATIONS

Original: Though parameter reusing by component-wise partitioning in Quaternion space has demonstrated success (Parcollet et al., 2018b; Zhu et al., 2018; Parcollet et al., 2019; Tay et al., 2019), one key problem is that hypercomplex space only exists at very few predeﬁned dimensionalities, such as 4D (Quaternions), 8D (Octonions), and 16D (Sedonions).
Revised: Though parameter reusing by component-wise partitioning in Quaternion space has demonstrated success (Parcollet et al., 2018b; Zhu et al., 2018; Parcollet et al., 2019; Tay et al., 2019), one key problem is that hypercomplex space only exists at very few predeﬁned dimensionalities, such as 4D (Quaternions), 8D (Octonions), and 16D (Sedonions).

Original: Within the context of hypercomplex space, specialized multiplication rules, such as the Hamilton product, have to be devised and encoded in the network as a ﬁxed inductive bias.
Revised: Within the context of hypercomplex space, specialized multiplication rules, such as the Hamilton product, have to be devised and encoded in the network as a ﬁxed inductive bias.

Original: As described in Section 1, the very few choices over existing hypercomplex space restricts the ﬂexibility of networks that leverage hypercomplex multiplication.
Revised: As described in Section 1, the very few choices over existing hypercomplex space restricts the ﬂexibility of networks that leverage hypercomplex multiplication.

Original: In sharp contrast to relying on predeﬁned mathematical rules over limited dimensionality choices, the PHM layer treats the dimensionality n (number of Kronecker products) as a tunable hyperparameter and learns such specialized multiplication rules from data, as manifested in the parameterized matrices
Revised: In sharp contrast to relying on predeﬁned mathematical rules over limited dimensionality choices, the PHM layer treats the dimensionality n (number of Kronecker products) as a tunable hyperparameter and learns such specialized multiplication rules from data, as manifested in the parameterized matrices

Original: A i ( i = 1 , . . . , n ) in equation 3.3.
Revised: A i ( i = 1 , . . . , n ) in equation 3.3.

Original: On one hand, the PHM layer can express hypercomplex multiplications when A i are set to reﬂect those predeﬁned multiplication rules in hypercomplex space.
Revised: On one hand, the PHM layer can express hypercomplex multiplications when A i are set to reﬂect those predeﬁned multiplication rules in hypercomplex space.

Original: On the other hand, the PHM layer can be seen as a trainable and parameterized form of n D hypercomplex multiplications, where n can be values other than 4, 8, or 16.
Revised: On the other hand, the PHM layer can be seen as a trainable and parameterized form of n D hypercomplex multiplications, where n can be values other than 4, 8, or 16.

Original: Thus, the PHM layer generalizes multiplications in hypercomplex space.
Revised: Thus, the PHM layer generalizes multiplications in hypercomplex space.

Original: Since n can be 1, the PHM layer also offers a neat way to bridging multiplication between both real space and hypercomplex space.
Revised: Since n can be 1, the PHM layer also offers a neat way to bridging multiplication between both real space and hypercomplex space.

Original:  N EURAL M ODELS WITH PHM L AYERS
Revised:  N EURAL M ODELS WITH PHM L AYERS

Original: To demonstrate the applicability of the PHM layers, we develop PHM-LSTM and PHM-Transformer by equipping two popular neural network models, LSTM and Transformer, with PHM layers.
Revised: To demonstrate the applicability of the PHM layers, we develop PHM-LSTM and PHM-Transformer by equipping two popular neural network models, LSTM and Transformer, with PHM layers.

Original: 4. PHM-LSTM
Revised: 4. PHM-LSTM

Original: Recurrent neural networks such as LSTM (Hochreiter & Schmidhuber, 1997) are gated recurrent networks where the gating functions are parameterized by linear transformations.
Revised: Recurrent neural networks such as LSTM (Hochreiter & Schmidhuber, 1997) are gated recurrent networks where the gating functions are parameterized by linear transformations.

Original: We introduce
Revised: We introduce

Original: PHM-LSTM, which replaces such linear transformations in LSTM with PHM layers:
Revised: PHM-LSTM, which replaces such linear transformations in LSTM with PHM layers:

Original: [Equation]
Revised: [Equation]

Original: where σ s is the sigmoid activation function, σ t is the tanh activation function, φ : R 1 × d
Revised: where σ s is the sigmoid activation function, σ t is the tanh activation function, φ : R 1 × d

Original: → R 4 × dis a four-way split on the last dimension, and c t , h t are the cell state and the hidden state of the
Revised: → R 4 × dis a four-way split on the last dimension, and c t , h t are the cell state and the hidden state of the

Original: PHM-LSTM unit at any time step t .
Revised: PHM-LSTM unit at any time step t .

Original: 4. PHM-T RANSFORMER
Revised: 4. PHM-T RANSFORMER

Original: Transformer is a stacked neural network architecture that aggressively exploits linear transformations (Vaswani et al., 2017).
Revised: Transformer is a stacked neural network architecture that aggressively exploits linear transformations (Vaswani et al., 2017).

Original: Each self-attention layer comprises of Q (query), K (key), V (value) linear transformations, along with multiple heads.
Revised: Each self-attention layer comprises of Q (query), K (key), V (value) linear transformations, along with multiple heads.

Original: Each Transformer block also has a position-wise feed-forward network composed of two FC layers.
Revised: Each Transformer block also has a position-wise feed-forward network composed of two FC layers.

Original: Since a large majority of the Transformer parameters stem from linear transformations or FC layers, we introduce PHM-Transformer to replace all the linear transformations or FC layers with PHM layers.
Revised: Since a large majority of the Transformer parameters stem from linear transformations or FC layers, we introduce PHM-Transformer to replace all the linear transformations or FC layers with PHM layers.

Original: The single-head self-attention module is rewritten as:
Revised: The single-head self-attention module is rewritten as:

Original: [Equation]
Revised: [Equation]

Original: where d k is the key dimension, Φ : R 1 × d → R 3 ×
Revised: where d k is the key dimension, Φ : R 1 × d → R 3 ×

Original: d 3 is a three-way split on the last dimension, X is the input sequence, and A is the self-attentive representation.
Revised: d 3 is a three-way split on the last dimension, X is the input sequence, and A is the self-attentive representation.

Original: For multi-head attention, using PHM layers also enables weight sharing not only among the linear transformations of Q , K , V but also among the linear transformation of multiple heads:
Revised: For multi-head attention, using PHM layers also enables weight sharing not only among the linear transformations of Q , K , V but also among the linear transformation of multiple heads:

Original: [Equation]
Revised: [Equation]

Original: where N h is the number of heads and ( ; ) is the column-wise concatenation.
Revised: where N h is the number of heads and ( ; ) is the column-wise concatenation.

Original: Finally, the position-wise feed-forward network is now deﬁned as
Revised: Finally, the position-wise feed-forward network is now deﬁned as

Original: [Equation]
Revised: [Equation]

Original: , which transforms X with two PHM layers.
Revised: which transforms X with two PHM layers.

Original: PHM-LSTM reduces the parameters of the

Original: LSTM model and improves or partially matches

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We reiterate that our contribution is parameterization of hypercomplex multiplications rather than a model compression technique without requirements of generalizing hypercomplex multiplications.
Revised: We reiterate that our contribution is parameterization of hypercomplex multiplications rather than a model compression technique without requirements of generalizing hypercomplex multiplications.

Original: In the ﬁeld of representation learning using hypercomplex multiplications, Quaternion convolutional neural networks (Zhu et al., 2018), Quaternion recurrent neural networks (Parcollet et al., 2018a), and Quaternion Transformer (Tay et al., 2019) have all compared themselves with only real-valued counterparts.
Revised: In the ﬁeld of representation learning using hypercomplex multiplications, Quaternion convolutional neural networks (Zhu et al., 2018), Quaternion recurrent neural networks (Parcollet et al., 2018a), and Quaternion Transformer (Tay et al., 2019) have all compared themselves with only real-valued counterparts.

Original: Therefore, to support our contribution and be consistent with the rest of the literature, we evaluate PHM-LSTM and PHM-Transformer that are equipped with PHM layers
Revised: Therefore, to support our contribution and be consistent with the rest of the literature, we evaluate PHM-LSTM and PHM-Transformer that are equipped with PHM layers

Original: [Table]
Revised: [Table]

Original: and compare them with Quaternion LSTM, Quaternion Transformer, or real-valued LSTM or Transformer.
Revised: and compare them with Quaternion LSTM, Quaternion Transformer, or real-valued LSTM or Transformer.

Original: Both
Revised: Both

Original: Quaternion LSTM and Quaternion Transformer replace linear transformations with Hamilton products of Quaternions.
Revised: Quaternion LSTM and Quaternion Transformer replace linear transformations with Hamilton products of Quaternions.

Original: To demonstrate the architectural and effectiveness, we evaluate different settings of PHM-LSTM and PHM-Transformer to show that allowing for ﬂexible choices of the hyperparameter n in the
Revised: To demonstrate the architectural and effectiveness, we evaluate different settings of PHM-LSTM and PHM-Transformer to show that allowing for ﬂexible choices of the hyperparameter n in the

Original: PHM layer may lead to more effective performance.
Revised: PHM layer may lead to more effective performance.

Original: Details of the setup for the experiments are provided in the supplementary materials.
Revised: Details of the setup for the experiments are provided in the supplementary materials.

Original: 5. N ATURAL L ANGUAGE I NFERENCE
Revised: 5. N ATURAL L ANGUAGE I NFERENCE

Original: The task of natural language inference is to determine the logical relationship between two text sequences (MacCartney, 2009).
Revised: The task of natural language inference is to determine the logical relationship between two text sequences (MacCartney, 2009).

Original: It is a fundamental task pertaining to language understanding.
Revised: It is a fundamental task pertaining to language understanding.

Original: To this end, they serve as a suitable benchmark for evaluating recurrent models.
Revised: To this end, they serve as a suitable benchmark for evaluating recurrent models.

Original: We run experiments on ﬁve datasets: (i) MultiNLI
Revised: We run experiments on ﬁve datasets: (i) MultiNLI

Original: (Williams et al., 2017), (ii) QNLI (Quora) (Wang et al., 2017), (iii) SNLI (Bowman et al., 2015), (iv) Dialogue NLI (Welleck et al., 2018), and (v)
Revised: (Williams et al., 2017), (ii) QNLI (Quora) (Wang et al., 2017), (iii) SNLI (Bowman et al., 2015), (iv) Dialogue NLI (Welleck et al., 2018), and (v)

Original: SciTail (Science Entailment) (Khot et al., 2018).
Revised: SciTail (Science Entailment) (Khot et al., 2018).

Original: Table 1 reports the results on all these datasets.
Revised: Table 1 reports the results on all these datasets.

Original: All in all, such results show that the PHM layer can not only reduce the parameters but also improve performance with ﬂexible choices of n (four out of ﬁve datasets show reasonable improvement or partially match).
Revised: All in all, such results show that the PHM layer can not only reduce the parameters but also improve performance with ﬂexible choices of n (four out of ﬁve datasets show reasonable improvement or partially match).

Original: The only exception is on the QNLI dataset, where the performance drop is marginal ( < 1% ).
Revised: The only exception is on the QNLI dataset, where the performance drop is marginal ( < 1% ).

Original: This is still decent considering the parameter saving: the parameterization cost of PHMLSTM is in the order of O (1 /n ) of that of the standard LSTM, where settings of n = 5 and n =do not take values of power of 2.
Revised: This is still decent considering the parameter saving: the parameterization cost of PHMLSTM is in the order of O (1 /n ) of that of the standard LSTM, where settings of n = 5 and n =do not take values of power of 2.

Original: As detailed in the supplementary materials, since we use the 300D GloVe (Pennington et al., 2014) embeddings to represent input tokens, we choose multiples of 5 instead of 4 for ease of divisibility.
Revised: As detailed in the supplementary materials, since we use the 300D GloVe (Pennington et al., 2014) embeddings to represent input tokens, we choose multiples of 5 instead of 4 for ease of divisibility.

Original: It is also noteworthy that on the SNLI, Dialogue NLI, and
Revised: It is also noteworthy that on the SNLI, Dialogue NLI, and

Original: SciTail datasets, all the PHM-LSTM variants outperform the standard LSTM model.
Revised: SciTail datasets, all the PHM-LSTM variants outperform the standard LSTM model.

Original: We think that the element reusing properties of the Kronecker product operation, in addition to learning to share such reused components amongst recurrent gating functions, may contribute to both effective and efﬁcient representations.
Revised: We think that the element reusing properties of the Kronecker product operation, in addition to learning to share such reused components amongst recurrent gating functions, may contribute to both effective and efﬁcient representations.

Original: 5. M ACHINE T RANSLATION
Revised: 5. M ACHINE T RANSLATION

Original: Machine translation is concerned with translating between source-target language pairs.
Revised: Machine translation is concerned with translating between source-target language pairs.

Original: To this end, sequence transduction models are central to this problem domain.
Revised: To this end, sequence transduction models are central to this problem domain.

Original: In this experiment, the key goal is to compare PHM-Transformer against the standard and Quaternion Transformer models.
Revised: In this experiment, the key goal is to compare PHM-Transformer against the standard and Quaternion Transformer models.

Original: [Table]
Revised: [Table]

Original: We run experiments on seven datasets: (i) IWSLT’15 English-Vietnamese
Revised: We run experiments on seven datasets: (i) IWSLT’15 English-Vietnamese

Original: (En-Vi), (ii) IWSLT’
Revised: (En-Vi), (ii) IWSLT’

Original: English-Indonesian (En-Id), (iii) IWSLT’14 German-English (De-En), (iv) IWSLT’14 RomanianEnglish (Ro-En)
Revised: English-Indonesian (En-Id), (iii) IWSLT’14 German-English (De-En), (iv) IWSLT’14 RomanianEnglish (Ro-En)

Original: , (v) WMT’18 English-Estonian (En-Et), (vi) Setimes English-Macedonian (EnMk), and (vii) WMT’18 English-Romanian (En-Ro).
Revised: , (v) WMT’18 English-Estonian (En-Et), (vi) Setimes English-Macedonian (EnMk), and (vii) WMT’18 English-Romanian (En-Ro).

Original: Table 2 reports our results of the machine translation tasks.
Revised: Table 2 reports our results of the machine translation tasks.

Original: Overall, these empirical results with different settings demonstrate architectural ﬂexibility and effectiveness of the hypercomplex multiplication parameterization.
Revised: Overall, these empirical results with different settings demonstrate architectural ﬂexibility and effectiveness of the hypercomplex multiplication parameterization.

Original: First and foremost, across six out of seven benchmarks, PHM-Transformer at n = 4 makes reasonable gains over Quaternion Transformer, signifying that parameterization of hypercomplex multiplications by learning from data can be more effective than predeﬁning Hamilton product rules mathematically.
Revised: First and foremost, across six out of seven benchmarks, PHM-Transformer at n = 4 makes reasonable gains over Quaternion Transformer, signifying that parameterization of hypercomplex multiplications by learning from data can be more effective than predeﬁning Hamilton product rules mathematically.

Original: Second, though increasing n leads to more parameter savings, we observe that increasing n all the way to 16 does not cause signiﬁcant degradation in performance on datasets such as En-Vi.
Revised: Second, though increasing n leads to more parameter savings, we observe that increasing n all the way to 16 does not cause signiﬁcant degradation in performance on datasets such as En-Vi.

Original: Third, for most datasets, even with signiﬁcant parameter savings, we ﬁnd that the decrease in the BLEU score is mostly manageable ( ≈ 1–3 BLEU points).
Revised: Third, for most datasets, even with signiﬁcant parameter savings, we ﬁnd that the decrease in the BLEU score is mostly manageable ( ≈ 1–3 BLEU points).

Original: However, we also note a rare occurrence where n = 16 results in a signiﬁcant decrease in the BLEU score, such as on the En-Id dataset.
Revised: However, we also note a rare occurrence where n = 16 results in a signiﬁcant decrease in the BLEU score, such as on the En-Id dataset.

Original: Fourth, on several datasets, the PHM-Transformer model improves the performance of the standard Transformer model.
Revised: Fourth, on several datasets, the PHM-Transformer model improves the performance of the standard Transformer model.

Original: For example, on datasets such as En-Vi and En-Et, the
Revised: For example, on datasets such as En-Vi and En-Et, the

Original: PHM-Transformer model enjoys a performance boost of about 0 .
Revised: PHM-Transformer model enjoys a performance boost of about 0 .

Original: 8 BLEU point with n = 2 .
Revised: 8 BLEU point with n = 2 .

Original: Finally, by re-scaling with a factor of 2 (doubling the hidden size), we are able to improve the performance on three datasets: En-Vi, En-Id, and En-Mk.
Revised: Finally, by re-scaling with a factor of 2 (doubling the hidden size), we are able to improve the performance on three datasets: En-Vi, En-Id, and En-Mk.

Original: [Table]
Revised: [Table]

Original: Table 3 reports the training and inference time for Transformer variants.
Revised: Table 3 reports the training and inference time for Transformer variants.

Original: We observe that PHMTransformer with n = 8 has the fastest inference speed amongst all the variants, primarily due to a signiﬁcant reduction of parameters.
Revised: We observe that PHMTransformer with n = 8 has the fastest inference speed amongst all the variants, primarily due to a signiﬁcant reduction of parameters.

Original: All in all, the training speed is also approximately comparable.
Revised: All in all, the training speed is also approximately comparable.

Original: This ascertains that the PHM layer does not increase much computational cost in practice.
Revised: This ascertains that the PHM layer does not increase much computational cost in practice.

Original: 5. T EXT S TYLE T RANSFER
Revised: 5. T EXT S TYLE T RANSFER

Original: We continue to experiment with sequence transduction for text style transfer.
Revised: We continue to experiment with sequence transduction for text style transfer.

Original: The goal of this task is to convert text of a certain style to another style.
Revised: The goal of this task is to convert text of a certain style to another style.

Original: We use the Modern → Shakespeare corpus 1 in the experiments.
Revised: We use the Modern → Shakespeare corpus 1 in the experiments.

Original: Table 4 reports the results on this text style transfer task.
Revised: Table 4 reports the results on this text style transfer task.

Original: We observe that the best performance is achieved with PHM-Transformer ( n = 4 ).
Revised: We observe that the best performance is achieved with PHM-Transformer ( n = 4 ).

Original: Notably, all except the n = 16 variant increases or matches the performance of the standard Transformer model.
Revised: Notably, all except the n = 16 variant increases or matches the performance of the standard Transformer model.

Original: This ascertains architectural ﬂexibility and effectiveness of the proposed PHM layer.
Revised: This ascertains architectural ﬂexibility and effectiveness of the proposed PHM layer.

Original: This not only enables parameter savings but also improves the performance of Transformer.
Revised: This not only enables parameter savings but also improves the performance of Transformer.

Original: PHM-Transformer may reduce the parameters of the standard Transformer model and improve performance.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 5. S UBJECT V ERB A GREEMENT
Revised: 5. S UBJECT V ERB A GREEMENT

Original: We conduct additional experiments on the subject-verb agreement task (Linzen et al., 2016).
Revised: We conduct additional experiments on the subject-verb agreement task (Linzen et al., 2016).

Original: The task predicts if the sentence, e.g., ‘The keys to the cabinet .’
Revised: The task predicts if the sentence, e.g., ‘The keys to the cabinet .’

Original: is followed by a plural or a singular.
Revised: is followed by a plural or a singular.

Original: The used dataset can be found online (Linzen et al., 2016).
Revised: The used dataset can be found online (Linzen et al., 2016).

Original: Table 5 reports the results on the subject-verb agreement task.
Revised: Table 5 reports the results on the subject-verb agreement task.

Original: Results are promising, demonstrating that all variants with PHM layers outperform the standard and Quaternion Transformer models.
Revised: Results are promising, demonstrating that all variants with PHM layers outperform the standard and Quaternion Transformer models.

Original: The best performance peaks at n = 8 , despite a parameter saving to up to 1 / 8 .
Revised: The best performance peaks at n = 8 , despite a parameter saving to up to 1 / 8 .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: While neural networks have been a well-established line of research, progress on hypercomplex representations for deep learning is still in its infancy and most works on this topic are new (Gaudet & Maida, 2017; Parcollet et al., 2018a;b; Zhu et al., 2018; Tay et al., 2019).
Revised: While neural networks have been a well-established line of research, progress on hypercomplex representations for deep learning is still in its infancy and most works on this topic are new (Gaudet & Maida, 2017; Parcollet et al., 2018a;b; Zhu et al., 2018; Tay et al., 2019).

Original: The hypercomplex
Revised: The hypercomplex

Original: Hamilton product provides a greater extent of expressiveness, similar to the complex multiplication, albeit with a 4-fold increase in interactions between real and imaginary components.
Revised: Hamilton product provides a greater extent of expressiveness, similar to the complex multiplication, albeit with a 4-fold increase in interactions between real and imaginary components.

Original: In the case of
Revised: In the case of

Original: Quaternion representations, due to parameter savings in the Hamilton product, models also enjoy a 75% reduction in the parameter size (Parcollet et al., 2018a; Tay et al., 2019).
Revised: Quaternion representations, due to parameter savings in the Hamilton product, models also enjoy a 75% reduction in the parameter size (Parcollet et al., 2018a; Tay et al., 2019).

Original: A striking caveat is that all Quaternions are fundamentally limited to 4D hypercomplex space, which restricts architectural ﬂexibility.
Revised: A striking caveat is that all Quaternions are fundamentally limited to 4D hypercomplex space, which restricts architectural ﬂexibility.

Original: The other options would be to scale to Octonion (8D) or Sedonion (16D) space, given the predeﬁned multiplication rules in such space.
Revised: The other options would be to scale to Octonion (8D) or Sedonion (16D) space, given the predeﬁned multiplication rules in such space.

Original: To the best of our knowledge, there is no work that attempts to generalize arbitrary n D hypercomplex multiplications to allow for architectural ﬂexibility, where n can be speciﬁed or tuned by users.
Revised: To the best of our knowledge, there is no work that attempts to generalize arbitrary n D hypercomplex multiplications to allow for architectural ﬂexibility, where n can be speciﬁed or tuned by users.

Original: Our work can also be interpreted as a form of soft parameter sharing, albeit learned from data.
Revised: Our work can also be interpreted as a form of soft parameter sharing, albeit learned from data.

Original: Quaternion networks (Zhu et al., 2018; Parcollet et al., 2018b; 2019) are known to possess weight sharing properties via the Hamilton product operation and have demonstrated reasonable success despite having fewer parameters.
Revised: Quaternion networks (Zhu et al., 2018; Parcollet et al., 2018b; 2019) are known to possess weight sharing properties via the Hamilton product operation and have demonstrated reasonable success despite having fewer parameters.

Original: To the best of our knowledge, there has been no work that attempts to parameterize the hypercomplex Hamilton product for neural networks, i.e., enabling end-to-end learning of real and imaginary component interactions from data.
Revised: To the best of our knowledge, there has been no work that attempts to parameterize the hypercomplex Hamilton product for neural networks, i.e., enabling end-to-end learning of real and imaginary component interactions from data.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed parameterized hypercomplex multiplication (PHM) layers that learn and generalize hypercomplex multiplications.
Revised: We proposed parameterized hypercomplex multiplication (PHM) layers that learn and generalize hypercomplex multiplications.

Original: PHM layers arehighly modular and applicable to dominant models such as LSTM and Transformer.
Revised: PHM layers are applicable to dominant models such as LSTM and Transformer.

Original: We evaluated these models equipped by PHM layers on comprehensive tasks to demonstrate architectural ﬂexibility and effectiveness of the hypercomplex multiplication parameterization. 
Revised: We evaluated these models equipped by PHM layers on comprehensive tasks to show architectural ﬂexibility and effectiveness of the hypercomplex multiplication parameterization. Acknowledgements.

Revised: We thank the anonymous reviewers for insightful comments on this paper.