Original: T
Revised: T

Original: ENSORIZED
Revised: ENSORIZED

Original: E MBEDDING
Revised: E MBEDDING

Original: L AYERS
Revised: L AYERS

Original: FOR
Revised: FOR

Original: E FFICIENT
Revised: E FFICIENT

Original: M ODEL C OMPRESSION
Revised: M ODEL C OMPRESSION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing.
Revised: The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing.

Original: However, when the vocabulary is large, the corresponding weight matrices can be enormous, which precludes their deployment in a limited resource setting.
Revised: However, when the vocabulary is large, the corresponding weight matrices can be enormous, which precludes their deployment in a limited resource setting.

Original: We introduce a novel way of parametrizing embedding layers based on the Tensor Train (TT) decomposition, which allows compressing the model signiﬁcantly at the cost of a negligible drop or even a slight gain in performance.
Revised: We introduce a novel way of parametrizing embedding layers based on the Tensor Train (TT) decomposition, which allows compressing the model signiﬁcantly at the cost of a negligible drop or even a slight gain in performance.

Original: We evaluate our method on a wide range of benchmarks in natural language processing and analyze the trade-off between performance and compression ratios for a wide range of architectures, from MLPs to LSTMs and Transformers.
Revised: We evaluate our method on a wide range of benchmarks in natural language processing and analyze the trade-off between performance and compression ratios for a wide range of architectures, from MLPs to LSTMs and Transformers.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks (DNNs) typically used in natural language processing (NLP) employ large embeddings layers, which map the input words into continuous representations and usually have the form of lookup tables.
Revised: Deep neural networks (DNNs) typically used in natural language processing (NLP) employ large embeddings layers, which map the input words into continuous representations and usually have the form of lookup tables.

Original: Despite such simplicity and, arguably because of it, the resulting models are cumbersome, which may cause problems in training and deploying them in a limited resource setting.
Revised: Despite such simplicity and, arguably because of it, the resulting models are cumbersome, which may cause problems in training and deploying them in a limited resource setting.

Original: Thus, the compression of large neural networks and the development of novel lightweight architectures have become essential problems in NLP research.
Revised: Thus, the compression of large neural networks and the development of novel lightweight architectures have become essential problems in NLP research.

Original: One way to reduce the number of parameters in the trained model is to imply a speciﬁc structure on its weight matrices (e.g., assume that they are low-rank or can be well approximated by low-rank tensor networks).
Revised: One way to reduce the number of parameters in the trained model is to imply a speciﬁc structure on its weight matrices (e.g., assume that they are low-rank or can be well approximated by low-rank tensor networks).

Original: Such approaches are successful at compressing the pre-trained models, but they do not facilitate the training itself.
Revised: Such approaches are successful at compressing the pre-trained models, but they do not facilitate the training itself.

Original: Furthermore, they usually require an additional ﬁne-tuning stage to recover the performance of the original model.
Revised: Furthermore, they usually require an additional ﬁne-tuning stage to recover the performance of the original model.

Original: In this paper, we introduce a new, parameter efﬁcient embedding layer, termed TT–embedding, which can be plugged in into any model and trained end-to-end.
Revised: In this paper, we introduce a new, parameter efﬁcient embedding layer, termed TT–embedding, which can be plugged in into any model and trained end-to-end.

Original: The beneﬁts of our compressed TT–layer are twofold.
Revised: The beneﬁts of our compressed TT–layer are twofold.

Original: Firstly, instead of storing huge embedding matrix, we store a sequence of much smaller
Revised: Firstly, instead of storing huge embedding matrix, we store a sequence of much smaller

Original: -dimensional and 3-dimensional tensors, necessary for reconstructing the required embeddings, which allows compressing the model signiﬁcantly at the cost of a negligible performance drop.
Revised: -dimensional and 3-dimensional tensors, necessary for reconstructing the required embeddings, which allows compressing the model signiﬁcantly at the cost of a negligible performance drop.

Original: Secondly, the overall number of parameters can be relatively small (and constant) during the whole training stage, which allows to use larger batches or train efﬁciently in a case of limited resources.
Revised: Secondly, the overall number of parameters can be relatively small (and constant) during the whole training stage, which allows to use larger batches or train efﬁciently in a case of limited resources.

Original: To validate the efﬁciency of the proposed approach, we have tested it on several popular NLP tasks.
Revised: To validate the efﬁciency of the proposed approach, we have tested it on several popular NLP tasks.

Original: In our experiments, we have observed that the standard embeddings can be replaced by TT–embeddings with the compression ratio of 1 − 3 orders without any signiﬁcant drop (and sometimes even with a slight gain) of the metric of interest.
Revised: In our experiments, we have observed that the standard embeddings can be replaced by TT–embeddings with the compression ratio of 1 − 3 orders without any signiﬁcant drop (and sometimes even with a slight gain) of the metric of interest.

Original: Speciﬁcally, we report the following compression ratios of the embedding layers: 441 on the IMDB dataset
Revised: Speciﬁcally, we report the following compression ratios of the embedding layers: 441 on the IMDB dataset

Original: with
Revised: with

Original: % absolute increase in classiﬁcation accuracy;
Revised: % absolute increase in classiﬁcation accuracy;

Original: 15 on the WMT 

Revised: 15 on the WMT

Original: En–De dataset with 0 .
Revised: En–De dataset with 0 .

Original: 3 drop in the BLEU score
Revised: 3 drop in the BLEU score

Original: Additionally, we have also evaluated our algorithm on a task of binary classiﬁcation based on a large number of categorical features.
Revised: Additionally, we have also evaluated our algorithm on a task of binary classiﬁcation based on a large number of categorical features.

Original: More concretely, we applied TT–embedding to the click through rate (CTR) prediction problem, a crucial task in the ﬁeld of digital advertising.
Revised: More concretely, we applied TT–embedding to the click through rate (CTR) prediction problem, a crucial task in the ﬁeld of digital advertising.

Original: Neural networks, typically used for solving this problem, while being rather elementary, include a large number of embedding layers of signiﬁcant size.
Revised: Neural networks, typically used for solving this problem, while being rather elementary, include a large number of embedding layers of signiﬁcant size.

Original: As a result, a majority of model parameters that represent these layers, may occupy hundreds of gigabytes of space.
Revised: As a result, a majority of model parameters that represent these layers, may occupy hundreds of gigabytes of space.

Original: We show that TT–embedding not only considerably reduces the number of parameters in such models, but also sometimes improves their accuracy.
Revised: We show that TT–embedding not only considerably reduces the number of parameters in such models, but also sometimes improves their accuracy.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: In recent years, a large body of research was devoted to compressing and speeding up various components of neural networks used in NLP tasks.
Revised: In recent years, a large body of research was devoted to compressing and speeding up various components of neural networks used in NLP tasks.

Original: Joulin et al.
Revised: Joulin et al.

Original: (2016) adapted the framework of product quantization to reduce the number of parameters in linear models used for text classiﬁcation.
Revised: (2016) adapted the framework of product quantization to reduce the number of parameters in linear models used for text classiﬁcation.

Original: See et al.
Revised: See et al.

Original: (2016) proposed to compress LSTM-based neural machine translation models with pruning algorithms.
Revised: (2016) proposed to compress LSTM-based neural machine translation models with pruning algorithms.

Original: Lobacheva et al.
Revised: Lobacheva et al.

Original: (2017) showed that the recurrent models could be signiﬁcantly sparsiﬁed with the help of variational dropout (Kingma et al., 2015).
Revised: (2017) showed that the recurrent models could be signiﬁcantly sparsiﬁed with the help of variational dropout (Kingma et al., 2015).

Original: Chen et al.
Revised: Chen et al.

Original: (2018b) proposed more compact
Revised: (2018b) proposed more compact

Original: K-way D-dimensional discrete encoding scheme to replace the “one-hot” encoding of categorical features, such as words in NLP taks.
Revised: K-way D-dimensional discrete encoding scheme to replace the “one-hot” encoding of categorical features, such as words in NLP taks.

Original: Very recently, Chen et al.
Revised: Very recently, Chen et al.

Original: (2018a)
Revised: (2018a)

Original: and Variani et al.
Revised: and Variani et al.

Original: (2018) introduced GroupReduce and WEST, two very efﬁcient compression methods for the embedding and softmax layers, based on structured low-rank matrix approximation.
Revised: (2018) introduced GroupReduce and WEST, two very efﬁcient compression methods for the embedding and softmax layers, based on structured low-rank matrix approximation.

Original: Concurrently, Lam (2018) proposed the quantization algorithm for compressing word vectors and showed the superiority of the obtained embeddings on word similarity, word analogy, and question answering tasks.
Revised: Concurrently, Lam (2018) proposed the quantization algorithm for compressing word vectors and showed the superiority of the obtained embeddings on word similarity, word analogy, and question answering tasks.

Original: Tensor methods have also been already successfully applied to neural networks compression.
Revised: Tensor methods have also been already successfully applied to neural networks compression.

Original: Novikov et al.
Revised: Novikov et al.

Original: (2015) coined the idea of reshaping weights of fully-connected layers into high-dimensional tensors and representing them in Tensor Train (TT) (Oseledets, 2011) format.
Revised: (2015) coined the idea of reshaping weights of fully-connected layers into high-dimensional tensors and representing them in Tensor Train (TT) (Oseledets, 2011) format.

Original: This approach was later extended to convolutional (Garipov et al., 2016) and recurrent (Yang et al., 2017;
Revised: This approach was later extended to convolutional (Garipov et al., 2016) and recurrent (Yang et al., 2017;

Original: Tjandra et al., 2017; Yu et al., 2017) neural networks.
Revised: Tjandra et al., 2017; Yu et al., 2017) neural networks.

Original: Furthermore, Lebedev et al. (2015) showed that convolutional layers could be also compressed with canonical (CP) tensor decomposition (Carroll & Chang, 1970;
Revised: Furthermore, Lebedev et al. (2015) showed that convolutional layers could be also compressed with canonical (CP) tensor decomposition (Carroll & Chang, 1970;

Original: Harshman, 1970).
Revised: Harshman, 1970).

Original: Finally, Wang et al.
Revised: Finally, Wang et al.

Original: (2018) compressed both fully-connected and convolutional layers with Tensor Ring decomposition (Zhao et al., 2016).
Revised: (2018) compressed both fully-connected and convolutional layers with Tensor Ring decomposition (Zhao et al., 2016).

Original: While all these methods allowed to reduce the number of parameters in the networks dramatically, they mostly capitalized on heavy fullyconnected and convolutional layers (present in AlexNet (Krizhevsky et al., 2012) or VGG (Simonyan & Zisserman, 2014)), which became outdated in the following years.
Revised: While all these methods allowed to reduce the number of parameters in the networks dramatically, they mostly capitalized on heavy fullyconnected and convolutional layers (present in AlexNet (Krizhevsky et al., 2012) or VGG (Simonyan & Zisserman, 2014)), which became outdated in the following years.

Original: Recently, Ma et al.
Revised: Recently, Ma et al.

Original: (2019) succesfully applied Block-Term Tensor Decomposition to the compression of self-attention modules in the Transformer (Vaswani et al., 2017) architecture.
Revised: (2019) succesfully applied Block-Term Tensor Decomposition to the compression of self-attention modules in the Transformer (Vaswani et al., 2017) architecture.

Original: In this work, we show the beneﬁts of applying tensor machinery to the compression of embedding layers, which are still widely used in NLP.
Revised: In this work, we show the beneﬁts of applying tensor machinery to the compression of embedding layers, which are still widely used in NLP.

Original:  T ENSOR T RAIN EMBEDDING
Revised:  T ENSOR T RAIN EMBEDDING

Original: In this section, we brieﬂy introduce the necessary notation and present the algorithm for training the
Revised: In this section, we brieﬂy introduce the necessary notation and present the algorithm for training the

Original: TT–embedding layer.
Revised: TT–embedding layer.

Original: Hereinafter, by N -way tensor X
Revised: Hereinafter, by N -way tensor X

Original: we mean a multidimensional array:
Revised: we mean a multidimensional array:

Original: [Equation]
Revised: [Equation]

Original: with entries X ( i 1 , . . .
Revised: with entries X ( i 1 , . . .

Original: , i N ) , such that { 0 ≤ i k < I k } Nk =1 .
Revised: , i N ) , such that { 0 ≤ i k < I k } Nk =1 .

Original: 3. M OTIVATION
Revised: 3. M OTIVATION

Original: Since most of the parameters in the NLP models occupy the embedding layers, we can greatly reduce size of the entire model by compressing these layers.
Revised: Since most of the parameters in the NLP models occupy the embedding layers, we can greatly reduce size of the entire model by compressing these layers.

Original: Our goal is to replace the standard embedding matrix with a more compact, yet powerful and trainable, representation which would allow us to efﬁciently map words into vectors.
Revised: Our goal is to replace the standard embedding matrix with a more compact, yet powerful and trainable, representation which would allow us to efﬁciently map words into vectors.

Original: The simplest approach to compactly represent a matrix of a large size is to use the low–rank matrix factorization, which treats matrix E ∈ R
Revised: The simplest approach to compactly represent a matrix of a large size is to use the low–rank matrix factorization, which treats matrix E ∈ R

Original: I × J as a product of two matrices E = UV (cid:62) .
Revised: I × J as a product of two matrices E = UV (cid:62) .

Original: Here
Revised: Here

Original: U ∈ R I × R and V ∈ R J × R are much “thinner” matrices, and R is the rank hyperparameter.
Revised: U ∈ R I × R and V ∈ R J × R are much “thinner” matrices, and R is the rank hyperparameter.

Original: Note that rather than training the model with the standard embedding layer, and then trying to compress the obtained embedding, we can initially seek the embedding matrix in the described low–rank format.
Revised: Note that rather than training the model with the standard embedding layer, and then trying to compress the obtained embedding, we can initially seek the embedding matrix in the described low–rank format.

Original: Then, for evaluation and training, the individual word embedding E [ i, :] can be computed as a product
Revised: Then, for evaluation and training, the individual word embedding E [ i, :] can be computed as a product

Original: U [ i, :] V (cid:62) which does not require materializing the full matrix E .
Revised: U [ i, :] V (cid:62) which does not require materializing the full matrix E .

Original: This approach reduces the number of degrees of freedom in the embedding layer from IJ to ( I + J ) R .
Revised: This approach reduces the number of degrees of freedom in the embedding layer from IJ to ( I + J ) R .

Original: However, typically, in the NLP tasks the embedding dimension J is much smaller than the vocabulary size I , and obtaining signiﬁcant compression ratio using low-rank matrix factorization is problematic.
Revised: However, typically, in the NLP tasks the embedding dimension J is much smaller than the vocabulary size I , and obtaining signiﬁcant compression ratio using low-rank matrix factorization is problematic.

Original: In order to preserve the model performance, the rank R cannot be taken very small, and the compression ratio is bounded by
Revised: In order to preserve the model performance, the rank R cannot be taken very small, and the compression ratio is bounded by

Original: IJ ( I + J ) R ≤ JR , which is close to 1 for usually full-rank embedding matrix (see Figure 1 in Chen et al.
Revised: IJ ( I + J ) R ≤ JR , which is close to 1 for usually full-rank embedding matrix (see Figure 1 in Chen et al.

Original: (2018b)).
Revised: (2018b)).

Original: To overcome this bound and achieve signiﬁcant compression ratio even for matrices of disproportional dimensionalities, we reshape them into multidimensional tensors and apply the Tensor Train decomposition, which allows for more compact representation, where the number of parameters falls down to logarithmic with respect to I .
Revised: To overcome this bound and achieve signiﬁcant compression ratio even for matrices of disproportional dimensionalities, we reshape them into multidimensional tensors and apply the Tensor Train decomposition, which allows for more compact representation, where the number of parameters falls down to logarithmic with respect to I .

Original: 3. T ENSOR T RAIN DECOMPOSITION
Revised: 3. T ENSOR T RAIN DECOMPOSITION

Original: A tensor X is said to be represented in the Tensor Train (TT) format (Oseledets, 2011) if each element of X can be computed as:
Revised: A tensor X is said to be represented in the Tensor Train (TT) format (Oseledets, 2011) if each element of X can be computed as:

Original: [Equation]
Revised: [Equation]

Original: where the tensors G ( k ) ∈ R R
Revised: where the tensors G ( k ) ∈ R R

Original: k − 1 ×
Revised: k − 1 ×

Original: I k × R k are the so-called TT–cores and R 

Revised: I k × R k are the so-called TT–cores and R

Original: =
Revised: =

Original: R N = 1 by deﬁnition.
Revised: R N = 1 by deﬁnition.

Original: The minimal values of { R k } N − 1 k =1 for which the TT–decomposition exists are called TT–ranks .
Revised: The minimal values of { R k } N − 1 k =1 for which the TT–decomposition exists are called TT–ranks .

Original: Note, that the element X ( i 1 , i 2 . . .
Revised: Note, that the element X ( i 1 , i 2 . . .

Original: i N ) is just effectively the product of 2 vectors and N − 2 matrices:
Revised: i N ) is just effectively the product of 2 vectors and N − 2 matrices:

Original: [Equation]
Revised: [Equation]

Original: , where G ( k ) [: , i k , :] stands for the slice (a subset of a tensor with some indices ﬁxed) of the corresponding TT–core G ( k ) .
Revised: , where G ( k ) [: , i k , :] stands for the slice (a subset of a tensor with some indices ﬁxed) of the corresponding TT–core G ( k ) .

Original: The number of degrees of freedom in such a decomposition can be evaluated to be (cid:80) 
Revised: The number of degrees of freedom in such a decomposition can be evaluated as (cid:80) Nk

Original: Nk =1 R k {+− 
+}
Revised: =1 R k {+− 
+}

Original: I k R k 
Revised: I k R k .

Original: +1 .

Original: Thus, in the case of small ranks, the total number of parameters required to store a tensor in TT– representation is signiﬁcantly smaller than (cid:81) Nk
Revised: Thus, in the case of small ranks, the total number of parameters required to store a tensor in TT– representation is signiﬁcantly smaller than (cid:81) Nk

Original: =

Revised: =

Original: I k parameters required to store the full tensor of the corresponding size.
Revised: I k parameters required to store the full tensor of the corresponding size.

Original: This observation makes the application of the TT–decomposition appealing in many problems dealing with extremely large tensors.
Revised: This observation makes the application of the TT–decomposition appealing in many problems dealing with extremely large tensors.

Original: 3. TT– MATRIX
Revised: 3. TT– MATRIX

Original: Let X ∈ R
Revised: Let X ∈ R

Original: I × J be a matrix of size I × J .
Revised: I × J be a matrix of size I × J .

Original: Given two arbitrary factorizations of its dimensions into natural numbers, I = (cid:81)
Revised: Given two arbitrary factorizations of its dimensions into natural numbers, I = (cid:81)

Original: Nk
Revised: Nk

Original: =

Revised: =

Original: I k and J = (cid:81)
Revised: I k and J = (cid:81)

Original: Nk
Revised: Nk

Original: =1 J k , we can reshape 1 and transpose this matrix into an
Revised: =1 J k , we can reshape 1 and transpose this matrix into an

Original: N -way tensor X ∈ R
Revised: N -way tensor X ∈ R

Original: I 1 J 1 ×
Revised: I 1 J 1 ×

Original: I 2 J 2 ×···× I N J N and then apply the TT–decomposition to it, resulting in a more compact representation.
Revised: I 2 J 2 ×···× I N J N and then apply the TT–decomposition to it, resulting in a more compact representation.

Original: More concretely, deﬁne the bijections I ( i )
Revised: More concretely, deﬁne the bijections I ( i )

Original: =
Revised: =

Original: ( i 1 , . .
Revised: ( i 1 , . .

Original: , i N ) and J ( j )
Revised: , i N ) and J ( j )

Original: =
Revised: =

Original: ( j 1 , . . .
Revised: ( j 1 , . . .

Original: , j N ) that map row and column indices
Revised: , j N ) that map row and column indices

Original: i and j of the matrix X to the N -dimensional vector-indices such that 0 ≤ i k < I k , 0 ≤ j k < J k , ∀ k = 1 , . .
Revised: i and j of the matrix X to the N -dimensional vector-indices such that 0 ≤ i k < I k , 0 ≤ j k < J k , ∀ k = 1 , . .

Original: . , N .
Revised: . , N .

Original: From the matrix X we can form an N -way tensor
Revised: From the matrix X we can form an N -way tensor

Original: X whose k -th dimension is of length
Revised: X whose k -th dimension is of length

Original: I k J k
Revised: I k J k

Original: and is indexed by the tuple ( i k , j k ) .
Revised: and is indexed by the tuple ( i k , j k ) .

Original: This tensor is then represented in the TT–format:
Revised: This tensor is then represented in the TT–format:

Original: [Equation]
Revised: [Equation]

Original: Such representation of the matrix in the TT–format is called TT–matrix (Oseledets, 2010; Novikov et al., 2015) and is also known as Matrix Product Operator (Pirvu et al., 2010) in physics literature.
Revised: Such representation of the matrix in the TT–format is called TT–matrix (Oseledets, 2010; Novikov et al., 2015) and is also known as Matrix Product Operator (Pirvu et al., 2010) in physics literature.

Original: The factorizations ( I 1 , I 2 , . . .
Revised: The factorizations ( I 1 , I 2 , . . .

Original: I N ) × ( J 1 , J 2 , . . .
Revised: I N ) × ( J 1 , J 2 , . . .

Original: J N ) will be referred to as the shape of TT–matrix, or TT–shapes .
Revised: J N ) will be referred to as the shape of TT–matrix, or TT–shapes .

Original: The process of constructing the TT–matrix from the standard matrix is visualized in Figure 1 for the tensor of order 3 .
Revised: The process of constructing the TT–matrix from the standard matrix is visualized in Figure 1 for the tensor of order 3 .

Original: Note, that in this case the TT–cores are in fact 4 -th order tensors, but all the operations deﬁned for tensors in the TT–format are naturally extended to TT–matrices.
Revised: Note, that in this case the TT–cores are in fact 4 -th order tensors, but all the operations deﬁned for tensors in the TT–format are naturally extended to TT–matrices.

Original: 3. TT– EMBEDDING
Revised: 3. TT– EMBEDDING

Original: By TT–embedding , we call a layer with trainable parameters (TT–cores) represented as a TT–matrix
Revised: By TT–embedding , we call a layer with trainable parameters (TT–cores) represented as a TT–matrix

Original: E of the underlying tensor shape ( I 1 , I 2 , . . .
Revised: E of the underlying tensor shape ( I 1 , I 2 , . . .

Original: I N ) × ( J 1 , J 2 , . . .
Revised: I N ) × ( J 1 , J 2 , . . .

Original: J N ) , which can be transformed into a valid embedding layer E ∈
Revised: J N ) , which can be transformed into a valid embedding layer E ∈

Original: R
Revised: R

Original: I × J , with I = (cid:81)
Revised: I × J , with I = (cid:81)

Original: Nk
Revised: Nk

Original: =

Revised: =

Original: I k and J = (cid:81)
Revised: I k and J = (cid:81)

Original: Nk
Revised: Nk

Original: =1 J k .
Revised: =1 J k .

Original: To specify the shapes of
Revised: To specify the shapes of

Original: TT–cores one has also to provide the TT–ranks, which are treated as hyperparameters of the layer and explicitly deﬁne the total compression ratio.
Revised: TT–cores one has also to provide the TT–ranks, which are treated as hyperparameters of the layer and explicitly deﬁne the total compression ratio.

Original: [Figure]
Revised: [Figure]

Original: In order to compute the embedding for a particular word indexed i in the vocabulary, we ﬁrst map the row index i into the N -dimensional vector index ( i 1 , . .
Revised: In order to compute the embedding for a particular word indexed i in the vocabulary, we ﬁrst map the row index i into the N -dimensional vector index ( i 1 , . .

Original: , i N ) , and then calculate components of the embedding with formula (1).
Revised: , i N ) , and then calculate components of the embedding with formula (1).

Original: Note, that the computation of all its components is equivalent to selecting the particular slices in TT-cores (slices of shapes J 1 × R 1 in G (1) , R 1 × J 2 × R 2 in
Revised: Note, that the computation of all its components is equivalent to selecting the particular slices in TT-cores (slices of shapes J 1 × R 1 in G (1) , R 1 × J 2 × R 2 in

Original: G (2) and so on) and performing a sequence of matrix multiplications, which is executed efﬁciently in modern linear algebra packages, such as BLAS.
Revised: G (2) and so on) and performing a sequence of matrix multiplications, which is executed efﬁciently in modern linear algebra packages, such as BLAS.

Original: Pseudocode for the procedure of computing the mapping i
Revised: Pseudocode for the procedure of computing the mapping i

Original: → ( i 1 , . .
Revised: → ( i 1 , . .

Original: , i N ) is given in Appendix A.
Revised: , i N ) is given in Appendix A.

Original: In order to construct TT–embedding layer for a vocabulary of size I and embedding dimension J , and to train a model with such a layer, one has to perform the following steps.
Revised: In order to construct TT–embedding layer for a vocabulary of size I and embedding dimension J , and to train a model with such a layer, one has to perform the following steps.

Original: • Provide factorizations of I and J into factors
Revised: • Provide factorizations of I and J into factors

Original: I =
Revised: I =

Original: I 

Revised: I

Original: × I 2 × · · · × I N and J =
Revised: × I 2 × · · · × I N and J =

Original: J 1 × J 2 × · · · × J N , and specify the set of TT–ranks { R 1 , R 2 , . .
Revised: J 1 × J 2 × · · · × J N , and specify the set of TT–ranks { R 1 , R 2 , . .

Original: , R N − 1 } .
Revised: , R N − 1 } .

Original: • Initialize the set of parameters of the embedding Θ
Revised: • Initialize the set of parameters of the embedding Θ

Original: = { G ( k ) ∈ R R k
Revised: = { G ( k ) ∈ R R k

Original: [Equation]
Revised: [Equation]

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: R k } Nk =1 .
Revised: R k } Nk =1 .

Original: Concrete initialization scenarios are discussed further in the text.
Revised: Concrete initialization scenarios are discussed further in the text.

Original: • During training, given a batch of indices { i 1 , i 2 , . . .
Revised: • During training, given a batch of indices { i 1 , i 2 , . . .

Original: i b } , compute the corresponding embeddings { e 1 , e 2 , . . .
Revised: i b } , compute the corresponding embeddings { e 1 , e 2 , . . .

Original: , e b } using Eq.
Revised: , e b } using Eq.

Original: (1) and Algorithm 1.
Revised: (1) and Algorithm 1.

Original: • Computed embeddings can be followed by any standard layer such as LSTM (Hochreiter & Schmidhuber, 1997) or self-attention (Vaswani et al., 2017), and trained
Revised: • Computed embeddings can be followed by any standard layer such as LSTM (Hochreiter & Schmidhuber, 1997) or self-attention (Vaswani et al., 2017), and trained

Original: with backpropagation since they differentially depend on the parameters Θ .
Revised: with backpropagation since they differentially depend on the parameters Θ .

Original: TT–embedding implies a speciﬁc structure on the order of tokens in the vocabulary (the order of rows in the embedding matrix), and determining the optimal order is an appealing problem to solve.
Revised: TT–embedding implies a speciﬁc structure on the order of tokens in the vocabulary (the order of rows in the embedding matrix), and determining the optimal order is an appealing problem to solve.

Original: However, we leave this problem for future work and use the order produced by the standard tokenizer (sorted by frequency) in our current experiments.
Revised: However, we leave this problem for future work and use the order produced by the standard tokenizer (sorted by frequency) in our current experiments.

Original: We also experimented with  more general form of TT-decomposition, namely Tensor Ring decomposition (Zhao et al., 2016; Wang et al., 2018).
Revised: We also experimented with a more general form of TT-decomposition, namely Tensor Ring decomposition (Zhao et al., 2016; Wang et al., 2018).

Original: This decomposition by construction has the appealing property of being circular permutation invariant (and, thus, more robust with respect to the order of the tokens), which could have potentially provided an improvement over the TT-based models with simple frequency based ordering.
Revised: This decomposition by construction has the appealing property of being circular permutation invariant (and, thus, more robust with respect to the order of the tokens), which could have potentially provided an improvement over the TT-based models with simple frequency based ordering.

Original: In our experiments, however, the resulting TR–embeddings performed slightly worse than TT–embeddings with the same amount of parameters (Appendix B).
Revised: Our experiments with TR decomposition on Transformer for NMT can be found in Appendix B.

Original: Initialization The standard way to initialize an embedding matrix E ∈ R
Revised: Initialization The standard way to initialize an embedding matrix E ∈ R

Original: I × J is via, e.g., Glorot initializer (Glorot & Bengio, 2010), which initializes each element as E ( i, j ) ∼ N (cid:16) 0 , 2 I + J (cid:17) .
Revised: I × J is via, e.g., Glorot initializer (Glorot & Bengio, 2010), which initializes each element as E ( i, j ) ∼ N (cid:16) 0 , 2 I + J (cid:17) .

Original: For the TT–embedding, we can only initialize the TT–cores, and the distribution of the elements of the resulting matrix E is rather non–trivial.
Revised: For the TT–embedding, we can only initialize the TT–cores, and the distribution of the elements of the resulting matrix E is rather non–trivial.

Original: However, it is easy to verify that if we initialize each
Revised: However, it is easy to verify that if we initialize each

Original: TT–core element as G ( k ) ( r k − 1 , i k , r k ) ∼ N (0 , 1) , the resulting distribution of the matrix elements
Revised: TT–core element as G ( k ) ( r k − 1 , i k , r k ) ∼ N (0 , 1) , the resulting distribution of the matrix elements

Original: E ( i, j ) has the property that E [ E ( i, j )]
Revised: E ( i, j ) has the property that E [ E ( i, j )]

Original: = 

Revised: =

Original: and Var[ E ( i, j )]
Revised: and Var[ E ( i, j )]

Original: = (cid:81)
Revised: = (cid:81)

Original: Nk
Revised: Nk

Original: =1 R k
Revised: =1 R k

Original: = R 2 .
Revised: = R 2 .

Original: Capitalizing on this observation, in order to obtain the desired variance Var[ E ( i, j )]
Revised: Capitalizing on this observation, in order to obtain the desired variance Var[ E ( i, j )]

Original: = σ 2 while keeping E
Revised: = σ 2 while keeping E

Original: [ E ( i, j )] = 0 , we can simply initialize each TT–core as 
Revised: [ E ( i, j )] = 0 , we can simply initialize each TT–core as [Equation]

Original: [Equation]
Revised: [Equation]

Original: The resulting distribution is not Gaussian, however, it approaches the Gaussian distribution with the increase of the TT–rank (Figure 2).
Revised: The resulting distribution is not Gaussian, however, it approaches the Gaussian distribution with the increase of the TT–rank (Figure 2).

Original: In our experiments, we have used the modiﬁed Glorot initializer implemented by formula (2), which greatly improved performance, as opposed to initializing TT–cores simply via a standard normal distribution.
Revised: In our experiments, we have used the modiﬁed Glorot initializer implemented by formula (2), which greatly improved performance, as opposed to initializing TT–cores simply via a standard normal distribution.

Original: It is also possible to initialize TT–embedding layer by converting the learned embedding matrix into TT–format using the TT–SVD algorithm (Oseledets, 2011), however, this approach requires the pretrained embedding matrix and does not exhibit better performance in practice.
Revised: It is also possible to initialize TT–embedding layer by converting the learned embedding matrix into TT–format using the TT–SVD algorithm (Oseledets, 2011), however, this approach requires the pretrained embedding matrix and does not exhibit better performance in practice.

Original: Hyperparameter selection
Revised: Hyperparameter selection

Original: Our embedding layer introduces two additional structure-speciﬁc hyperparameters, namely TT–shapes and TT–ranks .
Revised: Our embedding layer introduces two additional structure-speciﬁc hyperparameters, namely TT–shapes and TT–ranks .

Original: TT–embedding does not require the vocabulary size I to be represented exactly as the product of factors I 1 , . .
Revised: TT–embedding does not require the vocabulary size I to be represented exactly as the product of factors I 1 , . .

Original: , I N , in fact, any factorization (cid:81) kk
Revised: , I N , in fact, any factorization (cid:81) kk

Original: =

Revised: =

Original: I k = (cid:101)
Revised: I k = (cid:101)

Original: I ≥ I will sufﬁce.
Revised: I ≥ I will sufﬁce.

Original: However, in order to achieve the highest possible compression ratio for a ﬁxed value of (cid:101) I , the factors { I k } Nk =1 should be as close to each other as possible.
Revised: However, in order to achieve the highest possible compression ratio for a ﬁxed value of (cid:101) I , the factors { I k } Nk =1 should be as close to each other as possible.

Original: Our implementation includes a simple automated procedure for selecting a good values of { I k } Nk =1 during TT–embedding initialization.
Revised: Our implementation includes a simple automated procedure for selecting a good values of { I k } Nk =1 during TT–embedding initialization.

Original: The factors J 1 , . . .
Revised: The factors J 1 , . . .

Original: , J N are deﬁned by the embedding dimensionality J which can be easily chosen to support good factorization, e.g., 512 = 8 × 8 × 8 or 480 = 6 × 5 × 4 × 4 .
Revised: , J N are deﬁned by the embedding dimensionality J which can be easily chosen to support good factorization, e.g., 512 = 8 × 8 × 8 or 480 = 6 × 5 × 4 × 4 .

Original: The values of TT–ranks directly deﬁne the compression ratio, so choosing them to be too small or too large will result into either signiﬁcant performance drop or little reduction of the number of parameters.
Revised: The values of TT–ranks directly deﬁne the compression ratio, so choosing them to be too small or too large will result into either signiﬁcant performance drop or little reduction of the number of parameters.

Original: In our experiments, we set all TT–ranks to 16 for the problems with small vocabularies and 64 − 192 for the problems with larger vocabularies, which allowed us to achieve signiﬁcant compression of the embedding layer, at the cost of a tiny sacriﬁce in the metrics of interest.
Revised: In our experiments, we set all TT–ranks to 16 for the problems with small vocabularies and 64 − 192 for the problems with larger vocabularies, which allowed us to achieve signiﬁcant compression of the embedding layer, at the cost of a tiny sacriﬁce in the metrics of interest.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Code We have implemented TT–embeddings described in Section 3 in Python using PyTorch (Paszke et al., 2017).
Revised: Code We have implemented TT–embeddings described in Section 3 in Python using PyTorch (Paszke et al., 2017).

Original: The code is available at the anonymous repository https://github.com/ttembedding/tt-embeddings.
Revised: The code is available at the anonymous repository https://github.com/ttembedding/tt-embeddings.

Original: Experimental setup We tested our approach on several popular NLP tasks:
Revised: Experimental setup We tested our approach on several popular NLP tasks:

Original: • Sentiment analysis — as a starting point in our experiments, we test TT–embeddings on a rather simple task of predicting polarity of a sentence.
Revised: • Sentiment analysis — as a starting point in our experiments, we test TT–embeddings on a rather simple task of predicting polarity of a sentence.

Original: • Neural Machine Translation (NMT) — to verify the applicability of TT–embeddings in more practical problems, we test it on a more challenging task of machine translation.
Revised: • Neural Machine Translation (NMT) — to verify the applicability of TT–embeddings in more practical problems, we test it on a more challenging task of machine translation.

Original: • Language Modeling (LM) — then, we evaluate TT–embeddings on language modeling tasks in the case of extremely large vocabularies.
Revised: • Language Modeling (LM) — then, we evaluate TT–embeddings on language modeling tasks in the case of extremely large vocabularies.

Original: • Click Through Rate (CTR) prediction — ﬁnally, we show that TT–embeddings can be applied for the binary classiﬁcation with categorical features of signiﬁcant cardinality.
Revised: • Click Through Rate (CTR) prediction — ﬁnally, we show that TT–embeddings can be applied for the binary classiﬁcation with categorical features of signiﬁcant cardinality.

Original: To prove the generality and wide applicability of the proposed approach, we tested it on various architectures, such as MLPs (CTR), LSTMs (sentiment analysis), and Transformers (NMT, LM).
Revised: To prove the generality and wide applicability of the proposed approach, we tested it on various architectures, such as MLPs (CTR), LSTMs (sentiment analysis), and Transformers (NMT, LM).

Original: Note that Transformers in LM and NMT use the same weight matrix for their embedding and softmax layers (Press & Wolf, 2016; Inan et al., 2016) which already signiﬁcantly reduces model size.
Revised: Note that Transformers in LM and NMT use the same weight matrix for their embedding and softmax layers (Press & Wolf, 2016; Inan et al., 2016) which already signiﬁcantly reduces model size.

Original: Untying weights and tensorizing the embedding layer only will lead to the increase in the number of parameters instead of compression.
Revised: Untying weights and tensorizing the embedding layer only will lead to the increase in the number of parameters instead of compression.

Original: In our experiments, we use two separate TT-decompositions of the same shape for embedding and softmax layers and report the compression ratios as | V |× d model  × TT-params .
Revised: In our experiments, we use two separate TT-decompositions of the same shape for embedding and softmax layers and report the compression ratios as | V |× d model 2 × TT-params .

Original: 4. S ENTIMENT ANALYSIS
Revised: 4. S ENTIMENT ANALYSIS

Original: For this experiment, we have used the IMDB dataset (Maas et al., 2011) with two categories, and the
Revised: For this experiment, we have used the IMDB dataset (Maas et al., 2011) with two categories, and the

Original: Stanford Sentiment Treebank (SST)  with ﬁve categories.
Revised: Stanford Sentiment Treebank (SST) (Socher et al., 2013) with ﬁve categories.

Original: We have taken the most frequentwords for the IMDB dataset and 17200 for SST, embedded them into a J –dimensional space using either standard embedding or TT–embedding layer, and performed classiﬁcation using a standard bidirectional two–layer LSTM with hidden size h = 128 , and dropout rate P drop = 0 .
Revised: We have taken the most frequent 25000 words for the IMDB dataset and 17200 for SST, embedded them into a J –dimensional space using either standard embedding or TT–embedding layer, and performed classiﬁcation using a standard bidirectional two–layer LSTM with hidden size h = 128 , and dropout rate P drop = 0 .

Original: Our ﬁndings are summarized in Table 1.
Revised: Our ﬁndings are summarized in Table 1.

Original: We observe that the models with largely compressed embedding layers can perform equally or even better than the full uncompressed models.
Revised: We observe that the models with largely compressed embedding layers can perform equally or even better than the full uncompressed models.

Original: This suggests that learning individual independent embeddings for each particular word is superﬂuous, as the expressive power of LSTM is sufﬁcient to make use of these intertwined, yet more compact embeddings.
Revised: This suggests that learning individual independent embeddings for each particular word is superﬂuous, as the expressive power of LSTM is sufﬁcient to make use of these intertwined, yet more compact embeddings.

Original: Moreover, slightly better test accuracy of the compressed models in certain cases (e.g., for the SST dataset of a rather small size) insinuates that imposing speciﬁc tensorial low–rank structure on the embedding matrix can be viewed as a special form of regularization , thus potentially improving model generalization.
Revised: Moreover, slightly better test accuracy of the compressed models in certain cases (e.g., for the SST dataset of a rather small size) insinuates that imposing speciﬁc tensorial low–rank structure on the embedding matrix can be viewed as a special form of regularization , thus potentially improving model generalization.

Original: A detailed and comprehensive test of this hypothesis goes beyond the scope of this paper, and we leave it for future work.
Revised: A detailed and comprehensive test of this hypothesis goes beyond the scope of this paper, and we leave it for future work.

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 4. N EURAL M ACHINE T RANSLATION
Revised: N EURAL M ACHINE T RANSLATION

Original: For this experiment
Revised: For this experiment

Original: [Table]
Revised: [Table]

Original: we have trained the Transformer-big model ( d model = 1024 , d ff = 4096 , h = 16 ) from (Vaswani et al., 2017) on WMT 2014 English–German dataset consisting of roughly 4 . 5 million sentence pairs.
Revised: we have trained the Transformer-big model ( d model = 1024 , d ff = 4096 , h = 16 ) from (Vaswani et al., 2017) on WMT 2014 English–German dataset consisting of roughly 4 . 5 million sentence pairs.

Original: We evaluated on newstest2014 dataset using beam search with a beam size of 4 and no length penalty.
Revised: We evaluated on newstest2014 dataset using beam search with a beam size of 4 and no length penalty.

Original: We did not employ checkpoint averaging and used the last checkpoint to compute the BLEU score.
Revised: We did not employ checkpoint averaging and used the last checkpoint to compute the BLEU score.

Original: Sentences were tokenized with YouTokenToMe 2 byte-pair-encodings, resulting in a joint vocabulary of 32768 tokens.
Revised: Sentences were tokenized with YouTokenToMe 2 byte-pair-encodings, resulting in a joint vocabulary of 32768 tokens.

Original: For the full list of hyperparameters, see Appendix C.
Revised: For the full list of hyperparameters, see Appendix C.

Original: Our results are summarized in Table 2.
Revised: Our results are summarized in Table 2.

Original: We observe that even in this rather challenging task, both embedding and softmax layers can be compressed signiﬁcantly, at the cost of a small drop in the
Revised: We observe that even in this rather challenging task, both embedding and softmax layers can be compressed signiﬁcantly, at the cost of a small drop in the

Original: BLEU score.
Revised: BLEU score.

Original: However, with the increase of compression factor, the performance deteriorates rapidly.
Revised: However, with the increase of compression factor, the performance deteriorates rapidly.

Original: Compared to the sentiment analysis, NMT is a much more complex task which beneﬁts more from additional capacity (in the form of more powerful RNN or more transformer blocks) rather than regularization (Bahdanau et al., 2014; Vaswani et al., 2017; Wu et al., 2019), which may explain why we did not manage to improve the model by regularizing its embedding layers.
Revised: Compared to the sentiment analysis, NMT is a much more complex task which beneﬁts more from additional capacity (in the form of more powerful RNN or more transformer blocks) rather than regularization (Bahdanau et al., 2014; Vaswani et al., 2017; Wu et al., 2019), which may explain why we did not manage to improve the model by regularizing its embedding layers.

Original: TT-embeddings induce 8% training iteration time overhead if compared to the baseline Transformerbig due to our current implementation heavy relying on slow torch.einsum function while standard embedding and softmax layers make use of fast and highly-optimized Tensor Cores for mixed-precision training.
Revised: TT-embeddings induce 8% training iteration time overhead if compared to the baseline Transformerbig due to our current implementation heavy relying on slow torch.einsum function while standard embedding and softmax layers make use of fast and highly-optimized Tensor Cores for mixed-precision training.

Original: We expect a dedicated CUDA kernel to be much more efﬁcient
Revised: We expect a dedicated CUDA kernel to be much more efﬁcient

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: L ANGUAGE MODELING
Revised: L ANGUAGE MODELING

Original: We took the Transformer-XL (Dai et al., 2019), an open source 3 state-of-the-art language modeling architecture at the time of this writing, and replaced its embedding and softmax layers with TT– factorizations.
Revised: We took the Transformer-XL (Dai et al., 2019), an open source 3 state-of-the-art language modeling architecture at the time of this writing, and replaced its embedding and softmax layers with TT– factorizations.

Original: Then, we tested different model conﬁgurations on the WikiText–103 (Merity et al., 2016) dataset and reported the results in Table 3.
Revised: Then, we tested different model conﬁgurations on the WikiText–103 (Merity et al., 2016) dataset and reported the results in Table 3.

Original: For the full list of hyperparameters, see Appendix C.
Revised: For the full list of hyperparameters, see Appendix C.

Original: Compared to sentiment analysis and NMT, we were not able to achieve that high compression ratios for embedding and softmax layers in LM.
Revised: Compared to sentiment analysis and NMT, we were not able to achieve that high compression ratios for embedding and softmax layers in LM.

Original: However, even moderate 3 . 8 times compression allowed us to save 100 M of weights at the cost of ∼ 1 .
Revised: However, even moderate 3 . 8 times compression allowed us to save 100 M of weights at the cost of ∼ 1 .

Original: 5 perplexity drop.
Revised: 5 perplexity drop.

Original: [Table]
Revised: [Table]

Original: C LICK T HROUGH R ATE PREDICTION
Revised: C LICK T HROUGH R ATE PREDICTION

Original: Among other applications of the TT–embedding layer
Revised: Among other applications of the TT–embedding layer

Original: [Table]
Revised: [Table]

Original: we chose to focus on CTR prediction, a popular task in digital advertising (He et al., 2014).
Revised: we chose to focus on CTR prediction, a popular task in digital advertising (He et al., 2014).

Original: We consider open dataset provided by Criteo for
Revised: We consider open dataset provided by Criteo for

Original: Kaggle Display Advertising Challenge (Criteo Labs, 2014) which consists of 39 categorical features, 45 .
Revised: Kaggle Display Advertising Challenge (Criteo Labs, 2014) which consists of 39 categorical features, 45 .

Original: 8 M samples and is binary labeled according to whether the user clicked on the given advertisement.
Revised: 8 M samples and is binary labeled according to whether the user clicked on the given advertisement.

Original: Unique values of categorical features are bijectively mapped into integers.
Revised: Unique values of categorical features are bijectively mapped into integers.

Original: To reduce the memory footprint, if the size of a corresponding vocabulary is immense (e.g., a cardinality of some features in this dataset is of order 10 6 ), these integers are further hashed by taking modulus with respect to some ﬁxed number such as 10 5 .
Revised: To reduce the memory footprint, if the size of a corresponding vocabulary is immense (e.g., a cardinality of some features in this dataset is of order 10 6 ), these integers are further hashed by taking modulus with respect to some ﬁxed number such as 10 5 .

Original: However, due to strong compression properties of TT–embeddings, this is not necessary for our approach, and we consider both full and hashed datasets in our experiments.
Revised: However, due to strong compression properties of TT–embeddings, this is not necessary for our approach, and we consider both full and hashed datasets in our experiments.

Original: CTR with the baseline algorithm
Revised: CTR with the baseline algorithm

Original: The task at hand can be treated as a binary classiﬁcation problem.
Revised: The task at hand can be treated as a binary classiﬁcation problem.

Original: As a baseline algorithm, we consider the neural network with the following architecture.
Revised: As a baseline algorithm, we consider the neural network with the following architecture.

Original: First, each of the categorical features is passed through a separate embedding layer with embedding size J .
Revised: First, each of the categorical features is passed through a separate embedding layer with embedding size J .

Original: After that, the embedded features are concatenated and passed through 4 fully-connected layers of 1024 neurons and ReLU activation functions.
Revised: After that, the embedded features are concatenated and passed through 4 fully-connected layers of 1024 neurons and ReLU activation functions.

Original: In all experiments, we used Adam optimizer with the learning rate equal to 0 . 0005 .
Revised: In all experiments, we used Adam optimizer with the learning rate equal to 0 . 0005 .

Original: Since many input features have a large number of unique values (e.g., 10131227 ) and storing the corresponding embedding matrices would be costly, we employ the hashing procedure mentioned earlier.
Revised: Since many input features have a large number of unique values (e.g., 10131227 ) and storing the corresponding embedding matrices would be costly, we employ the hashing procedure mentioned earlier.

Original: CTR with TT–embeddings We substitute the embedding layers with the TT–embedding layers.
Revised: CTR with TT–embeddings We substitute the embedding layers with the TT–embedding layers.

Original: Besides that, we leave the overall structure of the neural network unchanged with the same parameters as in the baseline approach.
Revised: Besides that, we leave the overall structure of the neural network unchanged with the same parameters as in the baseline approach.

Original: Table 4 presents the experimental results on the Criteo CTR dataset.
Revised: Table 4 presents the experimental results on the Criteo CTR dataset.

Original: To the best of our knowledge, our loss value is very close to the state-of-the-art result (Juan et al., 2016).
Revised: To the best of our knowledge, our loss value is very close to the state-of-the-art result (Juan et al., 2016).

Original: These experiments indicate that the substitution of large embedding layers with TT–embeddings leads to signiﬁcant compression ratios (up to 2011 times) with a slight improvement in the test loss, and up to 4200 with a small drop in the test loss.
Revised: These experiments indicate that the substitution of large embedding layers with TT–embeddings leads to signiﬁcant compression ratios (up to 2011 times) with a slight improvement in the test loss, and up to 4200 with a small drop in the test loss.

Original: The total size of the compressed model does not exceed 20 Mb, while the baseline model weighs about 160 Mb.
Revised: The total size of the compressed model does not exceed 20 Mb, while the baseline model weighs about 160 Mb.

Original: The obtained compression ratio suggests that the usage of TT–embedding layers may be beneﬁcial in CTR prediction tasks.
Revised: The obtained compression ratio suggests that the usage of TT–embedding layers may be beneﬁcial in CTR prediction tasks.

Original: [Table]
Revised: [Table]

Original:  D ISCUSSION AND FUTURE WORK
Revised:  D ISCUSSION AND FUTURE WORK

Original: We propose a novel embedding layer, the TT–embedding, for compressing huge lookup tables used for encoding categorical features of signiﬁcant cardinality, such as the index of a token in natural language processing tasks.
Revised: We propose a novel embedding layer, the TT–embedding, for compressing huge lookup tables used for encoding categorical features of signiﬁcant cardinality, such as the index of a token in natural language processing tasks.

Original: The proposed approach, based on the TT–decomposition, experimentally proved to be effective, as it heavily decreases the number of training parameters at the cost of a small deterioration in performance.
Revised: The proposed approach, based on the TT–decomposition, experimentally proved to be effective, as it heavily decreases the number of training parameters at the cost of a small deterioration in performance.

Original: In addition, our method can be easily integrated into any deep learning framework and trained via backpropagation, while capitalizing on reduced memory requirements and increased training batch size.
Revised: In addition, our method can be easily integrated into any deep learning framework and trained via backpropagation, while capitalizing on reduced memory requirements and increased training batch size.

Original: Our experimental results suggest several appealing directions for future work.
Revised: Our experimental results suggest several appealing directions for future work.

Original: First of all, TT– embeddings impose a concrete tensorial low-rank structure on the embedding matrix, which was shown to improve the generalization ability of the networks acting as a regularizer.
Revised: First of all, TT– embeddings impose a concrete tensorial low-rank structure on the embedding matrix, which was shown to improve the generalization ability of the networks acting as a regularizer.

Original: The properties and conditions of applicability of this regularizer are subject to more rigorous analysis.
Revised: The properties and conditions of applicability of this regularizer are subject to more rigorous analysis.

Original: Secondly, unlike standard embedding, we can introduce non-linearity into TT-cores to improve their expressive power (Khrulkov et al., 2019).
Revised: Secondly, unlike standard embedding, we can introduce non-linearity into TT-cores to improve their expressive power (Khrulkov et al., 2019).

Original: Additionally, it is important to understand how the order of tokens in the vocabulary affects the properties of the networks with TT–embedding.
Revised: Additionally, it is important to understand how the order of tokens in the vocabulary affects the properties of the networks with TT–embedding.

Original: We hypothesize that there exists the optimal order of tokens which better exploits the particular structure of TT–embedding and leads to a boost in performance and/or compression ratio.
Revised: We hypothesize that there exists the optimal order of tokens which better exploits the particular structure of TT–embedding and leads to a boost in performance and/or compression ratio.

Original: Finally, the idea of applying higher–order tensor decompositions to reduce the number of parameters in neural nets is complementary to more traditional methods such as pruning (Han et al., 2015) and quantization (Hubara et al., 2017; Xu et al., 2018).
Revised: Finally, the idea of applying higher–order tensor decompositions to reduce the number of parameters in neural nets is complementary to more traditional methods such as pruning (Han et al., 2015) and quantization (Hubara et al., 2017; Xu et al., 2018).

Original: Thus, it would be interesting to make a thorough comparison of all these methods and investigate whether their combination may lead to even stronger compression.
Revised: Thus, it would be interesting to make a thorough comparison of all these methods and investigate whether their combination may lead to even stronger compression.

Original: A M ULTIINDEX CONSTRUCTION
Revised: A M ULTIINDEX CONSTRUCTION

Original: Algorithm 

Revised: Algorithm

Original: The algorithm implementing the bijection I ( i ) as described in Section 3.3.
Revised: The algorithm implementing the bijection I ( i ) as described in Section 3.3.

Original: Require : I – vocabulary size, { I k } Nk =1 – an arbitrary factorization of I , i – index of the target word in vocabulary.
Revised: Require : I – vocabulary size, { I k } Nk =1 – an arbitrary factorization of I , i – index of the target word in vocabulary.

Original: Returns : I ( i ) =
Revised: Returns : I ( i ) =

Original: ( i 1 , . .
Revised: ( i 1 , . .

Original: , i N ) – N -dimensional index.
Revised: , i N ) – N -dimensional index.

Original: Initialize :
Revised: Initialize :

Original: L = { 1 , I 1 , I 1 I 2 , . . .
Revised: L = { 1 , I 1 , I 1 I 2 , . . .

Original: , I 1 I 2 . . .
Revised: , I 1 I 2 . . .

Original: I N − 1 } for k = N to 1 do i k ← floor ( i / L [ k ] )
Revised: I N − 1 } for k = N to 1 do i k ← floor ( i / L [ k ] )

Original: i ← i mod L [ k ] end for
Revised: i ← i mod L [ k ] end for

Original: {+Algorithm 
+}
Revised: {+Algorithm 
+}

Original: B T ENSOR R ING E MBEDDING
Revised: The algorithm implementing the bijection ( i 1 , . . , i N ) → i , inverse to I ( i ) . Require : I – vocabulary size, { I k } Nk =1 – an arbitrary factorization of I , ( i 1 , . . , i N ) – Ndimensional index.

Revised: Returns : i – index of the target word in vocabulary

Revised: Initialize :

Revised: L = { 1 , I 1 , I 1 I 2 , . . .

Revised: , I 1 I 2 . . .

Revised: I N − 1 } i ←for k = 1 to N do i ← i

Revised: + i

Revised: k

Revised: ×

Original:  Tensor Ring (TR) decomposition is a generalization to TT-decomposition where the ﬁrst and the last cores are 3 -dimensional tensors which corresponds to R 0 =
Revised: L [ k ] end for B T ENSOR R ING E MBEDDING Tensor Ring (TR) decomposition is a generalization to TT-decomposition where the ﬁrst and the last cores are 3 -dimensional tensors which corresponds to R 0 =

Original: R N > 1 .
Revised: R N > 1 .

Original: Formally, a tensor X is said to be represented in the TR format (Zhao et al., 2016)
Revised: Formally, a tensor X is said to be represented in the TR format (Zhao et al., 2016)

Original: if each element of X can be computed as:
Revised: if each element of X can be computed as:

Original: X ( i 1 , i 2 , . .
Revised: X ( i 1 , i 2 , . .

Original: , i d ) =
Revised: , i d ) =

Original: R 0 (cid:88) r 0 =
Revised: R 0 (cid:88) r 0 =

Original: R 1 (cid:88)
Revised: R 1 (cid:88)

Original: r 1 = · · ·
Revised: r 1 = · · ·

Original: R N − 1 (cid:88) r N − 1 =
Revised: R N − 1 (cid:88) r N − 1 =

Original: G (1) ( r 0 , i 1 , r 1 ) G (2) ( r 1 , i 2 , r 2 ) . . .
Revised: G (1) ( r 0 , i 1 , r 1 ) G (2) ( r 1 , i 2 , r 2 ) . . .

Original: G ( N ) ( r N − 1 , i N , r 0 ) .
Revised: G ( N ) ( r N − 1 , i N , r 0 ) .

Original: Similar to TT, we can deﬁne TR-matrix (see Figure 3) and corresponding TR-embedding layer.
Revised: Similar to TT, we can deﬁne TR-matrix (see Figure 3) and corresponding TR-embedding layer.

Original: [Figure]
Revised: [Figure]

Original: Table 5 shows the performance of different NMT models with both embedding and softmax layers replaced by either TT or TR factorizations.
Revised: Table 5 shows the performance of different NMT models with both embedding and softmax layers replaced by either TT or TR factorizations.

Original: To achieve the same compression factor as the corresponding TT models, TR models should have smaller ranks which negatively affects their performance.
Revised: To achieve the same compression factor as the corresponding TT models, TR models should have smaller ranks which negatively affects their performance.

Original: Furthermore, TR is more computationally heavy.
Revised: Furthermore, TR is more computationally heavy.

Original: [Table]

Original: [Table] [Table] [Table] 15 .179 M 1 .
Revised: C C OMPLETE LIST OF HYPERPARAMETERS

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 178 M

Original: [Table] [Table] [Table] [Table]
Revised: cosine weight decay 0 .batch size in tokens number of training steps number of warmup steps

Original: 177 M

Original: [Table] [Table] [Table] [Table]
Revised: Regularization global dropout, P drop 0 .label smoothing 0 .

Original: 179 M

Original: [Table]

Original: [Table] [Table] 177 M [Table]
Revised: Inference beam search beam size length penalty max source / target difference in tokens

Original: C C OMPLETE LIST OF HYPERPARAMETERS

Original: [Table]
Revised: [Table]

Original: [Table]