Original: T EXT TN: P ROBABILISTIC E NCODING OF L ANGUAGE ON T ENSOR N ETWORK
Revised: T EXT TN: P ROBABILISTIC E NCODING OF L ANGUAGE ON T ENSOR N ETWORK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: As a novel model that bridges machine learning and quantum theory, tensor network (TN) has recently gained increasing attention and successful applications for processing natural images.
Revised: As a novel model that bridges machine learning and quantum theory, tensor network (TN) has recently gained increasing attention and successful applications for processing natural images.

Original: However, for natural languages, it is unclear how to design a probabilistic encoding architecture to efﬁciently and accurately learn and classify texts based on TN.
Revised: However, for natural languages, it is unclear how to design a probabilistic encoding architecture to efﬁciently and accurately learn and classify texts based on TN.

Original: This paper proposes a general two-step scheme of text classiﬁcation based on Tensor Network, which is named as TextTN.
Revised: This paper proposes a general two-step scheme of text classiﬁcation based on Tensor Network, which is named as TextTN.

Original: TextTN ﬁrst encodes the word vectors in a probabilistic space by a generative TN (word-GTN), and then classiﬁes a text sentence using a discriminative TN (sentence-DTN).
Revised: TextTN ﬁrst encodes the word vectors in a probabilistic space by a generative TN (word-GTN), and then classiﬁes a text sentence using a discriminative TN (sentence-DTN).

Original: Moreover, in sentence-DTN, its hyper-parameter (i.e., bond-dimension) can be analyzed and selected by the theoretical property of TextTN’s expressive power.
Revised: Moreover, in sentence-DTN, its hyper-parameter (i.e., bond-dimension) can be analyzed and selected by the theoretical property of TextTN’s expressive power.

Original: In experiments, our TextTN also obtains the state-of-the-art result on SST-5 sentiment classiﬁcation task.
Revised: In experiments, our TextTN also obtains the state-of-the-art result on SST-5 sentiment classiﬁcation task.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Machine learning incorporating with the quantum mechanics forms a novel interdisciplinary ﬁeld known as quantum machine learning (Huggins et al., 2019; Ran et al., 2020).
Revised: Machine learning incorporating with the quantum mechanics forms a novel interdisciplinary ﬁeld known as quantum machine learning (Huggins et al., 2019; Ran et al., 2020).

Original: Tensor network (TN) as a novel model has become prominent in the ﬁeld of quantum machine learning (Biamonte et al., 2017).
Revised: Tensor network (TN) as a novel model has become prominent in the ﬁeld of quantum machine learning (Biamonte et al., 2017).

Original: On the one hand, tensor network can be used as mathematical tool to enhance the theoretical understanding of existing neural network methods (Levine et al., 2018; 2019).
Revised: On the one hand, tensor network can be used as mathematical tool to enhance the theoretical understanding of existing neural network methods (Levine et al., 2018; 2019).

Original: On the other hand, based on tensor network, new machine learning algorithms have been proposed, e.g., discriminative TN (DTN) (Stoudenmire & Schwab, 2016) for supervised tasks and generative TN (GTN) (Han et al., 2018) for unsupervised scenarios (Han et al., 2018).
Revised: On the other hand, based on tensor network, new machine learning algorithms have been proposed, e.g., discriminative TN (DTN) (Stoudenmire & Schwab, 2016) for supervised tasks and generative TN (GTN) (Han et al., 2018) for unsupervised scenarios (Han et al., 2018).

Original: Based on the natural analogy between the quantum concepts (e.g., quantum many-body system (Levine et al., 2018)) and the image representation, many studies and applications are conducted for processing and learning natural pictures (Stoudenmire & Schwab, 2016;
Revised: Based on the natural analogy between the quantum concepts (e.g., quantum many-body system (Levine et al., 2018)) and the image representation, many studies and applications are conducted for processing and learning natural pictures (Stoudenmire & Schwab, 2016;

Original: Sun et al., 2020; Liu et al., 2019).
Revised: Sun et al., 2020; Liu et al., 2019).

Original: However, for natural languages, it remains unclear how to design an efﬁcient and effective TN approach, which can accurately learn and classify texts.
Revised: However, for natural languages, it remains unclear how to design an efﬁcient and effective TN approach, which can accurately learn and classify texts.

Original: In the ﬁeld of natural language processing (NLP), researchers have realized the analogy between the quantum many-body wave function and the word interactions (by the tensor product) in a text sentence, and developed a quantum-inspired language representation (Zhang et al., 2018).
Revised: In the ﬁeld of natural language processing (NLP), researchers have realized the analogy between the quantum many-body wave function and the word interactions (by the tensor product) in a text sentence, and developed a quantum-inspired language representation (Zhang et al., 2018).

Original: Based on the quantum many-body physics and tensor decomposition techniques, Zhang et al.
Revised: Based on the quantum many-body physics and tensor decomposition techniques, Zhang et al.

Original: (2018) provided a mathematical understanding of existing convolution neural network (CNN) based text classiﬁcation methods.
Revised: (2018) provided a mathematical understanding of existing convolution neural network (CNN) based text classiﬁcation methods.

Original: Similarly, a tensor space language model (TSLM) has been built based on the tensor network formulation (Zhang et al., 2019).
Revised: Similarly, a tensor space language model (TSLM) has been built based on the tensor network formulation (Zhang et al., 2019).

Original: This work shows that TSLM is a more generalized language model compared with n − gram and recurrent neural network (RNN) based language models.
Revised: This work shows that TSLM is a more generalized language model compared with n − gram and recurrent neural network (RNN) based language models.

Original: In implementation, however, TSLM did not provide a tensor network algorithm.
Revised: In implementation, however, TSLM did not provide a tensor network algorithm.

Original: The challenge lies in the high dimensionality of each word vector, which is much higher than the dimensionality of each pixel representation in image scenarios.
Revised: The challenge lies in the high dimensionality of each word vector, which is much higher than the dimensionality of each pixel representation in image scenarios.

Original: After the tensor product of a number of word vectors, the resulting high-order tensors will become computationally intractable.
Revised: After the tensor product of a number of word vectors, the resulting high-order tensors will become computationally intractable.

Original: More recently, a tensor network algorithm, namely uniform matrix product state (u-MPS) model, has been proposed for probabilistic modeling of a text sequence (Miller et al., 2020).
Revised: More recently, a tensor network algorithm, namely uniform matrix product state (u-MPS) model, has been proposed for probabilistic modeling of a text sequence (Miller et al., 2020).

Original: u-MPS is evaluated on a context-free language task, which uses an synthetic data set.
Revised: u-MPS is evaluated on a context-free language task, which uses an synthetic data set.

Original: However, u-MPS has not been applied in a real-world NLP task, e.g., typical language modeling or text classiﬁcation task.
Revised: However, u-MPS has not been applied in a real-world NLP task, e.g., typical language modeling or text classiﬁcation task.

Original: In addition, the expressive power of u-MPS has not been investigated.
Revised: In addition, the expressive power of u-MPS has not been investigated.

Original: The expressive power of tensor network is a fundamental property of various TNs and has been systematically studied for tensor network factorizations of multivariate probability distributions (Glasser et al., 2019).
Revised: The expressive power of tensor network is a fundamental property of various TNs and has been systematically studied for tensor network factorizations of multivariate probability distributions (Glasser et al., 2019).

Original: This motivates us to make use of the theoretical property of TN’s expressive power, for developing a tensor network based probabilistic model for natural language representation and classiﬁcation 1 .
Revised: This motivates us to make use of the theoretical property of TN’s expressive power, for developing a tensor network based probabilistic model for natural language representation and classiﬁcation 1 .

Original: To build such a text tensor network, we need to address two research problems in this paper.
Revised: To build such a text tensor network, we need to address two research problems in this paper.

Original: First, how to design a probabilistic encoding architecture to efﬁciently and effectively learn and classify the text.
Revised: First, how to design a probabilistic encoding architecture to efﬁciently and effectively learn and classify the text.

Original: Second, how to analyse its expressive power, and make use of such analyses for more theoretical understanding and practical effectiveness of the text tensor network.
Revised: Second, how to analyse its expressive power, and make use of such analyses for more theoretical understanding and practical effectiveness of the text tensor network.

Original: In this paper, we propose a novel tensor network architecture, named as TextTN.
Revised: In this paper, we propose a novel tensor network architecture, named as TextTN.

Original: TextTN encodes each word vector in word-GTNs and classiﬁes the sentence in a sentence-DTN.
Revised: TextTN encodes each word vector in word-GTNs and classiﬁes the sentence in a sentence-DTN.

Original: First, the proposed word-GTNs train a TN for each word and treat each element of a word vector as a node.
Revised: First, the proposed word-GTNs train a TN for each word and treat each element of a word vector as a node.

Original: In this manner, the word-GTNs ﬁrstly map a high-dimensional word vector a low-dimensional linear space by the tensor network operators.
Revised: In this manner, the word-GTNs ﬁrstly map a high-dimensional word vector a low-dimensional linear space by the tensor network operators.

Original: Then, the second layer tensor network, called as sentence-DTN, trains a TN for each sentence, by regarding the low-dimensional word vector obtained by word-GTNs as its input.
Revised: Then, the second layer tensor network, called as sentence-DTN, trains a TN for each sentence, by regarding the low-dimensional word vector obtained by word-GTNs as its input.

Original: In TextTN, a sentence is represented by the tensor product among word vectors.
Revised: In TextTN, a sentence is represented by the tensor product among word vectors.

Original: Therefore, the interaction information among different word vectors and different dimensions are both modeled in TextTN.
Revised: Therefore, the interaction information among different word vectors and different dimensions are both modeled in TextTN.

Original: Such interactions, are encoded in the high-order weighted tensors, which represent a high-order semantic space.
Revised: Such interactions, are encoded in the high-order weighted tensors, which represent a high-order semantic space.

Original: In both word-GTNs and sentence-DTN, the high-order tensor can be solved by the tensor network, i.e., a matrix product state model, which uses the idea of low-rank approximation that can conquer the exponential wall problem (Watson & Dunn, 2010).
Revised: In both word-GTNs and sentence-DTN, the high-order tensor can be solved by the tensor network, i.e., a matrix product state model, which uses the idea of low-rank approximation that can conquer the exponential wall problem (Watson & Dunn, 2010).

Original: In sentence-DTN, the bond-dimension is an important hyper-parameter and reﬂects the expressive power of TextTN.
Revised: In sentence-DTN, the bond-dimension is an important hyper-parameter and reﬂects the expressive power of TextTN.

Original: In this paper, we analyze the upper and lower bounds of the bond-dimension.
Revised: In this paper, we analyze the upper and lower bounds of the bond-dimension.

Original: Particularly, its lower bound can be determined by the entanglement entropy, which can be considered as a measurement of the communication information encoded in tensor network.
Revised: Particularly, its lower bound can be determined by the entanglement entropy, which can be considered as a measurement of the communication information encoded in tensor network.

Original: A reference bonddimension can be set as this lower bound, as we assume that a larger value means an information redundancy and a smaller value indicates an insufﬁciency of the TN’s expressive power.
Revised: A reference bonddimension can be set as this lower bound, as we assume that a larger value means an information redundancy and a smaller value indicates an insufﬁciency of the TN’s expressive power.

Original: In the experiments, such a reference bond-dimension can achieve effective classiﬁcation results, which indicates the TextTN’s practical advantage in its potential to save hyper-parameter tuning efforts.
Revised: In the experiments, such a reference bond-dimension can achieve effective classiﬁcation results, which indicates the TextTN’s practical advantage in its potential to save hyper-parameter tuning efforts.

Original: Moreover, the word interaction has been taken into account in sentence-DTN by the joint effect of different words for the later class predication by the loss functions.
Revised: Moreover, the word interaction has been taken into account in sentence-DTN by the joint effect of different words for the later class predication by the loss functions.

Original: For the learning algorithm, we observe that different word positions have different weights in a sentence, so that the one-function (for a speciﬁc position) training in the original DTN is inappropriate.
Revised: For the learning algorithm, we observe that different word positions have different weights in a sentence, so that the one-function (for a speciﬁc position) training in the original DTN is inappropriate.

Original: Therefore, we propose an all-function training process in the sentence-DTN to improve the stability of TextTN.
Revised: Therefore, we propose an all-function training process in the sentence-DTN to improve the stability of TextTN.

Original: We have evaluated TextTN in four major text classiﬁcation datasets (MR, Subj, CR and MPQA).
Revised: We have evaluated TextTN in four major text classiﬁcation datasets (MR, Subj, CR and MPQA).

Original: The results show that TextTN outperforms convolutional neural network (CNN) on all the datasets.
Revised: The results show that TextTN outperforms convolutional neural network (CNN) on all the datasets.

Original: This departs from vision tasks where according to the recent literature, a tensor network has not been reported to outperform CNN (Kim, 2014).
Revised: This departs from vision tasks where according to the recent literature, a tensor network has not been reported to outperform CNN (Kim, 2014).

Original: In addition, based on the word vectors from the pre-trained model BERT, the TextTN has better results than the BERT model on SST-2 and SST-5 tasks, and the accuracy of BERT+TextTN is comparable with the state of the art (SOTA) result on SST-5 dataset.
Revised: In addition, based on the word vectors from the pre-trained model BERT, the TextTN has better results than the BERT model on SST-2 and SST-5 tasks, and the accuracy of BERT+TextTN is comparable with the state of the art (SOTA) result on SST-5 dataset.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We now provide the background of Matrix product States (MPS), a family of tensor networks.
Revised: We now provide the background of Matrix product States (MPS), a family of tensor networks.

Original: MPS (Schollwock, 2011) is also known as the tensor train decomposition (Oseledets, 2011).
Revised: MPS (Schollwock, 2011) is also known as the tensor train decomposition (Oseledets, 2011).

Original: Because of the low degree of freedom, the research based on MPS is developing rapidly.
Revised: Because of the low degree of freedom, the research based on MPS is developing rapidly.

Original: At present, the tensor network based on MPS can be roughly divided into two categories.
Revised: At present, the tensor network based on MPS can be roughly divided into two categories.

Original: One is the Generative Tensor
Revised: One is the Generative Tensor

Original: Network (GTN) (Han et al., 2018; Sun et al., 2020), and the other one is the supervised tensor network (also named as Discriminative Tensor Network, DTN) (Stoudenmire & Schwab, 2016).
Revised: Network (GTN) (Han et al., 2018; Sun et al., 2020), and the other one is the supervised tensor network (also named as Discriminative Tensor Network, DTN) (Stoudenmire & Schwab, 2016).

Original: Then, we brieﬂy describe existing GTN and DTN models for image classiﬁcation tasks.
Revised: Then, we brieﬂy describe existing GTN and DTN models for image classiﬁcation tasks.

Original: GTNs are used to model the joint probability distribution of given data.
Revised: GTNs are used to model the joint probability distribution of given data.

Original: For a picture X with n pixels, each pixel is encoded into a two-dimensional vector x i
Revised: For a picture X with n pixels, each pixel is encoded into a two-dimensional vector x i

Original: =
Revised: =

Original: ( p i , 1 − p i ) T by a feature mapping from a
Revised: ( p i , 1 − p i ) T by a feature mapping from a

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: pixel’s value
Revised: pixel’s value

Original: The representation of the picture X can be give out Φ( X ) through the operator of tensor product between these vectors x i
Revised: The representation of the picture X can be give out Φ( X ) through the operator of tensor product between these vectors x i

Original: A joint probability distribution of the picture X is computed by the GTNs
Revised: A joint probability distribution of the picture X is computed by the GTNs

Original: [Equation]
Revised: [Equation]

Original: where P j ( X ) represents the probability of a picture X with respect to the category j ( j ∈ { 1 , .
Revised: where P j ( X ) represents the probability of a picture X with respect to the category j ( j ∈ { 1 , .

Original: , m } ), m is the number of the categories on an image classiﬁcation dataset, and • is the operator of tensor contraction.
Revised: , m } ), m is the number of the categories on an image classiﬁcation dataset, and • is the operator of tensor contraction.

Original: The MPS decomposition of the j th n -order weight tensor W j can be written as:
Revised: The MPS decomposition of the j th n -order weight tensor W j can be written as:

Original: [Equation]
Revised: [Equation]

Original: In the left of the Figure 1, we give out an illustrative GTNs in two categories (i.e., m = 2 ).
Revised: In the left of the Figure 1, we give out an illustrative GTNs in two categories (i.e., m = 2 ).

Original: Second or third order tensors
Revised: Second or third order tensors

Original: A s i ( i ∈ { 1 , . . . , n } ) are from the decomposition of the weight tensor
Revised: A s i ( i ∈ { 1 , . . . , n } ) are from the decomposition of the weight tensor

Original: W j .
Revised: W j .

Original: Each ’virtual’ indicator α k ( k ∈ { 1 , . .
Revised: Each ’virtual’ indicator α k ( k ∈ { 1 , . .

Original: . , n − 1 } ) is the rank obtained from the tensor-train decomposition (Oseledets, 2011), and the ’physical’ indicator s i is the dimension of the pixel x i .
Revised: . , n − 1 } ) is the rank obtained from the tensor-train decomposition (Oseledets, 2011), and the ’physical’ indicator s i is the dimension of the pixel x i .

Original: DTN is used to identify the class or label of a picture and computes a conditional probability distribution P ( y | X ) .
Revised: DTN is used to identify the class or label of a picture and computes a conditional probability distribution P ( y | X ) .

Original: In DTN, the conditional probability distribution P ( y | X ) is computed as follows.
Revised: In DTN, the conditional probability distribution P ( y | X ) is computed as follows.

Original: [Equation]
Revised: [Equation]

Original: where the ( n + 1 )-order weight tensor W l is decomposed into a MPS and l is an extra order/index representing the label:
Revised: where the ( n + 1 )-order weight tensor W l is decomposed into a MPS and l is an extra order/index representing the label:

Original: [Equation]
Revised: [Equation]

Original: where P ( y | X ) is a vector and encodes the conditional probability distribution of outputting y given a picture X .
Revised: where P ( y | X ) is a vector and encodes the conditional probability distribution of outputting y given a picture X .

Original: Eq.
Revised: Eq.

Original: 3 is deﬁned to classify the input X by choosing the label for which the value in the vector P ( y | X ) is largest.
Revised: 3 is deﬁned to classify the input X by choosing the label for which the value in the vector P ( y | X ) is largest.

Original: In practice, the MPS shown in the right of Figure 1 is a supervised learning model (Stoudenmire & Schwab, 2016), and these rank values { α k } are set to be equal as the hyper-parameter (bond-dimension) in TN.
Revised: In practice, the MPS shown in the right of Figure 1 is a supervised learning model (Stoudenmire & Schwab, 2016), and these rank values { α k } are set to be equal as the hyper-parameter (bond-dimension) in TN.

Original:  T ENSOR N ETWORK FOR L ANGUAGE E NCODING AND C LASSIFICATION
Revised:  T ENSOR N ETWORK FOR L ANGUAGE E NCODING AND C LASSIFICATION

Original: Problem setting Our goal is to develop a tensor network architecture to encode and classify a sentence of words in a TN’s probabilistic manner.
Revised: Problem setting Our goal is to develop a tensor network architecture to encode and classify a sentence of words in a TN’s probabilistic manner.

Original: For a sentence with n words (i.e, S =( w 1 , . . . , w n )), the tensor representation of the sentence can be written as ( w 1 ⊗ . . .
Revised: For a sentence with n words (i.e, S =( w 1 , . . . , w n )), the tensor representation of the sentence can be written as ( w 1 ⊗ . . .

Original: ⊗ w i ⊗ . . .
Revised: ⊗ w i ⊗ . . .

Original: ⊗ w n ) (Zhang et al., 2019), where w i are the word vectors.
Revised: ⊗ w n ) (Zhang et al., 2019), where w i are the word vectors.

Original: However, as aforementioned in Introduction, because of the dimensional disaster, it is infeasible to directly input such as a sentence representation into a tensor network.
Revised: However, as aforementioned in Introduction, because of the dimensional disaster, it is infeasible to directly input such as a sentence representation into a tensor network.

Original: In order to solve this problem and still model the word interaction on a high-order tensor (denoted as W l ), we can formalize the TN model as follows:
Revised: In order to solve this problem and still model the word interaction on a high-order tensor (denoted as W l ), we can formalize the TN model as follows:

Original: [Equation]
Revised: [Equation]

Original: where f is an operator to encode a sentence in a low-dimensional probabilistic space.
Revised: where f is an operator to encode a sentence in a low-dimensional probabilistic space.

Original: In our work, word-GTNs are used to encode the word vectors into a low-dimensional space, and then the new representation of word can be efﬁciently inputted to the tensor network.
Revised: In our work, word-GTNs are used to encode the word vectors into a low-dimensional space, and then the new representation of word can be efﬁciently inputted to the tensor network.

Original: Speciﬁcally, the function f in Eq 5 is embodied as word-GTNs (in Section 3.1), and W l is embodied as sentenceDTN (in Section 3.2).
Revised: Speciﬁcally, the function f in Eq 5 is embodied as word-GTNs (in Section 3.1), and W l is embodied as sentenceDTN (in Section 3.2).

Original: In the process of text classiﬁcation, the word-GTNs and sentence-DTN are uniﬁed into a new TN framework (TextTN), as shown in Figure 2.
Revised: In the process of text classiﬁcation, the word-GTNs and sentence-DTN are uniﬁed into a new TN framework (TextTN), as shown in Figure 2.

Original: Bond-dimension is an important hyper-parameter which reﬂects the expressive power of TN model and inﬂuences the effectiveness of text classiﬁcation.
Revised: Bond-dimension is an important hyper-parameter which reﬂects the expressive power of TN model and inﬂuences the effectiveness of text classiﬁcation.

Original: In Section 3.2, we propose a reference bond-dimension that can be computed based on the entanglement entropy.
Revised: In Section 3.2, we propose a reference bond-dimension that can be computed based on the entanglement entropy.

Original: Besides, to improve the stability of sentence-DTN, all-function learning on sentence-DTN is proposed in Section 3.3.
Revised: Besides, to improve the stability of sentence-DTN, all-function learning on sentence-DTN is proposed in Section 3.3.

Original: 3. W ORD E NCODING B ASED ON WORD -GTN S
Revised: 3. W ORD E NCODING B ASED ON WORD -GTN S

Original: We unfasten TextTN step by step in the sentence representation and classiﬁcation problem.
Revised: We unfasten TextTN step by step in the sentence representation and classiﬁcation problem.

Original: First, m word-GTNs are used to encode word vectors to a low-dimensional space, where m ( m > 1) corresponds to the dimension of the low-dimensional space.
Revised: First, m word-GTNs are used to encode word vectors to a low-dimensional space, where m ( m > 1) corresponds to the dimension of the low-dimensional space.

Original: In Figure 2, as an example, two wordGTNs are used to encode each word in a sentence, with shared parameters for each word.
Revised: In Figure 2, as an example, two wordGTNs are used to encode each word in a sentence, with shared parameters for each word.

Original: Firstly, each element of word embedding vector is mapped by a feature mapping.
Revised: Firstly, each element of word embedding vector is mapped by a feature mapping.

Original: Then, each word is represented as a d -order tensor by calculating the tensor product of each element after feature mapping.
Revised: Then, each word is represented as a d -order tensor by calculating the tensor product of each element after feature mapping.

Original: For example, for a word vector w i =
Revised: For example, for a word vector w i =

Original: ( θ 1 , . .
Revised: ( θ 1 , . .

Original: , θ d ) T , the tensor product representation is shown as follows:
Revised: , θ d ) T , the tensor product representation is shown as follows:

Original: [Equation]
Revised: [Equation]

Original: where ⊗ is the tensor product, and φ ( w i ) is the tensor product representation of word w i .
Revised: where ⊗ is the tensor product, and φ ( w i ) is the tensor product representation of word w i .

Original: The feature mapping (i.e., cos ( · ) and sin ( · ) ) is motivated by “spin” vectors encoded in quantum systems, which is usually used in TN (Stoudenmire & Schwab, 2016; Han et al., 2018).
Revised: The feature mapping (i.e., cos ( · ) and sin ( · ) ) is motivated by “spin” vectors encoded in quantum systems, which is usually used in TN (Stoudenmire & Schwab, 2016; Han et al., 2018).

Original: Then, two word-GTNs are used to encode the tensor Φ( w i ) and get the probability distribution v i , i ∈ { 1 , . .
Revised: Then, two word-GTNs are used to encode the tensor Φ( w i ) and get the probability distribution v i , i ∈ { 1 , . .

Original: . , n } (see Figure 2).
Revised: . , n } (see Figure 2).

Original: Speciﬁcally, for word w i , each word-GTN can encode the word into a probability value P ij , j ∈ { 1 , 2 } .
Revised: Speciﬁcally, for word w i , each word-GTN can encode the word into a probability value P ij , j ∈ { 1 , 2 } .

Original: In the following, v i is represented by a 2 -dimensional vector ( P i 1 , P i 2 ) T which is calculated by:
Revised: In the following, v i is represented by a 2 -dimensional vector ( P i 1 , P i 2 ) T which is calculated by:

Original: [Equation]
Revised: [Equation]

Original: where • is the operator of tensor contraction, W 1 and W 2 are two weight tensors to be learned in word-GTNs, and shared for all word vectors in a sentence.
Revised: where • is the operator of tensor contraction, W 1 and W 2 are two weight tensors to be learned in word-GTNs, and shared for all word vectors in a sentence.

Original: The probability condition of P i 1 + P i 2 = 1 is then achieved by the softmax in practice.
Revised: The probability condition of P i 1 + P i 2 = 1 is then achieved by the softmax in practice.

Original: word-GTNs are different from the existing GTNs shown in Background section.
Revised: word-GTNs are different from the existing GTNs shown in Background section.

Original: The word-GTNs are used to learning the new word representation (rather than text representation), and the existing
Revised: The word-GTNs are used to learning the new word representation (rather than text representation), and the existing

Original: GTNs are used to classify the pictures (not for pixels).
Revised: GTNs are used to classify the pictures (not for pixels).

Original: Since a word does not have a class or label, the output of word-GTNs corresponds to the probabilities in a low-dimensional feature space.
Revised: Since a word does not have a class or label, the output of word-GTNs corresponds to the probabilities in a low-dimensional feature space.

Original: Regarding the dimension of low-dimensional space of word-GTNs.
Revised: Regarding the dimension of low-dimensional space of word-GTNs.

Original: i.e, m , we will show that the upper bound of the bond dimension of TextTN is m (cid:98) n/ 2 (cid:99) in Theorem 1.
Revised: i.e, m , we will show that the upper bound of the bond dimension of TextTN is m (cid:98) n/ 2 (cid:99) in Theorem 1.

Original: The expressive power of the model will increase exponentially along with the increasing m (Khrulkov et al., 2018).
Revised: The expressive power of the model will increase exponentially along with the increasing m (Khrulkov et al., 2018).

Original: Therefore, the dimension m should be set as a small value (e.g., 2 in this section) to prevent a too large expressive power.
Revised: Therefore, the dimension m should be set as a small value (e.g., 2 in this section) to prevent a too large expressive power.

Original: This observation is also supported by our experiments.
Revised: This observation is also supported by our experiments.

Original: 3. A NALYSIS OF B OND - DIMENSION ON S ENTENCE -DTN
Revised: 3. A NALYSIS OF B OND - DIMENSION ON S ENTENCE -DTN

Original: After we use word-GTNs, the new representation of a sentence S can be written as ( v 1 ⊗ . . .
Revised: After we use word-GTNs, the new representation of a sentence S can be written as ( v 1 ⊗ . . .

Original: ⊗ v n ) .
Revised: ⊗ v n ) .

Original: which is a n -order tensor.
Revised: which is a n -order tensor.

Original: We now introduce a sentence-DTN to classify sentences.
Revised: We now introduce a sentence-DTN to classify sentences.

Original: This sentenceDTN is a MPS which is from the tensor-train decomposition (Oseledets, 2011) of high-order tensor.
Revised: This sentenceDTN is a MPS which is from the tensor-train decomposition (Oseledets, 2011) of high-order tensor.

Original: By inputting the sentence vector obtained by word-GTNs, the sentenc-DTN can be formalized as:
Revised: By inputting the sentence vector obtained by word-GTNs, the sentenc-DTN can be formalized as:

Original: [Equation]
Revised: [Equation]

Original: which is a n + 1 -order MPS.
Revised: which is a n + 1 -order MPS.

Original: As we introduced in Background, we can set the rank values as an equal value (bond-dimension), i.e., α 1 = . . .
Revised: As we introduced in Background, we can set the rank values as an equal value (bond-dimension), i.e., α 1 = . . .

Original: =
Revised: =

Original: α j = . . .
Revised: α j = . . .

Original: =
Revised: =

Original: α n − 1 .
Revised: α n − 1 .

Original: In other words, the bond dimension is set as r
Revised: In other words, the bond dimension is set as r

Original: ⋯ ⋯ 𝒍𝒍 word [Figure] GTNs sentence
Revised: ⋯ ⋯ 𝒍𝒍 word-GTNs sentence-DTN

Original: [Figure]

Original: DTN

Original: ProbabilityEncoding ⋯⋯ ⋯ ⋯⋯ ⋯ ⋯⋯ 𝑤𝑤𝑤𝑤𝑤𝑤 𝑛𝑛 ⋯ 𝐴𝐴 𝑠𝑠𝐴𝐴 𝑠𝑠 2 𝑙𝑙 𝐴𝐴 𝑠𝑠 𝑛𝑛 𝒗𝒗𝒗𝒗 𝑛𝑛 𝛼𝛼𝛼𝛼𝛼𝛼 𝑛𝑛− 𝜙(𝒘 1 ) 𝜙(𝒘 2 ) 𝜙(𝒘 n ) 𝒗𝒗⋯ ⋯⋯ 𝐴𝐴 𝑠𝑠 𝑛𝑛 𝒗𝒗 𝑛𝑛− 𝛼𝛼 𝑛𝑛− 𝑤𝑤 𝑛𝑛− ( 𝒔𝒔 𝑫𝑫𝑫𝑫𝑫𝑫 ) ( [Figure] 𝑮𝑮𝑫𝑫𝑫𝑫𝒔𝒔 ) 𝜙(𝒘 n-1 ) -
Revised: ProbabilityEncoding ⋯⋯ ⋯ ⋯⋯ ⋯ ⋯⋯ 𝑤𝑤𝑤𝑤𝑤𝑤 𝑛𝑛 ⋯ 𝐴𝐴 𝑠𝑠𝐴𝐴 𝑠𝑠 2 𝑙𝑙 𝐴𝐴 𝑠𝑠 𝑛𝑛 𝒗𝒗𝒗𝒗 𝑛𝑛 𝛼𝛼𝛼𝛼𝛼𝛼 𝑛𝑛− 𝜙(𝒘 1 ) 𝜙(𝒘 2 ) 𝜙(𝒘 n ) 𝒗𝒗⋯ ⋯⋯ 𝐴𝐴 𝑠𝑠 𝑛𝑛 𝒗𝒗 𝑛𝑛− 𝛼𝛼 𝑛𝑛− 𝑤𝑤 𝑛𝑛− ( 𝒔𝒔 𝑫𝑫𝑫𝑫𝑫𝑫 ) ( 𝒘𝒘 𝑮𝑮𝑫𝑫𝑫𝑫𝒔𝒔 ) 𝜙(𝒘 n-1 ) -

Original: ( r = α j ).
Revised: ( r = α j ).

Original: Then, each element of W l can be calculated as follows,
Revised: Then, each element of W l can be calculated as follows,

Original: [Equation]
Revised: [Equation]

Original: The bond-dimension r is the hyper-parameter which can reﬂect the expressive power of a tensor network.
Revised: The bond-dimension r is the hyper-parameter which can reﬂect the expressive power of a tensor network.

Original: Inspired by (Deng et al., 2017), we analyze the bond-dimension  by the entanglement entropy,  as deﬁned in Deﬁnition 1.
Revised: We provide an approach to a reference bond-dimension, which is an optimal parameter setting reported in our experiments. In the following, we will derive the bound of the bond-dimension and show that its lower bound is the reference or recommended bond-dimension. This lower-bound, is calculated by the entanglement entropy, which will be described as follows.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: For an n +1-order weight tensor W l decomposed to a matrix product states (MPS), λ ( λ ≤ r ) singular values can be given out in the (cid:98) n 2 (cid:99) -th node of MPS.
Revised: For an n +1-order weight tensor W l decomposed to a matrix product states (MPS), λ ( λ ≤ r ) singular values can be given out in the (cid:98) n 2 (cid:99) -th node of MPS.

Original: The entanglement entropy is calculated by the singular-value vector g k as
Revised: The entanglement entropy is calculated by the singular-value vector g k as

Original: [Equation]
Revised: [Equation]

Original:  where λ is the number of non-zero singular values, log is the log to the base 2 , and the normalization condition of the singular-value vector (cid:80) Rk
Revised: (10) where λ is the number of non-zero singular values, log is the log to the base 2 , and the normalization condition of the singular-value vector (cid:80) Rk

Original: =1 g 2 k
Revised: =1 g 2 k

Original: = 1 is satisﬁed.
Revised: = 1 is satisﬁed.

Original: Entanglement entropy measures the amount of communication information in TN (Levine et al., 2019). We can derive the lower bond of the bond dimension based on Entanglement entropy, and the upper bond based on the length of sentence for TextTN.
Revised: Entanglement entropy measures the amount of communication information in TN (Levine et al., 2019), and we can derive the lower bound of the bond-dimension based on entanglement entropy, as shown in Theorem 1.

Original: According to the research (Khrulkov et al., 2018), the upper bond of the bond-dimension can be given out and is m (cid:98) n/ 2 (cid:98) , where m is the inputting dimension of each node in MPS, and n is the number of the nodes, i.e., the length of sentence.

Original: Based on the above ideas, we introduce the Theorem 1 as follows.

Original: Theorem 1.
Revised: Theorem 1.

Original: For the inputof the sentence-DTN { v 1 , . . . , v n } , the bond-dimension is bounded as
Revised: For the input { v 1 , . . . , v n } of the sentence-DTN, the bond-dimension is bounded as

Original: [Equation]
Revised: [Equation]

Original: where E is the entanglement entropy of TextTN, m is the inputting dimension of sentence-DTN,  and (cid:98) (cid:99) indicates rounding down, the reference bond-dimension is equal to (cid:98) 2 E (cid:99) .
Revised: where E is the entanglement entropy of TextTN, m is the inputting dimension of sentence-DTN, n is the length of a sentence, and (cid:98) (cid:99) indicates rounding down.

Original: Proof.
Revised: Proof.

Original: The proof can be found in Appendix B.
Revised: The proof can be found in Appendix B.

Original: Based on the lower-bound of the bond dimension, we propose to identify the reference bonddimension of TextTN.

Original: According to Eq.
Revised: According to Eq.

Original: 11, the lower bond can be determined by the entanglement entropy.
Revised: 11, the lower bound of bond-dimension can be determined by the entanglement entropy E .

Original: If the bond-dimension is less than (cid:98) 2 E (cid:99) , TN model can noteffectively model sufﬁcient information measured by the entanglement entropy.
Revised: If the bond-dimension is less than (cid:98) 2 E (cid:99) , TextTN can not model sufﬁcient information measured by the entanglement entropy E .

Original: If otherwise the bond-dimension is larger than (cid:98) 2 E (cid:99) , although TextTN’s expressive power increases, it can not model more information, since the additional information/features learned are corresponding to zero singular values.
Revised: Compared with the lower bound (cid:98) 2 E (cid:99) , choosing a larger value which satisﬁes Eq.11 will increase the number of parameters and the whole computational cost of the TextTN, but only gain little information from the data.

Original:  Therefore, a reference  bond-dimension can be estimated by the lower bond of bond-dimension in Eq.
Revised: In addition, the increase of the bond-dimension will increase the model complexity, possibly leading to the overﬁtting problem. Therefore, we recommend the lower bound (cid:98) 2 E (cid:99) as the reference bond-dimension. About calculating the required entanglement entropy E , we ﬁrst need to use a sufﬁciently large bond-dimension. Since the initial bond-dimension is large enough and the entanglement entropy ﬁnally converges, the E measures the amount of information that the network can capture from a certain data set (Levine et al., 2019).

Original: 11, and we will evaluate such as  reference  bond-dimension  in experiments.
Revised: After calculating the entanglement entropy E , according to the proof of the Theorem 1, we get the lower bound of bond-dimension shown as Eq.11, which is the reference bond-dimension. Moreover, we can derive the upper bound of the bond-dimension. According to the research (Khrulkov et al., 2018), the upper bound of the bond-dimension can be given out and is m (cid:98) n/ 2 (cid:99) , where m is the inputting dimension of each node in MPS, and n is the number of the nodes, i.e., the length of sentence in TextTN. If the bond-dimension is greater than m (cid:98) n/ 2 (cid:99) , then the network structure will not meet the constraints of Tensor-Train decomposition (Oseledets, 2011) ( Appendix B), then TextTN will be rank deﬁcient.

Original: 3. A LL -F UNCTION L EARNING ON SENTENCE -DTN
Revised: 3. A LL -F UNCTION L EARNING ON SENTENCE -DTN

Original: In the original DTN, the position of l in Figure 2 can be moved from A S 1 to A S n in the process of training.
Revised: In the original DTN, the position of l in Figure 2 can be moved from A S 1 to A S n in the process of training.

Original: It means that any position can be selected for  l , leading to a variance for different positions in the processes of training and prediction.
Revised: It means that any position can be selected for the label bound l , leading to a variance for different positions in the processes of training and prediction.

Original: The following Claim 1 shows that the original DTN  corresponds toone probability function from a functions set.
Revised: The following Claim 1 shows that the original DTN calculates a conditional probability by putting the label bound in a certain of the model, which corresponds to a certain certain probability function.

Revised: However, the effects of putting the label bound in different positions should be taken into consideration.

Original: Claim 1.
Revised: Claim 1.

Original: For an input S = { v 1 , v 2 , . . .
Revised: For an input S = { v 1 , v 2 , . . .

Original: , v n } , the original DTN (Stoudenmire & Schwab, 2016) computes the conditional probability P ( y | S ) , which can be considered as a function from a set T of functions which compute all the possible conditional probabilities.
Revised: , v n } , the original DTN (Stoudenmire & Schwab, 2016) computes the conditional probability P ( y | S ) by putting the label bound l in a certain position, which can be considered as a function from a set T of functions which compute all the possible conditional probabilities.

Original: Such a set is deﬁned as:
Revised: Such a set is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where i the location of segmentation for the input S , and is also a position of label l .
Revised: where i the location of segmentation for the input S , and is also a position of label l .

Original: Proof.
Revised: Proof.

Original: The proof can be found in Appendix C.
Revised: The proof can be found in Appendix C.

Original: In the original DTN (Stoudenmire & Schwab, 2016), the loss function is deﬁned as L ( W l ) =
Revised: In the original DTN (Stoudenmire & Schwab, 2016), the loss function is deﬁned as L ( W l ) =

Original: (cid:80)
Revised: (cid:80)

Original: Nj =1 l ( (cid:104) f
Revised: Nj =1 l ( (cid:104) f

Original: ( S ) , W l (cid:105) , y )
Revised: ( S ) , W l (cid:105) , y )

Original: based on the  l in a ﬁxed position, which only corresponds to one probability function.
Revised: based on the label bound l in a certain position, which only corresponds to a certain probability function.

Original: If one function is learned, the different aspects probability distribution for the tensor decomposition in different position can not be considered.
Revised: If one function is learned, the different aspects probability distribution for the label bound l in different position can not be considered.

Original: To solve this problem, we propose to model all functions in the set T in Eq.
Revised: To solve this problem, we propose to model all functions in the set T in Eq.

Original: 12, with a new loss deﬁned as follows:
Revised: 12, with a new loss deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: 13) where CE is cross-entropy, y is the label that is represented by a one-hot vector, N is the number of samples, n is the length of the sentence, g is a parameter vector and g i is a value in the vector.
Revised: where CE is cross-entropy, y is the label that is represented by a one-hot vector, N is the number of samples, n is the length of the sentence, g is a parameter vector and g i is a value in the vector.

Original: (cid:104) f
Revised: (cid:104) f

Original: ( S ) , W l i (cid:105) is one function from the function set T .
Revised: ( S ) , W l i (cid:105) is one function from the function set T .

Original: The sentence-DTN with all-function learning is expected to achieve higher accuracy than the original DTN.
Revised: The sentence-DTN with all-function learning is expected to achieve higher accuracy than the original DTN.

Revised: Finally, we give a Algorithm to describe our training method of overall TextTN given a sentence S with n words in the Algorithm 1 .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we divide the experiment into two parts based on the theories presented above.
Revised: In this section, we divide the experiment into two parts based on the theories presented above.

Original: The ﬁrst set of experiments is to verify the effectiveness of TextTN.
Revised: The ﬁrst set of experiments is to verify the effectiveness of TextTN.

Original: The second set of experiment is to verify that the effectiveness of the reference bond-dimension, which can be computed by Eq.based on the entanglement entropy for different datasets.
Revised: The second set of experiment is to verify that the effectiveness of the reference bond-dimension, which can be computed by Eq.based on the entanglement entropy for different datasets.

Original: A Python implementation of the proposed algorithm is placed in Supplementary Material.
Revised: A Python implementation of the proposed algorithm is placed in Supplementary Material.

Original: 4. T EXT C LASSIFICATION E VALUATION OF T EXT TN

Original: Datasets and Benchmarks : Six text classiﬁcation datasets are used in our experiments.
Revised: Datasets and Benchmarks : Six text classiﬁcation datasets are used in our experiments.

Original: MR (Pang & Lee, 2004b): Movie reviews are divided into positive and negative categories; CR (Hu & Liu, 2004): Customer reviews set where the task is to predict positive or negative product reviews;
Revised: MR (Pang & Lee, 2004b): Movie reviews are divided into positive and negative categories; CR (Hu & Liu, probability codings v W 1 • φ w

Original: Subj : Subjectivity dataset where the target is to classify a text as being subjective or objective;
Revised: | W 2 • φ w v ∈ subjective objective

Original: MPQA (Wiebe et al., 2005): Opinion polarity detection subtask; SST-5 (Pang & Lee, 2004a):
Revised: et al 2004a):

Original: The movie reviews in the Stanford Sentiment Treebank, which is a ﬁne-grained classiﬁcation task (negative, somewhat negative, neutral, somewhat positive, positive); SST-2 (Socher et al., 2013):
Revised: The movie reviews in the Stanford Sentiment Treebank, which is a ﬁne-grained classiﬁcation task (negative, somewhat negative, neutral, somewhat positive, positive); SST-2 (Socher et al., 2013):

Original: Stanford Sentiment Treebank with binary classes.
Revised: Stanford Sentiment Treebank with binary classes.

Original: The evaluation metric is "Accuracy" of all tasks.
Revised: The evaluation metric is "Accuracy" of all tasks.

Original: For CR, MR, MPQA, and Subj tasks, we use publicly word vectors (Word2vec) (Pennington et al., 2014) with 300 dimensions.
Revised: For CR, MR, MPQA, and Subj tasks, we use publicly word vectors (Word2vec) (Pennington et al., 2014) with 300 dimensions.

Original: For SST-2 and SST-5, the Bert-Large pre-trained model (Devlin et al., 2019) is used to obtain word vectors, and the dimension of the word vector from the BERT is 1024 .
Revised: For SST-2 and SST-5, the Bert-Large pre-trained model (Devlin et al., 2019) is used to obtain word vectors, and the dimension of the word vector from the BERT is 1024 .

Original: More dataset statistics and experimental settings  Appendix D.1.
Revised: More dataset statistics and experimental settings are given out in the Appendix D.1.

Revised: Accuracy is the evaluation metric.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Ten benchmarks are used to compare with TextTN.
Revised: Ten benchmarks are used to compare with TextTN.

Original: In Table 1, CNN (Kim, 2014) is used for classiﬁcation; DiSAN (Shen et al., 2017) is a self-attention based model; Capsule-B (Yang et al., 2018) is a capsule neural network; SGC (Wu et al., 2019) proposes a graph neural network; MPSAN (Dai et al., 2020) designs a novel self-attention network.
Revised: In Table 1, CNN (Kim, 2014) is used for classiﬁcation; DiSAN (Shen et al., 2017) is a self-attention based model; Capsule-B (Yang et al., 2018) is a capsule neural network; SGC (Wu et al., 2019) proposes a graph neural network; MPSAN (Dai et al., 2020) designs a novel self-attention network.

Original: In Table 2 , BCN+ELMo (Peters et al., 2018) is a a deep bidirectional language model; Star-Transformer (Guo et al., 2019) is a improved Transformer network; BERT (Devlin et al., 2019) , CNN_Large (Baevski et al., 2019) and FLOATER (Liu et al., 2020) are complicated neural network models based on the BERT_Large pre-trained model.
Revised: In Table 2 , BCN+ELMo (Peters et al., 2018) is a a deep bidirectional language model; Star-Transformer (Guo et al., 2019) is a improved Transformer network; BERT (Devlin et al., 2019) , CNN_Large (Baevski et al., 2019) and FLOATER (Liu et al., 2020) are complicated neural network models based on the BERT_Large pre-trained model.

Original: Experimental Results and Analysis : First, TextTN is experimented on four datasets.
Revised: Experimental Results and Analysis : First, TextTN is experimented on four datasets.

Original: The results in
Revised: The results in

Original: Table 1 show that TextTN has better results than CNN (Kim, 2014) on MR (+0.7%), CR (+0.7%), 
Revised: Table 1 show that TextTN has better results than CNN (Kim, 2014) on MR (+0.7%), CR (+0.7%), [Figure]

Revised: g l e m e n t E n t r o p y

Revised: CR    80 Epoch 4.

Revised: E n t a n g l e m e n t E n t r o p y

Original: Subj (+1.9%) and MPQA    (+0.8%).
Revised: MPQA    80 Epoch 4. E n t a n g l e m e n t E n t r o p y Subj MR

Revised: CR

Original:  Moreover, compared with the state-of-the-art self-attention based model MPSAN (Dai et al., 2020), TextTN outperforms its results on CR by 0.3%, Subj by 0.7%, and obtains a same result 90.4% on MPQA.
Revised: MPQA S ubj Subj (+1.9%) and MPQA (+0.8%). Moreover, compared with the state-of-the-art self-attention based model MPSAN (Dai et al., 2020), TextTN outperforms its results on CR by 0.3%, Subj by 0.7%, and obtains a same result 90.4% on MPQA.

Original: After that, TextTN is experimented with the pre-training word vectors from BERT on SST-2 and
Revised: After that, TextTN is experimented with the pre-training word vectors from BERT on SST-2 and

Original: SST-5.
Revised: SST-5.

Original: The results are shown in Table 2.
Revised: The results are shown in Table 2.

Original: In this experiment, BERT+TextTN achieves the comparable result with the model ELECTRA (Clark et al., 2020), and obtains higher results than BERT (Devlin et al., 2019) by 0.7% on SST-2.
Revised: In this experiment, BERT+TextTN achieves the comparable result with the model ELECTRA (Clark et al., 2020), and obtains higher results than BERT (Devlin et al., 2019) by 0.7% on SST-2.

Original: For SST-5 dataset, BERT+TextTN outperforms the state-of-the-art results 54.7 by 0.1%, and is higher than BERT (Devlin et al., 2019) by 1.9%.
Revised: For SST-5 dataset, BERT+TextTN outperforms the state-of-the-art results 54.7 by 0.1%, and is higher than BERT (Devlin et al., 2019) by 1.9%.

Original: In order to verify the effectiveness of all-function learning and word-GTNs, we perform ablation experiments shown as Table 3.
Revised: In order to verify the effectiveness of all-function learning and word-GTNs, we perform ablation experiments shown as Table 3.

Original: First, when TextTN is without the all-function learning (TextTN w/o all-function), the classiﬁcation accuracy declines to varying degrees on the four data sets.
Revised: First, when TextTN is without the all-function learning (TextTN w/o all-function), the classiﬁcation accuracy declines to varying degrees on the four data sets.

Original: The accuracy values on MR, CR, Subj and MPQA drop by 1.7%, 1.6%, 0.9% and 0.6%, respectively.
Revised: The accuracy values on MR, CR, Subj and MPQA drop by 1.7%, 1.6%, 0.9% and 0.6%, respectively.

Original: The results verify that the method of all-function learning for text classiﬁcation is effective and useful.
Revised: The results verify that the method of all-function learning for text classiﬁcation is effective and useful.

Original: In addition, in Table 3, we evaluate the effectiveness of word-GTNs by ablation experiments.
Revised: In addition, in Table 3, we evaluate the effectiveness of word-GTNs by ablation experiments.

Original: TextTN w/o w-GTNs means that we directly average the word vectors of words in a sentence to obtain the sentence representation.
Revised: TextTN w/o w-GTNs means that we use a linear function of the word embeddings to obtain the low-dimensional word representations, which are inptted to the sentence-DTN.

Original: On the contrary, TextTN uses word-GTNs to generate sentence representations.
Revised: On the contrary, TextTN uses word-GTNs to generate sentence representations.

Original: Compared with TextTN, the accuracy of TextTN w/o w-GTNs are largely reduced on four dataset.
Revised: Compared with TextTN, the accuracy of TextTN w/o w-GTNs are largely reduced on four dataset.

Original: In particular, CR drops by 11.4%, MR, MPQA, and Subj decrease by 7.1%, 5.3%, and 1.6%, respectively.
Revised: In particular, CR drops by 5.5%, MR, MPQA, and Subj decrease by 3.6%, 3.1%, and 2.4%, respectively.

Original: This shows that word-GTNs can effectively model word dependence in sentences and provide more effective semantic information for the classiﬁer.
Revised: This shows that word-GTNs can effectively model word dependence in sentences and provide more effective semantic information for the classiﬁer.

Original: Finally, we analyze the impact of different word vector dimensions on the performance of TextTN, and the results are shown in the Appendix D.2.
Revised: Finally, we analyze the impact of different word vector dimensions on the performance of TextTN, and the results are shown in the Appendix D.2.

Original: In addition, in the Section 3.1, the dimension m of probability encoding is a hyper-parameter set to 2 .
Revised: In addition, in the Section 3.1, the dimension m of probability encoding is a hyper-parameter set to 2 .

Original: In order to verify this choice, we conduct a comparative experiment, and the results in Appendix D.3 illustrate that compared with m = 2 , the accuracy of classiﬁcation with m > 2 has not been improved, or even decreased.[Figure] g l e m e n t E n t r o p y CR    80 Epoch 4.
Revised: In order to verify this choice, we conduct a comparative experiment, and the results in Appendix D.3 illustrate that compared with m = 2 , the accuracy of classiﬁcation with m > 2 has not been improved, or even decreased.

Original: E n t a n g l e m e n t E n t r o p y

Original: MPQA    80 Epoch 4.

Original: E n t a n g l e m e n t E n t r o p y

Original: Subj

Original: MR

Original: CR MPQA S ubj 4. E VALUATION OF THE R EFERENCE B OND - DIMENSION OF T EXT TN
Revised: 4. E VALUATION OF THE R EFERENCE B OND - DIMENSION OF T EXT TN

Original: In Theorem 1, we show that the lower-bound of the bond-dimension can be derived by the entanglement entropy, and then consider such a lower bound can be the reference bond-dimension of TextTN.
Revised: In Theorem 1, we show that the lower-bound of the bond-dimension can be derived by the entanglement entropy, and then consider such a lower bound can be the reference bond-dimension of TextTN.

Original: In order to evaluate such a reference hyper-parameter value, we designed experiments as follows.
Revised: In order to evaluate such a reference hyper-parameter value, we designed experiments as follows.

Original: First, we can compute the entanglement entropy through the initialized training of TextTN on text classiﬁcation datasets (see Appendix D.4 for details).
Revised: First, we can compute the entanglement entropy through the initialized training of TextTN on text classiﬁcation datasets (see Appendix D.4 for details).

Original: After that, based on the entanglement entropy, the reference bond-dimension can be computed.
Revised: After that, based on the entanglement entropy, the reference bond-dimension can be computed.

Original: For MR, CR, MPQA and Subj datasets, their accuracy scores are highest when the bond-dimensions of sentence-DTN are 21 , 18 , 19 and 20 , respectively.
Revised: For MR, CR, MPQA and Subj datasets, their accuracy scores are highest when the bond-dimensions of sentence-DTN are 21 , 18 , 19 and 20 , respectively.

Original: These values (i.e., 21 , 18 , 19 and 20 ), are actually the reference bond-dimensions, which are computed based on the entanglement entropy from the initialized training.
Revised: These values (i.e., 21 , 18 , 19 and 20 ), are actually the reference bond-dimensions, which are computed based on the entanglement entropy from the initialized training.

Original: Speciﬁcally, these values can be computed by Eq.
Revised: Speciﬁcally, these values can be computed by Eq.

Original: 11, i.e., (cid:98) 2 4 .
Revised: 11, i.e., (cid:98) 2 4 .

Original: 40 (cid:99) = 21 , (cid:98) 2 4 .
Revised: 40 (cid:99) = 21 , (cid:98) 2 4 .

Original: 20 (cid:99) = 18 , (cid:98) 2 4 .
Revised: 20 (cid:99) = 18 , (cid:98) 2 4 .

Original: 32 (cid:99) =
Revised: 32 (cid:99) =

Original: 19 and (cid:98) 2 4 .
Revised: 19 and (cid:98) 2 4 .

Original: 37 (cid:99) = 20 , respectively.
Revised: 37 (cid:99) = 20 , respectively.

Original: In Figure 3, four charts show that the accuracy values reach the highest point when the reference bond-dimensions are computed as 21 (MR), 18 (CR), 19 (MPQA) and 20 (Subj).
Revised: In Figure 3, four charts show that the accuracy values reach the highest point when the reference bond-dimensions are computed as 21 (MR), 18 (CR), 19 (MPQA) and 20 (Subj).

Original: As shown in Figure 3, by reducing and increasing the bond-dimension with respect to its reference value, we also observe interesting results about the accuracy.
Revised: As shown in Figure 3, by reducing and increasing the bond-dimension with respect to its reference value, we also observe interesting results about the accuracy.

Original: For example, in the MR task, when bond dimension ≤ 21 accuracy is less than the accuracy when the bond-dimension is the reference value 21 .
Revised: For example, in the MR task, when bond dimension ≤ 21 accuracy is less than the accuracy when the bond-dimension is the reference value 21 .

Original: If bond dimension ≥ 21 , the accuracy begins to decline, but the updated entanglement entropy value (computed by the non-zero single values of sentence-DTN) remains stably.
Revised: If bond dimension ≥ 21 , the accuracy begins to decline, but the updated entanglement entropy value (computed by the non-zero single values of sentence-DTN) remains stably.

Original: The same results are shown on the other three data sets, i.e., CR, MPQA and Subj.
Revised: The same results are shown on the other three data sets, i.e., CR, MPQA and Subj.

Original: In summary, the above results show the effectiveness of the reference bond-dimension.
Revised: In summary, the above results show the effectiveness of the reference bond-dimension.

Original: However, a more rigorous analysis about the reference bond-dimension and the classiﬁcation effectiveness of
Revised: However, a more rigorous analysis about the reference bond-dimension and the classiﬁcation effectiveness of

Original: TextTN is still an open research question.
Revised: TextTN is still an open research question.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this paper, we propose a text tensor network, named as TextTN, which aims at encoding each word (in a sentence) from a high dimensional vector to a low-dimensional vector by word-GTNs.
Revised: In this paper, we propose a text tensor network, named as TextTN, which aims at encoding each word (in a sentence) from a high dimensional vector to a low-dimensional vector by word-GTNs.

Original: After that, the new sentence representations can be inputted to the sentence-DTN to carry out the sentence classiﬁcation.
Revised: After that, the new sentence representations can be inputted to the sentence-DTN to carry out the sentence classiﬁcation.

Original: The proposed TextTN achieves better results than CNN, and even better experimental results than some neural network models, such as HCapsNet.
Revised: The proposed TextTN achieves better results than CNN, and even better experimental results than some neural network models, such as HCapsNet.

Original: In addition, TextTN also achieves the experimental results that largely exceeds GTNs in all datasets.
Revised: In addition, TextTN also achieves the experimental results that largely exceeds GTNs in all datasets.

Original: The entanglement entropy is deﬁned in TextTN and is helpful to identify the the reference bond-dimension which usually achieves the best accuracy in our experiments.
Revised: The entanglement entropy is deﬁned in TextTN and is helpful to identify the the reference bond-dimension which usually achieves the best accuracy in our experiments.

Original: In the future, we would like to further investigate more theoretical evidence and understanding of using of the entanglement entropy to set an optimal bound dimension.
Revised: In the future, we would like to further investigate more theoretical evidence and understanding of using of the entanglement entropy to set an optimal bound dimension.

Original: In the practical point of view, we can apply TextTN on more NLP tasks and test the effectiveness of both TextTN and its reference hyper-parameters.
Revised: In the practical point of view, we can apply TextTN on more NLP tasks and test the effectiveness of both TextTN and its reference hyper-parameters.

Original: A B ASIC K NOWLEDGE ABOUT T ENSOR N ETWORK (TN)
Revised: A B ASIC K NOWLEDGE ABOUT T ENSOR N ETWORK (TN)

Original: We now describe some basic knowledge on tensor and tensor network, as well as the matrix product states (MPS).
Revised: We now describe some basic knowledge on tensor and tensor contraction as well as the matrix product states (MPS).

Original: Tensor :
Revised: A. T ENSOR AND T ENSOR C ONTRACTION

Original: A tensor can simply be thought of as multi-dimensional array.
Revised: A tensor can simply be thought of as multi-dimensional array.

Original: The order of a tensor is deﬁned to the number of indexing entries in the array.
Revised: The order of a tensor is deﬁned to the number of indexing entries in the array.

Original: For example, in Figure 4 ( a ) , the vector v with the index i ∈[ m] 2  is a one-order tensor, and the matrix A with two indexes (or nodes) i ∈ 
Revised: For example, in Figure 4a the vector v with the index i ∈ m 2 , is a one order tensor and the matrix A with two indexes (or nodes) i ∈ m and j ∈ n is a two-order tensor.

Original: [ m ] and j ∈ [ n ] is a two-order tensor.
Revised: After that, a n-order tensor is drawn in Figure with n nodes, d 1 , d 2 , ..., and d n , where each node corresponds to an input on TN. Tensors have some basic operations. The most fundamental one is tensor contradiction, which means summing over same indices of two tensors. Two examples are presented here in order to illustrate the operation more directly.

Original: After that, a n -order tensor is drawn in Figure 4 ( a ) with n nodes, d 1 , d 2 , . . .
Revised: In Figure 4a, we show the process of a matrix (2-order tensor) contracting with a vector (1-order tensor). In Figure 4b, another contraction process between two matrix A and

Original: and d n , where each node corresponds to an input on TN.
Revised: B is demonstrated. And those high-order tensors, the tensor contraction operations between them follow the same rules with the above examples.

Original:  Tensor Network(TN): A TN as a weighted graph, where each node corresponds to a tensor.
Revised: A Tensor Network as a weighted graph, where each node corresponds to a tensor.

Original: The order of the tensor is the degree of the node in the graph.
Revised: It has a linear architecture with a list of nodes and edges, and the order of the tensor is the degree of the node in the graph.

Original: The weight of each edge is referred to as its bond dimension.
Revised: The weight of each edge is referred to as its bond dimension.

Original: A fundamental operator in tensor analysis is the tensor product, which is denoted as ⊗ .
Revised: In (Novikov et al., 2016; Han et al., 2018)), MPS is a kind of TN, which is decomposed by a high-order tensor through Tensor-Train decomposition (Oseledets, 2011). First, as shown in Figure 5a, the high-order tensor

Original: It is an operator that intakes two tensors (or two vectors).
Revised: W is decomposed into multiple low-order tensors.

Original: Matrix Product State : MPS is a kind of TN, which is showed in Figure 4( b ). A n -order weight tensor W in Figure 4( b ) can be decomposed into a MPS, which has n nodes.
Revised: Then, through tensor contraction, the low-order tensors represented by nodes can be contracted into a MPS network in Figure 5b. The formula is as follows: [Equation]

Original: In the MPS, α 1 , α 2 , . . .
Revised: In the MPS, α 1 , α 2 , . . .

Original: , α n − 1 are the indexes of bond dimensions (also named as TT-rank in some works (Novikov et al., 2016; Han et al., 2018)).
Revised: , α N − 1 are the indexes of bond_dimensions (also named as TT-rank in some works), d i ∈ R m ( i ∈ [ N ]) .

Revised: If the low-order tensors is not trimmed in the process of TT decomposition, that is, the full rank is guaranteed, then α 1 ∈ R m , α 2 ∈ R m 2 , α (cid:98) N/ 2 (cid:99) ∈ R m (cid:98) N/ 2 (cid:99) , . . .

Revised: , α N ∈ R m .

Revised: That is, the maximum bond-dimension is m (cid:98) N/ 2 (cid:99) .

Revised: Therefore, we avoid directly using the high-order tensor and Tensor-Train decomposition due to its exponentially increasing parameters.

Original: B P ROOF OF T HEOREM
Revised: During the practical process, we do not construct a highorder tensor, nor do we perform tensor decomposition. Instead, we usually initialize an MPS architecture with random generated low-order tensors, optimizing them while training in order to better approximate feature distribution of data (which can be understood as a high-order tensor). Within the allowable deviation range, the amount of parameters will be dramatically reduced from m N to N × m × r × r , and r is the value of bond-dimension. [Equation]

Revised: [Figure]

Revised: W

Revised: [Figure]

Revised: ⋯ 𝑑𝑑𝑑 𝑁 𝛼𝛼𝛼 𝑁− 𝐴𝐴𝐴 𝑁

Revised: MPS ⋯ 𝑑𝑑𝑑 𝑁 𝛼𝛼𝛼𝛼 𝑁−1 𝐴𝐴𝐴 𝑁 ⋯ 𝑑𝑑𝑑 𝑁 𝛼𝛼𝛼𝛼 𝑁−1 𝐴𝐴𝐴 𝑁 ⋯ 𝑑

Revised: [Figure]

Revised: [Figure]

Original: In TextTN, the bond-dimension is an important hyper-parameter which directly inﬂuences the expressive power of tensor network models (i.e., TextTN).
Revised: [Figure] 𝟐𝟐′ ( 𝑱𝑱 [Figure] 𝒈𝒈 𝟑𝟑′ ( 𝑱𝑱 ) 𝒈𝒈 𝟒𝟒′ ( 𝑱𝑱 )

Original: In general, the larger bond-dimension a TN model has, the stronger expressive power the TN model will be.
Revised: [Figure] 𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄 ∗ 𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄

Original: However, TN models with too large expressive power are not to improve the effective of model.{+[Figure] 𝒏𝒏 /𝑔𝑔𝑔𝑔𝑔𝑔𝑔𝑔0 00   
+}
Revised: {+[Figure] 𝒏𝒏 /𝑔𝑔𝑔𝑔𝑔𝑔𝑔𝑔0 00   
+}

Original: In order to ﬁnd a proper bond-dimension to model TextTN for achieving a higher accuracy, we can compute thereference bond-dimension by the entanglement entropy.
Revised: In TextTN, the bond-dimension is an important hyper-parameter which directly inﬂuences the expressive power of tensor network models (i.e., TextTN).

Original: The entanglement entropy in TextTN have been deﬁned in Deﬁnition 1. The entanglement entropy can be computed in the process of TextTN training.
Revised: An reference bond-dimension can be given out based on the entanglement entropy which is deﬁned in Deﬁnition 1. ,

Original: After that, we need to ﬁnd the smallest bond-dimension that can effectively characterize the entanglement entropy. The smallest bond-dimension is a reference bond-dimension in TextTN. The Theorem 1 shows how to compute a reference bond-dimension. 
Revised: Therefore, The Theorem 1 shows the relation between the entanglement entropy and the bond-dimension, and gives out the lower-bond and the upper-bond of the bond-dimension. The proof of inequality 15 is showed here.

Original: Theorem 1.
Revised: Theorem 1.

Original: For the input of the sentence-DTN { v 1 , . . .
Revised: For the input of the sentence-DTN { v 1 , . . .

Original: , v n } , the bond- dimension is bounded as
Revised: , v n } , the bond- dimension is bounded as

Original: [Equation]
Revised: [Equation]

Original: ∈ { 1 , 2 , . . . , m } [Figure] 𝐴𝐴 𝐖𝐖 𝑖𝑖 ∈ [ 𝑚𝑚 ]
Revised: where E is the entanglement entropy of TextTN, m is the inputting dimension of sentence-DTN, n is the length of a sentence, and (cid:98) (cid:99) indicates rounding down.

Original: [Figure]

Original: ∈

Original: [Figure]

Original: 𝑛𝑛 ] 𝑖𝑖 ∈

Original: [ 𝑚𝑚 ]

Original: [Figure]

Original: ⋯ 𝐖𝐖

Original: [Figure] ≈ 𝑑𝑑 [Figure] ⋯ ⋯
Revised: Proof.

Original: 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴 𝑛𝑛 𝛼𝛼 [Figure] 𝛼𝛼 [Figure]
Revised: As for the upper bound of the Eq. 15, as shown in Appendix A.2, sentence-DTN as a

Original: 𝛼𝛼

Original: [Figure] [Figure] [Figure] [Figure]
Revised: MPS tensor network can be represented by a decomposed n-order tensor.

Original: 𝟐𝟐 ( 𝑰𝑰 ) 𝒈𝒈 𝟑𝟑 ( 𝑰𝑰 ) 𝒈𝒈 𝟒𝟒 ( 𝑰𝑰 ) 𝒈𝒈 𝟏𝟏′ ( 𝑱𝑱 ) [Figure] ⋯ [Figure]
Revised: If no rank is cut during decomposition, the bond-dimension (TT-rank) of sentence-DTN is the same as the n -order tensor.

Original: ∗ 𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄 [Figure] ⋯ [Figure]
Revised: Suppose the formula of n-order tensor is:

Original: where E is the entanglement entropy of TextTN, m is the inputting dimension of sentence-DTN, and (cid:98) (cid:99) indicates rounding down, the reference bond-dimension is equal to (cid:98) 2 E (cid:99) .

Original: Proof.
Revised: [Equation]

Original: As shown in Eq.
Revised: 16) v i is the word probability encodings shown in Eq.7. T is a m n tensor. From the matrix maximum rank theorem, the maximum rank in T appears at the position (cid:98) n/ 2 (cid:99) , that is, the maximum rank is m (cid:98) n/ 2 (cid:99) . Thus, the upper bound of bond-dimension in sentence-DTN is also m (cid:98) n/ 2 (cid:99) .

Original: 14, to calculate the reference bond dimension for TextTN, the entanglement entropy  is ﬁrst computed, which is deﬁned in the Deﬁnition 1.
Revised: As for the lower bound of Eq. 15, the entanglement entropy E is ﬁrst computed by the ﬁrst initialization training.

Original: Practically, the singular values between v (cid:98) n/ 2 (cid:99) and v (cid:98) n/ 2 (cid:99) +1 in Figure 5 can be obtained in the training process of TextTN, when TextTN converges.
Revised: For the lower bond of bond-dimension, practically, the singular values between v (cid:98) n/ 2 (cid:99) and v (cid:98) n/ 2 (cid:99) +1 in Figure 6 can be obtained in the training process of TextTN, when TextTN converges.

Original: Assume that bond − dimension of TextTN is equal to k .
Revised: Assume that bond − dimension of TextTN is equal to k .

Original: The maximal entanglement entropy is computed when the k singular values in TextTN are equal.
Revised: The maximal entanglement entropy is computed when the k singular values in TextTN are equal.

Original: Eq.
Revised: Eq.

Original: 15 gives out the maximal entanglement entropy based on a ﬁxed bond-dimension k .
Revised: 17 gives out the maximal entanglement entropy based on a ﬁxed bond-dimension k .

Original: [Equation]
Revised: [Equation]

Original: On both sides of the equation, doing the exponential calculations in base 2 on both sides of the equation, the new equation can be computed as follows, 
Revised: On both sides of the equation, doing the exponential calculations in base 2 on both sides of the equation, the new equation can be computed as follows, [Equation]

Original: [Equation]
Revised: [Equation]

Original: When k singular values are not equal, the inequality can be written as
Revised: When k singular values are not equal, the inequality can be written as

Original: [Equation]
Revised: [Equation]

Revised: To ensure that bond − dimension represented by k , is an integer, the inequality can rewritten as k ≥ (cid:98)

Original: [Equation]
Revised: [Equation]

Original: To ensure that bond − dimension represented by k , is an integer, the inequality can rewritten as k ≥ (cid:98) 2 E (cid:99) , where (cid:98) (cid:99) indicates rounding down.
Revised: (cid:99) [Equation] where (cid:98) (cid:99) indicates rounding down.

Original: Therefore, the inequality bond − dimension ≥ (cid:98) 2 E (cid:99) in Eq 11 can be established.
Revised: Therefore, the inequality bond − dimension ≥ (cid:98) 2 E (cid:99) in Eq 15 can be established.

Original: Recall that the entanglement entropy E computed by Eq.
Revised: Recall that the entanglement entropy E computed by Eq.

Original: 15 represents the maximal amount of information that
Revised: 17 represents the maximal amount of information that

Original: TextTN can model.
Revised: TextTN can model.

Original: Eq.
Revised: Eq.

Original: 17 implies that the maximal entanglement entropy E determines the lower bound of the bond-dimension of TextTN.
Revised: 19 implies that the maximal entanglement entropy E determines the lower bound of the bond-dimension of TextTN.

Original: If the bond-dimension of TextTN is less than (cid:98) 2 E (cid:99) , the  maximal amount of information cannot be achieved by TextTN. If the bond-dimension of TextTN is greater than (cid:98)  2E (cid:99) , TextTN will not model more information than E , since E is the maximal entanglement entropy; In this case, the TextTN will be more complicated.
Revised: For the upper bond, a parameter tensor W ∈ R m n can be viewed as a matrix with the shape m i × m n − i . The maximal rank can be obtained when the parameter tensor W is viewed as the matrix with the shape m (cid:98) n/ 2 (cid:99) × m n −(cid:98)

Original: Then, with no more information modeled but more complicated model, it is possible to cause the overﬁtting problem. Therefore, the reference bond-dimension is equal to  (cid:98)  2E (cid:99) .
Revised: n/ 2 (cid:99) . In this case, the value of maximal rank can be equal to m (cid:98) n/ 2 (cid:99) , which is the upper bond of bond-dimension. Therefore, the upper bond and lower bond of bond-dimension be obtained.

Original: Theorem 1 has also been validated in practice.
Revised: Theorem 1 has also been validated in practice.

Original: In the process of TextTN training, we can compute the entanglement entropy of TextTN according to the Deﬁnition 1.
Revised: In the process of TextTN training, we can compute the entanglement entropy of TextTN according to the Deﬁnition 1.

Original: When this training process converges, the maximal entanglement entropy is obtained.
Revised: When this training process converges, the maximal entanglement entropy is obtained.

Original: In addition, TextTNs with different bond-dimensions are experimented.
Revised: In addition, TextTNs with different bond-dimensions are experimented.

Original: At the beginning, the classiﬁcation accuracy of TextTN increases with the increasing of the bond-dimension.
Revised: At the beginning, the classiﬁcation accuracy of TextTN increases with the increasing of the bond-dimension.

Original: When the bond-dimension increases to a certain value, the accuracy of TextTN begins to decline slowly.
Revised: When the bond-dimension increases to a certain value, the accuracy of TextTN begins to decline slowly.

Original: Such a certain value is the reference bond-dimension and can be computed by (cid:98) 2 E (cid:99) , when the training process converges.
Revised: Such a certain value is the reference bond-dimension and can be computed by (cid:98) 2 E (cid:99) , when the training process converges.

Original: C P ROOF OF C LAIM
Revised: C P ROOF OF C LAIM

Original: Claim 1 For an input S
Revised: Claim 1 For an input S

Original: = { v 1 , v 2 , . .
Revised: = { v 1 , v 2 , . .

Original: . , v n } , the original DTN (Stoudenmire & Schwab, 2016) computes the conditional probability P ( y | S ) , which can be considered as a  function from a set T of functions which compute all the possible conditional probabilities.
Revised: . , v n } , the original DTN (Stoudenmire & Schwab, 2016) computes the conditional probability P ( y | S ) by putting the label bound l in a certain position, which can be considered as a certain function from a set T of functions which compute all the possible conditional probabilities.

Original: Such a set is deﬁned as:
Revised: Such a set is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where i the location of segmentation for the input S , and is also a position of label  l .
Revised: where i the location of segmentation for the input S , and is also a position of label bound l .

Original: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: For an input S = { v 1 , . . .
Revised: For an input S = { v 1 , . . .

Original: , v n } , which is from the output of word-DTNs, a n -order tensor, i.e., the input of the sentence-DTN, is given by the operator of tensor product between vectors, v i , i ∈ { 1 , . . . , n } .
Revised: , v n } , which is from the output of word-DTNs, a n -order tensor, i.e., the input of the sentence-DTN, is given b y the operator of tensor product between vectors, v i , i ∈ { 1 , . . . , n } .

Original: The parameter tensor W l in DTN is a ( n + 1) -order tensor.
Revised: The parameter tensor W l in DTN is a ( n + 1) -order tensor.

Original: There is a mode (or an index) in tensor W l to control the number of class in the classiﬁcation task, which is l .
Revised: There is a mode (or an index) in tensor W l to control the number of class in the classiﬁcation task, which is l .

Original: As shown in
Revised: As shown in

Original: Figure 6 ( a ) , in the processing of training, the index l is moved from S 1 to S n .
Revised: Figure 7 ( a ) , in the processing of training, the index l is moved from S 1 to S n .

Original: When the position of l is S 1 , the input S is split two parts, which are I = { v 1 } and J = { v 2 , . .
Revised: When the position of l is S 1 , the input S is split two parts, which are I = { v 1 } and J = { v 2 , . .

Original: . , v n } , the basic events are I and J , the classiﬁcation function f 1 ( S ) can be represented by condition probability as follows,[Equation]
Revised: . , v n } , the basic events are I and J , the classiﬁcation function f 1 ( S ) can be represented by condition probability as follows,

Original: [Equation]
Revised: [Equation]

Original: When the position of separation i is 1 < i < n , the input S is separated into two parts, which are
Revised: When the position of separation i is 1 < i < n , the input S is separated into two parts, which are

Original: I = { v 1 , . .
Revised: I = { v 1 , . .

Original: . , v i } and J = { v i +1 , . . .
Revised: . , v i } and J = { v i +1 , . . .

Original: , v n } , the function f 1 ( S ) is represented
Revised: , v n } , the function f 1 ( S ) is represented

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: When the position of separation i is i = n − 1 , the function f 1 ( S ) is denoted Eq.
Revised: When the position of separation i is i = n − 1 , the function f 1 ( S ) is denoted Eq.

Original: 21 similar to Eq.and Eq.
Revised: 23 similar to Eq.and Eq.

Original: [Equation]
Revised: [Equation]

Original: For different functions f i ( S ) , they represent different condition probabilities.
Revised: For different functions f i ( S ) , they represent different condition probabilities.

Original: In the existing work (Stoudenmire & Schwab, 2016), i.e., original DTN, one function is used to classify the input S .
Revised: In the existing work (Stoudenmire & Schwab, 2016), i.e., original DTN, one function is used to classify the input S .

Original: In order to model all functions in the process of training, and one function learning cannot model all conditional probabilities in the input S , we propose all-function learning method for text classiﬁcation, which is showed in Figure 6 ( b ) .
Revised: In order to model all functions in the process of training, and one function learning cannot model all conditional probabilities in the input S , we propose all-function learning method for text classiﬁcation, which is showed in Figure 7 ( b ) .

Original: We adopt weighted summation for these functions (i.e., f ( S ) =
Revised: We adopt weighted summation for these functions (i.e., f ( S ) =

Original: (cid:80) ni
Revised: (cid:80) ni

Original: =

Revised: =

Original: g i f i ( S ) , where g is a parameter vector, g i is an element from the vector g ).
Revised: g i f i ( S ) , where g is a parameter vector, g i is an element from the vector g ).

Original: D E XPERIMENTAL R ESULTS AND A NALYSIS
Revised: D E XPERIMENTAL R ESULTS AND A NALYSIS

Original: In this section, we ﬁrst describe the datasets used in our experiments.
Revised: In this section, we ﬁrst describe the datasets used in our experiments.

Original: D. D ATASETS
Revised: D. D ATASETS

Original: The statistics of six datasets are showed in Table 4.
Revised: The statistics of six datasets are showed in Table 4.

Original: For CR, MR, MPQA and Subj datasets, they all have a total ﬁle.
Revised: For CR, MR, MPQA and Subj datasets, they all have a total ﬁle.

Original: Therefore, we randomly select 10% of the ﬁle as the validation set and test set, respectively, and select 80% of the ﬁle as the train set.
Revised: Therefore, we randomly select 10% of the ﬁle as the validation set and test set, respectively, and select 80% of the ﬁle as the train set.

Original: The evaluating indicator of all text classiﬁcation tasks is the "Accuracy".
Revised: The evaluating indicator of all text classiﬁcation tasks is the "Accuracy".

Original: For CR, MR, MPQA and Subj tasks, we implement our model with Pytorch-1.20, and train them on a Nvidia P40 GPU.
Revised: For CR, MR, MPQA and Subj tasks, we implement our model with Pytorch-1.20, and train them on a Nvidia P40 GPU.

Original: As for learning method, we use the Adam optimizer Kingma & Ba (2015) and an exponentially decaying learning rate with a linear warm up.
Revised: As for learning method, we use the Adam optimizer Kingma & Ba (2015) and an exponentially decaying learning rate with a linear warm up.

Original: For SST-2 and SST-5 dataset, the initialized learning rate is 5e-5. 
Revised: For SST-2 and SST-5 dataset, the initialized learning rate is 5e-5. [Table] [Figure] n

Original:  D. C OMPARISON BETWEEN D IFFERENT D IMENSIONS OF P RE - TRAINED W ORD V ECTORS
Revised: [Figure] n [Figure] D. C OMPARISON BETWEEN D IFFERENT D IMENSIONS OF P RE - TRAINED W ORD V ECTORS

Original: As shown in Table 5, we compare the accuracy of TextTN using word2vec (300-dimension) and
Revised: As shown in Table [Figure] we compare the accuracy of TextTN using word2vec (300-dimension) and

Original: Bert_large (1024-dimension) pre-trained word vectors,respectively, for nine text classiﬁcation tasks (e.g., MR, CR and SST-5).
Revised: Bert_large (1024-dimension) pre-trained word vectors,respectively, for nine text classiﬁcation tasks (e.g., MR, CR and SST-5).

Original: The results show that the accuracy for six tasks will have a signiﬁcantly increase as the dimension of word representations increases in TextTN.
Revised: The results show that the accuracy for six tasks will have a signiﬁcantly increase as the dimension of word representations increases in TextTN.

Original: Especially for SST-5 and MR, the results of 1024-dimensional word embedding has over 5% increases compared with the results of 300-dimensional word embedding.
Revised: Especially for SST-5 and MR, the results of 1024-dimensional word embedding has over 5% increases compared with the results of 300-dimensional word embedding.

Original: Comparative experiments show that TextTN has better learning effects for high-dimensional word representations containing more semantic information.[Table] [Figure] n t r o p
Revised: Comparative experiments show that TextTN has better learning effects for high-dimensional word representations containing more semantic information.

Original: [Figure]
Revised: [Table]

Original: g l e m e n t E n t r o p y

Original: CR    80 Epoch 4.

Original: E n t a n g l e m e n t E n t r o p y

Original: MPQA    80 Epoch 4.

Original: E n t a n g l e m e n t E n t r o p y

Original: Subj

Original: [Table]
Revised: [Table]

Original: D. A E XPERIMENT ABOUT T HE D IMENSION OF P ROBABILITY E NCODING
Revised: D. A E XPERIMENT ABOUT T HE D IMENSION OF P ROBABILITY E NCODING

Original: In Section 3.1, we analyze that the dimension m of probability encoding can not exceed 2.
Revised: In Section 3.1, we analyze that the dimension m of probability encoding can not exceed 2.

Original: In this experiment, we evaluate the conclusion.
Revised: In this experiment, we evaluate the conclusion.

Original: As shown in Table 6, we compare the accuracy of the TextTN with different probability encoding dimensions set from 2 to 10.
Revised: As shown in Table 6, we compare the accuracy of the TextTN with different probability encoding dimensions set from 2 to 10.

Original: In addition, reported results are the average of 10 runs.
Revised: In addition, reported results are the average of 10 runs.

Original: The results illustrate that when m = 2 , the accuracy of classiﬁcation is 95.3, and when m > 2 , the accuracy has a signiﬁcant decrease.
Revised: The results illustrate that when m = 2 , the accuracy of classiﬁcation is 95.3, and when m > 2 , the accuracy has a signiﬁcant decrease.

Original: In particular, the accuracy only achieves 94.
Revised: In particular, the accuracy only achieves 94.

Original: when m = 5 , dropping by 1.1%.
Revised: when m = 5 , dropping by 1.1%.

Original: The Experimental results verify the effectiveness of m = 2 .
Revised: The Experimental results verify the effectiveness of m = 2 .

Original: D. T HE I NITIALIZED T RAINING ON T EXT TN
Revised: D. T HE I NITIALIZED T RAINING ON T EXT TN

Original: In order to get the entanglement entropy  in the initialized training, we set 80 epochs to training the TextTN.
Revised: In order to get the entanglement entropy shown as Eq.10(in this paper) in the initialized training, we set 80 epochs to training the TextTN on four text classiﬁcation tasks (i.e., MR, CR, MPQA and Subj).

Original: In this process, the entanglement entropy, which is computed according to Determinate 1, gradually converges to a determinate value as the increasing of training epoch.
Revised: In this process, the entanglement entropy, which is computed according to Deﬁnition 1, gradually converges to a determinate value as the increasing of training epoch, which shown as Figure 8.