Original: B AG OF REGION EMBEDDINGS VIA LOCAL CONTEXT UNIT FOR TEXT CLASSIFICATION
Revised: B AG OF REGION EMBEDDINGS VIA LOCAL CONTEXT UNITS FOR TEXT CLASSIFICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Contextual information and word orders are proved valuable for text classiﬁcation task.
Revised: Contextual information and word orders are proved valuable for text classiﬁcation task.

Original: To make use of local word order information, n-grams are commonly used features in several models, such as linear models.
Revised: To make use of local word order information, n-grams are commonly used features in several models, such as linear models.

Original: However, these models commonly suffer the data sparsity problem and are difﬁcult to represent large size region.
Revised: However, these models commonly suffer the data sparsity problem and are difﬁcult to represent large size region.

Original: The discrete or distributed representations of n-grams can be regarded as region embeddings, which are representations of ﬁxed size regions.
Revised: The discrete or distributed representations of n-grams can be regarded as region embeddings, which are representations of ﬁxed size regions.

Original: In this paper, we propose two novel text classiﬁcation models that learn task speciﬁc region embeddings without hand crafted features, hence the drawbacks of n-grams can be overcome.
Revised: In this paper, we propose two novel text classiﬁcation models that learn task speciﬁc region embeddings without hand crafted features, hence the drawbacks of n-grams can be overcome.

Original: In our model, each word has two attributes, a commonly used word embedding, and an additional local context unit which is used to interact with the word’s local context.
Revised: In our model, each word has two attributes, a commonly used word embedding, and an additional local context unit which is used to interact with the word’s local context.

Original: Both the units and word embeddings are used to generate representations of regions, and are learned as model parameters.
Revised: Both the units and word embeddings are used to generate representations of regions, and are learned as model parameters.

Original: Finally, bag of region embeddings of a document is fed to a linear classiﬁer.
Revised: Finally, bag of region embeddings of a document is fed to a linear classiﬁer.

Original: Experimental results show that our proposed methods achieve state-of-the-art performance on several benchmark datasets.
Revised: Experimental results show that our proposed methods achieve state-of-the-art performance on several benchmark datasets.

Original: We provide visualizations and analysis illustrating that our proposed local context unit can capture the syntactic and semantic information.
Revised: We provide visualizations and analysis illustrating that our proposed local context unit can capture the syntactic and semantic information.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Text classiﬁcation is an important task in natural language processing, which has been studied for years with applications such as topic categorization, web search, sentiment classiﬁcation.
Revised: Text classiﬁcation is an important task in natural language processing, which has been studied for years with applications such as topic categorization, web search, sentiment classiﬁcation.

Original: A simple yet effective approach for text classiﬁcation is to represent documents as bag-of-words, and train a linear classiﬁer, e.g., a logistic regression, a support vector machines (Joachims, 1998; Fan et al., 2008) or a naive bayes (McCallum et al., 1998).
Revised: A simple yet effective approach for text classiﬁcation is to represent documents as bag-of-words, and train a linear classiﬁer, e.g., a logistic regression, a support vector machines (Joachims, 1998; Fan et al., 2008) or a naive bayes (McCallum et al., 1998).

Original: Although bag-of-words methods are efﬁcient and popular, they lose the local ordering information of words which has been proved useful particularly on sentiment classiﬁcation (Pang et al., 2002).
Revised: Although bag-of-words methods are efﬁcient and popular, they lose the local ordering information of words which has been proved useful particularly on sentiment classiﬁcation (Pang et al., 2002).

Original: To beneﬁt from local word order for text classiﬁcation, n-grams which is a simple statistics of ordered word combinations usually performs good (Pang et al., 2002;
Revised: To beneﬁt from local word order for text classiﬁcation, n-grams which is a simple statistics of ordered word combinations usually performs good (Pang et al., 2002;

Original: Wang & Manning, 2012; Joulin et al., 2016).
Revised: Wang & Manning, 2012; Joulin et al., 2016).

Original: With a n-gram vocabulary, Wang & Manning (2012) use one-hot vectors as discrete embedding of n-grams.
Revised: With a n-gram vocabulary, Wang & Manning (2012) use one-hot vectors as discrete embedding of n-grams.

Original: Recently, FastText (Joulin et al., 2016) directly learns distributed embeddings of n-grams and words for the task of interest.
Revised: Recently, FastText (Joulin et al., 2016) directly learns distributed embeddings of n-grams and words for the task of interest.

Original: In FastText, learning the embedding of n-gram can be regarded as learning low-dimensional vector representation of a small ﬁxed size region, i.e, region embedding.
Revised: In FastText, learning the embedding of n-gram can be regarded as learning low-dimensional vector representation of a small ﬁxed size region, i.e, region embedding.

Original: Although the n-grams was widely used, two caveats come with n-grams: 1) the vocabulary size increases exponentially with n , and yielding a high-dimensional feature space, which makes it difﬁcult to apply large region size(e.g., n > 4 ), such as People ’s Republic of China 2)
Revised: Although the n-grams was widely used, two caveats come with n-grams: 1) the vocabulary size increases exponentially with n , and yielding a high-dimensional feature space, which makes it difﬁcult to apply large region size(e.g., n > 4 ), such as People ’s Republic of China 2)

Original: Since the representation of each n-gram is looked up by an unique id, the data sparsity problem cannot be avoided, for example, the similar n-grams a very good movie and an excellent movie are trained separately.
Revised: Since the representation of each n-gram is looked up by an unique id, the data sparsity problem cannot be avoided, for example, the similar n-grams a very good movie and an excellent movie are trained separately.

Original: In this paper, we focus on learning task speciﬁc region embedding for text classiﬁcation, while Johnson & Zhang (2015) learn the task independent region embedding from unlabeled data.
Revised: In this paper, we focus on learning task speciﬁc region embedding for text classiﬁcation, while Johnson & Zhang (2015) learn the task independent region embedding from unlabeled data.

Original: Instead of using hand crafted n-grams features, we directly learn the region embedding based on word sequence, thus the limitations of n-grams can be avoided.
Revised: Instead of using hand crafted n-grams features, we directly learn the region embedding based on word sequence, thus the limitations of n-grams can be avoided.

Original: Intuitively, given a word sequence in a ﬁxed size region, both words and words’ relative positions are the key information to generate the semantic of the region.
Revised: Intuitively, given a word sequence in a ﬁxed size region, both words and words’ relative positions are the key information to generate the semantic of the region.

Original: In this work, each word has two components, a commonly used word embedding, and an additional local context unit.
Revised: In this work, each word has two components, a commonly used word embedding, and an additional local context unit.

Original: To utilize the information of words’ relative positions, the local context unit of a word is parameterized by a matrix, and each column is used to interact with the local context word in corresponding relative position, which can be also regarded as a linear projection function on the embedding of corresponding word.
Revised: To utilize the information of words’ relative positions, the local context unit of a word is parameterized by a matrix, and each column is used to interact with the local context word in corresponding relative position, which can be also regarded as a linear projection function on the embedding of corresponding word.

Original: The local context units and word embeddings are used to generate representations of regions from two perspectives.
Revised: The local context units and word embeddings are used to generate representations of regions from two perspectives.

Original: For text classiﬁcation task, our models just use bag of region embeddings to represent the document, and feed the document representation to a fully connected linear classiﬁer.
Revised: For text classiﬁcation task, our models just use bag of region embeddings to represent the document, and feed the document representation to a fully connected linear classiﬁer.

Original: The word embeddings and local context units are jointly trained on the classiﬁcation task.
Revised: The word embeddings and local context units are jointly trained on the classiﬁcation task.

Original: Recently, several neural models are proposed to make use of local ordered information in word sequence, and convolutional neural networks(CNN) (Kim, 2014) (Johnson & Zhang, 2014) are most similar with our works.
Revised: Recently, several neural models are proposed to make use of local ordered information in word sequence, and convolutional neural networks(CNN) (Kim, 2014) (Johnson & Zhang, 2014) are most similar with our works.

Original: The essence of CNN is to learn embeddings for small ﬁxed size regions, and each kernel of the convolutional layer tries to capture a speciﬁc semantic or structural feature.
Revised: The essence of CNN is to learn embeddings for small ﬁxed size regions, and each kernel of the convolutional layer tries to capture a speciﬁc semantic or structural feature.

Original: The kernel of CNN is shared globally, and usually several kernels are needed to preserve task related features, while the proposed local context unit is word dependent and can be applied to capture the particular semantic of the word to its local context.
Revised: The kernel of CNN is shared globally, and usually several kernels are needed to preserve task related features, while the proposed local context unit is word dependent and can be applied to capture the particular semantic of the word to its local context.

Original: Moreover, the convolution kernels extract the predictive feature by applying convolution operation on the word sequence, while our local context units are used as distinct linear projection functions on context words in corresponding relative positions.
Revised: Moreover, the convolution kernels extract the predictive feature by applying convolution operation on the word sequence, while our local context units are used as distinct linear projection functions on context words in corresponding relative positions.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Text classiﬁcation has been studied for years, traditional approaches focused on feature engineering and using different types of machine learning algorithms.
Revised: Text classiﬁcation has been studied for years, traditional approaches focused on feature engineering and using different types of machine learning algorithms.

Original: For feature engineering, bag-of-words features are efﬁcient and popular.
Revised: For feature engineering, bag-of-words features are efﬁcient and popular.

Original: In addition, the hand-crafted n-grams or phrases are added to make use of word order in text data, which has been shown effective in Wang & Manning (2012).
Revised: In addition, the hand-crafted n-grams or phrases are added to make use of word order in text data, which has been shown effective in Wang & Manning (2012).

Original: For machine learning algorithms, linear classiﬁers are widely used, such as naive bayes (McCallum et al., 1998), logistic regression and support vector machines (Joachims, 1998; Fan et al., 2008).
Revised: For machine learning algorithms, linear classiﬁers are widely used, such as naive bayes (McCallum et al., 1998), logistic regression and support vector machines (Joachims, 1998; Fan et al., 2008).

Original: However, these models commonly suffer the data sparsity problem.
Revised: However, these models commonly suffer the data sparsity problem.

Original: Recently, several neural models have been proposed, the pre-trained word embeddings of word2vec (Mikolov et al., 2013) have been widely used as the inputs to deep neural models such as recursive tensor networks (Socher et al., 2013).
Revised: Recently, several neural models have been proposed, the pre-trained word embeddings of word2vec (Mikolov et al., 2013) have been widely used as the inputs to deep neural models such as recursive tensor networks (Socher et al., 2013).

Original: On the other hand, models that can directly learn task speciﬁc word embeddings have been proposed recently, such as FastText (Joulin et al., 2016) and CNN (Kim, 2014), which both have been successfully applied on text classiﬁcation tasks.
Revised: On the other hand, models that can directly learn task speciﬁc word embeddings have been proposed recently, such as FastText (Joulin et al., 2016) and CNN (Kim, 2014), which both have been successfully applied on text classiﬁcation tasks.

Original: In the rest of this section, we brieﬂy introduce FastText and CNN, the two approaches that are most similar to ours.
Revised: In the rest of this section, we brieﬂy introduce FastText and CNN, the two approaches that are most similar to ours.

Original: FastText FastText averages the word embeddings to represent a document, and uses a full connected linear layer as the classiﬁer.
Revised: FastText FastText averages the word embeddings to represent a document, and uses a full connected linear layer as the classiﬁer.

Original: The word embeddings are trained for each task speciﬁcally.
Revised: The word embeddings are trained for each task speciﬁcally.

Original: To utilize the local word order information of small regions, FastText uses hand-crafted n-grams as features in addition to single words.
Revised: To utilize the local word order information of small regions, FastText uses hand-crafted n-grams as features in addition to single words.

Original: With the simple architecture, FastText has been proved to be effective and highly efﬁcient on text classiﬁcation tasks.
Revised: With the simple architecture, FastText has been proved to be effective and highly efﬁcient on text classiﬁcation tasks.

Original: Similarly, our models use bag of region embeddings to represent a document, and use the same linear classiﬁer.
Revised: Similarly, our models use bag of region embeddings to represent a document, and use the same linear classiﬁer.

Original: Differently, our models directly learn the semantics of regions based on word sequence, hand-crafted features is not required.
Revised: Differently, our models directly learn the semantics of regions based on word sequence, hand-crafted features is not required.

Original: CNN CNN is a feed-forward network with convolution layers interleaved with pooling layers, which are originally used for image processing tasks.
Revised: CNN CNN is a feed-forward network with convolution layers interleaved with pooling layers, which are originally used for image processing tasks.

Original: For natural language processing, words are commonly converted to vectors.
Revised: For natural language processing, words are commonly converted to vectors.

Original: CNN directly applies convolutional layer on word vectors, both word vectors and the shared(word independent) kernels are the parameters of CNN, which can be learned to capture the predictive structures of small regions.
Revised: CNN directly applies convolutional layer on word vectors, both word vectors and the shared(word independent) kernels are the parameters of CNN, which can be learned to capture the predictive structures of small regions.

Original: Our purpose is similar with CNN, which tries to learn task speciﬁc representations for regions.
Revised: Our purpose is similar with CNN, which tries to learn task speciﬁc representations for regions.

Original: Unlike CNN, we apply local context units on word vectors, our units are word dependent.
Revised: Unlike CNN, we apply local context units on word vectors, our units are word dependent.

Original:  M ETHOD
Revised:  M ETHOD

Original: Several previous works have been proposed to show the effectiveness of word order on text classiﬁcation task, such as statistical n-gram features, or apply CNNs to learn an embedding of a ﬁxed size region.
Revised: Several previous works have been proposed to show the effectiveness of word order on text classiﬁcation task, such as statistical n-gram features, or apply CNNs to learn an embedding of a ﬁxed size region.

Original: In this paper we also focus on learning the representations of small text regions which preserve the local internal structure information for text classiﬁcation.
Revised: In this paper we also focus on learning the representations of small text regions which preserve the local internal structure information for text classiﬁcation.

Original: The regions in document can be treat as ﬁxed length continues subsequence of the document.
Revised: The regions in document can be treat as ﬁxed length continues subsequence of the document.

Original: More speciﬁcally, with w i stands for the i -th(starting from 0) word of the document, we use region ( i, c ) to denote the region with middle word w i , where c is the region size.
Revised: More speciﬁcally, with w i stands for the i -th(starting from 0) word of the document, we use region ( i, c ) to denote the region with middle word w i , where c is the region size.

Original: For instance, given a sentence such as The food is not very good in this hotel , region (3 , 5) means the subsequence food is not very good .
Revised: For instance, given a sentence such as The food is not very good in this hotel , region (3 , 5) means the subsequence food is not very good .

Original: In this work, each word has two components, a word embedding and an additional local context unit.
Revised: In this work, each word has two components, a word embedding and an additional local context unit.

Original: With the units and embeddings of words in a ﬁxed size region, the representation of the region can be generated from two perspectives.
Revised: With the units and embeddings of words in a ﬁxed size region, the representation of the region can be generated from two perspectives.

Original: In the rest of this section, we will introduce the local context units ﬁrstly, and two new architectures to generate the region embeddings with local context units will be introduced, ﬁnally we will introduce how we use the region embeddings on text classiﬁcation.
Revised: In the rest of this section, we will introduce the local context units ﬁrstly, and two new architectures to generate the region embeddings with local context units will be introduced, ﬁnally we will introduce how we use the region embeddings on text classiﬁcation.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: In natural language processing tasks, words are commonly converted to low dimensional vectors as the inputs to neural networks.
Revised: In natural language processing tasks, words are commonly converted to low dimensional vectors as the inputs to neural networks.

Original: More formally, the embedding e w of word w is represented by a column in a matrix E ∈ R h × v with a look up layer, where v is the size of vocabulary, h is the embedding size.
Revised: More formally, the embedding e w of word w is represented by a column in a matrix E ∈ R h × v with a look up layer, where v is the size of vocabulary, h is the embedding size.

Original: To utilize information of words’ relative positions, we learn a local context unit for each word in addition to word embedding, which both the unit and word embedding can be learned as model parameters.
Revised: To utilize information of words’ relative positions, we learn a local context unit for each word in addition to word embedding, which both the unit and word embedding can be learned as model parameters.

Original: Formally, we deﬁne the local context unit K w
Revised: Formally, we deﬁne the local context unit K w

Original: i ∈
Revised: i ∈

Original: R h × c of w i as a matrix which can be looked up in the matrix U ∈ R
Revised: R h × c of w i as a matrix which can be looked up in the matrix U ∈ R

Original: h × c × v by w i ’s index in vocabulary.
Revised: h × c × v by w i ’s index in vocabulary.

Original: Each column in K w i can be used to interact with the local context word in corresponding relative position of w i .
Revised: Each column in K w i can be used to interact with the local context word in corresponding relative position of w i .

Original: In fact, the columns of a unit matrix can be regarded as distinction linear projection functions on the embeddings of words in the local context.
Revised: In fact, the columns of a unit matrix can be regarded as distinction linear projection functions on the embeddings of words in the local context.

Original: The parameters of these projection functions(i.e., columns of each unit matrix) can be learned to capture the semantic and syntactic inﬂuence from the word to its context.
Revised: The parameters of these projection functions(i.e., columns of each unit matrix) can be learned to capture the semantic and syntactic inﬂuence from the word to its context.

Original: Word embeddings are used as inputs to the projection functions, and we call the outputs projected word embeddings.
Revised: Word embeddings are used as inputs to the projection functions, and we call the outputs projected word embeddings.

Original: Formally, given the unit K w i of w i and the embedding e w i + t of w i + t , the ( c 2 + t ) -th column in K w i is denoted by K w i ,t , where − c 2 < =
Revised: Formally, given the unit K w i of w i and the embedding e w i + t of w i + t , the ( c 2 + t ) -th column in K w i is denoted by K w i ,t , where − c 2 < =

Original: t < =
Revised: t < =

Original: c 2 .
Revised: c 2 .

Original: We apply a element-wise multiplication(denoted by (cid:12) ) to compute the projected word embedding e iw i + t of w i + t as:
Revised: We apply a element-wise multiplication(denoted by (cid:12) ) to compute the projected word embedding e iw i + t of w i + t as:

Original: [Equation]
Revised: [Equation]

Original: Note that the middle column K w i , c 2 of K w i can be regarded as a linear projection function on e w i itself, which transforms e iw i to the same feature space as other projected embeddings’.
Revised: Note that the middle column K w i , c 2 of K w i can be regarded as a linear projection function on e w i itself, which transforms e iw i to the same feature space as other projected embeddings’.

Original: For a local context word in the particular relative position of w i , there is a corresponding linear projection function(a particular column of K w i ), thus our proposed local context units can utilize the local ordered word information in a novel way.
Revised: For a local context word in a particular relative position of w i , there is a corresponding linear projection function(a particular column of K w i ), thus our proposed local context units can utilize the local ordered word information in a novel way.

Original: 3. W ORD - CONTEXT REGION EMBEDDING
Revised: 3. W ORD - CONTEXT REGION EMBEDDING

Original: We proposed two architectures to perform the region embedding from different perspectives.
Revised: We proposed two architectures to perform the region embedding from different perspectives.

Original: In the ﬁrst proposed architecture, we focus on addressing the inﬂuence from the middle word to the context words.
Revised: In the ﬁrst proposed architecture, we focus on addressing the inﬂuence from the middle word to the context words.

Original: For example, in the sentence The food is not very good in this hotel , the occurrence of word not might bring a semantic reversal to the local region.
Revised: For example, in the sentence The food is not very good in this hotel , the occurrence of word not might bring a semantic reversal to the local region.

Original: Hence we compose the region embedding with word embeddings of all words in the region as well as the local context unit of the central word.
Revised: Hence we compose the region embedding with word embeddings of all words in the region as well as the local context unit of the central word.

Original: we call this model word-context region embedding.
Revised: we call this model word-context region embedding.

Original: Formally, for a region ( i, c ) , given the context unit K w i of center word w i and the embeddings of words in the region, the projected embeddings are computed as the following matrix:
Revised: Formally, for a region ( i, c ) , given the context unit K w i of center word w i and the embeddings of words in the region, the projected embeddings are computed as the following matrix:

Original: [Equation]
Revised: [Equation]

Original: To compose the representation of a region region ( i, c ) noted as r i,c , we apply a max pooling on the row dimension of the projected embedding matrix.
Revised: To compose the representation of a region region ( i, c ) noted as r i,c , we apply a max pooling on the row dimension of the projected embedding matrix.

Original: Figure 1a shows the details of the ﬁrst model architecture.
Revised: Figure 1a shows the details of the ﬁrst model architecture.

Original: For instance, in the sentence The food is not very good in this hotel , the projected word embeddings in the region (3 , 5) are composed by the element-wise multiplications between columns in the local context unit of not and word embeddings of food , is , not , very and good .
Revised: For instance, in the sentence The food is not very good in this hotel , the projected word embeddings in the region (3 , 5) are composed by the element-wise multiplications between columns in the local context unit of not and word embeddings of food , is , not , very and good .

Original: The embedding r 3 , 5 of region (3 , 5) can be obtained by max pooling on projected word embedding matrix.
Revised: The embedding r 3 , 5 of region (3 , 5) can be obtained by max pooling on projected word embedding matrix.

Original: 3. C ONTEXT - WORD REGION EMBEDDING
Revised: 3. C ONTEXT - WORD REGION EMBEDDING

Original: The second architecture takes as a different view, which address the local context words’ inﬂuence on the center word in the region, and we call this model context-word region embedding.
Revised: The second architecture takes as a different view, which address the local context words’ inﬂuence on the center word in the region, and we call this model context-word region embedding.

Original: Similarly, for a region ( i, c ) , the projected embeddings are represent as the following matrix:
Revised: Similarly, for a region ( i, c ) , the projected embeddings are represent as the following matrix:

Original: [Equation]
Revised: [Equation]

Original: Figure 1b shows the details of the second model architecture, the difference between our two models is that inputs of max pooling layer are different, the Word-Context model uses the projected embedding produced by the context unit of central word and word embedding of context words, while the Context-Word model just the opposite.
Revised: Figure 1b shows the details of the second model architecture, the difference between our two models is that inputs of max pooling layer are different, the Word-Context model uses the projected embeddings produced by the context unit of central word and word embeddings of context words, while the Context-Word model just the opposite.

Original: 3. R EGION EMBEDDING FOR TEXT CLASSIFICATION
Revised: 3. R EGION EMBEDDING FOR TEXT CLASSIFICATION

Original: On text classiﬁcation task, the documents commonly are of variable-length, which need to be represented as ﬁxed size vectors.
Revised: On text classiﬁcation task, the documents commonly are of variable-length, which need to be represented as ﬁxed size vectors.

Original: In order to show the effectiveness of our proposed region embedding models, we just sum up the embeddings of all regions to represent a document, and feed it an upper
Revised: In order to show the effectiveness of our proposed region embedding models, we just sum up the embeddings of all regions to represent a document, and feed it an upper

Original: Full-Connected layer for text classiﬁcation task.
Revised: Full-Connected layer for text classiﬁcation task.

Original: Formally, the model can be represented as following:
Revised: Formally, the model can be represented as following:

Original: [Equation]
Revised: [Equation]

Original: where x denotes the input text sequence, W and b denote the weight matrix and bias of the fully connected layer respectively, g denotes the softmax function of the output layer, σ denotes the softsign function and n denotes the number of regions in the document, r is the region embedding which can be computed by the two proposed methods.
Revised: where x denotes the input text sequence, W and b denote the weight matrix and bias of the fully connected layer respectively, g denotes the softmax function of the output layer, σ denotes the softsign function and n denotes the number of regions in the document, r is the region embedding which can be computed by the two proposed methods.

Original: E , U , W and b can be updated in the training period.
Revised: E , U , W and b can be updated in the training period.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We report the experiments with proposed models in comparison with previous models.
Revised: We report the experiments with proposed models in comparison with previous models.

Original: The Code is publicly available on the Internet.
Revised: The Code is publicly available on the Internet.

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: We use publicly available datasets from Zhang et al.
Revised: We use publicly available datasets from Zhang et al.

Original: (2015) to evaluate our models.
Revised: (2015) to evaluate our models.

Original: There are total 8 text classiﬁcation datasets, corresponding to sentiment analysis, news topics, question-answer, ontology extraction, respectively.
Revised: There are total 8 text classiﬁcation datasets, corresponding to sentiment analysis, news topics, question-answer, ontology extraction, respectively.

Original: Table 1 shows the descriptive statistics of datasets used in our experiments.
Revised: Table 1 shows the descriptive statistics of datasets used in our experiments.

Original: To guarantee comparable indications, same evaluation protocol of Zhang et al.
Revised: To guarantee comparable indications, same evaluation protocol of Zhang et al.

Original: (2015) is employed.
Revised: (2015) is employed.

Original: [Table]
Revised: [Table]

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: Our models are compared with several widely used supervised text classiﬁcation models.
Revised: Our models are compared with several widely used supervised text classiﬁcation models.

Original: We report the n-grams and TFIDF baselines from Zhang et al.
Revised: We report the n-grams and TFIDF baselines from Zhang et al.

Original: (2015), as well as the character level convolutional model (char-CNN) of Zhang & LeCun (2015), the word level convolutional model (wordCNN) of Zhang & LeCun (2015), the character based convolution recurrent network (char-CRNN) of Xiao & Cho (2016), the very deep convolutional network (VDCNN) of Conneau et al.
Revised: (2015), as well as the character level convolutional model (char-CNN) of Zhang & LeCun (2015), the word level convolutional model (wordCNN) of Zhang & LeCun (2015), the character based convolution recurrent network (char-CRNN) of Xiao & Cho (2016), the very deep convolutional network (VDCNN) of Conneau et al.

Original: (2016), the Discriminative LSTM (D-LSTM) of Yogatama et al.
Revised: (2016), the Discriminative LSTM (D-LSTM) of Yogatama et al.

Original: (2017) and the bigram FastText (bigramFastText) of Joulin et al. (2016).
Revised: (2017) and the bigram FastText (bigramFastText) of Joulin et al. (2016).

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: For data preprocessing, all the texts of datasets are tokenized by Stanford tokenizer and all words are converted to lower case.
Revised: For data preprocessing, all the texts of datasets are tokenized by Stanford tokenizer and all words are converted to lower case.

Original: Words that appear only once are treated as out-of-vocabulary (OOV) items, and any stop words or symbols are kept.
Revised: Words that appear only once are treated as out-of-vocabulary (OOV) items, and any stop words or symbols are kept.

Original: Additionally, padding length of c 2 are added to both the head and tail of each document.
Revised: Additionally, padding length of c 2 are added to both the head and tail of each document.

Original: Optimal hyperparameters are tuned with 10 % of the training set on Yelp Review Full dataset, and identical hyperparameters are applied to all datasets: the dimension of word embedding is 128 , the region size is 7 which means the shape of local context unit matrix of each word is 128 × 7 , the learning rate is set to 1 × 10 − 4 , and the batch size is 16 .
Revised: Optimal hyperparameters are tuned with 10 % of the training set on Yelp Review Full dataset, and identical hyperparameters are applied to all datasets: the dimension of word embedding is 128 , the region size is 7 which means the shape of local context unit matrix of each word is 128 × 7 , the learning rate is set to 1 × 10 − 4 , and the batch size is 16 .

Original: For optimization, the embeddings of words and the units are randomly initialized with Gaussian Distribution.
Revised: For optimization, the embeddings of words and the units are randomly initialized with Gaussian Distribution.

Original: Adam (Kingma & Ba, 2014) are used as the optimizer.
Revised: Adam (Kingma & Ba, 2014) are used as the optimizer.

Original: We do not use any extra regularization methods, such as L2 normalization or dropout.
Revised: We do not use any extra regularization methods, such as L2 normalization or dropout.

Original: Algorithms are entirely implemented with TensorFlow and trained on NVIDIA Tesla P
Revised: Algorithms are entirely implemented with TensorFlow and trained on NVIDIA Tesla P

Original: GPUs.
Revised: GPUs.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: Table 2 is the summary of our experimental results.
Revised: Table 2 is the summary of our experimental results.

Original: performance summary of compared methods on listed datasets.
Revised: performance summary of compared methods on listed datasets.

Original: We use underscores to represent the best known published results, and bold the best records.
Revised: We use underscores to represent the best known published results, and bold the best records.

Original: On six datasets of eight, our models beat or match the state-of-the-art with a performance gain highest to 0.7(%).
Revised: On six datasets of eight, our models beat or match the state-of-the-art with a performance gain highest to 0.7(%).

Original: We beat all the previous models on all datasets except VDCNN, while the latter performs almost best on all classiﬁcation tasks before.
Revised: We beat all the previous models on all datasets except VDCNN, while the latter performs almost best on all classiﬁcation tasks before.

Original: As a result, we slightly win VDCNN on six datasets and lost in two of Amazon datasets.
Revised: As a result, we slightly win VDCNN on six datasets and lost in two of Amazon datasets.

Original: Although the slight advantage, our methods are remarkable on account of its simplicity along with divergence.
Revised: Although the slight advantage, our methods are remarkable on account of its simplicity along with divergence.

Original: Particularly, the upper layer structure of our models only uses a summing up operation,
Revised: Particularly, the upper layer structure of our models only uses a summing up operation,

Original: [Table]
Revised: [Table]

Original: which is more concise and robust than any other deep or complex models.
Revised: which is more concise and robust than any other deep or complex models.

Original: In fact
Revised: In fact

Original: [Table]
Revised: [Table]

Original: both of our two proposed models are effective.
Revised: both of our two proposed models are effective.

Original: 4. E XPLORATORY E XPERIMENTS
Revised: 4. E XPLORATORY E XPERIMENTS

Original: In this subsection we are going to do a set of exploratory experiments to study the relative effect of each component in our model.
Revised: In this subsection we are going to do a set of exploratory experiments to study the relative effect of each component in our model.

Original: Typical cases will be analyzed to validate properties of various aspects of our models.
Revised: Typical cases will be analyzed to validate properties of various aspects of our models.

Original: With the limitation of paper space, we only analyzed the Word-Context region embedding model in our exploratory experiments.
Revised: With the limitation of paper space, we only analyzed the Word-Context region embedding model in our exploratory experiments.

Original: 4.5. E FFECT OF REGION SIZE AND EMBEDDING SIZE
Revised: 4.5. E FFECT OF REGION SIZE AND EMBEDDING SIZE

Original: Our method uses a ﬁxed size of region as contextual information just like CNN.
Revised: Our method uses a ﬁxed size of region as contextual information just like CNN.

Original: So the selection of region size really matters.
Revised: So the selection of region size really matters.

Original: A small region may lose some long-distance patterns, whereas large regions will bring into more noises.
Revised: A small region may lose some long-distance patterns, whereas large regions will bring into more noises.

Original: Luckily, our models seem to be fairly insensitive towards kinds of datasets.
Revised: Luckily, our models seem to be fairly insensitive towards kinds of datasets.

Original: Actually, we just select consistent conﬁguration of size 7 for entire datasets and it is able to outperform best published results ever.
Revised: Actually, we just select consistent conﬁguration of size 7 for entire datasets and it is able to outperform best published results ever.

Original: Figure 2 describes the performance on Yelp Review Full with different region sizes, and when the size equals to 1 , the result is quite close to unigram FastText(Accuracy 60.7%), but still get 0.6(%) promotion.
Revised: Figure 2 describes the performance on Yelp Review Full with different region sizes, and when the size equals to 1 , the result is quite close to unigram FastText(Accuracy 60.7%), but still get 0.6(%) promotion.

Original: Intuitively, the central word can not inﬂuence other words except itself when the size equals to 1 .
Revised: Intuitively, the central word can not inﬂuence other words except itself when the size equals to 1 .

Original: The performance increases with the growth of region size up to 7.
Revised: The performance increases with the growth of region size up to 7.

Original: To further study the effectiveness of region sizes, we experiment our models with the combination of multi region sizes.
Revised: To further study the effectiveness of region sizes, we experiment our models with the combination of multi region sizes.

Original: In ﬁgure 2, the combination of multi region sizes 3,5,7 is slightly better than the best single region size 7.
Revised: In ﬁgure 2, the combination of multi region sizes 3,5,7 is slightly better than the best single region size 7.

Original: The effectiveness of multi-size combination can be explained by the difference of inﬂuence ranges between words, for example, word very only emphasizes the next word while however may lay stress on a wide range of the following words.
Revised: The effectiveness of multi-size combination can be explained by the difference of inﬂuence ranges between words, for example, word very only emphasizes the next word while however may lay stress on a wide range of the following words.

Original: In addition to the analysis of region sizes, we further study the inﬂuence of word embedding dimensions.
Revised: In addition to the analysis of region sizes, we further study the inﬂuence of word embedding dimensions.

Original: Figure 3 lists the comparative results on Yelp Review Full with different embedding dimensions.
Revised: Figure 3 lists the comparative results on Yelp Review Full with different embedding dimensions.

Original: The result shows that our model is less prone to be over-ﬁtting than FastText and CNN with the word embedding dimension increasing.
Revised: The result shows that our model is less prone to be over-ﬁtting than FastText and CNN with the word embedding dimension increasing.

Original: In fact, the amount of parameters in our models is relatively large.
Revised: In fact, the amount of parameters in our models is relatively large.

Original: Since we learn a speciﬁc unit for each word, under the same word embedding dimension, our parameter size has been expanded by region size times.
Revised: Since we learn a speciﬁc unit for each word, under the same word embedding dimension, our parameter size has been expanded by region size times.

Original: Notice that the sizes of parameters are relatively consistent among 1024 in FastText , 1024 in CNN and 128 in ours
Revised: Notice that the sizes of parameters are relatively consistent among 1024 in FastText , 1024 in CNN and 128 in ours

Original: 4.5. E FFECT OF C ONTEXT UNIT
Revised: 4.5. E FFECT OF C ONTEXT UNIT

Original: In this section, we explore some comparative experiments to show the effectiveness of our proposed word speciﬁc context unit.
Revised: In this section, we explore some comparative experiments to show the effectiveness of our proposed word speciﬁc context unit.

Original: The experiments are employed based on unigram FastText baseline, which has similar upper layer structure with our models.
Revised: The experiments are employed based on unigram FastText baseline, which has similar upper layer structure with our models.

Original: Table 3 illustrates the interrelated results.
Revised: Table 3 illustrates the interrelated results.

Original: For FastText(Unigram), embedding dimension is 10.
Revised: For FastText(Unigram), embedding dimension is 10.

Original: For FastText(Win-pool), W.C.region.emb(Scalar) and
Revised: For FastText(Win-pool), W.C.region.emb(Scalar) and

Original: W.C.region.emb(our model), region size is 7 and embedding dimension is
Revised: W.C.region.emb(our model), region size is 7 and embedding dimension is

Original: [Table]
Revised: [Table]

Original: Firstly, we remove the entire context units from our model, which means it is just a variant version of unigram FastText, we call it FastText(Win-pool).
Revised: Firstly, we remove the entire context units from our model, which means it is just a variant version of unigram FastText, we call it FastText(Win-pool).

Original: The difference is that FastText sums up the word embeddings directly while FastText(Win-pool) sums up the window pooled embeddings in a stride of 1.
Revised: The difference is that FastText sums up the word embeddings directly while FastText(Win-pool) sums up the window pooled embeddings in a stride of 1.

Original: It yields a slightly accuracy gain of (0.28%) than unigram FastText.
Revised: It yields a slightly accuracy gain of (0.28%) than unigram FastText.

Original: Secondly, we apply a simpliﬁed scalar version of context units to FastText(Win-pool).
Revised: Secondly, we apply a simpliﬁed scalar version of context units to FastText(Win-pool).

Original: Distinguishable, the context unit of each word has the shape with 1 × c , hence it can be regarded as a broadcasting operation on corresponding word embeddings of its local context.
Revised: Distinguishable, the context unit of each word has the shape with 1 × c , hence it can be regarded as a broadcasting operation on corresponding word embeddings of its local context.

Original: We name this method
Revised: We name this method

Original: W.C.region.emb(Scalar).
Revised: W.C.region.emb(Scalar).

Original: Compared to the non-scalar method, it yields a huge parameter size reduction.
Revised: Compared to the non-scalar method, it yields a huge parameter size reduction.

Original: but it already yields a signiﬁcant gain of 2.45(%).
Revised: but it already yields a signiﬁcant gain of 2.45(%).

Original: Furthermore, W.C.region.emb(our model) is the variant version of W.C.region.emb(Scalar) where the each column of scalar context unit is expanded to a dense vector.
Revised: Furthermore, W.C.region.emb(our model) is the variant version of W.C.region.emb(Scalar) where the each column of scalar context unit is expanded to a dense vector.

Original: Each word’s context unit has a shape with h × c .
Revised: Each word’s context unit has a shape with h × c .

Original: Adding the low dimensional dense context unit improves performance by 4.17(%).
Revised: Adding the low dimensional dense context unit improves performance by 4.17(%).

Original: We can sense much from the procedure of decomposition, with the help of context unit, even a simpler scalar version promotes a lot.
Revised: We can sense much from the procedure of decomposition, with the help of context unit, even a simpler scalar version promotes a lot.

Original: To have a better understanding of what context unit actually are adversarial like however , but and modiﬁers like very , good , bad are listed in Figure 4.
Revised: To have a better understanding of what context unit actually are adversarial like however , but and modiﬁers like very , good , bad are listed in Figure 4.

Original: For each row of the ﬁgure, the intensity of the surrounding color box reﬂects the emphasis degree in the view of the central word.
Revised: For each row of the ﬁgure, the intensity of the surrounding color box reﬂects the emphasis degree in the view of the central word.

Original: Qualitative but not fully rigorous, a normalized L2-norm is used to render the shade.
Revised: Qualitative but not fully rigorous, a normalized L2-norm is used to render the shade.

Original: Window size 7 is adopted default, annotation l i (i ≤ 3 ) is denoted as left columns of the speciﬁc context unit, while r i (i ≤ 3 ) denoted as the right part.
Revised: Window size 7 is adopted default, annotation l i (i ≤ 3 ) is denoted as left columns of the speciﬁc context unit, while r i (i ≤ 3 ) denoted as the right part.

Original: What the ﬁgure reﬂects are consistent with intuitive priors of human beings.
Revised: What the ﬁgure reﬂects are consistent with intuitive priors of human beings.

Original: In the perspective of however , right contexts play the key role for classiﬁcation polarity because of the emotional turnaround, the color is indeed deeper in r i than l i , so does but .
Revised: In the perspective of however , right contexts play the key role for classiﬁcation polarity because of the emotional turnaround, the color is indeed deeper in r i than l i , so does but .

Original: For word very , r 1 is more prominent than the rest of all, which capture some modiﬁed patterns like very happy or very sad .
Revised: For word very , r 1 is more prominent than the rest of all, which capture some modiﬁed patterns like very happy or very sad .

Original: For word good , tendencies will be completely different for pattern not good , very good and not that good , which are intensive negative, intensive positive and slightly hesitated, separately, the position of lwill be strengthened as a result, so does word bad .
Revised: For word good , tendencies will be completely different for pattern not good , very good and not that good , which are intensive negative, intensive positive and slightly hesitated, separately, the position of lwill be strengthened as a result, so does word bad .

Original: Actually, from the motivation of word speciﬁed context units, we would like to believe this feature help capture syntactic and semantic inﬂuence on the relative position of its contexts.
Revised: Actually, from the motivation of word speciﬁed context units, we would like to believe this feature help capture syntactic and semantic inﬂuence on the relative position of its contexts.

Original: It can be regarded as a way to make use of word order in context unit covered regions.
Revised: It can be regarded as a way to make use of word order in context unit covered regions.

Original: 4.5. V ISUALIZATION
Revised: 4.5. V ISUALIZATION

Original: In this subsection, we will try to visualize the contribution of each word and selected phrase to the classiﬁcation.
Revised: In this subsection, we will try to visualize the contribution of each word and selected phrase to the classiﬁcation.

Original: Detailed visualization techniques have been introduced in Li et al.
Revised: Detailed visualization techniques have been introduced in Li et al.

Original: Here we generalize it the to the color rendering of multi-category version.
Revised: Here we generalize it the to the color rendering of multi-category version.

Original: Notice that not the self embedding acts here, but the projected embedding of one on words’
Revised: Notice that not the self embedding acts here, but the projected embedding of one on words’

Original: [Table]
Revised: [Table]

Original: With
Revised: With

Original: [Table]
Revised: [Table]

Original: remarkable ,
Revised: remarkable ,

Original: [Table]
Revised: [Table]

Original: For clarity, we choose a binary classiﬁcation task of sentiment analysis.
Revised: For clarity, we choose a binary classiﬁcation task of sentiment analysis.

Original: In Table 4, we list two real case samples in Yelp Review Polarity dataset, in which our model behave as expected.
Revised: In Table 4, we list two real case samples in Yelp Review Polarity dataset, in which our model behave as expected.

Original: Words or artiﬁcially selected phrases are highlighted green if they are positive factors, red if they are negative
Revised: Words or artiﬁcially selected phrases are highlighted green if they are positive factors, red if they are negative

Original: The intensity of the color indicates the extends of the polarity.
Revised: The intensity of the color indicates the extends of the polarity.

Original: To have a better comparison, the methods of with and without context unit are both visualized.
Revised: To have a better comparison, the methods of with and without context unit are both visualized.

Original: We abbreviate them as With C-unit and Without C-unit , respectively.
Revised: We abbreviate them as With C-unit and Without C-unit , respectively.

Original: For sentence get your wallet ready, the prices are crazy high , if no context unit is adopted, the word color will be reﬂected by its only embedding.
Revised: For sentence get your wallet ready, the prices are crazy high , if no context unit is adopted, the word color will be reﬂected by its only embedding.

Original: The word crazy is colored positive, and high colored negative.
Revised: The word crazy is colored positive, and high colored negative.

Original: Because the intensity of crazy is higher than high , the entire polarity of phrase prices are crazy high is totally positive, which is a complete mistake.
Revised: Because the intensity of crazy is higher than high , the entire polarity of phrase prices are crazy high is totally positive, which is a complete mistake.

Original: But with context unit, things have changed quite a bit.
Revised: But with context unit, things have changed quite a bit.

Original: Because of the exist of high as the context, the positive polarity of crazy vanishes.
Revised: Because of the exist of high as the context, the positive polarity of crazy vanishes.

Original: The negative polarity of high is stronger, phrase prices are crazy high performs positive overall.
Revised: The negative polarity of high is stronger, phrase prices are crazy high performs positive overall.

Original: For another case nothing remarkable, but not bad either , things seem more interesting.
Revised: For another case nothing remarkable, but not bad either , things seem more interesting.

Original: If without context-unit , remarkable will be positive, while nothing, not, bad perform negative, respectively.
Revised: If without context-unit , remarkable will be positive, while nothing, not, bad perform negative, respectively.

Original: But with context-unit, the polarity of the part ahead of but is reduced, meanwhile the polarity of not and bad ﬂip attractively.
Revised: But with context-unit, the polarity of the part ahead of but is reduced, meanwhile the polarity of not and bad ﬂip attractively.

Original: As a result, phrase but not bad either performs positive overall.
Revised: As a result, phrase but not bad either performs positive overall.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper propose two novel architectures for text classiﬁcation tasks that learn task speciﬁc region embeddings without hand crafted features.
Revised: This paper propose two novel architectures for text classiﬁcation tasks that learn task speciﬁc region embeddings without hand crafted features.

Original: To utilize the information of words’ relative positions, we learn a local context unit for each word in addition to word embedding.
Revised: To utilize the information of words’ relative positions, we learn a local context unit for each word in addition to word embedding.

Original: Our models achieve state-ofthe-art performances on six benchmark text classiﬁcation datasets, and the visualization experiments show that our proposed local context unit can capture the semantic and synthetic information for each word.
Revised: Our models achieve state-ofthe-art performances on six benchmark text classiﬁcation datasets, and the visualization experiments show that our proposed local context unit can capture the semantic and synthetic information for each word.

Original: Noticed that the power of the local context unit on learning task related region embeddings, we are interested in its ability to semi-supervised learning.
Revised: Noticed that the power of the local context unit on learning task related region embeddings, we are interested in its ability to semi-supervised learning.

Original: At the same time, we are also curious about whether we can do more natural language processing tasks by adding more complex and deep upper layers.
Revised: At the same time, we are also curious about whether we can do more natural language processing tasks by adding more complex and deep upper layers.