Original: A NEW METHOD OF REGION EMBEDDING FOR TEXT CLASSIFICATION
Revised: A NEW METHOD OF REGION EMBEDDING FOR TEXT CLASSIFICATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: To represent a text as a bag of properly identiﬁed “phrases” and use the representation for processing the text is proved to be useful.
Revised: To represent a text as a bag of properly identiﬁed “phrases” and use the representation for processing the text is proved to be useful.

Original: The key question here is how to identify the phrases and represent them.
Revised: The key question here is how to identify the phrases and represent them.

Original: The traditional method of utilizing n-grams can be regarded as an approximation of the approach.
Revised: The traditional method of utilizing n-grams can be regarded as an approximation of the approach.

Original: Such a method can suffer from data sparsity, however, particularly when the length of n-gram is large.
Revised: Such a method can suffer from data sparsity, however, particularly when the length of n-gram is large.

Original: In this paper, we propose a new method of learning and utilizing task-speciﬁc distributed representations of n-grams, referred to as “region embeddings”.
Revised: In this paper, we propose a new method of learning and utilizing task-speciﬁc distributed representations of n-grams, referred to as “region embeddings”.

Original: Without loss of generality we address text classiﬁcation.
Revised: Without loss of generality we address text classiﬁcation.

Original: We speciﬁcally propose two models for region embeddings.
Revised: We speciﬁcally propose two models for region embeddings.

Original: In our models, the representation of a word has two parts, the embedding of the word itself, and a weighting matrix to interact with the local context, referred to as local context unit.
Revised: In our models, the representation of a word has two parts, the embedding of the word itself, and a weighting matrix to interact with the local context, referred to as local context unit.

Original: The region embeddings are learned and used in the classiﬁcation task, as parameters of the neural network classiﬁer.
Revised: The region embeddings are learned and used in the classiﬁcation task, as parameters of the neural network classiﬁer.

Original: Experimental results show that our proposed method outperforms existing methods in text classiﬁcation on several benchmark datasets.
Revised: Experimental results show that our proposed method outperforms existing methods in text classiﬁcation on several benchmark datasets.

Original: The results also indicate that our method can indeed capture the salient phrasal expressions in the texts.
Revised: The results also indicate that our method can indeed capture the salient phrasal expressions in the texts.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Text classiﬁcation is an important task for many applications, including topic categorization, search query classiﬁcation, and sentiment analysis, which has been studied for years.
Revised: Text classiﬁcation is an important task for many applications, including topic categorization, search query classiﬁcation, and sentiment analysis, which has been studied for years.

Original: A simple yet effective approach for text classiﬁcation is to represent documents as bag-of-words, and train a classiﬁer on the basis of the representations using methods such as logistic regression, support vector machines (Joachims, 1998;
Revised: A simple yet effective approach for text classiﬁcation is to represent documents as bag-of-words, and train a classiﬁer on the basis of the representations using methods such as logistic regression, support vector machines (Joachims, 1998;

Original: Fan et al., 2008), and naive Bayes (McCallum et al., 1998).
Revised: Fan et al., 2008), and naive Bayes (McCallum et al., 1998).

Original: Although bag-of-words methods are effective and efﬁcient, they also have limitations.
Revised: Although bag-of-words methods are effective and efﬁcient, they also have limitations.

Original: The representations do not take into account the word order information which has been proved to be useful at least in some applications such as sentiment analysis (Pang et al., 2002).
Revised: The representations do not take into account the word order information which has been proved to be useful at least in some applications such as sentiment analysis (Pang et al., 2002).

Original: To make effective use of word order information for text classiﬁcation, people have traditionally exploited n-grams, i.e., short sequences of words in the texts.
Revised: To make effective use of word order information for text classiﬁcation, people have traditionally exploited n-grams, i.e., short sequences of words in the texts.

Original: Previous work shows that the use of n-grams is effective in the text classiﬁcation task (Pang et al., 2002; Wang & Manning, 2012;
Revised: Previous work shows that the use of n-grams is effective in the text classiﬁcation task (Pang et al., 2002; Wang & Manning, 2012;

Original: Joulin et al., 2016).
Revised: Joulin et al., 2016).

Original: Although n-grams are very useful, they have certain limitations.
Revised: Although n-grams are very useful, they have certain limitations.

Original: The number of n-grams increases exponentially when the length of n-gram n increases.
Revised: The number of n-grams increases exponentially when the length of n-gram n increases.

Original: This makes it difﬁcult to exploit large n-grams (e.g., n > 4 ).
Revised: This makes it difﬁcult to exploit large n-grams (e.g., n > 4 ).

Original: Since the number of parameters in an n-gram model is very large, the estimation of the parameters usually suffers from the data sparsity problem.
Revised: Since the number of parameters in an n-gram model is very large, the estimation of the parameters usually suffers from the data sparsity problem.

Original: Recently, the method of FastText has been proposed (Joulin et al., 2016), which can learn and use distributed embeddings of n-grams.
Revised: Recently, the method of FastText has been proposed (Joulin et al., 2016), which can learn and use distributed embeddings of n-grams.

Original: More speciﬁcally, the embedding of an n-gram is deﬁned as a low-dimensional vector representation of the n-gram.
Revised: More speciﬁcally, the embedding of an n-gram is deﬁned as a low-dimensional vector representation of the n-gram.

Original: Note that the n-grams in a vocabulary can also be represented as one-hot vectors Wang & Manning (2012).
Revised: Note that the n-grams in a vocabulary can also be represented as one-hot vectors Wang & Manning (2012).

Original: In this paper, we propose to learn embeddings of n-grams for a speciﬁc task (e.g., classiﬁcation), which are more compact and thus easy to obtain.
Revised: In this paper, we propose to learn embeddings of n-grams for a speciﬁc task (e.g., classiﬁcation), which are more compact and thus easy to obtain.

Original: We call the embeddings region embeddings, following the work in Johnson & Zhang (2015).
Revised: We call the embeddings region embeddings, following the work in Johnson & Zhang (2015).

Original: Our method signiﬁcantly differs from their method, however, in the sense that the region embeddings in our method are task-dependent and acquired from supervised learning, while those in their method are task-independent and acquired from unsupervised learning.
Revised: Our method signiﬁcantly differs from their method, however, in the sense that the region embeddings in our method are task-dependent and acquired from supervised learning, while those in their method are task-independent and acquired from unsupervised learning.

Original: Our method is also largely different from FastText, as it learns richer models for region embeddings.
Revised: Our method is also largely different from FastText, as it learns richer models for region embeddings.

Original: Under a

Original: Intuitively, the meaning of a word is deﬁned by the meaning of itself as well as the meanings of words in the surrounding context.
Revised: Intuitively, the meaning of a word is deﬁned by the meaning of itself as well as the meanings of words in the surrounding context.

Original: The extended embedding of a word in an n-gram thus consists of two parts, the embedding of the word itself and a matrix to interact with the local context, named “local context unit”.
Revised: The extended embedding of a word in an n-gram thus consists of two parts, the embedding of the word itself and a matrix to interact with the local context, named “local context unit”.

Original: The embedding of a word is a column vector, and the local context unit of a word is a matrix in which the columns are used to interact with words in the local context.
Revised: The embedding of a word is a column vector, and the local context unit of a word is a matrix in which the columns are used to interact with words in the local context.

Original: The region embedding of an n-gram is then constructed by the extended embeddings of all words in the n-gram.
Revised: The region embedding of an n-gram is then constructed by the extended embeddings of all words in the n-gram.

Original: In this paper, we introduce two models for region embeddings.
Revised: In this paper, we introduce two models for region embeddings.

Original: For the text classiﬁcation task, a document is viewed as a bag of region embeddings, and the bag of region embeddings is fed into a classiﬁer.
Revised: For the text classiﬁcation task, a document is viewed as a bag of region embeddings, and the bag of region embeddings is fed into a classiﬁer.

Original: The parameters of the local context units and word embeddings are trained together with the parameters of the classiﬁer which is a fully connected neural network.
Revised: The parameters of the local context units and word embeddings are trained together with the parameters of the classiﬁer which is a fully connected neural network.

Original: Our models achieve better results than the state-of-the-art methods on several benchmark datasets of text classiﬁcation.
Revised: Our models achieve better results than the state-of-the-art methods on several benchmark datasets of text classiﬁcation.

Original: Experiments show that our proposed models can really capture important information for the task.
Revised: Experiments show that our proposed models can really capture important information for the task.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Text classiﬁcation has been studied for years, traditional approaches focused on feature engineering and using different types of machine learning algorithms.
Revised: Text classiﬁcation has been studied for years, traditional approaches focused on feature engineering and using different types of machine learning algorithms.

Original: For feature engineering, bag-of-words features are efﬁcient and popular.
Revised: For feature engineering, bag-of-words features are efﬁcient and popular.

Original: In addition, the hand-crafted n-grams or phrases are added to make use of word order in text data, which has been shown effective on Wang & Manning (2012).
Revised: In addition, the hand-crafted n-grams or phrases are added to make use of word order in text data, which has been shown effective on Wang & Manning (2012).

Original: For machine learning algorithms, linear classiﬁers are widely used, such as naive bayes (McCallum et al., 1998), logistic regression and support vector machines (Joachims, 1998; Fan et al., 2008).
Revised: For machine learning algorithms, linear classiﬁers are widely used, such as naive bayes (McCallum et al., 1998), logistic regression and support vector machines (Joachims, 1998; Fan et al., 2008).

Original: However, these models commonly suffer the data sparsity problem.
Revised: However, these models commonly suffer the data sparsity problem.

Original: Recently, several neural models have been proposed, the pre-trained word embeddings of word2vec (Mikolov et al., 2013) have been widely used as inputs to deep neural models such as recursive tensor networks (Socher et al., 2013).
Revised: Recently, several neural models have been proposed, the pre-trained word embeddings of word2vec (Mikolov et al., 2013) have been widely used as inputs to deep neural models such as recursive tensor networks (Socher et al., 2013).

Original: On the other hand, some simple and efﬁcient models which can directly learn task speciﬁc word embeddings or ﬁne-tune on pre-trained word embeddings have been proposed recently, such as Deep Averaging Networks (Iyyer et al., 2015),
Revised: On the other hand, some simple and efﬁcient models which can directly learn task speciﬁc word embeddings or ﬁne-tune on pre-trained word embeddings have been proposed recently, such as Deep Averaging Networks (Iyyer et al., 2015),

Original: FastText (Joulin et al., 2016).
Revised: FastText (Joulin et al., 2016).

Original: Several neural models have been proposed to make use of word order information, most models are based on convolutional neural network (CNN)
Revised: Several neural models have been proposed to make use of word order information, most models are based on convolutional neural network (CNN)

Original: (Kim, 2014; Johnson & Zhang, 2014; Zhang et al., 2015) and recurrent neural network (RNN) (Tang et al., 2015; Lai et al., 2015; Yogatama et al., 2017).
Revised: (Kim, 2014; Johnson & Zhang, 2014; Zhang et al., 2015) and recurrent neural network (RNN) (Tang et al., 2015; Lai et al., 2015; Yogatama et al., 2017).

Original: More recently, the Transformer (Vaswani et al., 2017), a sequence transduction model based solely on attention mechanisms has been proposed.
Revised: More recently, the Transformer (Vaswani et al., 2017), a sequence transduction model based solely on attention mechanisms has been proposed.

Original: Although
Revised: Although

Original: Transformer was not designed for the text classiﬁcation task, it has similarities with our work.
Revised: Transformer was not designed for the text classiﬁcation task, it has similarities with our work.

Original: In the rest of this section, we will brieﬂy introduce FastText, CNN and Transformer, which are the most relevant to our work.
Revised: In the rest of this section, we will brieﬂy introduce FastText, CNN and Transformer, which are the most relevant to our work.

Original: FastText FastText averages the word embeddings to represent a document, and uses a full connected linear layer as the classiﬁer.
Revised: FastText FastText averages the word embeddings to represent a document, and uses a full connected linear layer as the classiﬁer.

Original: The word embeddings are trained for each task speciﬁcally.
Revised: The word embeddings are trained for each task speciﬁcally.

Original: To utilize the local word order information of small regions, FastText uses hand-crafted n-grams as features in addition to single words.
Revised: To utilize the local word order information of small regions, FastText uses hand-crafted n-grams as features in addition to single words.

Original: With the simple architecture, FastText has been proved to be effective and highly efﬁcient on text classiﬁcation tasks.
Revised: With the simple architecture, FastText has been proved to be effective and highly efﬁcient on text classiﬁcation tasks.

Original: Similarly, our models use bag of region embeddings to represent a document, and use the same linear classiﬁer.
Revised: Similarly, our models use bag of region embeddings to represent a document, and use the same linear classiﬁer.

Original: Differently, our models directly learn the semantics of regions based on word sequence, hand-crafted features are not required.
Revised: Differently, our models directly learn the semantics of regions based on word sequence, hand-crafted features are not required.

Original: CNN CNN is a feed-forward network with convolutional layers interleaved with pooling layers, which are originally used for image processing tasks.
Revised: CNN CNN is a feed-forward network with convolutional layers interleaved with pooling layers, which are originally used for image processing tasks.

Original: For natural language processing, words are commonly converted to vectors.
Revised: For natural language processing, words are commonly converted to vectors.

Original: CNN directly applies convolutional layer on word vectors, both word vectors and the shared (word independent) kernels are the parameters of CNN, which can be learned to capture the predictive structures of small regions.
Revised: CNN directly applies convolutional layer on word vectors, both word vectors and the shared (word independent) kernels are the parameters of CNN, which can be learned to capture the predictive structures of small regions.

Original: The essence of CNN is to learn embeddings for small ﬁxed size regions, each kernel of the convolutional layer tries to capture a speciﬁc semantic or structural feature.
Revised: The essence of CNN is to learn embeddings for small ﬁxed size regions, each kernel of the convolutional layer tries to capture a speciﬁc semantic or structural feature.

Original: Our purpose is similar with CNN, which tries to learn task speciﬁc representations of regions.
Revised: Our purpose is similar with CNN, which tries to learn task speciﬁc representations of regions.

Original: Unlike CNN, we apply local context units on word vectors, which are word dependent, moreover, the convolution kernels extract the predictive features by applying convolution operation on word sequences, while we use local context units as distinct linear projection functions on context words in corresponding relative positions to get region representations.Under review as a conference paper at
Revised: Unlike CNN, we apply local context units on word vectors, which are word dependent, moreover, the convolution kernels extract the predictive features by applying convolution operation on word sequences, while we use local context units as distinct linear projection functions on context words in corresponding relative positions to get region representations.

Original: Transformer
Revised: Transformer

Original: Vaswani et al.
Revised: Vaswani et al.

Original: (2017) proposed a sequence transduction model, the Transformer, based solely on attention mechanisms.
Revised: (2017) proposed a sequence transduction model, the Transformer, based solely on attention mechanisms.

Original: Both Transformer and our method can capture word order information without any CNN or RNN component, and the scalar form of context units (introduced in our ablation experiments) can be regarded as a kind of local attention.
Revised: Both Transformer and our method can capture word order information without any CNN or RNN component, and the scalar form of context units (introduced in our ablation experiments) can be regarded as a kind of local attention.

Original: There are also some differences here: the motivation we proposed local context units is to address word speciﬁc inﬂuence between word and its context, while Vaswani et al.
Revised: There are also some differences here: the motivation we proposed local context units is to address word speciﬁc inﬂuence between word and its context, while Vaswani et al.

Original: (2017) has proposed a parallelable sequence transduction framework based entirely on attention; To utilize position information, in our method, words are interacted with context words at different relative positions by corresponding columns in their context units, while Transformer use ﬁxed sin and cos function based position encoding.
Revised: (2017) has proposed a parallelable sequence transduction framework based entirely on attention; To utilize position information, in our method, words are interacted with context words at different relative positions by corresponding columns in their context units, while Transformer use ﬁxed sin and cos function based position encoding.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this paper we focus on learning the representations of small text regions which preserve the local internal structure information for text classiﬁcation.
Revised: In this paper, we focus on learning the representations of small text regions which preserve the local internal structural information for text classiﬁcation.

Original: The regions in a document can be treated as ﬁxed length contiguous subsequences of the document.
Revised: The regions in a document can be considered as ﬁxed length contiguous subsequences of the document.

Original: More speciﬁcally, with w i standing for the i -th(starting from 0) word of the document, we use region ( i, c ) to denote the 2 ⇥ c +1 length region with middle word w i .
Revised: More speciﬁcally, with w i standing for the i -th(starting from 0) word of the document, we use region ( i, c ) to denote the 2 ⇥ c + 1 length region with middle word w i .

Original: For instance, given a sentence such as The food is not very good in this hotel , region (3 , 2) means the subsequence food is not very good .
Revised: For instance, given a sentence such as The food is not very good in this hotel , region (3 , 2) means the subsequence food is not very good .

Original: In this work, we use the interactions between words and their local context based on word embeddings as well as the local context units to produce region embeddings.
Revised: In this work, we use the interactions between words and their local context based on word embeddings as well as the local context units to produce region embeddings.

Original: In the rest of this section, we will introduce the local context units ﬁrstly, and two architectures to generate the region embeddings through local context units will be introduced, ﬁnally we will introduce how we use the region embeddings on text classiﬁcation.
Revised: In the rest of this section, we will introduce the local context units ﬁrstly, and two architectures to generate the region embeddings through local context units will be introduced, ﬁnally we will introduce how we use the region embeddings on text classiﬁcation.

Original: [Figure]
Revised: [Figure]

Original: L 
Revised: L OCAL CONTEXT UNIT

Original: [Figure]

Original: In natural language processing, words are commonly converted to low dimensional vectors(word embeddings) as the inputs to neural networks.
Revised: In natural language processing, words are commonly converted to low dimensional vectors(word embeddings) as the inputs to neural networks.

Original: More formally, the embedding e w of word w is represented by a column in a matrix E 2 R
Revised: More formally, the embedding e w of word w is represented by a column in a matrix E 2 R

Original: h ⇥ v with a look up layer, where v is the size of the vocabulary, h is the embedding size.
Revised: h ⇥ v with a look up layer, where v is the size of the vocabulary, h is the embedding size.

Original: To utilize the information of words’ relative positions and local context, we learn a local context unit for each word in addition to the word embedding, and both the unit and word embedding are learned as model parameters.
Revised: To utilize the information of words’ relative positions and local context, we learn a local context unit for each word in addition to the word embedding, and both the unit and word embedding are learned as model parameters.

Original: Formally, we deﬁne the local context unit K w i
Revised: Formally, we deﬁne the local context unit K w i

Original: R h ⇥ (2 ⇥ c +1) of w i as a matrix which can be looked up in the tensor
Revised: R h ⇥ (2 ⇥ c +1) of w i as a matrix which can be looked up in the tensor

Original: U 2 R h ⇥ (2 ⇥ c +1) ⇥ v by w i ’s index in the vocabulary.
Revised: U 2 R h ⇥ (2 ⇥ c +1) ⇥ v by w i ’s index in the vocabulary.

Original: Under

Original: Each column in K w i can be used to interact with the context word in corresponding relative position of w i .
Revised: Each column in K w i can be used to interact with the context word in corresponding relative position of w i .

Original: In fact, the columns of a unit matrix can be regarded as distinction linear projection functions on the embeddings of words in the local context.
Revised: In fact, the columns of a unit matrix can be regarded as distinctive linear projection functions on the embeddings of words in the local context.

Original: The parameters of these projection functions(i.e., columns of each unit matrix) can be learned to capture the semantic and syntactic inﬂuence of the word to its context.
Revised: The parameters of these projection functions(i.e., columns of each unit matrix) can be learned to capture the semantic and syntactic inﬂuence of the word to its context.

Original: Word embeddings are used as inputs to the projection functions, and we call the outputs projected word embeddings.
Revised: Word embeddings are used as inputs to the projection functions, and we call the outputs projected word embeddings.

Original: Formally, let p iw
Revised: Formally, let p iw

Original: i + t be the projected word embedding of w i + t in i -th word’s view, and K w i ,t be the ( c + t ) -th column in K w i
Revised: i + t be the projected word embedding of w i + t in i -th word’s view, and K w i ,t be the ( c + t ) -th column in K w i

Original: (  c < = t < = c ), given the unit K w i of w i and the embedding e w i + t of w i + t , we use an element-wise multiplication(denoted by  ) to compute p iw i + t :
Revised: (  c < = t < = c ), given the unit K w i of w i and the embedding e w i + t of w i + t , we use an element-wise multiplication(denoted by  ) to compute p iw i + t :

Original: [Equation]
Revised: [Equation]

Original: For a context word in a particular relative position of w i , there is a corresponding linear projection function(a particular column of K w i ), thus our proposed local context units can utilize the local ordered word information in a novel way.
Revised: For a context word in a particular relative position of w i , there is a corresponding linear projection function(a particular column of K w i ), thus our proposed local context units can utilize the local ordered word information in a novel way.

Original: Note that the middle column K w i , 0 of K w i can be regarded as a linear projection function on e w i itself, which transforms e w i to the same space as other projected embeddings.
Revised: Note that the middle column K w i , 0 of K w i can be regarded as a linear projection function on e w i itself, which transforms e w i to the same space as other projected embeddings.

Original: 3. W ORD - CONTEXT REGION EMBEDDING
Revised: 3. W ORD - CONTEXT REGION EMBEDDING

Original: We proposed two architectures to perform the region embedding from different perspectives.
Revised: We proposed two architectures to perform the region embedding from different perspectives.

Original: We consider the semantic of a given region is derived from the mutual inﬂuences of the words in this region.
Revised: We consider the semantics of a given region is derived from the mutual inﬂuences of the words in this region.

Original: In this paper, the regions can be regarded as snapshots of a window sliding on a document, whose middle words are contiguous, hence we can compose the semantic of a give region only by the middle word’s inﬂuences on the context words, or the context words’ inﬂuences on the middle word.
Revised: In this paper, the regions can be regarded as snapshots of a window sliding on a document, whose middle words are contiguous, hence we can compose the semantics of a give region only by the middle word’s inﬂuences on the context words, or the context words’ inﬂuences on the middle word.

Original: In the ﬁrst proposed architecture, we focus on addressing the middle word’s inﬂuences on the context words.
Revised: In the ﬁrst proposed architecture, we focus on addressing the middle word’s inﬂuences on the context words.

Original: For example, in the sentence The food is not very good in this hotel , the occurrence of word not might bring a semantic reversal to the local region.
Revised: For example, in the sentence The food is not very good in this hotel , the occurrence of word not might bring a semantic reversal to the local region.

Original: We use the local context unit of the middle word and the original word embeddings in a region to perform the projected embeddings in a word-to-context view, where the projected embeddings can reﬂect the middle word’s inﬂuences on the context words.
Revised: We use the local context unit of the middle word and the original word embeddings in a region to perform the projected embeddings in a word-to-context view, where the projected embeddings can reﬂect the middle word’s inﬂuences on the context words.

Original: Once the projected embeddings are obtained, a max pooling operation is applied to extract the most predictive features in the region.
Revised: Once the projected embeddings are obtained, a max pooling operation is applied to extract the most predictive features in the region.

Original: The output of the max pooling operation can be regarded as a task related region embedding in a word-to-context view, i.e. Word-Context region embedding.
Revised: The output of the max pooling operation can be regarded as a task related region embedding in a word-to-context view, i.e. Word-Context region embedding.

Original: Formally, we use the context unit K w i of middle word w i and embeddings of all words in a region region ( i, c ) to compute the projected embedding matrix by equation (1), then the Word-Context region embedding r ( i,c ) can be obtained through a max pooling operation on the projected embedding matrix:
Revised: Formally, we use the context unit K w i of middle word w i and embeddings of all words in a region region ( i, c ) to compute the projected embedding matrix by equation (1), then the Word-Context region embedding r ( i,c ) can be obtained through a max pooling operation on the projected embedding matrix:

Original: [Equation]
Revised: [Equation]

Original: where max standing for the max pooling operation on the column dimension of the input matrix.
Revised: where max standing for the max pooling operation on the column dimension of the input matrix.

Original: Finally, we get r ( i,c ) as a vector representation of region ( i, c ) with dimension h .
Revised: Finally, we get r ( i,c ) as a vector representation of region ( i, c ) with dimension h .

Original: Figure 1a shows the details of the ﬁrst model architecture.
Revised: Figure 1a shows the details of the ﬁrst model architecture.

Original: For instance, in the sentence The food is not very good in this hotel , the projected word embeddings in the region (3 , 2) are composed by the element-wise multiplications between columns in the local context unit of not and word embeddings of food , is , not , very and good .
Revised: For instance, in the sentence The food is not very good in this hotel , the projected word embeddings in the region (3 , 2) are composed by the element-wise multiplications between columns in the local context unit of not and word embeddings of food , is , not , very and good .

Original: The embedding r 3 , 2 of region (3 , 2) can be obtained by max pooling on the projected word embedding matrix.
Revised: The embedding r 3 , 2 of region (3 , 2) can be obtained by max pooling on the projected word embedding matrix.

Original: 3. C ONTEXT - WORD REGION EMBEDDING
Revised: 3. C ONTEXT - WORD REGION EMBEDDING

Original: The second architecture goes as a different view, which addresses the local context words’ inﬂuences on the middle word in the region, and we call this Context-Word region embedding.
Revised: The second architecture goes as a different view, which addresses the local context words’ inﬂuences on the middle word in the region, and we call this Context-Word region embedding.

Original: Similarly, for a region ( i, c ) , the projected embeddings are computed by the original word embedding of the middleUnder review as a conference paper word and the context units of all words in the region, then the Context-Word region embedding can be obtained by a max pooling operation through the column dimension of the projected embedding matrix:
Revised: Similarly, for a region ( i, c ) , the projected embeddings are computed by the original word embedding of the middle word and the context units of all words in the region, then the Context-Word region embedding can be obtained by a max pooling operation through the column dimension of the projected embedding matrix:

Original: [Equation]
Revised: [Equation]

Original: Figure 1b shows the details of the second model architecture.
Revised: Figure 1b shows the details of the second model architecture.

Original: our two models take different ways to produce the projected word embeddings, the Word-Context model uses context units of middle words and word embeddings of context words, while the Context-Word model uses context units of context words and word embeddings of  middle word.
Revised: our two models take different ways to produce the projected word embeddings, the Word-Context model uses context units of middle words and word embeddings of context words, while the Context-Word model uses context units of context words and word embeddings of the middle word.

Original: 3. R EGION EMBEDDING FOR TEXT CLASSIFICATION
Revised: 3. R EGION EMBEDDING FOR TEXT CLASSIFICATION

Original: For text classiﬁcation, documents are usually variable-sized, which need to be represented as ﬁxed size vectors.
Revised: For text classiﬁcation, documents are usually variable-sized, which need to be represented as ﬁxed size vectors.

Original: In order to show the effectiveness of our proposed region embedding models, we just sum up the embeddings of all regions to represent a document, and feed it to an upper FullConnected layer for text classiﬁcation task.
Revised: In order to show the effectiveness of our proposed region embedding models, we just sum up the embeddings of all regions to represent a document, and feed it to an upper FullConnected layer for text classiﬁcation task.

Original: Formally, the model can be represented as following:
Revised: Formally, the model can be represented as following:

Original: [Equation]
Revised: [Equation]

Original: where x denotes the input text sequence, W and b denote the weight matrix and bias of the fully connected layer respectively, g denotes the softmax function of the output layer,  denotes the softsign function and n denotes the number of regions in a document, r is the region embedding which can be computed by the equation (2) or (3).
Revised: where x denotes the input text sequence, W and b denote the weight matrix and bias of the fully connected layer respectively, g denotes the softmax function of the output layer,  denotes the softsign function and n denotes the number of regions in a document, r is the region embedding which can be computed by the equation (2) or (3).

Original: E , U , W and b can be updated in the training period.
Revised: E , U , W and b can be updated in the training period.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We reportthe experiments with proposed models in comparison with previous models.
Revised: We report experiments with proposed models in comparison with previous models.

Original: The code is publicly available on the Internet.

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: We use publicly available datasets from Zhang et al.
Revised: We use publicly available datasets from Zhang et al.

Original: (2015) to evaluate our models.
Revised: (2015) to evaluate our models.

Original: There are in total 8 text classiﬁcation datasets, corresponding to sentiment analysis,news classiﬁcation , questionanswer, ontology extraction tasks, respectively.
Revised: There are in total 8 text classiﬁcation datasets, corresponding to sentiment analysis, news classiﬁcation, questionanswer, ontology extraction tasks, respectively.

Original: Table 1 shows the descriptive statistics of datasets used in our experiments.
Revised: Table 1 shows the descriptive statistics of datasets used in our experiments.

Original: To guarantee comparable indications, same evaluation protocol of Zhang et al.
Revised: To guarantee comparable indications, same evaluation protocol of Zhang et al.

Original: (2015) is employed
Revised: (2015) is employed.

Original: [Table]

Original: [Table]

Original: Under review as a conference paper at ICLR

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: Our models are compared with several widely used supervised text classiﬁcation models.
Revised: Our models are compared with several widely used supervised text classiﬁcation models.

Original: We report the n-grams and TFIDF baselines from Zhang et al.
Revised: We report the n-grams and TFIDF baselines from Zhang et al.

Original: (2015), as well as the character level convolutional model (char-CNN) of Zhang & LeCun (2015), the character based convolution recurrent network (char-CRNN) of Xiao & Cho (2016), the very deep convolutional network (VDCNN) of Conneau et al.
Revised: (2015), as well as the character level convolutional model (char-CNN) of Zhang & LeCun (2015), the character based convolution recurrent network (char-CRNN) of Xiao & Cho (2016), the very deep convolutional network (VDCNN) of Conneau et al.

Original: (2016), the Discriminative LSTM (D-LSTM) of Yogatama et al.
Revised: (2016), the Discriminative LSTM (D-LSTM) of Yogatama et al.

Original: (2017) and the bigram FastText (bigram-FastText) of Joulin et al. 
Revised: (2017) and the bigram FastText (bigram-FastText) of Joulin et al. [Table]

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: For data preprocessing, all the texts of datasets are tokenized by Stanford tokenizer and all words are converted to lower case.
Revised: For data preprocessing, all the texts of datasets are tokenized by Stanford tokenizer and all words are converted to lower case.

Original: Words that appear only in one document are treated as out-of-vocabulary (OOV) items, and all stop words as well as symbols are kept.
Revised: Words that appear only in one document are treated as out-of-vocabulary (OOV) items, and all stop words as well as symbols are kept.

Original: Additionally, length of c padding are added to both the head and tail of each document.
Revised: Additionally, length of c padding are added to both the head and tail of each document.

Original: For our models, optimal hyperparameters are tuned with 10 % of the training set on Yelp Review Full dataset, and identical hyperparameters are applied to all datasets: the dimension of word embedding is 128 , the region size is 7 which means the shape of local context unit matrix of each word is 128 ⇥ 7 , the initial learning rate is set to 1 ⇥ 10  4 , and the batch size is 16 .
Revised: For our models, optimal hyperparameters are tuned with 10 % of the training set on Yelp Review Full dataset, and identical hyperparameters are applied to all datasets: the dimension of word embedding is 128 , the region size is 7 which means the shape of local context unit matrix of each word is 128 ⇥ 7 , the initial learning rate is set to 1 ⇥ 10  4 , and the batch size is 16 .

Original: For optimization, the embeddings of words and the units are randomly initialized with Gaussian Distribution.
Revised: For optimization, the embeddings of words and the units are randomly initialized with Gaussian Distribution.

Original: Adam (Kingma & Ba, 2014) is used as the optimizer.
Revised: Adam (Kingma & Ba, 2014) is used as the optimizer.

Original: We do not use any extra regularization methods, like L2 normalization or dropout.
Revised: We do not use any extra regularization methods, like L2 normalization or dropout.

Original: Algorithms are entirely implemented with TensorFlow and trained on NVIDIA Tesla
Revised: Algorithms are entirely implemented with TensorFlow and trained on NVIDIA Tesla

Original: P40 GPUs.
Revised: P40 GPUs.

Revised: The code 1 is publicly available on the Internet.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: Model

Original: Yelp

Original: [Table]
Revised: [Table]

Original: Yelp

Original: [Table]

Original: Amz.

Original: P. Amz [Table] Table 2 is the summary of the experimental results.
Revised: Table 2 is the summary of the experimental results.

Original: We use underscores to represent the best published results, and bold the best records.
Revised: We use underscores to represent the best published results, and bold the best records.

Original: On six datasets of eight, our models beat or match the state-of-the-art with a performance gain highest to 0.7%.
Revised: On six datasets of eight, our models beat or match the state-of-the-art with a performance gain highest to 0.7%.

Original: We beat all the previous models on all datasets except VDCNN, while the latter performs almost best on all classiﬁcation tasks before.
Revised: We beat all the previous models on all datasets except VDCNN, while the latter performs almost best on all classiﬁcation tasks before.

Original: As a result, we slightly win VDCNN on six datasets and lost in two of Amazon datasets.
Revised: As a result, we slightly win VDCNN on six datasets and lost in two of Amazon datasets.

Original: Detailed experimental results including best performance epoch and training time for all listed datasets are reported in Appendix A.
Revised: Detailed experimental results including best performance epoch and training time for all listed datasets are reported in Appendix A.

Original: Furthermore, the upper layer structure of our models only uses a summing up operation, which is more concise and robust than any other deep or complex models.
Revised: Furthermore, the upper layer structure of our models only uses a summing up operation, which is more concise and robust than any other deep or complex models.

Original: In fact, both of our two proposed models are effective against previous models.
Revised: In fact, both of our two proposed models are effective against previous models.

Original: 4. E XPLORATORY E XPERIMENTS
Revised: 4. E XPLORATORY E XPERIMENTS

Original: In this subsection we are going to do a set of exploratory experiments to study the effect of each component in our model.
Revised: In this subsection, we are going to do a set of exploratory experiments to study the effect of each component of our model.

Original: Typical cases will be analyzed to validate properties of various aspects ofUnder as a our models.
Revised: Typical cases will be analyzed to validate properties of various aspects of our models.

Original: Considering the limitation of paper space, we only analyzed the Word-Context region embedding model in our exploratory experiments.
Revised: Considering the limitation of paper space, we only analyzed the Word-Context region embedding model in our exploratory experiments.

Original: 4.5. E FFECT OF REGION SIZE AND EMBEDDING SIZE
Revised: 4.5. E FFECT OF REGION SIZE AND EMBEDDING SIZE

Original: Our method uses a ﬁxed size of region as contextual information just like CNN.
Revised: Our method uses a ﬁxed size of region as contextual information just like CNN.

Original: So the selection of region size really matters.
Revised: So the selection of region size really matters.

Original: A small region may lose some long distance patterns, whereas large regions will bring into more noises.
Revised: A small region may lose some long distance patterns, whereas large regions will bring into more noises.

Original: Luckily, our models seem to be fairly insensitive towards kinds of datasets.
Revised: Luckily, our models seem to be fairly insensitive towards kinds of datasets.

Original: Actually, we just use identical region size 7 for all datasets and it is able to outperform  best published results ever.
Revised: Actually, we just use identical region size 7 for all datasets and it is able to outperform the best published results ever.

Original: Figure 2a describes the performance on Yelp Review Full with different region sizes, and when the size equals to 1 , the result is quite close to unigram FastText(accuracy 60.7%), but still gets a 0.6% promotion.
Revised: Figure 2a describes the performance on Yelp Review Full with different region sizes, and when the size equals to 1 , the result is quite close to unigram FastText(accuracy 60.7%), but still gets a 0.6% promotion.

Original: Intuitively, the middle word can not inﬂuence other words except itself when the size equals to 1 .
Revised: Intuitively, the middle word cannot inﬂuence other words except itself when the size equals to 1 .

Original: The performance increases with the growth of region size up to 7.
Revised: The performance increases with the growth of region size up to 7.

Original: [Figure]
Revised: [Figure]

Original: Furthermore, we experiment our models with the combination of multi region sizes.
Revised: Furthermore, we experiment our models with the combination of multi region sizes.

Original: Here we use the approach of sharing context units among each region, where the parameters of local context units of smaller regions are just the slice of the longest one.
Revised: Here we use the approach of sharing context units among each region, where the parameters of local context units of smaller regions are just the slice of the longest one.

Original: Region embeddings of different sizes are concatenated for ﬁnal classiﬁcation.
Revised: Region embeddings of different sizes are concatenated for ﬁnal classiﬁcation.

Original: In ﬁgure 2a, the combination of multi region sizes 3,5,7 is slightly better than the best single region size 7.
Revised: In ﬁgure 2a, the combination of multi region sizes 3,5,7 is slightly better than the best single region size 7.

Original: The effectiveness of multi-size combination can be explained by the difference of inﬂuence ranges between words.
Revised: The effectiveness of multi-size combination can be explained by the difference of inﬂuence ranges between words.

Original: For example, in sentiment analysis, word very only emphasizes the next word while however may lay stress on a wide range of the following words.
Revised: For example, in sentiment analysis, word very only emphasizes the next word while however may lay stress on a wide range of the following words.

Original: In addition to the analysis of region sizes, we further study the inﬂuence of word embedding dimensions.
Revised: In addition to the analysis of region sizes, we further study the inﬂuence of word embedding dimensions.

Original: Figure 2b lists the comparative results on Yelp Review Full with different embedding dimensions.
Revised: Figure 2b lists the comparative results on Yelp Review Full with different embedding dimensions.

Original: The result shows that our model is more robust to overﬁtting than FastText and CNN with the word embedding dimension increasing.
Revised: The result shows that our model is more robust to overﬁtting than FastText and CNN with the word embedding dimension increasing.

Original: In fact, the amount of parameters in our models is relatively large.
Revised: In fact, the amount of parameters in our models is relatively large.

Original: Since we learn a speciﬁc unit for each word, under the same word embedding dimension, our parameter size has been expanded by region size times, the parameters number is v ⇥ h + v ⇥ (2 ⇥ c + 1) ⇥
Revised: Since we learn a speciﬁc unit for each word, under the same word embedding dimension, our parameter size has been expanded by region size times, the parameters number is v ⇥ h + v ⇥ (2 ⇥ c + 1) ⇥

Original: h
Revised: h

Original: + h ⇥ m
Revised: + h ⇥ m

Original: + m , where m is the number of classes.
Revised: + m , where m is the number of classes.

Original: Speciﬁc numbers parameters for different region sizes are listed in Appendix A. Notice that the sizes of parameters are relatively consistent among 1024 in FastText , 1024 in CNN and 128 in ours .
Revised: Speciﬁc numbers parameters for different region sizes are listed in Appendix A. Notice that the sizes of parameters are relatively consistent among 1024 in FastText , 1024 in CNN and 128 in ours .

Original: 4.5. E FFECT OF C ONTEXT UNIT
Revised: 4.5. E FFECT OF C ONTEXT UNIT

Original: In this section, we explore some comparative experiments to show the effectiveness of our proposed word speciﬁc context unit.
Revised: In this section, we explore some comparative experiments to show the effectiveness of our proposed word speciﬁc context unit.

Original: The experiments are employed based on unigram FastText baseline, which has similar upper layer structure with our models.
Revised: The experiments are employed based on unigram FastText baseline, which has similar upper layer structure with our models.

Original: Table 3 illustrates the results.
Revised: Table 3 illustrates the results.

Original: Firstly, we remove the entire context units from our model, which means it is just a variant version of unigram FastText, we call it FastText(Win-pool).

Original: The difference is that FastText sums up the word

Original: [Table]
Revised: [Table]

Original: Text(Unigram), embedding dimension is 10.
Revised: Text(Unigram), embedding dimension is 10.

Original: For FastText(Win-pool), W.C.region.emb(Scalar) and
Revised: For FastText(Win-pool), [Table] C.region.emb(Scalar) and

Original: W.C.region.emb(our model), region size is 7 and embeddingdimension is.
Revised: [Table] .C.region.emb(our model), region size [Table] 7 and embedding

Original: [Table]
Revised: [Table]

Revised: Firstly, we remove the entire context units from our model, which means it is just a variant version of unigram FastText, we call it FastText(Win-pool).

Original:  embeddings directly while FastText(Win-pool) sums up the window pooled embeddings in a stride of 1.
Revised: The difference is that FastText sums up the word embeddings directly while FastText(Win-pool) sums up the window pooled embeddings in a stride of 1.

Original: It yields a slightly accuracy gain of 0.28% than unigram FastText.
Revised: It yields a slightly accuracy gain of 0.28% than unigram FastText.

Original: Secondly, we apply a simpliﬁed scalar version of context units to FastText(Win-pool).
Revised: Secondly, we apply a simpliﬁed scalar version of context units to FastText(Win-pool).

Original: Distinguishable, the context unit of each word has the shape with 1 ⇥ (2 ⇥ c + 1) , hence it can be regarded as a broadcasting operation on corresponding word embeddings of its local context.
Revised: Distinguishable, the context unit of each word has the shape with 1 ⇥ (2 ⇥ c + 1) , hence it can be regarded as a broadcasting operation on corresponding word embeddings of its local context.

Original: We name this method W.C.region.emb(Scalar).
Revised: We name this method W.C.region.emb(Scalar).

Original: Compared to the non-scalar method, it yields a huge parameter size reduction, but it already yields a signiﬁcant gain of 2.45%.
Revised: Compared to the non-scalar method, it yields a huge parameter size reduction, but it already yields a signiﬁcant gain of 2.45%.

Original: Furthermore, W.C.region.emb(our model) is the variant version of W.C.region.emb(Scalar) where each column of scalar context unit is expanded to a dense vector.
Revised: Furthermore, W.C.region.emb(our model) is the variant version of W.C.region.emb(Scalar) where each column of scalar context unit is expanded to a dense vector.

Original: Each word’s context unit has a shape with h ⇥ (2 ⇥ c +1) .
Revised: Each word’s context unit has a shape with h ⇥ (2 ⇥ c +1) .

Original: Adding the low dimensional dense context unit improves the performance by 4.17%.
Revised: Adding the low dimensional dense context unit improves the performance by 4.17%.

Original: We can sense much from the procedure of decomposition, with the help of context unit, even a simpler scalar version promotes a lot.
Revised: We can sense much from the procedure of decomposition, with the help of context unit, even a simpler scalar version promotes a lot.

Original: To have a better understanding of what context unit actually capture, heat maps are plotted for chosen word samples.
Revised: To have a better understanding of what context unit actually capture, heat maps are plotted for chosen word samples.

Original: Representative adversarial conjunctions like however , but , modiﬁers like very , good , bad and nouns like food , morning are listed in Figure 3.
Revised: Representative adversarial conjunctions like however , but , modiﬁers like very , good , bad and nouns like food , morning are listed in Figure 3.

Original: For each row of the ﬁgure, the intensity of the surrounding color box reﬂects the emphasis degree in the view of the middle word.
Revised: For each row of the ﬁgure, the intensity of the surrounding color box reﬂects the emphasis degree in the view of the middle word.

Original: Qualitative but not fully rigorous, a normalized L2-norm of each column in context unit is used to render the shade.
Revised: Qualitative but not fully rigorous, a normalized L2-norm of each column in context unit is used to render the shade.

Original: Region size 7 is adopted default, annotation l i (i  3 ) is denoted as left columns of the speciﬁc context unit, while r i (i  3 ) denoted as the right part.
Revised: Region size 7 is adopted default, annotation l i (i  3 ) is denoted as left columns of the speciﬁc context unit, while r i (i  3 ) denoted as the right part.

Original: What the ﬁgure reﬂects are consistent with intuitive priors of human beings.
Revised: What the ﬁgure reﬂects are consistent with intuitive priors of human beings.

Original: In the perspective of however , right contexts play the key role for classiﬁcation polarity because of the emotional reversal, the color is indeed deeper in r i than l i , so does but .
Revised: In the perspective of however , right contexts play the key role for classiﬁcation polarity because of the emotional reversal, the color is indeed deeper in r i than l i , so does but .

Original: For word very , r 1 is more prominent than the rest of all, which captures some modiﬁed patterns like very happy or very sad .
Revised: For word very , r 1 is more prominent than the rest of all, which captures some modiﬁed patterns like very happy or very sad .

Original: For word good , tendencies will be completely different for patterns like not good , very good and not that good , which are intensive negative, intensive positive and slightly hesitated, separately, the position of lwill be strengthened as a result, so does word bad .
Revised: For word good , tendencies will be completely different for patterns like not good , very good and not that good , which are intensive negative, intensive positive and slightly hesitated, separately, the position of lwill be strengthened as a result, so does word bad .

Original: There are signiﬁcant differences between two nouns food and morning .
Revised: There are signiﬁcant differences between two nouns food and morning .

Original: The heat map of word food implies patterns like delicious food or food was mediocre , while the word morning has fewer valuable patterns for classiﬁcation.
Revised: The heat map of word food implies patterns like delicious food or food was mediocre , while the word morning has fewer valuable patterns for classiﬁcation.

Original: Actually, from the motivation of word speciﬁed context units, we would like to believe this feature helps capture syntactic and semantic inﬂuences of words on surrounding words at relative positions.
Revised: Actually, from the motivation of word speciﬁed context units, we would like to believe this feature helps capture syntactic and semantic inﬂuences of words on surrounding words at relative positions.

Original: 4.5. V ISUALIZATION
Revised: 4.5. V ISUALIZATION

Original: In this subsection, we will try to visualize the contribution of each word and selected phrase to classiﬁcation.
Revised: In this subsection, we will try to visualize the contribution of each word and selected phrase to classiﬁcation.

Original: Detailed visualization techniques have been introduced in Li et al.
Revised: Detailed visualization techniques have been introduced in Li et al.

Original: Here we generalize it to the color rendering of multi-category version.
Revised: Here we generalize it to the color rendering of multi-category version.

Original: Notice that for our model, not the original embedding acts here, but the accumulation of the projected embedding of each word on its surrounding words.
Revised: Notice that for our model, not the original embedding acts here, but the accumulation of the projected embedding of each word on its surrounding words.

Original: For clarity, we choose a binary classiﬁcation task of sentiment analysis.
Revised: [Table] With C-unit get your wallet ready , the prices are crazy high [Table] No C-unit

Original: In Table 4, we list two cases in Yelp Review Polarity dataset, in which our model behaves as
Revised: [Table] remarkable , [Table] not

Original: [Table]
Revised: [Table]

Original: With C-unit get your wallet ready , the prices are
Revised: bad

Original: [Table]
Revised: [Table]

Original: high
Revised: With C-unit nothing remarkable , but not bad either but not bad either For clarity, we choose a binary classiﬁcation task of sentiment analysis.

Original: [Table]
Revised: In Table 4, we list two cases in Yelp Review Polarity dataset, in which our model behaves as expected. Words and artiﬁcially selected phrases are highlighted green if they are positive factors, red if they are negative. The intensity of the color indicates the degree of the polarity.

Original: To have a better comparison, the results of with and without context unit methods are both visualized.
Revised: To have a better comparison, the results of with and without context unit methods are both visualized.

Original: We abbreviate them as With C-unit and No C-unit , respectively.
Revised: We abbreviate them as With C-unit and No C-unit , respectively.

Original: For sentence get your wallet ready, the prices are crazy high , if no context unit is adopted, the word color reﬂects its word embedding, which is context-free.
Revised: For sentence get your wallet ready, the prices are crazy high , if no context unit is adopted, the word color reﬂects its word embedding, which is context-free.

Original: The polarity of crazy is positive, and high is negative.
Revised: The polarity of crazy is positive, and high is negative.

Original: Because the intensity of crazy is higher than high , the polarity of phrase prices are crazy high is totally positive, which is a mistake.
Revised: Because the intensity of crazy is higher than high , the polarity of phrase prices are crazy high is totally positive, which is a mistake.

Original: But with context unit, things have changed quite a bit, the polarities of words are context dependent.
Revised: But with context unit, things have changed quite a bit, the polarities of words are context dependent.

Original: Under the inﬂuence of high , the positive polarity of crazy vanishes and phrase prices are crazy high performs negative overall.
Revised: Under the inﬂuence of high , the positive polarity of crazy vanishes and phrase prices are crazy high performs negative overall.

Original: For another case nothing remarkable, but not bad either , things seem more interesting.
Revised: For another case nothing remarkable, but not bad either , things seem more interesting.

Original: Without context-unit , remarkable is positive, while nothing, not, bad perform negative, respectively.
Revised: Without context-unit , remarkable is positive, while nothing, not, bad perform negative, respectively.

Original: But with context unit, the polarity of the part ahead of but weakens, meanwhile the polarities of not and bad ﬂips.
Revised: But with context unit, the polarity of the part ahead of but weakens, meanwhile the polarities of not and bad ﬂips.

Original: As a result, phrase but not bad either performs positive overall.
Revised: As a result, phrase but not bad either performs positive overall.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper proposed two novel architectures for text classiﬁcation tasks, which learn task speciﬁc region embeddings without hand crafted features.
Revised: This paper proposed two novel architectures for text classiﬁcation tasks, which learn task speciﬁc region embeddings without hand crafted features.

Original: To utilize the word speciﬁc inﬂuences of each word on its context words, a local context unit for each word is learned in addition to word embedding.
Revised: To utilize the word speciﬁc inﬂuences of each word on its context words, a local context unit for each word is learned in addition to word embedding.

Original: Our models achieve state-of-the-art performances on six benchmark text classiﬁcation datasets, and the visualization experiments show that our proposed local context unit can capture the semantic and syntactic information for each word.
Revised: Our models achieve state-of-the-art performances on six benchmark text classiﬁcation datasets, and the visualization experiments show that our proposed local context unit can capture the semantic and syntactic information for each word.

Original: Noticed the power of the local context unit on learning task related region embeddings, we are interested in its ability to unsupervised and semi-supervised learning.
Revised: Noticed the power of the local context unit on learning task related region embeddings, we are interested in its ability to unsupervised and semi-supervised learning.

Original: At the same time, we are also curious about whether we can achieve better results by introducing more complex upper layers on text classiﬁcation, and other natural language processing tasks. 
Revised: At the same time, we are also curious about whether we can achieve better results by introducing more complex upper layers on text classiﬁcation, and other natural language processing tasks. A CKNOWLEDGMENTS

Revised: This paper is supported by National Basic Research Program of China (973 program

Revised: No.2014CB340505).

Revised: We gratefully thank the anonymous reviewers for their insightful comments.

Original: APPENDIX A D ETAILED EXPERIMENTAL RESULTS
Revised: APPENDIX A D ETAILED EXPERIMENTAL RESULTS

Original: To have a further insight about our reported results, we list the training time and best testing performance epoch for kinds of region sizes( 5,7,9 ) in detail.
Revised: To have a further insight about our reported results, we list the training time and best testing performance epoch for kinds of region sizes( 5,7,9 ) in detail.

Original: For all the 8 datasets in Table 5, hyperparameters are kept in line with section 4.3.
Revised: For all the 8 datasets in Table 5, hyperparameters are kept in line with section 4.3.

Original: The dimension of word embedding is 128 , the region size is 7 , the initial learning rate is set to 1 ⇥ 10  4 , and the batch size is 16 .
Revised: The dimension of word embedding is 128 , the region size is 7 , the initial learning rate is set to 1 ⇥ 10  4 , and the batch size is 16 .

Original: Epoch index starts from 0 and training time is reported per epoch.
Revised: Epoch index starts from 0 and training time is reported per epoch.

Original: We choose  model of word-context region embedding here.
Revised: We choose the model of word-context region embedding here.

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] Yah.

Revised: A.

Original: [Table]
Revised: [Table]

Original: We also report results of several repeated runs in Table 6 to exclude effects of randomness and ensure reproducibility.
Revised: We also report results of several repeated runs in Table 6 to exclude the effect of randomness and ensure reproducibility.

Original: Five independent tries are conducted on each dataset of Yelp.
Revised: Five independent runs are conducted on each dataset of Yelp.

Original: P and Yelp.
Revised: P and Yelp.

Original: F, where both performance variances are within 0.11% on accuracy.
Revised: F, where both performance variances are within 0.11% on accuracy.

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]