Original: B AG OF REGION EMBEDDINGS VIA LOCAL CONTEXT UNITS FOR TEXT CLASSIFICATION
Revised: A NEW METHOD OF REGION EMBEDDING FOR TEXT CLASSIFICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Contextual information and word orders are proved valuable for  text classiﬁcation task.
Revised: To represent a text as a bag of properly identiﬁed “phrases” and use the representation for processing the text is proved to be useful.

Original: To make use of local word order information, n-grams are commonly used features in several models, such as linear models. However, these models commonly suffer the  data sparsity problem and are difﬁcult to represent large size region.
Revised: The key question here is how to identify the phrases and represent them. The traditional method of utilizing n-grams can be regarded as an approximation of the approach. Such a method can suffer from data sparsity, however, particularly when the length of n-gram is large.

Original: The discrete or distributed representations of n-grams can be regarded as region embeddings, which are representations of ﬁxed size regions. In this paper, we  propose twonovel text classiﬁcation models that learn task speciﬁc region embeddings without hand crafted features, hence the drawbacks of n-grams can be overcome.
Revised: In this paper, we propose a new method of learning and utilizing task-speciﬁc distributed representations of n-grams, referred to as “region embeddings”. Without loss of generality we address text classiﬁcation. We speciﬁcally propose two models for region embeddings.

Original: In our model, each word has two attributes, a commonly used word embedding, and an additional local context unit which is used to interact with theword’s local context.
Revised: In our models, the representation of a word has two parts, the embedding of the word itself, and a weighting matrix to interact with the local context, referred to as local context unit.

Original: Both the units and word embeddings areused to generate representations of regions, and are learned  as model parameters. Finally, bag of region embeddings of a document is fed to a linear classiﬁer.
Revised: The region embeddings are learned and used in the classiﬁcation task, as parameters of the neural network classiﬁer.

Original: Experimental results show that our proposed  methods achieve state-of-the-art performance on several benchmark datasets.
Revised: Experimental results show that our proposed method outperforms existing methods in text classiﬁcation on several benchmark datasets.

Original: We provide visualizations and analysis illustrating that our proposed local context unit can  capture the syntactic and semantic information.
Revised: The results also indicate that our method can indeed capture the salient phrasal expressions in the texts.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Text classiﬁcation is an important task in natural language processing, which has been studied for years with applications such as topic categorization, web search, sentiment classiﬁcation.
Revised: Text classiﬁcation is an important task for many applications, including topic categorization, search query classiﬁcation, and sentiment analysis, which has been studied for years.

Original: A simple yet effective approach for text classiﬁcation is to represent documents as bag-of-words, and train a linear classiﬁer, e.g., a logistic regression,a support vector machines (Joachims, 1998; Fan et al., 2008) or a naive bayes (McCallum et al., 1998).
Revised: A simple yet effective approach for text classiﬁcation is to represent documents as bag-of-words, and train a classiﬁer on the basis of the representations using methods such as logistic regression, support vector machines (Joachims, 1998; Fan et al., 2008), and naive Bayes (McCallum et al., 1998).

Original: Although bag-of-words methods are efﬁcient and popular, they lose the local ordering informationof words which has been proved  useful particularly on sentiment classiﬁcation (Pang et al., 2002).
Revised: Although bag-of-words methods are effective and efﬁcient, they also have limitations. The representations do not take into account the word order information which has been proved to be useful at least in some applications such as sentiment analysis (Pang et al., 2002).

Original: To beneﬁt from local word order  for text classiﬁcation,  n-gramswhich is a simple statistics of ordered word combinations usually performs good (Pang et al., 2002; 
Revised: To make effective use of word order information for text classiﬁcation, people have traditionally exploited n-grams, i.e., short sequences of words in the texts. Previous work shows that the use of n-grams is effective in the text classiﬁcation task (Pang et al., 2002; Wang & Manning, 2012;

Original: Wang & Manning, 2012; Joulin et al., 2016).
Revised: Joulin et al., 2016).

Original: With a n-gram vocabulary, Wang & Manning (2012) use one-hot vectors as discrete embedding of n-grams.
Revised: Although n-grams are very useful, they have certain limitations. The number of n-grams increases exponentially when the length of n-gram n increases. This makes it difﬁcult to exploit large n-grams (e.g., n > 4 ). Since the number of parameters in an n-gram model is very large, the estimation of the parameters usually suffers from the data sparsity problem.

Original: Recently,  FastText  (Joulin et al., 2016) directly learns distributed embeddings of n-grams and words for the task of interest.
Revised: Recently, the method of FastText has been proposed (Joulin et al., 2016), which can learn and use distributed embeddings of n-grams.

Original: In FastText, learning the embedding of  n-gram can be regarded as learning low-dimensional vector representation of a small ﬁxed size region, i.e, region embedding.
Revised: More speciﬁcally, the embedding of an n-gram is deﬁned as a low-dimensional vector representation of the n-gram.

Original: Although the n-grams was widely used, two caveats come with n-grams: 1) the vocabulary size increases exponentially with n , and yielding a high-dimensional feature space, which makes it difﬁcult to apply large region size(e.g., n > 4 ), such as People ’s Republic of China 2) Since the representation ofeach n-gram is looked up by an unique id, the data sparsity problem cannot be avoided, for example, the similar n-grams  a very good movie and an excellent movie are trained separately.
Revised: Note that the n-grams in a vocabulary can also be represented as one-hot vectors Wang & Manning (2012). In this paper, we propose to learn embeddings of n-grams for a speciﬁc task (e.g., classiﬁcation), which are more compact and thus easy to obtain.

Original: In this paper, we focus on learning task speciﬁc region embedding for text classiﬁcation, while Johnson & Zhang (2015) learn thetask independent region embedding from unlabeled data. Instead of using hand crafted n-grams features, we directly learn the region embedding based on word sequence, thus the limitations of n-grams can be avoided.
Revised: We call the embeddings region embeddings, following the work in Johnson & Zhang (2015). Our method signiﬁcantly differs from their method, however, in the sense that the region embeddings in our method are task-dependent and acquired from supervised learning, while those in their method are task-independent and acquired from unsupervised learning. Our method is also largely different from FastText, as it learns richer models for region embeddings.

Original:  Intuitively, given a word sequence in a ﬁxed size region, both words and words’ relative positions are the key information to generate the semantic of  the region.
Revised: Under a Intuitively, the meaning of a word is deﬁned by the meaning of itself as well as the meanings of words in the surrounding context.

Original: In this work, each word has two components, a commonly used word embedding, and an additional local  context unit.
Revised: The extended embedding of a word in an n-gram thus consists of two parts, the embedding of the word itself and a matrix to interact with the local context, named “local context unit”.

Original: To utilize the information of words’ relative positions, the local context unit of a word isparameterized by a matrix, and each column is used to interact with the local context word incorresponding relative position, which can be also regarded as a linear projection function on the  embedding of corresponding word.
Revised: The embedding of a word is a column vector, and the local context unit of a word is a matrix in which the columns are used to interact with words in the local context. The region embedding of an n-gram is then constructed by the extended embeddings of all words in the n-gram.

Original: The local context units and word embeddings are used to generate representations of regions from two perspectives.
Revised: In this paper, we introduce two models for region embeddings.

Original: For  text classiﬁcation task, our models just use bag of region embeddings to represent the document, andfeed the document representation to afully connected linear classiﬁer.
Revised: For the text classiﬁcation task, a document is viewed as a bag of region embeddings, and the bag of region embeddings is fed into a classiﬁer.

Original: The word embeddings and local context units  arejointly trained on the classiﬁcation task.
Revised: The parameters of the local context units and word embeddings are trained together with the parameters of the classiﬁer which is a fully connected neural network.

Original: Recently, several neural models are proposed to make use of local ordered information in word sequence, and convolutional neural networks(CNN) (Kim, 2014) (Johnson & Zhang, 2014) are most similar with our works.
Revised: Our models achieve better results than the state-of-the-art methods on several benchmark datasets of text classiﬁcation.

Original: The essence of CNN is to learn embeddings for small ﬁxed size regions, and each kernel of the convolutional layer tries to capture a speciﬁc semantic or structural feature. The kernel of CNN is shared globally, and usually several kernels are needed to preserve task related features, while the proposed local context unit is word dependent and can be applied to capture  the particular semantic of the word to its local context. Moreover, the convolution kernels extract the predictive feature by applying convolution operation on the word sequence, while our local context units are used as distinct linear projection functions on context words in corresponding relative positions.
Revised: Experiments show that our proposed models can really capture important information for the task.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORK

Original: Text classiﬁcation has been studied for years, traditional approaches focused on feature engineering and using different types of machine learning algorithms.
Revised: Text classiﬁcation has been studied for years, traditional approaches focused on feature engineering and using different types of machine learning algorithms.

Original: For feature engineering, bag-of-words features are efﬁcient and popular.
Revised: For feature engineering, bag-of-words features are efﬁcient and popular.

Original: In addition, the hand-crafted n-grams or phrases are added to make use of word order in text data, which has been shown effective in Wang & Manning (2012).
Revised: In addition, the hand-crafted n-grams or phrases are added to make use of word order in text data, which has been shown effective on Wang & Manning (2012).

Original: For machine learning algorithms, linear classiﬁers are widely used, such as naive bayes (McCallum et al., 1998), logistic regression and support vector machines (Joachims, 1998; Fan et al., 2008).
Revised: For machine learning algorithms, linear classiﬁers are widely used, such as naive bayes (McCallum et al., 1998), logistic regression and support vector machines (Joachims, 1998; Fan et al., 2008).

Original: However, these models commonly suffer the data sparsity problem.
Revised: However, these models commonly suffer the data sparsity problem.

Original: Recently, several neural models have been proposed, the pre-trained word embeddings of word2vec (Mikolov et al., 2013) have been widely used asthe inputs to deep neural models such as recursive tensor networks (Socher et al., 2013).
Revised: Recently, several neural models have been proposed, the pre-trained word embeddings of word2vec (Mikolov et al., 2013) have been widely used as inputs to deep neural models such as recursive tensor networks (Socher et al., 2013).

Original: On the other hand,  models that can directly learn task speciﬁc word embeddings  have been proposed recently, such as  FastText (Joulin et al., 2016) and CNN (Kim, 2014), which both have been successfully applied on text classiﬁcation tasks.
Revised: On the other hand, some simple and efﬁcient models which can directly learn task speciﬁc word embeddings or ﬁne-tune on pre-trained word embeddings have been proposed recently, such as Deep Averaging Networks (Iyyer et al., 2015), FastText (Joulin et al., 2016). Several neural models have been proposed to make use of word order information, most models are based on convolutional neural network (CNN) (Kim, 2014; Johnson & Zhang, 2014; Zhang et al., 2015) and recurrent neural network (RNN) (Tang et al., 2015; Lai et al., 2015; Yogatama et al., 2017).

Original:  In the rest of this section, we  brieﬂy introduce FastText and CNN, the two approaches that are  most similar to ours.
Revised: More recently, the Transformer (Vaswani et al., 2017), a sequence transduction model based solely on attention mechanisms has been proposed. Although Transformer was not designed for the text classiﬁcation task, it has similarities with our work. In the rest of this section, we will brieﬂy introduce FastText, CNN and Transformer, which are the most relevant to our work.

Original: FastText FastText averages the word embeddings to represent a document, and uses a full connected linear layer as the classiﬁer.
Revised: FastText FastText averages the word embeddings to represent a document, and uses a full connected linear layer as the classiﬁer.

Original: The word embeddings are trained for each task speciﬁcally.
Revised: The word embeddings are trained for each task speciﬁcally.

Original: To utilize the local word order information of small regions, FastText uses hand-crafted n-grams as features in addition to single words.
Revised: To utilize the local word order information of small regions, FastText uses hand-crafted n-grams as features in addition to single words.

Original: With the simple architecture, FastText has been proved to be effective and highly efﬁcient on text classiﬁcation tasks.
Revised: With the simple architecture, FastText has been proved to be effective and highly efﬁcient on text classiﬁcation tasks.

Original: Similarly, our models use bag of region embeddings to represent a document, and use the same linear classiﬁer.
Revised: Similarly, our models use bag of region embeddings to represent a document, and use the same linear classiﬁer.

Original: Differently, our models directly learn the semantics of regions based on word sequence, hand-crafted features is not required.
Revised: Differently, our models directly learn the semantics of regions based on word sequence, hand-crafted features are not required.

Original: CNN CNN is a feed-forward network with convolution layers interleaved with pooling layers, which are originally used for image processing tasks.
Revised: CNN CNN is a feed-forward network with convolutional layers interleaved with pooling layers, which are originally used for image processing tasks.

Original: For natural language processing, words are commonly converted to vectors.
Revised: For natural language processing, words are commonly converted to vectors.

Original: CNN directly applies convolutional layer on word vectors, both word vectors and the shared(word independent) kernels are the parameters of CNN, which can be learned to capture the predictive structures of small regions.
Revised: CNN directly applies convolutional layer on word vectors, both word vectors and the shared (word independent) kernels are the parameters of CNN, which can be learned to capture the predictive structures of small regions.

Revised: The essence of CNN is to learn embeddings for small ﬁxed size regions, each kernel of the convolutional layer tries to capture a speciﬁc semantic or structural feature.

Original: Our purpose is similar with CNN, which tries to learn task speciﬁc representations for regions.
Revised: Our purpose is similar with CNN, which tries to learn task speciﬁc representations of regions.

Original: Unlike CNN, we apply local context units on word vectors, our units are word dependent.
Revised: Unlike CNN, we apply local context units on word vectors, which are word dependent, moreover, the convolution kernels extract the predictive features by applying convolution operation on word sequences, while we use local context units as distinct linear projection functions on context words in corresponding relative positions to get region representations. Under review as a conference paper at

Original: M ETHOD
Revised: Transformer

Original: Several previous works have been proposed to show the effectiveness of word order on text classiﬁcation task, such as statistical n-gram features, or apply CNNs to learn an embedding of  a ﬁxed size region.
Revised: Vaswani et al. (2017) proposed a sequence transduction model, the Transformer, based solely on attention mechanisms. Both Transformer and our method can capture word order information without any CNN or RNN component, and the scalar form of context units (introduced in our ablation experiments) can be regarded as a kind of local attention. There are also some differences here: the motivation we proposed local context units is to address word speciﬁc inﬂuence between word and its context, while Vaswani et al.

Revised: (2017) has proposed a parallelable sequence transduction framework based entirely on attention; To utilize position information, in our method, words are interacted with context words at different relative positions by corresponding columns in their context units, while Transformer use ﬁxed sin and cos function based position encoding.

Revised:  M ETHOD

Original: In this paper wealso focus on learning the representations of small text regions which preserve the local internal structure information for text classiﬁcation.
Revised: In this paper we focus on learning the representations of small text regions which preserve the local internal structure information for text classiﬁcation.

Original: The regions in  document can be treat as ﬁxed length continues subsequence of the document.
Revised: The regions in a document can be treated as ﬁxed length contiguous subsequences of the document.

Original: More speciﬁcally, with w i stands for the i -th(starting from 0) word of the document, we use region ( i, c ) to denote the  region with middle word w i , where c is the region size.
Revised: More speciﬁcally, with w i standing for the i -th(starting from 0) word of the document, we use region ( i, c ) to denote the 2 ⇥ c +1 length region with middle word w i .

Original: For instance, given a sentence such as The food is not very good in this hotel , region (3 , 5) means the subsequence food is not very good .
Revised: For instance, given a sentence such as The food is not very good in this hotel , region (3 , 2) means the subsequence food is not very good .

Original: In this work, each word has two components, a word embedding and an additional local context unit. With the units and embeddings of words in a ﬁxed size region, the representation of the  region can be generated from two perspectives.
Revised: In this work, we use the interactions between words and their local context based on word embeddings as well as the local context units to produce region embeddings.

Original: In the rest of this section, we will introduce the local context units ﬁrstly, and twonew architectures to generate the region embeddings with local context units will be introduced, ﬁnally we will introduce how we use the region embeddings on text classiﬁcation.
Revised: In the rest of this section, we will introduce the local context units ﬁrstly, and two architectures to generate the region embeddings through local context units will be introduced, ﬁnally we will introduce how we use the region embeddings on text classiﬁcation.

Original: [Figure]
Revised: [Figure]

Revised: L

Original: [Figure]
Revised: [Figure]

Original: In natural language processing tasks, words are commonly converted to low dimensional vectors as the inputs to neural networks.
Revised: In natural language processing, words are commonly converted to low dimensional vectors(word embeddings) as the inputs to neural networks.

Original: More formally, the embedding e w of word w is represented by a column in a matrix E ∈ R h × v with a look up layer, where v is the size of  vocabulary, h is the embedding size.
Revised: More formally, the embedding e w of word w is represented by a column in a matrix E 2 R h ⇥ v with a look up layer, where v is the size of the vocabulary, h is the embedding size.

Original: To utilize  information of words’ relative positions, we learn a local context unit for each word in addition to  word embedding, which both the unit and word embedding can be learned as model parameters.
Revised: To utilize the information of words’ relative positions and local context, we learn a local context unit for each word in addition to the word embedding, and both the unit and word embedding are learned as model parameters.

Original: Formally, we deﬁne the local context unit K w 
Revised: Formally, we deﬁne the local context unit K w i

Original: i ∈ R h × c  of w i as a matrix which can be looked up in the matrix U ∈ R h × c × v by w i ’s index in  vocabulary.
Revised: R h ⇥ (2 ⇥ c +1) of w i as a matrix which can be looked up in the tensor U 2 R h ⇥ (2 ⇥ c +1) ⇥ v by w i ’s index in the vocabulary.

Original:  Each column in K w i can be used to interact with thelocal context word in corresponding relative position of w i .
Revised: Under Each column in K w i can be used to interact with the context word in corresponding relative position of w i .

Original: In fact, the columns of a unit matrix can be regarded as distinction linear projection functions on the embeddings of words in the local context.
Revised: In fact, the columns of a unit matrix can be regarded as distinction linear projection functions on the embeddings of words in the local context.

Original: The parameters of these projection functions(i.e., columns of each unit matrix) can be learned to capture the semantic and syntactic inﬂuence from the word to its context.
Revised: The parameters of these projection functions(i.e., columns of each unit matrix) can be learned to capture the semantic and syntactic inﬂuence of the word to its context.

Original: Word embeddings are used as inputs to the projection functions, and we call the outputs projected word embeddings.
Revised: Word embeddings are used as inputs to the projection functions, and we call the outputs projected word embeddings.

Original: Formally, given the unit K w i of w i and the  embedding e w i + t of w i + t , the ( c2 + t ) -th column in K w iis denoted by K w i ,t , where − c 2 < = t < =
Revised: Formally, let p iw i + t be the projected word embedding of w i + t in i -th word’s view, and K w i ,t be the ( c + t ) -th column in K w i

Original:  c 2 . We apply a element-wise multiplication(denoted by (cid:12) ) to compute theprojected word embedding e iw i + t of w i + t as:
Revised: (  c < = t < = c ), given the unit K w i of w i and the embedding e w i + t of w i + t , we use an element-wise multiplication(denoted by  ) to compute p iw i + t :

Original: [Equation]
Revised: [Equation]

Original: Note that the middle column K w i , c 2 of K w i can be regarded as a linear projection function on e w i itself, which transforms e iw i to the same feature space as other projected embeddings’. For alocal context word in a particular relative position of w i , there is a corresponding linear projection function(a particular column of K w i ), thus our proposed local context units can utilize the local ordered word information in a novel way. 
Revised: For a context word in a particular relative position of w i , there is a corresponding linear projection function(a particular column of K w i ), thus our proposed local context units can utilize the local ordered word information in a novel way. Note that the middle column K w i , 0 of K w i can be regarded as a linear projection function on e w i itself, which transforms e w i to the same space as other projected embeddings.

Original: 3. W ORD - CONTEXT REGION EMBEDDING
Revised: 3. W ORD - CONTEXT REGION EMBEDDING

Original: We proposed two architectures to perform the region embedding from different perspectives.
Revised: We proposed two architectures to perform the region embedding from different perspectives.

Revised: We consider the semantic of a given region is derived from the mutual inﬂuences of the words in this region.

Revised: In this paper, the regions can be regarded as snapshots of a window sliding on a document, whose middle words are contiguous, hence we can compose the semantic of a give region only by the middle word’s inﬂuences on the context words, or the context words’ inﬂuences on the middle word.

Original: In the ﬁrst proposed architecture, we focus on addressing theinﬂuence from the middle word to the context words.
Revised: In the ﬁrst proposed architecture, we focus on addressing the middle word’s inﬂuences on the context words.

Original: For example, in the sentence The food is not very good in this hotel , the occurrence of word not might bring a semantic reversal to the local region.
Revised: For example, in the sentence The food is not very good in this hotel , the occurrence of word not might bring a semantic reversal to the local region.

Original: Hence we compose the region embedding with word embeddingsof all words in the region as well as the local context unit of the central word.
Revised: We use the local context unit of the middle word and the original word embeddings in a region to perform the projected embeddings in a word-to-context view, where the projected embeddings can reﬂect the middle word’s inﬂuences on the context words.

Original: we call this model word-context region embedding.
Revised: Once the projected embeddings are obtained, a max pooling operation is applied to extract the most predictive features in the region. The output of the max pooling operation can be regarded as a task related region embedding in a word-to-context view, i.e. Word-Context region embedding.

Original: Formally, for a region ( i, c ) , given the context unit K w i of center word w i andthe embeddings of  words in the region, the projected embeddings are computed as the following matrix:
Revised: Formally, we use the context unit K w i of middle word w i and embeddings of all words in a region region ( i, c ) to compute the projected embedding matrix by equation (1), then the Word-Context region embedding r ( i,c ) can be obtained through a max pooling operation on the projected embedding matrix:

Original: [Equation]
Revised: [Equation]

Original: To compose therepresentation of a region region ( i, c ) noted as r i,c , we apply a max pooling  on the row dimension of the projected embedding matrix. 
Revised: where max standing for the max pooling operation on the column dimension of the input matrix. Finally, we get r ( i,c ) as a vector representation of region ( i, c ) with dimension h .

Original: Figure 1a shows the details of the ﬁrst model architecture.
Revised: Figure 1a shows the details of the ﬁrst model architecture.

Original: For instance, in the sentence The food is not very good in this hotel , the projected word embeddings in the region (3 , 5) are composed by the element-wise multiplications between columns in the local context unit of not and word embeddings of food , is , not , very and good .
Revised: For instance, in the sentence The food is not very good in this hotel , the projected word embeddings in the region (3 , 2) are composed by the element-wise multiplications between columns in the local context unit of not and word embeddings of food , is , not , very and good .

Original: The embedding r 3 , 5 of region (3 , 5) can be obtained by max pooling on  projected word embedding matrix.
Revised: The embedding r 3 , 2 of region (3 , 2) can be obtained by max pooling on the projected word embedding matrix.

Original: 3. C ONTEXT - WORD REGION EMBEDDING
Revised: 3. C ONTEXT - WORD REGION EMBEDDING

Original: The second architecture takes as a different view, which address the local context words’ inﬂuence on the center word in the region, and we call this model context-word region embedding.
Revised: The second architecture goes as a different view, which addresses the local context words’ inﬂuences on the middle word in the region, and we call this Context-Word region embedding.

Original: Similarly, for a region ( i, c ) , the projected embeddings are represent as the following matrix:
Revised: Similarly, for a region ( i, c ) , the projected embeddings are computed by the original word embedding of the middle

Revised: Under review as a conference paper word and the context units of all words in the region, then the Context-Word region embedding can be obtained by a max pooling operation through the column dimension of the projected embedding matrix:

Original: [Equation]
Revised: [Equation]

Original: Figure 1b shows the details of the second model architecture, the difference between our two models is that inputs of max pooling layer are different, the Word-Context model usesthe projected embeddings produced by the context unit of central word and word embeddings of context words, while the Context-Word model just the opposite.
Revised: Figure 1b shows the details of the second model architecture. our two models take different ways to produce the projected word embeddings, the Word-Context model uses context units of middle words and word embeddings of context words, while the Context-Word model uses context units of context words and word embeddings of middle word.

Original: 3. R EGION EMBEDDING FOR TEXT CLASSIFICATION
Revised: 3. R EGION EMBEDDING FOR TEXT CLASSIFICATION

Original: On text classiﬁcation task, the documentscommonly are of variable-length, which need to be represented as ﬁxed size vectors.
Revised: For text classiﬁcation, documents are usually variable-sized, which need to be represented as ﬁxed size vectors.

Original: In order to show the effectiveness of our proposed region embedding models, we just sum up the embeddings of all regions to represent a document, and feed it  an upper Full-Connected layer for text classiﬁcation task.
Revised: In order to show the effectiveness of our proposed region embedding models, we just sum up the embeddings of all regions to represent a document, and feed it to an upper FullConnected layer for text classiﬁcation task.

Original: Formally, the model can be represented as following:
Revised: Formally, the model can be represented as following:

Original: [Equation]
Revised: [Equation]

Original: where x denotes the input text sequence, W and b denote the weight matrix and bias of the fully connected layer respectively, g denotes the softmax function of the output layer, σ denotes the softsign function and n denotes the number of regions in the document, r is the region embedding which can be computed by the two proposed methods.
Revised: where x denotes the input text sequence, W and b denote the weight matrix and bias of the fully connected layer respectively, g denotes the softmax function of the output layer,  denotes the softsign function and n denotes the number of regions in a document, r is the region embedding which can be computed by the equation (2) or (3).

Original: E , U , W and b can be updated in the training period.
Revised: E , U , W and b can be updated in the training period.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We report the experiments with proposed models in comparison with previous models.
Revised: We report the experiments with proposed models in comparison with previous models.

Original: The Code is publicly available on the Internet.
Revised: The code is publicly available on the Internet.

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: We use publicly available datasets from Zhang et al.
Revised: We use publicly available datasets from Zhang et al.

Original: (2015) to evaluate our models.
Revised: (2015) to evaluate our models.

Original: There are  total 8 text classiﬁcation datasets, corresponding to sentiment analysis, news topics, question-answer, ontology extraction, respectively.
Revised: There are in total 8 text classiﬁcation datasets, corresponding to sentiment analysis,news classiﬁcation , questionanswer, ontology extraction tasks, respectively.

Original: Table 1 shows the descriptive statistics of datasets used in our experiments.
Revised: Table 1 shows the descriptive statistics of datasets used in our experiments.

Original: To guarantee comparable indications, same evaluation protocol of Zhang et al.
Revised: To guarantee comparable indications, same evaluation protocol of Zhang et al.

Original: (2015) is employed.
Revised: (2015) is employed

Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: Under review as a conference paper at ICLR

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: Our models are compared with several widely used supervised text classiﬁcation models.
Revised: Our models are compared with several widely used supervised text classiﬁcation models.

Original: We report the n-grams and TFIDF baselines from Zhang et al.
Revised: We report the n-grams and TFIDF baselines from Zhang et al.

Original: (2015), as well as the character level convolutional model (char-CNN) of Zhang & LeCun (2015), theword level convolutional model (wordCNN) of Zhang & LeCun (2015), the character based convolution recurrent network (char-CRNN) of Xiao & Cho (2016), the very deep convolutional network (VDCNN) of Conneau et al.
Revised: (2015), as well as the character level convolutional model (char-CNN) of Zhang & LeCun (2015), the character based convolution recurrent network (char-CRNN) of Xiao & Cho (2016), the very deep convolutional network (VDCNN) of Conneau et al.

Original: (2016), the Discriminative LSTM (D-LSTM) of Yogatama et al.
Revised: (2016), the Discriminative LSTM (D-LSTM) of Yogatama et al.

Original: (2017) and the bigram FastText (bigramFastText) of Joulin et al.(2016).
Revised: (2017) and the bigram FastText (bigram-FastText) of Joulin et al.

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: For data preprocessing, all the texts of datasets are tokenized by Stanford tokenizer and all words are converted to lower case.
Revised: For data preprocessing, all the texts of datasets are tokenized by Stanford tokenizer and all words are converted to lower case.

Original: Words that appear only once are treated as out-of-vocabulary (OOV) items, and any stop words or symbols are kept.
Revised: Words that appear only in one document are treated as out-of-vocabulary (OOV) items, and all stop words as well as symbols are kept.

Original: Additionally,padding length of c 2 are added to both the head and tail of each document.
Revised: Additionally, length of c padding are added to both the head and tail of each document.

Original: Optimal hyperparameters are tuned with 10 % of the training set on Yelp Review Full dataset, and identical hyperparameters are applied to all datasets: the dimension of word embedding is 128 , the region size is 7 which means the shape of local context unit matrix of each word is 128 × 7 , the  learning rate is set to 1 × 10 − 4 , and the batch size is 16 .
Revised: For our models, optimal hyperparameters are tuned with 10 % of the training set on Yelp Review Full dataset, and identical hyperparameters are applied to all datasets: the dimension of word embedding is 128 , the region size is 7 which means the shape of local context unit matrix of each word is 128 ⇥ 7 , the initial learning rate is set to 1 ⇥ 10  4 , and the batch size is 16 .

Original: For optimization, the embeddings of words and the units are randomly initialized with Gaussian Distribution.
Revised: For optimization, the embeddings of words and the units are randomly initialized with Gaussian Distribution.

Original: Adam (Kingma & Ba, 2014) are used as the optimizer.
Revised: Adam (Kingma & Ba, 2014) is used as the optimizer.

Original: We do not use any extra regularization methods, such as L2 normalization or dropout.
Revised: We do not use any extra regularization methods, like L2 normalization or dropout.

Original: Algorithms are entirely implemented with TensorFlow and trained on NVIDIA TeslaP
Revised: Algorithms are entirely implemented with TensorFlow and trained on NVIDIA Tesla

Original:  GPUs.
Revised: P40 GPUs.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Revised: Model

Revised: Yelp

Original: Table 2 is the summary of our experimental results.
Revised: [Table] Yelp [Table] Amz.

Original: performance summary of compared methods on listed datasets.
Revised: P. Amz [Table] Table 2 is the summary of the experimental results.

Original: We use underscores to represent the bestknown published results, and bold the best records.
Revised: We use underscores to represent the best published results, and bold the best records.

Original: On six datasets of eight, our models beat or match the state-of-the-art with a performance gain highest to 0.7(%).
Revised: On six datasets of eight, our models beat or match the state-of-the-art with a performance gain highest to 0.7%.

Original: We beat all the previous models on all datasets except VDCNN, while the latter performs almost best on all classiﬁcation tasks before.
Revised: We beat all the previous models on all datasets except VDCNN, while the latter performs almost best on all classiﬁcation tasks before.

Original: As a result, we slightly win VDCNN on six datasets and lost in two of Amazon datasets.
Revised: As a result, we slightly win VDCNN on six datasets and lost in two of Amazon datasets.

Original: Although the slight advantage, our methods are remarkable on account of its simplicity along with divergence.
Revised: Detailed experimental results including best performance epoch and training time for all listed datasets are reported in Appendix A.

Original: Particularly, the upper layer structure of our models only uses a summing up operation,[Table] which is more concise and robust than any other deep or complex models.
Revised: Furthermore, the upper layer structure of our models only uses a summing up operation, which is more concise and robust than any other deep or complex models.

Original: In fact [Table] both of our two proposed models are effective.
Revised: In fact, both of our two proposed models are effective against previous models.

Original: 4. E XPLORATORY E XPERIMENTS
Revised: 4. E XPLORATORY E XPERIMENTS

Original: In this subsection we are going to do a set of exploratory experiments to study therelative effect of each component in our model.
Revised: In this subsection we are going to do a set of exploratory experiments to study the effect of each component in our model.

Original: Typical cases will be analyzed to validate properties of various aspects of  our models.
Revised: Typical cases will be analyzed to validate properties of various aspects of Under as a our models.

Original: With the limitation of paper space, we only analyzed the Word-Context region embedding model in our exploratory experiments.
Revised: Considering the limitation of paper space, we only analyzed the Word-Context region embedding model in our exploratory experiments.

Original: 4.5. E FFECT OF REGION SIZE AND EMBEDDING SIZE
Revised: 4.5. E FFECT OF REGION SIZE AND EMBEDDING SIZE

Original: Our method uses a ﬁxed size of region as contextual information just like CNN.
Revised: Our method uses a ﬁxed size of region as contextual information just like CNN.

Original: So the selection of region size really matters.
Revised: So the selection of region size really matters.

Original: A small region may lose some long-distance patterns, whereas large regions will bring into more noises.
Revised: A small region may lose some long distance patterns, whereas large regions will bring into more noises.

Original: Luckily, our models seem to be fairly insensitive towards kinds of datasets.
Revised: Luckily, our models seem to be fairly insensitive towards kinds of datasets.

Original: Actually, we just select consistent conﬁguration of size 7 for entire datasets and it is able to outperform best published results ever.
Revised: Actually, we just use identical region size 7 for all datasets and it is able to outperform best published results ever.

Original: Figure 2 describes the performance on Yelp Review Full with different region sizes, and when the size equals to 1 , the result is quite close to unigram FastText(Accuracy 60.7%), but still get 0.6(%) promotion.
Revised: Figure 2a describes the performance on Yelp Review Full with different region sizes, and when the size equals to 1 , the result is quite close to unigram FastText(accuracy 60.7%), but still gets a 0.6% promotion.

Original: Intuitively, the central word can not inﬂuence other words except itself when the size equals to 1 .
Revised: Intuitively, the middle word can not inﬂuence other words except itself when the size equals to 1 .

Original: The performance increases with the growth of region size up to 7.
Revised: The performance increases with the growth of region size up to 7.

Original: To further study the effectiveness of region sizes, we experiment our models with the combination of multi region sizes. 
Revised: [Figure] Furthermore, we experiment our models with the combination of multi region sizes. Here we use the approach of sharing context units among each region, where the parameters of local context units of smaller regions are just the slice of the longest one. Region embeddings of different sizes are concatenated for ﬁnal classiﬁcation.

Original: In ﬁgure 2, the combination of multi region sizes 3,5,7 is slightly better than the best single region size 7.
Revised: In ﬁgure 2a, the combination of multi region sizes 3,5,7 is slightly better than the best single region size 7.

Original: The effectiveness of multi-size combination can be explained by the difference of inﬂuence ranges between words, for example,  word very only emphasizes the next word while however may lay stress on a wide range of the following words.
Revised: The effectiveness of multi-size combination can be explained by the difference of inﬂuence ranges between words. For example, in sentiment analysis, word very only emphasizes the next word while however may lay stress on a wide range of the following words.

Original: In addition to the analysis of region sizes, we further study the inﬂuence of word embedding dimensions.
Revised: In addition to the analysis of region sizes, we further study the inﬂuence of word embedding dimensions.

Original: Figure 3 lists the comparative results on Yelp Review Full with different embedding dimensions.
Revised: Figure 2b lists the comparative results on Yelp Review Full with different embedding dimensions.

Original: The result shows that our model is less prone to be over-ﬁtting than FastText and CNN with the word embedding dimension increasing.
Revised: The result shows that our model is more robust to overﬁtting than FastText and CNN with the word embedding dimension increasing.

Original: In fact, the amount of parameters in our models is relatively large.
Revised: In fact, the amount of parameters in our models is relatively large.

Original: Since we learn a speciﬁc unit for each word, under the same word embedding dimension, our parameter size has been expanded by region size times.
Revised: Since we learn a speciﬁc unit for each word, under the same word embedding dimension, our parameter size has been expanded by region size times, the parameters number is v ⇥ h + v ⇥ (2 ⇥ c + 1) ⇥ h + h ⇥ m + m , where m is the number of classes.

Original:  Notice that the sizes of parameters are relatively consistent among 1024 in FastText , 1024 in CNN and 128 in ours 
Revised: Speciﬁc numbers parameters for different region sizes are listed in Appendix A. Notice that the sizes of parameters are relatively consistent among 1024 in FastText , 1024 in CNN and 128 in ours .

Original: 4.5. E FFECT OF C ONTEXT UNIT
Revised: 4.5. E FFECT OF C ONTEXT UNIT

Original: In this section, we explore some comparative experiments to show the effectiveness of our proposed word speciﬁc context unit.
Revised: In this section, we explore some comparative experiments to show the effectiveness of our proposed word speciﬁc context unit.

Original: The experiments are employed based on unigram FastText baseline, which has similar upper layer structure with our models.
Revised: The experiments are employed based on unigram FastText baseline, which has similar upper layer structure with our models.

Original: Table 3 illustrates theinterrelated results.
Revised: Table 3 illustrates the results.

Revised: Firstly, we remove the entire context units from our model, which means it is just a variant version of unigram FastText, we call it FastText(Win-pool).

Original: For FastText(Unigram), embedding dimension is 10.
Revised: The difference is that FastText sums up the word [Table] Text(Unigram), embedding dimension is 10.

Original: For FastText(Win-pool), W.C.region.emb(Scalar) and
Revised: For FastText(Win-pool), W.C.region.emb(Scalar) and

Original: W.C.region.emb(our model), region size is 7 and embedding dimension is
Revised: W.C.region.emb(our model), region size is 7 and embedding dimension is.

Original: [Table]
Revised: [Table]

Original: Firstly, we remove the entire context units from our model, which means it is just a variant version of unigram FastText, we call it FastText(Win-pool).

Original: The difference is that FastText sums up the word embeddings directly while FastText(Win-pool) sums up the window pooled embeddings in a stride of 1.
Revised: embeddings directly while FastText(Win-pool) sums up the window pooled embeddings in a stride of 1.

Original: It yields a slightly accuracy gain of (0.28%) than unigram FastText.
Revised: It yields a slightly accuracy gain of 0.28% than unigram FastText.

Original: Secondly, we apply a simpliﬁed scalar version of context units to FastText(Win-pool).
Revised: Secondly, we apply a simpliﬁed scalar version of context units to FastText(Win-pool).

Original: Distinguishable, the context unit of each word has the shape with 1 × c  , hence it can be regarded as a broadcasting operation on corresponding word embeddings of its local context.
Revised: Distinguishable, the context unit of each word has the shape with 1 ⇥ (2 ⇥ c + 1) , hence it can be regarded as a broadcasting operation on corresponding word embeddings of its local context.

Original: We name this method W.C.region.emb(Scalar).
Revised: We name this method W.C.region.emb(Scalar).

Original: Compared to the non-scalar method, it yields a huge parameter size reduction. but it already yields a signiﬁcant gain of 2.45(%).
Revised: Compared to the non-scalar method, it yields a huge parameter size reduction, but it already yields a signiﬁcant gain of 2.45%.

Original: Furthermore, W.C.region.emb(our model) is the variant version of W.C.region.emb(Scalar) wherethe each column of scalar context unit is expanded to a dense vector.
Revised: Furthermore, W.C.region.emb(our model) is the variant version of W.C.region.emb(Scalar) where each column of scalar context unit is expanded to a dense vector.

Original: Each word’s context unit has a shape with h × c  .
Revised: Each word’s context unit has a shape with h ⇥ (2 ⇥ c +1) .

Original: Adding the low dimensional dense context unit improves  performance by 4.17(%).
Revised: Adding the low dimensional dense context unit improves the performance by 4.17%.

Original: We can sense much from the procedure of decomposition, with the help of context unit, even a simpler scalar version promotes a lot.
Revised: We can sense much from the procedure of decomposition, with the help of context unit, even a simpler scalar version promotes a lot.

Original: To have a better understanding of what context unit actually  are  adversarial  like however , but and modiﬁers like very , good , bad  are listed in Figure 4.
Revised: To have a better understanding of what context unit actually capture, heat maps are plotted for chosen word samples. Representative adversarial conjunctions like however , but , modiﬁers like very , good , bad and nouns like food , morning are listed in Figure 3.

Original: For each row of the ﬁgure, the intensity of the surrounding color box reﬂects the emphasis degree in the view of the central word.
Revised: For each row of the ﬁgure, the intensity of the surrounding color box reﬂects the emphasis degree in the view of the middle word.

Original: Qualitative but not fully rigorous, a normalized L2-norm  is used to render the shade.
Revised: Qualitative but not fully rigorous, a normalized L2-norm of each column in context unit is used to render the shade.

Original: Window size 7 is adopted default, annotation l i (i ≤ 3 ) is denoted as left columns of the speciﬁc context unit, while r i (i ≤ 3 ) denoted as the right part.
Revised: Region size 7 is adopted default, annotation l i (i  3 ) is denoted as left columns of the speciﬁc context unit, while r i (i  3 ) denoted as the right part.

Original: What the ﬁgure reﬂects are consistent with intuitive priors of human beings.
Revised: What the ﬁgure reﬂects are consistent with intuitive priors of human beings.

Original: In the perspective of however , right contexts play the key role for classiﬁcation polarity because of the emotional turnaround, the color is indeed deeper in r i than l i , so does but .
Revised: In the perspective of however , right contexts play the key role for classiﬁcation polarity because of the emotional reversal, the color is indeed deeper in r i than l i , so does but .

Original: For word very , r 1 is more prominent than the rest of all, which capture some modiﬁed patterns like very happy or very sad .
Revised: For word very , r 1 is more prominent than the rest of all, which captures some modiﬁed patterns like very happy or very sad .

Original: For word good , tendencies will be completely different for pattern not good , very good and not that good , which are intensive negative, intensive positive and slightly hesitated, separately, the position of lwill be strengthened as a result, so does word bad . 
Revised: For word good , tendencies will be completely different for patterns like not good , very good and not that good , which are intensive negative, intensive positive and slightly hesitated, separately, the position of lwill be strengthened as a result, so does word bad . There are signiﬁcant differences between two nouns food and morning .

Original:  Actually, from the motivation of word speciﬁed context units, we would like to believe this feature help capture syntactic and semantic inﬂuence on the relative position of its contexts. It can be regarded as a way to make use of word order in context unit covered regions.
Revised: The heat map of word food implies patterns like delicious food or food was mediocre , while the word morning has fewer valuable patterns for classiﬁcation. Actually, from the motivation of word speciﬁed context units, we would like to believe this feature helps capture syntactic and semantic inﬂuences of words on surrounding words at relative positions.

Original: 4.5. V ISUALIZATION
Revised: 4.5. V ISUALIZATION

Original: In this subsection, we will try to visualize the contribution of each word and selected phrase tothe classiﬁcation.
Revised: In this subsection, we will try to visualize the contribution of each word and selected phrase to classiﬁcation.

Original: Detailed visualization techniques have been introduced in Li et al.
Revised: Detailed visualization techniques have been introduced in Li et al.

Original: Here we generalize itthe to the color rendering of multi-category version.
Revised: Here we generalize it to the color rendering of multi-category version.

Original: Notice that  not the self embedding acts here, but the  projected embedding of one on words’ [Table]
Revised: Notice that for our model, not the original embedding acts here, but the accumulation of the projected embedding of each word on its surrounding words.

Original: With
Revised: For clarity, we choose a binary classiﬁcation task of sentiment analysis. In Table 4, we list two cases in Yelp Review Polarity dataset, in which our model behaves as

Original: [Table]
Revised: [Table]

Original: remarkable , 
Revised: With C-unit get your wallet ready , the prices are

Original: [Table]
Revised: [Table]

Original: For clarity, we choose a binary classiﬁcation task of sentiment analysis.
Revised: high

Original: In Table 4, we list two real case samples in Yelp Review Polarity dataset, in which our model behave as expected. Words or artiﬁcially selected phrases are highlighted green if they are positive factors, red if they are negative The intensity of the color indicates the extends of the polarity.
Revised: [Table]

Original: To have a better comparison, the methods of with and without context unit  are both visualized.
Revised: To have a better comparison, the results of with and without context unit methods are both visualized.

Original: We abbreviate them as With C-unit and Without C-unit , respectively.
Revised: We abbreviate them as With C-unit and No C-unit , respectively.

Original: For sentence get your wallet ready, the prices are crazy high , if no context unit is adopted, the word color will be reﬂected by its only embedding.
Revised: For sentence get your wallet ready, the prices are crazy high , if no context unit is adopted, the word color reﬂects its word embedding, which is context-free.

Original: The word crazy iscolored positive, and high colored negative.
Revised: The polarity of crazy is positive, and high is negative.

Original: Because the intensity of crazy is higher than high , theentire polarity of phrase prices are crazy high is totally positive, which is acomplete mistake.
Revised: Because the intensity of crazy is higher than high , the polarity of phrase prices are crazy high is totally positive, which is a mistake.

Original: But with context unit, things have changed quite a bit.
Revised: But with context unit, things have changed quite a bit, the polarities of words are context dependent.

Original: Because of the exist of high as the context, the positive polarity of crazy vanishes. The negative polarity of high is stronger, phrase prices are crazy high performs positive overall.
Revised: Under the inﬂuence of high , the positive polarity of crazy vanishes and phrase prices are crazy high performs negative overall.

Original: For another case nothing remarkable, but not bad either , things seem more interesting.
Revised: For another case nothing remarkable, but not bad either , things seem more interesting.

Original: If without context-unit , remarkable will be positive, while nothing, not, bad perform negative, respectively.
Revised: Without context-unit , remarkable is positive, while nothing, not, bad perform negative, respectively.

Original: But with context-unit, the polarity of the part ahead of but is reduced, meanwhile the polarity of not and bad ﬂip attractively.
Revised: But with context unit, the polarity of the part ahead of but weakens, meanwhile the polarities of not and bad ﬂips.

Original: As a result, phrase but not bad either performs positive overall.
Revised: As a result, phrase but not bad either performs positive overall.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper propose two novel architectures for text classiﬁcation tasks that learn task speciﬁc region embeddings without hand crafted features.
Revised: This paper proposed two novel architectures for text classiﬁcation tasks, which learn task speciﬁc region embeddings without hand crafted features.

Original: To utilize the information of words’ relative positions, we learn a local context unit for each word  in addition to word embedding.
Revised: To utilize the word speciﬁc inﬂuences of each word on its context words, a local context unit for each word is learned in addition to word embedding.

Original: Our models achieve state-ofthe-art performances on six benchmark text classiﬁcation datasets, and the visualization experiments show that our proposed local context unit can capture the semantic and synthetic information for each word.
Revised: Our models achieve state-of-the-art performances on six benchmark text classiﬁcation datasets, and the visualization experiments show that our proposed local context unit can capture the semantic and syntactic information for each word.

Original: Noticedthat the power of the local context unit on learning task related region embeddings, we are interested in its ability to  semi-supervised learning.
Revised: Noticed the power of the local context unit on learning task related region embeddings, we are interested in its ability to unsupervised and semi-supervised learning.

Original: At the same time, we are also curious about whether we can do more natural language processing tasks by adding more complexand deep upper layers.
Revised: At the same time, we are also curious about whether we can achieve better results by introducing more complex upper layers on text classiﬁcation, and other natural language processing tasks.

Revised: APPENDIX A D ETAILED EXPERIMENTAL RESULTS

Revised: To have a further insight about our reported results, we list the training time and best testing performance epoch for kinds of region sizes( 5,7,9 ) in detail.

Revised: For all the 8 datasets in Table 5, hyperparameters are kept in line with section 4.3.

Revised: The dimension of word embedding is 128 , the region size is 7 , the initial learning rate is set to 1 ⇥ 10  4 , and the batch size is 16 .

Revised: Epoch index starts from 0 and training time is reported per epoch.

Revised: We choose model of word-context region embedding here.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: We also report results of several repeated runs in Table 6 to exclude effects of randomness and ensure reproducibility.

Revised: Five independent tries are conducted on each dataset of Yelp.

Revised: P and Yelp.

Revised: F, where both performance variances are within 0.11% on accuracy.

Revised: [Table]