Original: B AG OF REGION EMBEDDINGS VIA LOCAL CONTEXT UNIT FOR TEXT CLASSIFICATION
Revised: B AG OF REGION EMBEDDINGS VIA LOCAL CONTEXT UNIT FOR TEXT CLASSIFICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Contextual information and word orders are proved valuable for text classiﬁcation task.
Revised: Contextual information and word orders are proved valuable for text classiﬁcation task.

Original: To make use of local word order information, n-grams are commonly used features in several models, such as linear models.
Revised: To make use of local word order information, n-grams are commonly used features in several models, such as linear models.

Original: However, these models commonly suffer the data sparsity problem and are difﬁcult to represent large size region.
Revised: However, these models commonly suffer the data sparsity problem and are difﬁcult to represent large size region.

Original: The discrete or distributed representations of n-grams can be regarded as region embeddings, which are representations of ﬁxed size regions.
Revised: The discrete or distributed representations of n-grams can be regarded as region embeddings, which are representations of ﬁxed size regions.

Original: In this paper, we propose two novel text classiﬁcation models that learn task speciﬁc region embeddings without hand crafted features, hence the drawbacks of n-grams can be overcome.
Revised: In this paper, we propose two novel text classiﬁcation models that learn task speciﬁc region embeddings without hand crafted features, hence the drawbacks of n-grams can be overcome.

Original: In our model, each word has two attributes, a commonly used word embedding, and an additional local context unit which is used to interact with the word’s local context.
Revised: In our model, each word has two attributes, a commonly used word embedding, and an additional local context unit which is used to interact with the word’s local context.

Original: Both the units and word embeddings are used to generate representations of regions, and are learned as model parameters.
Revised: Both the units and word embeddings are used to generate representations of regions, and are learned as model parameters.

Original: Finally, bag of region embeddings of a document is fed to a linear classiﬁer.
Revised: Finally, bag of region embeddings of a document is fed to a linear classiﬁer.

Original: Experimental results show that our proposed methods achieve state-of-the-art performance on several benchmark datasets.
Revised: Experimental results show that our proposed methods achieve state-of-the-art performance on several benchmark datasets.

Original: We provide visualizations and analysis illustrating that our proposed local context unit can capture the syntactic and semantic information.
Revised: We provide visualizations and analysis illustrating that our proposed local context unit can capture the syntactic and semantic information.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Text classiﬁcation is an important task in natural language processing, which has been studied for years with applications such as topic categorization, web search, sentiment classiﬁcation.
Revised: Text classiﬁcation is an important task in natural language processing, which has been studied for years with applications such as topic categorization, web search, sentiment classiﬁcation.

Original: A simple yet effective approach for text classiﬁcation is to represent documents as bag-of-words, and train a linear classiﬁer, e.g., a logistic regression, a support vector machines (Joachims, 1998; Fan et al., 2008) or a naive bayes (McCallum et al., 1998).
Revised: A simple yet effective approach for text classiﬁcation is to represent documents as bag-of-words, and train a linear classiﬁer, e.g., a logistic regression, a support vector machines (Joachims, 1998; Fan et al., 2008) or a naive bayes (McCallum et al., 1998).

Original: Although bag-of-words methods are efﬁcient and popular, they lose the local ordering information of words which has been proved useful particularly on sentiment classiﬁcation (Pang et al., 2002).
Revised: Although bag-of-words methods are efﬁcient and popular, they lose the local ordering information of words which has been proved useful particularly on sentiment classiﬁcation (Pang et al., 2002).

Original: To beneﬁt from local word order for text classiﬁcation, n-grams which is a simple statistics of ordered word combinations usually performs good (Pang et al., 2002;
Revised: To beneﬁt from local word order for text classiﬁcation, n-grams which is a simple statistics of ordered word combinations usually performs good (Pang et al., 2002;

Original: Wang & Manning, 2012; Joulin et al., 2016).
Revised: Wang & Manning, 2012; Joulin et al., 2016).

Original: With a n-gram vocabulary, Wang & Manning (2012) use one-hot vectors as discrete embedding of n-grams.
Revised: With a n-gram vocabulary, Wang & Manning (2012) use one-hot vectors as discrete embedding of n-grams.

Original: Recently, FastText (Joulin et al., 2016) directly learns distributed embeddings of n-grams and words for the task of interest.
Revised: Recently, FastText (Joulin et al., 2016) directly learns distributed embeddings of n-grams and words for the task of interest.

Original: In FastText, learning the embedding of n-gram can be regarded as learning low-dimensional vector representation of a small ﬁxed size region, i.e, region embedding.
Revised: In FastText, learning the embedding of n-gram can be regarded as learning low-dimensional vector representation of a small ﬁxed size region, i.e, region embedding.

Original: Although the n-grams was widely used, two caveats come with n-grams: 1) the vocabulary size increases exponentially with n , and yielding a high-dimensional feature space, which makes it difﬁcult to apply large region size(e.g., n > 4 ), such as People ’s Republic of China 2)  the representation of each n-gram is looked up by an unique id, the data sparsity problem cannot be avoided, for example, the similar n-grams a very good movie and an excellent moive are trained seperately.
Revised: Although the n-grams was widely used, two caveats come with n-grams: 1) the vocabulary size increases exponentially with n , and yielding a high-dimensional feature space, which makes it difﬁcult to apply large region size(e.g., n > 4 ), such as People ’s Republic of China 2) Since the representation of each n-gram is looked up by an unique id, the data sparsity problem cannot be avoided, for example, the similar n-grams a very good movie and an excellent movie are trained separately.

Original: In this paper, we focus on learning task speciﬁc region embedding for text classiﬁcation, while Johnson & Zhang (2015) learn the task independent region embedding from unlabeled data.
Revised: In this paper, we focus on learning task speciﬁc region embedding for text classiﬁcation, while Johnson & Zhang (2015) learn the task independent region embedding from unlabeled data.

Original: Instead of using hand crafted n-grams features, we directly learn the region embedding based on word sequence, thus the limitations of n-grams can be avoided.
Revised: Instead of using hand crafted n-grams features, we directly learn the region embedding based on word sequence, thus the limitations of n-grams can be avoided.

Original: Intuitively, given a word sequence in a ﬁxed size region, both words and words’ relative positions are the key information to generate the semantic of the region.
Revised: Intuitively, given a word sequence in a ﬁxed size region, both words and words’ relative positions are the key information to generate the semantic of the region.

Original: In this work, each word has two components, a commonly used word embedding, and an additional local context unit.
Revised: In this work, each word has two components, a commonly used word embedding, and an additional local context unit.

Original: The local context unit of a word is parameterized by a matrix, and each column is used to interact with another local context word in particular relative position, hence the  local context unit utilizes the information of words’ relative positions.
Revised: To utilize the information of words’ relative positions, the local context unit of a word is parameterized by a matrix, and each column is used to interact with the local context word in corresponding relative position, which can be also regarded as a linear projection function on the embedding of corresponding word. The local context units and word embeddings are used to generate representations of regions from two perspectives. For text classiﬁcation task, our models just use bag of region embeddings to represent the document, and feed the document representation to a fully connected linear classiﬁer. The word embeddings and local context units are jointly trained on the classiﬁcation task.

Original: The convolutional networks(CNN) were proposed to make use of internal structure (word order) implicitly for text classiﬁcation (Kim, 2014) (Johnson & Zhang, 2014).
Revised: Recently, several neural models are proposed to make use of local ordered information in word sequence, and convolutional neural networks(CNN) (Kim, 2014) (Johnson & Zhang, 2014) are most similar with our works.

Original: The essence of CNN is to learn embeddings for small ﬁxed size regions(e.g., ”not good” in a document), and each kernel of the convolutional layer tries to capture a speciﬁc semantic or structural feature.
Revised: The essence of CNN is to learn embeddings for small ﬁxed size regions, and each kernel of the convolutional layer tries to capture a speciﬁc semantic or structural feature.

Original: The kernel of CNN is shared globally, thus commonly several kernels are needed to preserve task related features, while the proposed local context kernel is word speciﬁc and forced to capture the particular semantic of the word to its local context.
Revised: The kernel of CNN is shared globally, and usually several kernels are needed to preserve task related features, while the proposed local context unit is word dependent and can be applied to capture the particular semantic of the word to its local context.

Original: Moreover, the convolution kernels extract the predictive feature by applying convolution operation on the words vector sequence, while the local context kernels are used as distinct linear projection functions onthe context words in different relative position. For text classiﬁcation, we simply use bag of region embeddings to represent the document, and apply an additional non-linear activation function and a fully connected layer. The word embeddings and local context kernels are jointly trained on the classiﬁcation task.
Revised: Moreover, the convolution kernels extract the predictive feature by applying convolution operation on the word sequence, while our local context units are used as distinct linear projection functions on context words in corresponding relative positions.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Text classiﬁcation has been studied for years, traditional approaches focused on feature engineering and using different types of machine learning algorithms.
Revised: Text classiﬁcation has been studied for years, traditional approaches focused on feature engineering and using different types of machine learning algorithms.

Original: For feature engineering, bag-of-words features are efﬁcient and popular.
Revised: For feature engineering, bag-of-words features are efﬁcient and popular.

Original: In addition, the hand-crafted n-grams or phrases are added to make use of word order in text data, which has been shown effectiveness in Wang & Manning (2012).
Revised: In addition, the hand-crafted n-grams or phrases are added to make use of word order in text data, which has been shown effective in Wang & Manning (2012).

Original: For machine learning algorithms, linear classiﬁers are widely used, such as naive bayes (McCallum et al., 1998), logistic regression and support vector machines (Joachims, 1998; Fan et al., 2008).
Revised: For machine learning algorithms, linear classiﬁers are widely used, such as naive bayes (McCallum et al., 1998), logistic regression and support vector machines (Joachims, 1998; Fan et al., 2008).

Original: Howerve, these models commonly suffer the data sparsity problem.
Revised: However, these models commonly suffer the data sparsity problem.

Original: Recently, several neural models have been proposed, the pre-trained word embeddings of word2vec (Mikolov et al., 2013) have been widely used as the inputs to deep neural models such as recursive tensor networks (Socher et al., 2013).
Revised: Recently, several neural models have been proposed, the pre-trained word embeddings of word2vec (Mikolov et al., 2013) have been widely used as the inputs to deep neural models such as recursive tensor networks (Socher et al., 2013).

Original: On the other hand, models that can directly learn the word embeddingsspeciﬁcally for the task have been proposed recently, such as FastText (Joulin et al., 2016) and CNN (Kim, 2014), which both have been successfully applied on text classiﬁcation tasks.
Revised: On the other hand, models that can directly learn task speciﬁc word embeddings have been proposed recently, such as FastText (Joulin et al., 2016) and CNN (Kim, 2014), which both have been successfully applied on text classiﬁcation tasks.

Original: In the rest of this section, we brieﬂy introduce FastText and CNN, the two approaches that are most similar to ours.
Revised: In the rest of this section, we brieﬂy introduce FastText and CNN, the two approaches that are most similar to ours.

Original: FastText FastTextsimply averages the word embeddings to represent the document, and uses a full connected linear layer as the classiﬁer.
Revised: FastText FastText averages the word embeddings to represent a document, and uses a full connected linear layer as the classiﬁer.

Original: The word embeddings are trained for each task speciﬁcally.
Revised: The word embeddings are trained for each task speciﬁcally.

Original: To utilize the local word order  of small regions, FastText adds hand-crafted n-grams features.
Revised: To utilize the local word order information of small regions, FastText uses hand-crafted n-grams as features in addition to single words.

Original: With the simplemodel architecture, FastText has been proved to be effective and highly efﬁcient on text classiﬁcation tasks.
Revised: With the simple architecture, FastText has been proved to be effective and highly efﬁcient on text classiﬁcation tasks.

Original: Similarly, our models use bag of region embeddings to represent the document, and followed by the same full-connected layer.
Revised: Similarly, our models use bag of region embeddings to represent a document, and use the same linear classiﬁer.

Original: Differently, our models directly learn the semantics of regions based on words, hand-crafted features is not required.
Revised: Differently, our models directly learn the semantics of regions based on word sequence, hand-crafted features is not required.

Original: CNN CNN is a feed-forward network with convolution layers interleaved with pooling layers, which are originally used for image processing tasks.
Revised: CNN CNN is a feed-forward network with convolution layers interleaved with pooling layers, which are originally used for image processing tasks.

Original: For natural language processing, words are commonly converted to vectors.
Revised: For natural language processing, words are commonly converted to vectors.

Original: CNN directly applies convolutional layer on word vectors, both word vectors and the shared(word independent) kernles are the parameters of CNN, which can be learned to capture the predictive structures of small regions.
Revised: CNN directly applies convolutional layer on word vectors, both word vectors and the shared(word independent) kernels are the parameters of CNN, which can be learned to capture the predictive structures of small regions.

Original: Our purpose is similar with CNN, which tries to learn task speciﬁc representations for regions.
Revised: Our purpose is similar with CNN, which tries to learn task speciﬁc representations for regions.

Original: We apply local context kernels on word vectors,unlike CNN, our kernels are word dependent and the sepciﬁc kernel is learned for each word.
Revised: Unlike CNN, we apply local context units on word vectors, our units are word dependent.

Original:  M ETHOD
Revised:  M ETHOD

Original: Several previous works have been proposed to show the effectiveness of word order on text classiﬁcation task, such as statistical n-gram features, or apply CNNs to learn an embedding of a ﬁxed size region.
Revised: Several previous works have been proposed to show the effectiveness of word order on text classiﬁcation task, such as statistical n-gram features, or apply CNNs to learn an embedding of a ﬁxed size region.

Original: In this paper we also focus on learning the representations of small text regions which preserve the local internal structure information for text classiﬁcation.
Revised: In this paper we also focus on learning the representations of small text regions which preserve the local internal structure information for text classiﬁcation.

Original: The regions in document can be treat as ﬁxed length continues subsequence of the document.
Revised: The regions in document can be treat as ﬁxed length continues subsequence of the document.

Original: More speciﬁcally, with w i stands for the i -th(starting from 0) word of the document, we use region ( i, c ) to denote the region with middle word w i , where c is the region size.
Revised: More speciﬁcally, with w i stands for the i -th(starting from 0) word of the document, we use region ( i, c ) to denote the region with middle word w i , where c is the region size.

Original: For instance, given a sentence such as The food is not very good in this hotel , region (3 , 5) means the subsequence food is not very good .
Revised: For instance, given a sentence such as The food is not very good in this hotel , region (3 , 5) means the subsequence food is not very good .

Original: Considering the semantic inﬂuences from words to their context are diverse from each other, we take a different way to learn effective region embeddings on text classiﬁcation task.

Original: In our approach, besides the word embedding, each word has  an additional  context unit to interact with the context.
Revised: In this work, each word has two components, a word embedding and an additional local context unit.

Original: With the local context unit to each word, different semantic inﬂuences from the word to its context words can be addressed. In the ability to capture the inﬂuences through the local context units, we proposed two methods to perform the regional embeddings on text classiﬁcation from different views.
Revised: With the units and embeddings of words in a particular region, the representation of the region can be generated from two perspectives.

Original: In the rest of this section, we will introduce the local context units ﬁrstly, and two new architectures to produce the region embeddings using local context units will be introduced, ﬁnally we will introduce how we use the region embeddings on text classiﬁcation.
Revised: In the rest of this section, we will introduce the local context units ﬁrstly, and two new architectures to generate the region embeddings with local context units will be introduced, ﬁnally we will introduce how we use the region embeddings on text classiﬁcation.

Original: [Figure]
Revised: [Figure]

Original: 3. L OCAL CONTEXT UNITS
Revised: 3. L OCAL CONTEXT UNITS

Original: Words are commonly converted to low dimensional vectors when apply neural network model on natural language processing tasks.
Revised: In natural language processing tasks, words are commonly converted to low dimensional vectors as the inputs to neural networks.

Original: More formally, the embedding e w of word w is represented by a column in a matrix E ∈ R h × v with a look up layer, where v is the size of vocabulary, h is the embedding size.
Revised: More formally, the embedding e w of word w is represented by a column in a matrix E ∈ R h × v with a look up layer, where v is the size of vocabulary, h is the embedding size.

Original: Under the consideration that the occurrence of a word inﬂuences differently on the semantics of context words, we apply an additional local context unit for each word as well as the  word embedding.
Revised: To utilize information of words’ relative position, we learn a local context unit for each word in addition to word embedding, which both the unit and word embedding can be learned as model parameters.

Original: We deﬁne the local context unit K w
Revised: Formally, we deﬁne the local context unit K w

Original: i ∈
Revised: i ∈

Original: R h × c of w i as a matrix which can be looked uplayer in a third-order matrix U ∈ R
Revised: R h × c of w i as a matrix which can be looked up in the matrix U ∈ R

Original: h × c × v bythe index of w i  inthe vocabulary.
Revised: h × c × v by w i ’s index in vocabulary.

Original: Each column in the unit matrix of a given word can  interact  the the context words in a speciﬁc relative position.
Revised: Each column in K w i can be used to interact with the local context word in corresponding relative position of w i .

Original: In fact, the columns in the local context unit  can be regarded as diverse linear projection functions on the embeddings of words in the local region.
Revised: In fact, the columns of a unit matrix can be regarded as distinction linear projection functions on the embeddings of words in the local context.

Original: The parameters of these projection functions(i.e., columns of the unit matrix) can be learned to capture the semantic and syntactic inﬂuence from the word to its context.
Revised: The parameters of these projection functions(i.e., columns of each unit matrix) can be learned to capture the semantic and syntactic inﬂuence from the word to its context.

Original: We call theoutput of these projection functions projected word embeddings, which represented the new semantics of words in the local region under the inﬂuence of the center word.
Revised: Word embeddings are used as inputs to the projection functions, and we call the outputs projected word embeddings.

Original: Formally, given the unit K w i of w i and the embedding e w i + t of w i + t , the ( c 2 + t ) -th column in K w i is denoted by K w i ,t , where − c 2 < =
Revised: Formally, given the unit K w i of w i and the embedding e w i + t of w i + t , the ( c 2 + t ) -th column in K w i is denoted by K w i ,t , where − c 2 < =

Original: t < =
Revised: t < =

Original: c 2 .
Revised: c 2 .

Original: We apply a element-wise multiplication(denoted by (cid:12) ) to compute the projected word embedding e iw i + t of w i + t as:
Revised: We apply a element-wise multiplication(denoted by (cid:12) ) to compute the projected word embedding e iw i + t of w i + t as:

Original: [Equation]
Revised: [Equation]

Original: Note that the middle column K w i , c 2 of K w i can be regarded as a linear projection function on e w i itself, which makes e iw i in the same  space with other projected embeddings.
Revised: Note that the middle column K w i , c 2 of K w i can be regarded as a linear projection function on e w i itself, which transforms e iw i to the same feature space as other projected embeddings’.

Original: Since context words in different relative position are projected by different linear functions, the proposed local context units can utilize the  word order in a different way.
Revised: For a local context word in the particular relative position of w i , there is a corresponding linear projection function(a particular column of K w i ), thus our proposed local context units can utilize the local ordered word information in a novel way.

Original: 3. W ORD - CONTEXT REGION EMBEDDING
Revised: 3. W ORD - CONTEXT REGION EMBEDDING

Original: We proposed two architectures to perform the region embedding from different perspectives.
Revised: We proposed two architectures to perform the region embedding from different perspectives.

Original: In the ﬁrst proposed architecture, we focus on addressing the inﬂuence from the middle word to the context words.
Revised: In the ﬁrst proposed architecture, we focus on addressing the inﬂuence from the middle word to the context words.

Original: For example, in the sentence The food is not very good in this hotel , the occurrence of word not might bring a semantic reversal to the local region.
Revised: For example, in the sentence The food is not very good in this hotel , the occurrence of word not might bring a semantic reversal to the local region.

Original: Hence we compose the region embedding with word embeddings of all words in the region as well as the local context unit of the central word.
Revised: Hence we compose the region embedding with word embeddings of all words in the region as well as the local context unit of the central word.

Original: we call this model word-context region embedding.
Revised: we call this model word-context region embedding.

Original: Formally, for a region ( i, c ) , given the context unit K w i of center word w i and the embeddings of words in the region, the projected embeddings can be computed as the following matrix:
Revised: Formally, for a region ( i, c ) , given the context unit K w i of center word w i and the embeddings of words in the region, the projected embeddings are computed as the following matrix:

Original: [Equation]
Revised: [Equation]

Original: To compose the representation of a region region ( i, c ) noted as r i,c , we apply a max pooling on the column dimension of the projected embedding matrix.
Revised: To compose the representation of a region region ( i, c ) noted as r i,c , we apply a max pooling on the row dimension of the projected embedding matrix.

Original: Figure 1a shows the details of the ﬁrst model architecture.
Revised: Figure 1a shows the details of the ﬁrst model architecture.

Original: For instance, in the sentence The food is not very good in this hotel , the projected word embeddings in the region (3 , 5) are produced by the element-wise multiplications between columns in the local context unit of not and word embeddings ofwords food , is , not , very and good .
Revised: For instance, in the sentence The food is not very good in this hotel , the projected word embeddings in the region (3 , 5) are composed by the element-wise multiplications between columns in the local context unit of not and word embeddings of food , is , not , very and good .

Original: The embedding r 3 , 5 of region (3 , 5) can be obtained by max pooling on projected word embedding matrix.
Revised: The embedding r 3 , 5 of region (3 , 5) can be obtained by max pooling on projected word embedding matrix.

Original: 3. C ONTEXT - WORD REGION EMBEDDING
Revised: 3. C ONTEXT - WORD REGION EMBEDDING

Original: The second architecture takes as a different view, which address the local context words’ inﬂuence on the center word in the region, and we call this model context-word region embedding.
Revised: The second architecture takes as a different view, which address the local context words’ inﬂuence on the center word in the region, and we call this model context-word region embedding.

Original: Similarly, for a region ( i, c ) , the projected embeddings can be represent as the following matrix:
Revised: Similarly, for a region ( i, c ) , the projected embeddings are represent as the following matrix:

Original: [Equation]
Revised: [Equation]

Original: Since the units of words in the local context contains semantic information as well as the word embeddings, the region embedding r i,c can be composed with a max pooling on the column dimension of the projected embedding matrix.

Original: Figure 1b shows the details of the second model architecture. The difference between our two models is thatthe region embedding is produced by different inputs the max pooling operation applied on, the Word-Context model uses the projected embedding produced by the context unit of central word and word embedding of context words, while the
Revised: Figure 1b shows the details of the second model architecture, the difference between our two models is that inputs of max pooling layer are different, the Word-Context model uses the projected embedding produced by the context unit of central word and word embedding of context words, while the

Original: Context-Word  just the opposite.
Revised: Context-Word model just the opposite.

Original: 3. R EGION EMBEDDING FOR TEXT CLASSIFICATION
Revised: 3. R EGION EMBEDDING FOR TEXT CLASSIFICATION

Original: On text classiﬁcation task, the documents commonly are of variable-length, which need to be represented as ﬁxed size vectors.
Revised: On text classiﬁcation task, the documents commonly are of variable-length, which need to be represented as ﬁxed size vectors.

Original: In order to show the effectiveness of our proposed region embedding models, we simply sum  the embeddings of all regionsin one document with an additional nonlinear activation function to represent the document, and  an upper Full-Connected layeris used for text classiﬁcation.
Revised: In order to show the effectiveness of our proposed region embedding models, we just sum up the embeddings of all regions to represent a document, and feed it an upper Full-Connected layer for text classiﬁcation task.

Original: Formally, the model can be represented as following:
Revised: Formally, the model can be represented as following:

Original: [Equation]
Revised: [Equation]

Original: where x denotes the input text sequence, W and b denote the weight matrix and bias of the fully connected layer respectively, g denotes the softmax function of the output layer, σ denotes the softsign function and n denotes the number of regions in the document, r is the region embedding which can be computed by the two proposed methods.
Revised: where x denotes the input text sequence, W and b denote the weight matrix and bias of the fully connected layer respectively, g denotes the softmax function of the output layer, σ denotes the softsign function and n denotes the number of regions in the document, r is the region embedding which can be computed by the two proposed methods.

Original: E , U , W and b can be updated in the training period.
Revised: E , U , W and b can be updated in the training period.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We report the experiments with proposed models in comparison with previous models.
Revised: We report the experiments with proposed models in comparison with previous models.

Original: The Code is publicly available on the Internet.
Revised: The Code is publicly available on the Internet.

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: We use publicly available datasets from Zhang et al.
Revised: We use publicly available datasets from Zhang et al.

Original: (2015) to evaluate our models.
Revised: (2015) to evaluate our models.

Original: There are total 8 text classiﬁcation datasets, corresponding to sentiment analysis, news topics, question-answer, ontology extraction, respectively.
Revised: There are total 8 text classiﬁcation datasets, corresponding to sentiment analysis, news topics, question-answer, ontology extraction, respectively.

Original: Table 1 shows the descriptive statistics of datasets used in our experiments.
Revised: Table 1 shows the descriptive statistics of datasets used in our experiments.

Original: To guarantee comparable indications, same evaluation protocol of Zhang et al.
Revised: To guarantee comparable indications, same evaluation protocol of Zhang et al.

Original: (2015) is employed.
Revised: (2015) is employed.

Original: [Table]
Revised: [Table]

Revised: 4. B ASELINES

Original: Our models are compared with several widely used supervised text classiﬁcation models.
Revised: Our models are compared with several widely used supervised text classiﬁcation models.

Original: We report the n-grams and TFIDF baselines from Zhang et al.
Revised: We report the n-grams and TFIDF baselines from Zhang et al.

Original: (2015), as well as the character level convolutional model (char-CNN) of Zhang & LeCun (2015), the word level convolutional model (wordCNN) of Zhang & LeCun (2015), the character based convolution recurrent network (char-CRNN) of Xiao & Cho (2016), the very deep convolutional network (VDCNN) of Conneau et al.
Revised: (2015), as well as the character level convolutional model (char-CNN) of Zhang & LeCun (2015), the word level convolutional model (wordCNN) of Zhang & LeCun (2015), the character based convolution recurrent network (char-CRNN) of Xiao & Cho (2016), the very deep convolutional network (VDCNN) of Conneau et al.

Original: (2016), the Discriminative LSTM (D-LSTM) of Yogatama et al.
Revised: (2016), the Discriminative LSTM (D-LSTM) of Yogatama et al.

Original: (2017) and the bigram Fasttext (bigramFasttext) of Joulin et al.
Revised: (2017) and the bigram Fasttext (bigramFasttext) of Joulin et al.

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: For data preprocessing, all the texts of datasets are tokenized by Stanford tokenizer and all words are converted to lower case.
Revised: For data preprocessing, all the texts of datasets are tokenized by Stanford tokenizer and all words are converted to lower case.

Original: Words that appear only once are treated as out-of-vocabulary (OOV) items, and any stop words or symbols are kept.
Revised: Words that appear only once are treated as out-of-vocabulary (OOV) items, and any stop words or symbols are kept.

Original: Additionally, padding lenght of c 2 are added to both the head and tail of each document.
Revised: Additionally, padding length of c 2 are added to both the head and tail of each document.

Original: Optimal hyperparameters are tuned with 10 % of the training set on Yelp Review Full dataset, and identical hyperparameters are applied to all datasets: the dimension of word embedding is 128 , the region size is 7 which means the shape of local context kernel matrix of each word is 128 × 7 , the learning rate is set  1 × 10 − 4 , and the batch size is 16 .
Revised: Optimal hyperparameters are tuned with 10 % of the training set on Yelp Review Full dataset, and identical hyperparameters are applied to all datasets: the dimension of word embedding is 128 , the region size is 7 which means the shape of local context unit matrix of each word is 128 × 7 , the learning rate is set to 1 × 10 − 4 , and the batch size is 16 .

Original: For optimization, the embeddings of words and kernels are randomly initialized with Gaussian Distribution.
Revised: For optimization, the embeddings of words and the units are randomly initialized with Gaussian Distribution.

Original: Adam (Kingma & Ba, 2014) are used as the optimizer, Cross-entropy loss with softmax is applied meanwhile.
Revised: Adam (Kingma & Ba, 2014) are used as the optimizer.

Original: We do not use any extra regularization methods, such as L2 normalization or dropout.
Revised: We do not use any extra regularization methods, such as L2 normalization or dropout.

Original: Algorithms are entirely implemented with TensorFlow and trained on NVIDIA Tesla P40 GPUs.
Revised: Algorithms are entirely implemented with TensorFlow and trained on NVIDIA Tesla P GPUs.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: [Table] Table 2 is a performace summary of compared methods on listed datasets.
Revised: Table 2 is the summary of our experimental results. performance summary of compared methods on listed datasets.

Original: We use underscores to represent the best known published results, anduse the boldto indicate the new records.
Revised: We use underscores to represent the best known published results, and bold the best records.

Original: On six datasets of eight, our models achieve new state-of-art or equally best with a performamce gain highest to 0.7(%).
Revised: On six datasets of eight, our models beat or match the state-of-the-art with a performance gain highest to 0.7(%).

Original: We beat all the previous models on all datasets except VDCNN, while the latter performes almost best on all classiﬁcation tasks before the proposing of our methods.
Revised: We beat all the previous models on all datasets except VDCNN, while the latter performs almost best on all classiﬁcation tasks before.

Original: As a result, we slightly win VDCNN on six datasets and lost in two of Amazon datasets.
Revised: As a result, we slightly win VDCNN on six datasets and lost in two of Amazon datasets.

Original: Although the slight advantage, our methods are remarkable on account of its simplicity along with divergence.
Revised: Although the slight advantage, our methods are remarkable on account of its simplicity along with divergence.

Original: Concretly, The upper layer structure of our models only uses a summing up operation,  which is more concise and robust than any other deep or complex models.
Revised: Particularly, the upper layer structure of our models only uses a summing up operation, [Table] which is more concise and robust than any other deep or complex models.

Original: On the other hand, different with the fashionable convolutional kernels, which are shared across all the sequence, we use the self context-aided kernel for each word. To our best known, we are the ﬁrst to introduce word speciﬁc kernels techniques.
Revised: In fact

Original: In fact, we list the comparation of our two proposed models. Both of them perform fairly well.
Revised: [Table] both of our two proposed models are effective.

Original: In general, Word-Context model is slightly better than Context-Word model.

Original: The difference is that

Original: Context-Word model uses the same embedding of center word to get the projected word embeddings while Word-Context model uses different embeddings of words in one region, so the latter may use more decision units in each pooling operation than the former.

Original: 4. E XPLORATORY E XPERIMENTS
Revised: 4. E XPLORATORY E XPERIMENTS

Original: In this subsection we are going to do a set of exploratory experiments to study the relative effect of each component in our model.
Revised: In this subsection we are going to do a set of exploratory experiments to study the relative effect of each component in our model.

Original: Typical cases will be analyzed to validate properties of various aspects of our models.
Revised: Typical cases will be analyzed to validate properties of various aspects of our models.

Original: Without loss of generality, we only analyzed the Word-Context region embedding model in our exploratory experiments.
Revised: With the limitation of paper space, we only analyzed the Word-Context region embedding model in our exploratory experiments.

Original: 4.5. E FFECT OF REGION SIZE AND EMBEDDING SIZE
Revised: 4.5. E FFECT OF REGION SIZE AND EMBEDDING SIZE

Original: Our method uses a ﬁxed size of region as contextual information just like CNN.
Revised: Our method uses a ﬁxed size of region as contextual information just like CNN.

Original: So the selection of region size really matters.
Revised: So the selection of region size really matters.

Original: A small region may loseof some long-distance patterns, whereas large regions will lead into more noise.
Revised: A small region may lose some long-distance patterns, whereas large regions will bring into more noises.

Original: Luckily, our models seem to be fairly insensitive towards kinds of datasets.
Revised: Luckily, our models seem to be fairly insensitive towards kinds of datasets.

Original: Actually, we just select consistent conﬁguration of size 7 for entire datasets and it has been able to outperform best published results ever.
Revised: Actually, we just select consistent conﬁguration of size 7 for entire datasets and it is able to outperform best published results ever.

Original: Figure 2 describes the performance in Yelp Review Full with different region sizes, and when the size equals to 1 , the result is quite close to unigram Fasttext(Acc 60.7%), but still get 0.6%+ promotion.
Revised: Figure 2 describes the performance on Yelp Review Full with different region sizes, and when the size equals to 1 , the result is quite close to unigram Fasttext(Accuracy 60.7%), but still get 0.6(%) promotion.

Original: Intuitively, the central word can not inﬂuence other words except itself when the size equals to 1 .
Revised: Intuitively, the central word can not inﬂuence other words except itself when the size equals to 1 .

Original: The performance increases with the growth of region size up to 7, followed by a drop.
Revised: The performance increases with the growth of region size up to 7.

Original: To further study the effectiveness of region sizes, we experimented our models with the combination of multi region sizes.
Revised: To further study the effectiveness of region sizes, we experiment our models with the combination of multi region sizes.

Original: Figure 2 shows the comparisons between single region sizes and the multi-size combination of 3,5,7 , which shows that the combination of multi region sizes  is slightly better than the best single region size 7.
Revised: In ﬁgure 2, the combination of multi region sizes 3,5,7 is slightly better than the best single region size 7.

Original: The effectiveness of multi-size combination can be explained by the difference of inﬂuence ranges between words, such as very only emphasizes the next word while however may lay stress on a wide range of the following words.
Revised: The effectiveness of multi-size combination can be explained by the difference of inﬂuence ranges between words, for example, word very only emphasizes the next word while however may lay stress on a wide range of the following words.

Original: In addition to the analysis of region sizes, we further studied the inﬂuence of word embedding dimensions.
Revised: In addition to the analysis of region sizes, we further study the inﬂuence of word embedding dimensions.

Original: Figure 3 lists the comparative results on Yelp Review Full with different embedding dimensions.
Revised: Figure 3 lists the comparative results on Yelp Review Full with different embedding dimensions.

Original: The results show that our model is less prone to be overﬁtting than Fasttext and CNN with the word embedding dimension increasing.
Revised: The result shows that our model is less prone to be over-ﬁtting than Fasttext and CNN with the word embedding dimension increasing.

Original: In fact, the amount of parameters in our models is relatively large.
Revised: In fact, the amount of parameters in our models is relatively large.

Original: Since we learn a speciﬁc kernel for each word, under the same word embedding dimension, our parameter size has been expanded by region size times.
Revised: Since we learn a speciﬁc unit for each word, under the same word embedding dimension, our parameter size has been expanded by region size times.

Original: Notice that the sizes of parameters are relatively consistent among 1024 in Fasttext , 1024 in CNN and 128 in ours.
Revised: Notice that the sizes of parameters are relatively consistent among 1024 in Fasttext , 1024 in CNN and 128 in ours

Original: 4.5. E FFECT OF KERNELS
Revised: 4.5. E FFECT OF C ONTEXT UNIT

Original: In this section, we explored some comparative experiments to show the effectiveness of our proposed word kernels.
Revised: In this section, we explore some comparative experiments to show the effectiveness of our proposed word speciﬁc context unit.

Original: The experiments are employed based on unigram Fastext baseline, which has similar upper layer structure with our models.
Revised: The experiments are employed based on unigram Fasttext baseline, which has similar upper layer structure with our models.

Original: Table 3 illustrated the interrelated results.
Revised: Table 3 illustrates the interrelated results.

Original: Firstly, we remove the entire kernels from our model, which means it is just a variant version of unigram fasttext, we call it Fasttext(Win-pool).
Revised: For Fast (Unigram

Original: The difference isthat Fasttext sums up the word
Revised: , embedding dimension is Fasttext pool W region Scalar

Original: [Table]
Revised: [Table]

Revised: Firstly, we remove the entire context units from our model, which means it is just a variant version of unigram Fasttext, we call it Fasttext(Win-pool).

Original:  embeddings directly while Fasttext(Win-pool) sums up the window pooled embeddings in a stride of 1.
Revised: The difference is that Fasttext sums up the word embeddings directly while Fasttext(Win-pool) sums up the window pooled embeddings in a stride of 1.

Original: It yields a slightly accuracy gain of 0.28% than unigram fasttext.
Revised: It yields a slightly accuracy gain of (0.28%) than unigram Fasttext.

Original: Secondly, we apply a simpliﬁed scalar version of kernel to Fasttext(Win-pool).
Revised: Secondly, we apply a simpliﬁed scalar version of context units to Fasttext(Win-pool).

Original: Unlike our proposed kernels, the kernel of w i has the shape with 1 × c , hence the kernel can be regarded as scalling on corresponding word embeddings in its local context.
Revised: Distinguishable, the context unit of each word has the shape with 1 × c , hence it can be regarded as a broadcasting operation on corresponding word embeddings of its local context.

Original: We call this method C.W.region.emb(Scalar).
Revised: We name this method W.C.region.emb(Scalar).

Original: Compared to our models, it yields a huge parameter size reduction, however, it already yileds a signiﬁcant gain of 2.45% compared to unigram Fastext.
Revised: Compared to the non-scalar method, it yields a huge parameter size reduction. but it already yields a signiﬁcant gain of 2.45(%).

Original: Futermore, C.W.region.emb(our model) is the variant version of C.W.region.emb(Scalar) that the scala kernel is expanded to a dense vector, which the kernel of w i has  shape with h × c .
Revised: Furthermore, W.C.region.emb(our model) is the variant version of W.C.region.emb(Scalar) where the each column of scalar context unit is expanded to a dense vector. Each word’s context unit has a shape with h × c .

Original: Adding the low dimensional dense kernels improvesthe performance by 4.17% compared to unigram Fasttext and 1.72% to C.W.region.emb(Scalar).
Revised: Adding the low dimensional dense context unit improves performance by 4.17(%).

Original: The results proved that our proposed word kernels can considerably improves the performance compared to the unigram fasttext evenwith a simpler scalar version.
Revised: We can sense much from the procedure of decomposition, with the help of context unit, even a simpler scalar version promotes a lot.

Original: In order to further analyze what kernels actually capture, Figure 4 presents the heat maps of word kernels for chosen word samples, such as however ,  very , e.g. each rowrepresent a word kernel of the center word, r i denoted as i -th columns L norm on the right in the word kernel matrix and similar as l i .
Revised: To have a better understanding of what context unit actually are adversarial like however , but and modiﬁers like very , good , bad are listed in Figure 4. For each row of the ﬁgure, the intensity of the surrounding color box reﬂects the emphasis degree in the view of the central word.

Original: Qualitative but not fully rigorous a normalized L norm used to render shade intensity
Revised: Qualitative but not fully rigorous, a normalized L2-norm is used to render the shade.

Original:  i 
Revised: Window size 7 is adopted default, annotation l i (i ≤ 3 ) is denoted as left columns of the speciﬁc context unit, while r i (i ≤ 3 ) denoted as the right part.

Original: What the ﬁgure reﬂects keeps consistent with intuitive priors of human beings.
Revised: What the ﬁgure reﬂects are consistent with intuitive priors of human beings.

Original: In the perspective of however , right contexts play the key role for classiﬁcation polarity because of the emotional turnaround, the color is indeed deeper in r i than l i , so does but .
Revised: In the perspective of however , right contexts play the key role for classiﬁcation polarity because of the emotional turnaround, the color is indeed deeper in r i than l i , so does but .

Original: For word very , r 1 is more prominent than the rest of all, which capture some modiﬁed patterns like very happy or very sad .
Revised: For word very , r 1 is more prominent than the rest of all, which capture some modiﬁed patterns like very happy or very sad .

Original: For word good , tendencies will be completely different for pattern not good , very good and not that good , which are intensive negative, intensive positive and slightly hesitated, separately, the position of lwill be strengthened as a result, so does word bad .
Revised: For word good , tendencies will be completely different for pattern not good , very good and not that good , which are intensive negative, intensive positive and slightly hesitated, separately, the position of lwill be strengthened as a result, so does word bad .

Original: Actually, from the motivation of word speciﬁed kernels, we would like to believe this feature help capture syntactic and semantic inﬂuence on the relative position of its contexts.
Revised: Actually, from the motivation of word speciﬁed context units, we would like to believe this feature help capture syntactic and semantic inﬂuence on the relative position of its contexts.

Original: It can be regarded as a way to make use of word order in kernel-covered regions
Revised: It can be regarded as a way to make use of word order in context unit covered regions.

Original: 4.5. V ISUALIZATION
Revised: 4.5. V ISUALIZATION

Original: In this subsection, we will try to visualize the contribution of each word and selected phrase to the classiﬁcation.
Revised: In this subsection, we will try to visualize the contribution of each word and selected phrase to the classiﬁcation.

Original: Detailed visualization techniques have been introduced in Li et al.
Revised: Detailed visualization techniques have been introduced in Li et al.

Original: Here we generalize it the to the color rendering of multi-category version.
Revised: Here we generalize it the to the color rendering of multi-category version.

Original: Notice that not the self embedding acts here, but the projected embedding ofself one onsurrounding words’context unit.
Revised: Notice that not the self embedding acts here, but the projected embedding of one on words’

Original: [Table]
Revised: [Table]

Original: unit
Revised: With

Original: [Table]
Revised: [Table]

Original: With C-unit get your wallet ready ,the prices are crazy high
Revised: remarkable ,

Original: [Table]
Revised: [Table]

Original: No C-unit nothing remarkable , but not bad either but not bad either

Original: With C-unit nothing remarkable , but not bad either but not bad either

Original: For clarity, we choose a binary classiﬁcation task of sentiment analysis.
Revised: For clarity, we choose a binary classiﬁcation task of sentiment analysis.

Original: In Table 4, we list two real case samples in Yelp Review Polarity dataset, in which our model behave as expected.
Revised: In Table 4, we list two real case samples in Yelp Review Polarity dataset, in which our model behave as expected.

Original: Words or artiﬁcially selected phrases are highlighted green if they are positive factors, red if they are negative.
Revised: Words or artiﬁcially selected phrases are highlighted green if they are positive factors, red if they are negative

Original: The intensity of the color indicates the extends of the polarity.
Revised: The intensity of the color indicates the extends of the polarity.

Original: To have a better comparation, the methods of with and without context unit are both visualized.
Revised: To have a better comparison, the methods of with and without context unit are both visualized.

Original: We abbreviate them as With C-unit and Without C-unit , respectively.
Revised: We abbreviate them as With C-unit and Without C-unit , respectively.

Original: For sentence get your wallet ready, the prices are crazy high , if no context unit is adopted, the word color will be reﬂected by its only embedding.
Revised: For sentence get your wallet ready, the prices are crazy high , if no context unit is adopted, the word color will be reﬂected by its only embedding.

Original: The word crazy is colored positive, and high colored negative.
Revised: The word crazy is colored positive, and high colored negative.

Original: Because the intensity of crazy is higher than high , the entire polarity of phrase prices are crazy high is totally positive, which is a complete mistake.
Revised: Because the intensity of crazy is higher than high , the entire polarity of phrase prices are crazy high is totally positive, which is a complete mistake.

Original: But with context unit, things have changed quite a bit.
Revised: But with context unit, things have changed quite a bit.

Original: Because of the exist of high as the context, the positive polarity of crazy vanishes.
Revised: Because of the exist of high as the context, the positive polarity of crazy vanishes.

Original: The negative polarity of high is stronger, phrase prices are crazy high performs positive overall.
Revised: The negative polarity of high is stronger, phrase prices are crazy high performs positive overall.

Original: For another case nothing remarkable, but not bad either , things seem more interesting.
Revised: For another case nothing remarkable, but not bad either , things seem more interesting.

Original: If without context-unit , remarkable will be positive, while nothing, not, bad perform negative, respectively.
Revised: If without context-unit , remarkable will be positive, while nothing, not, bad perform negative, respectively.

Original: But with context-unit, the polarity of the part ahead of but is reduced, meanwhile the polarity of not and bad ﬂip attractively.
Revised: But with context-unit, the polarity of the part ahead of but is reduced, meanwhile the polarity of not and bad ﬂip attractively.

Original: As a result, phrase but not bad either performs positive overall.
Revised: As a result, phrase but not bad either performs positive overall.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper proposed two novel architectures for text classiﬁcation  that learns embeddings of small text regions by introduced local context kernels.
Revised: This paper propose two novel architectures for text classiﬁcation tasks that learn task speciﬁc region embeddings without hand crafted features.

Original: Without aid of additional resources , the embeddings and kernels are trained speciﬁcally for the task of interest.
Revised: To utilize the information of words’ relative positions, we learn a local context unit for each word in addition to word embedding.

Original: Beneﬁting the simplicity of our models, we achieved state-of-the-art performances on six benchmark text classiﬁcation datasets with identical hyperparameters .
Revised: Our models achieve state-ofthe-art performances on six benchmark text classiﬁcation datasets, and the visualization experiments show that our proposed local context unit can capture the semantic and synthetic information for each word.

Revised: Noticed that the power of the local context unit on learning task related region embeddings, we are interested in its ability to semi-supervised learning.

Revised: At the same time, we are also curious about whether we can do more natural language processing tasks by adding more complex and deep upper layers.