Original: R EGION EMBEDDING VIA LOCAL CONTEXT KERNEL FOR TEXT CLASSIFICATION
Revised: B AG OF REGION EMBEDDINGS VIA LOCAL CONTEXT UNIT FOR TEXT CLASSIFICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper proposed two novel text classiﬁcation model architectures with effective extracting of local predictive structures which can be represented as lowdimensional vector for small text regions, i.e., region embeddings. To produce the region embeddings we proposed the local context kernels, which are word speciﬁc parameter matrices.
Revised: Contextual information and word orders are proved valuable for text classiﬁcation task. To make use of local word order information, n-grams are commonly used features in several models, such as linear models. However, these models commonly suffer the data sparsity problem and are difﬁcult to represent large size region.

Original: The local context kernels can be learned to capture the speciﬁc relationships between words and their local context. With the kernels, we proposed two methods to produce the region embeddings from two different perspectives.
Revised: The discrete or distributed representations of n-grams can be regarded as region embeddings, which are representations of ﬁxed size regions. In this paper, we propose two novel text classiﬁcation models that learn task speciﬁc region embeddings without hand crafted features, hence the drawbacks of n-grams can be overcome. In our model, each word has two attributes, a commonly used word embedding, and an additional local context unit which is used to interact with the word’s local context.

Original: For text classiﬁcation, we simply sum the region embeddings to represent the text, and use an upper fully connected layer to achieve the classiﬁcation task. The local context kernels and word embeddings are trained as part of the models.
Revised: Both the units and word embeddings are used to generate representations of regions, and are learned as model parameters. Finally, bag of region embeddings of a document is fed to a linear classiﬁer.

Original: Experimental results show that the proposed methodscan achieve new state of art results on several benchmark text classiﬁcation tasks with identical hyperparameters setting.
Revised: Experimental results show that our proposed methods achieve state-of-the-art performance on several benchmark datasets.

Original: We provide visualizations and analysis showing that the learned kernels can capture  syntactic and semantic resemblance between words.
Revised: We provide visualizations and analysis illustrating that our proposed local context unit can capture the syntactic and semantic information.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Text classiﬁcation is an important task in natural language processing, which has been studied for years with applications such as topic categorization, web search, sentiment classiﬁcation.
Revised: Text classiﬁcation is an important task in natural language processing, which has been studied for years with applications such as topic categorization, web search, sentiment classiﬁcation.

Original: A simple yet effective approach for text classiﬁcation is to represent documents as bag-of-words, and train a linear classiﬁer, e.g., a logistic regression, a support vector machines (Joachims, 1998; Fan et al., 2008) or a naive bayes (McCallum et al., 1998).
Revised: A simple yet effective approach for text classiﬁcation is to represent documents as bag-of-words, and train a linear classiﬁer, e.g., a logistic regression, a support vector machines (Joachims, 1998; Fan et al., 2008) or a naive bayes (McCallum et al., 1998).

Original: Although bag-of-words methods are efﬁcient and popular, they lose the local ordering information of words which has been proved useful particularly on sentiment classiﬁcation (Pang et al., 2002).
Revised: Although bag-of-words methods are efﬁcient and popular, they lose the local ordering information of words which has been proved useful particularly on sentiment classiﬁcation (Pang et al., 2002).

Original: To beneﬁt from local word order for text classiﬁcation, n-grams which is a simple statistics of ordered word combinations usually performs good (Pang et al., 2002;
Revised: To beneﬁt from local word order for text classiﬁcation, n-grams which is a simple statistics of ordered word combinations usually performs good (Pang et al., 2002;

Original: Wang & Manning, 2012; Joulin et al., 2016).
Revised: Wang & Manning, 2012; Joulin et al., 2016).

Original: With a n-gram vocabulary, Wang & Manning (2012)simply use one-hot vectors as discrete embedding of n-grams.
Revised: With a n-gram vocabulary, Wang & Manning (2012) use one-hot vectors as discrete embedding of n-grams.

Original: Recently, FastText (Joulin et al., 2016) directly learns distributed embeddings of n-grams and words for the task of interest.
Revised: Recently, FastText (Joulin et al., 2016) directly learns distributed embeddings of n-grams and words for the task of interest.

Original: In FastText, learning the embedding of ngram can be regarded as learning low-dimensional vector representation of a small ﬁxed size region, i.e, region embedding.
Revised: In FastText, learning the embedding of n-gram can be regarded as learning low-dimensional vector representation of a small ﬁxed size region, i.e, region embedding.

Original: Although the n-grams was widely used, two caveats come with n-grams: 1) the vocabulary size increases exponentially with n , and yielding a high-dimensional feature space, which makes it difﬁcult to apply large region size(e.g., n > 4 ). A look up layer with an unique id is used to generate representation for each n-gram, thus the data sparsity problem cannot be avoided.
Revised: Although the n-grams was widely used, two caveats come with n-grams: 1) the vocabulary size increases exponentially with n , and yielding a high-dimensional feature space, which makes it difﬁcult to apply large region size(e.g., n > 4 ), such as People ’s Republic of China 2) the representation of each n-gram is looked up by an unique id, the data sparsity problem cannot be avoided, for example, the similar n-grams a very good movie and an excellent moive are trained seperately.

Original: In this article, we focus on learning task speciﬁc region embedding for text classiﬁcation, while Johnson & Zhang (2015) learn the task independent region embedding from unlabeled data.
Revised: In this paper, we focus on learning task speciﬁc region embedding for text classiﬁcation, while Johnson & Zhang (2015) learn the task independent region embedding from unlabeled data.

Original: Instead of using handcrafted n-grams features, we directly learn the region embedding based on word sequence inputs, which solves the data sparsity problem.
Revised: Instead of using hand crafted n-grams features, we directly learn the region embedding based on word sequence, thus the limitations of n-grams can be avoided.

Original: An additional advantage is that our model is ﬂexible for region sizes, which makes is easily to learn the embedding of large size region.

Original: (Intuitively, given a word sequence in a ﬁxed size region, both words and words’ relative position are the key informations to generate the semantic of the region.
Revised: Intuitively, given a word sequence in a ﬁxed size region, both words and words’ relative positions are the key information to generate the semantic of the region.

Original: Noticed that semantics of regions are composed of words in each region, nevertheless each word has a speciﬁc inﬂuence to its local context, which means semantics of regions are composed differently, we proposed the word speciﬁc local context kernels to capture the inﬂuences from words to their local context.

Original: In our approach, each word has  an additional local context kernel besides the original word embedding).
Revised: In this work, each word has two components, a commonly used word embedding, and an additional local context unit.

Original: To utilize the key infor- mations, we proposed an additional local context kernel for each word besides word embeddings. The local context kernel of a word is  a parameter matrix whose columns can be learned to address the different inﬂuence to the context words in different relative position. With local context kernels we can implicitly learn the region embeddings by ordered words inputs. Beneﬁted from the density of word embedding and the proposed context kernel, our method can apply larger region size and better generalization than n-grams.
Revised: The local context unit of a word is parameterized by a matrix, and each column is used to interact with another local context word in particular relative position, hence the local context unit utilizes the information of words’ relative positions.

Original: The convolutional networks(CNN) were proposed to make use of internal structure (word order) implicitly for text classiﬁcation (Kim, 2014)
Revised: The convolutional networks(CNN) were proposed to make use of internal structure (word order) implicitly for text classiﬁcation (Kim, 2014)

Original: (Johnson & Zhang, 2014).
Revised: (Johnson & Zhang, 2014).

Original: The essence of CNN is to learn embeddings for small ﬁxed size regions(e.g., ”not good” in a document), and each kernel of the convolutional layer tries to capture a speciﬁc semantic or structural feature.
Revised: The essence of CNN is to learn embeddings for small ﬁxed size regions(e.g., ”not good” in a document), and each kernel of the convolutional layer tries to capture a speciﬁc semantic or structural feature.

Original: The kernel of CNN is shared globally, thus commonly several kernels are needed to preserve task related features, while the proposed local context kernel is word speciﬁc and forced to capture the particular semantic of the word to its local context.
Revised: The kernel of CNN is shared globally, thus commonly several kernels are needed to preserve task related features, while the proposed local context kernel is word speciﬁc and forced to capture the particular semantic of the word to its local context.

Original: Moreover, the convolution kernels extract the predictive feature by applying convolution operation on the words vector sequence, while the local context kernels are used as distinct linear projection functions on the context words in different relative position.
Revised: Moreover, the convolution kernels extract the predictive feature by applying convolution operation on the words vector sequence, while the local context kernels are used as distinct linear projection functions on the context words in different relative position.

Original: For text classiﬁcation, we simply use bag of region embeddings to represent the document, and apply an additional non-linear activation function and a fully connected layer.
Revised: For text classiﬁcation, we simply use bag of region embeddings to represent the document, and apply an additional non-linear activation function and a fully connected layer.

Original: The word embeddings and local context kernels are jointly trained on the classiﬁcation task.
Revised: The word embeddings and local context kernels are jointly trained on the classiﬁcation task.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Text classiﬁcation has been studied for years, traditional approaches focused on feature engineering and using different types of machine learning algorithms.
Revised: Text classiﬁcation has been studied for years, traditional approaches focused on feature engineering and using different types of machine learning algorithms.

Original: For feature engineering, bag-of-words features are efﬁcient and popular.
Revised: For feature engineering, bag-of-words features are efﬁcient and popular.

Original: In addition, the hand-crafted n-grams or phrases are added to make use of word order in text data, which has been shown effectiveness in Wang & Manning (2012).
Revised: In addition, the hand-crafted n-grams or phrases are added to make use of word order in text data, which has been shown effectiveness in Wang & Manning (2012).

Original: For machine learning algorithms, linear classiﬁers are widely used, such as naive bayes (McCallum et al., 1998), logistic regression and support vector machines (Joachims, 1998; Fan et al., 2008).
Revised: For machine learning algorithms, linear classiﬁers are widely used, such as naive bayes (McCallum et al., 1998), logistic regression and support vector machines (Joachims, 1998; Fan et al., 2008).

Original: Howerve, these models commonly suffer the data sparsity problem.
Revised: Howerve, these models commonly suffer the data sparsity problem.

Original: Recently, several neural models have been proposed, the pre-trained word embeddings of word2vec (Mikolov et al., 2013) have been widely used as the inputs to deep neural models such as recursive tensor networks (Socher et al., 2013).
Revised: Recently, several neural models have been proposed, the pre-trained word embeddings of word2vec (Mikolov et al., 2013) have been widely used as the inputs to deep neural models such as recursive tensor networks (Socher et al., 2013).

Original: On the other hand, models that can directly learn the word embeddings speciﬁcally for the task have been proposed recently, such as FastText (Joulin et al., 2016) and CNN (Kim, 2014), which both have been successfully applied on text classiﬁcation tasks.
Revised: On the other hand, models that can directly learn the word embeddings speciﬁcally for the task have been proposed recently, such as FastText (Joulin et al., 2016) and CNN (Kim, 2014), which both have been successfully applied on text classiﬁcation tasks.

Original: In the rest of this section, we brieﬂy introduce FastText and CNN, the two approaches that are most similar to ours.
Revised: In the rest of this section, we brieﬂy introduce FastText and CNN, the two approaches that are most similar to ours.

Original: FastText FastText simply averages the word embeddings to represent the document, and uses a full connected linear layer as the classiﬁer.
Revised: FastText FastText simply averages the word embeddings to represent the document, and uses a full connected linear layer as the classiﬁer.

Original: The word embeddings are trained for each task speciﬁcally.
Revised: The word embeddings are trained for each task speciﬁcally.

Original: To utilize the local word order of small regions, FastText adds hand-crafted n-grams features.
Revised: To utilize the local word order of small regions, FastText adds hand-crafted n-grams features.

Original: With the simple model architecture, FastText has been proved to be effective and highly efﬁcient on text classiﬁcation tasks.
Revised: With the simple model architecture, FastText has been proved to be effective and highly efﬁcient on text classiﬁcation tasks.

Original: Similarly, our models use bag of region embeddings to represent the document, and followed by the same full-connected layer.
Revised: Similarly, our models use bag of region embeddings to represent the document, and followed by the same full-connected layer.

Original: Differently, our models directly learn the semantics of regions based on words, hand-crafted features is not required.
Revised: Differently, our models directly learn the semantics of regions based on words, hand-crafted features is not required.

Original: CNN CNN is a feed-forward network with convolution layers interleaved with pooling layers, which are originally used for image processing tasks.
Revised: CNN CNN is a feed-forward network with convolution layers interleaved with pooling layers, which are originally used for image processing tasks.

Original: For natural language processing, words are commonly converted to vectors.
Revised: For natural language processing, words are commonly converted to vectors.

Original: CNN directly applies convolutional layer on word vectors, both word vectors and the shared(word independent) kernles are the parameters of CNN, which can be learned to capture the predictive structures of small regions.
Revised: CNN directly applies convolutional layer on word vectors, both word vectors and the shared(word independent) kernles are the parameters of CNN, which can be learned to capture the predictive structures of small regions.

Original: Our purpose is similar with CNN, which tries to learn task speciﬁc representations for regions.
Revised: Our purpose is similar with CNN, which tries to learn task speciﬁc representations for regions.

Original: We apply local context kernels on word vectors, unlike CNN, our kernels are word dependent and the sepciﬁc kernel is learned for each word.
Revised: We apply local context kernels on word vectors, unlike CNN, our kernels are word dependent and the sepciﬁc kernel is learned for each word.

Original:  M ETHOD
Revised:  M ETHOD

Original: Several previous works have been proposed to show the effectiveness of word order on text classiﬁcation task, such as statistical n-gram features, or applying CNNs to learn an embedding of a ﬁxed size region.
Revised: Several previous works have been proposed to show the effectiveness of word order on text classiﬁcation task, such as statistical n-gram features, or apply CNNs to learn an embedding of a ﬁxed size region.

Original: In this article we also focused on learning the representations of small text regions which preserve the local internal structure information for text classiﬁcation.
Revised: In this paper we also focus on learning the representations of small text regions which preserve the local internal structure information for text classiﬁcation.

Original: The regions in text can be deﬁned as ﬁxed length substrings of the document.
Revised: The regions in document can be treat as ﬁxed length continues subsequence of the document.

Original: More speciﬁcally, with w i stands  the i -th word of the document, we use region ( i, c ) to denote the regtion with center word w i , where c is the region size.
Revised: More speciﬁcally, with w i stands for the i -th(starting from 0) word of the document, we use region ( i, c ) to denote the region with middle word w i , where c is the region size.

Original: For instance, given a sentence such as The food is not very good in this hotel , region (3 , 5) means the substring food is not very good .
Revised: For instance, given a sentence such as The food is not very good in this hotel , region (3 , 5) means the subsequence food is not very good .

Original: Considering the semantic inﬂuences from words to their context are diverse from each other, we take a different way to learning effective region representations on text classiﬁcation.
Revised: Considering the semantic inﬂuences from words to their context are diverse from each other, we take a different way to learn effective region embeddings on text classiﬁcation task.

Original: Instead of using n-grams or global convolutional kernels, we proposed the local context kernels to address the word speciﬁc inﬂuences to their context.

Original: In our approach, besides theoriginal word embedding, each word has a context kernel to interact with the context.
Revised: In our approach, besides the word embedding, each word has an additional context unit to interact with the context.

Original: The details of the local context kernels will be introduced later.
Revised: With the local context unit to each word, different semantic inﬂuences from the word to its context words can be addressed.

Original: In the ability to capture the inﬂuences through the local context kernels, we proposed two methods to perform the regional embeddings on text classiﬁcation from different views.
Revised: In the ability to capture the inﬂuences through the local context units, we proposed two methods to perform the regional embeddings on text classiﬁcation from different views.

Original: In the rest of this section, we will introduce the local context kernels ﬁrstly, and two new architectures to produce the region embeddings using local context kernels will be introduced, ﬁnally we will introduce how we use the region embeddings on text classiﬁcation.
Revised: In the rest of this section, we will introduce the local context units ﬁrstly, and two new architectures to produce the region embeddings using local context units will be introduced, ﬁnally we will introduce how we use the region embeddings on text classiﬁcation.

Original: [Figure]
Revised: [Figure]

Original:  L OCAL CONTEXT KERNELS
Revised: 3. L OCAL CONTEXT UNITS

Original: Words are commonly converted to low dimensionaldense vectors when apply neural network model on natural language processing tasks.
Revised: Words are commonly converted to low dimensional vectors when apply neural network model on natural language processing tasks.

Original: More formally, the embedding e wi of word wi is represented by a column in a matrix E ∈ R h × v with a look up layer, where v is the size of vocabulary, h is the embedding size.
Revised: More formally, the embedding e w of word w is represented by a column in a matrix E ∈ R h × v with a look up layer, where v is the size of vocabulary, h is the embedding size.

Original: The columns of E are indexed by positions of the words in the vocabulary.

Original: Under the considerationof that the occurrence of a word inﬂuences  the semantics of context words differently, we design an additionalspeciﬁc local context kernel for each word.
Revised: Under the consideration that the occurrence of a word inﬂuences differently on the semantics of context words, we apply an additional local context unit for each word as well as the word embedding.

Original: We deﬁne the local context kernel
Revised: We deﬁne the local context unit K w

Original: K w i ∈
Revised: i ∈

Original: R h × c of w i as a matrix which can be obtained through a loop up layer in  U ∈ R
Revised: R h × c of w i as a matrix which can be looked up layer in a third-order matrix U ∈ R

Original: h × c × v by the index of w i in the vocabulary.
Revised: h × c × v by the index of w i in the vocabulary.

Original: We hope that each column in the kernel matrix of a given word can address theparticular semantic and syntactic inﬂuence to the  words in a given relative position.
Revised: Each column in the unit matrix of a given word can interact the the context words in a speciﬁc relative position.

Original: The columns in the local context kernel of can be regarded as diverse linear projection functions on the embeddings of words in the local region.
Revised: In fact, the columns in the local context unit can be regarded as diverse linear projection functions on the embeddings of words in the local region.

Original: The parameters of these projection functions(i.e., columns of the kernel matrix) can be learned to capture the semantic and syntactic inﬂuence from the word to its context.
Revised: The parameters of these projection functions(i.e., columns of the unit matrix) can be learned to capture the semantic and syntactic inﬂuence from the word to its context.

Original: We call the output of these projection functions projected word embeddings, which represented the new semantics of words in the local region under the inﬂuence of the center word.
Revised: We call the output of these projection functions projected word embeddings, which represented the new semantics of words in the local region under the inﬂuence of the center word.

Original: Formally, given the kernel K w i of w i and the embedding e w i + t of w i + t , the ( c 2 + t ) -th column in K w i is denoted by K w i ,t ,which can be regarded as a linear projection function to on e w i + t , where − c 2 < =
Revised: Formally, given the unit K w i of w i and the embedding e w i + t of w i + t , the ( c 2 + t ) -th column in K w i is denoted by K w i ,t , where − c 2 < =

Original: t < =
Revised: t < =

Original: c 2 .
Revised: c 2 .

Original: We apply a element-wise multiply(denoted by (cid:12) ) to compute the projected word embedding e iw i + t of w i + t as:
Revised: We apply a element-wise multiplication(denoted by (cid:12) ) to compute the projected word embedding e iw i + t of w i + t as:

Original: [Equation]
Revised: [Equation]

Revised: Note that the middle column K w i , c 2 of K w i can be regarded as a linear projection function on e w i itself, which makes e iw i in the same space with other projected embeddings.

Original: Since word in different relative position is projected by different linear function, the proposed local context kernels can be regarded as utilized the word order in a different way.
Revised: Since context words in different relative position are projected by different linear functions, the proposed local context units can utilize the word order in a different way.

Original: 3. W ORD - CONTEXT REGION EMBEDDING
Revised: 3. W ORD - CONTEXT REGION EMBEDDING

Revised: We proposed two architectures to perform the region embedding from different perspectives.

Original: In the ﬁrst proposed architecture, we focus on addressing thecenter word’s inﬂuence on the words in its local context, e.g., in the sentence The food is not very good in this hotel , the occurrence of word not might bring a semantical reservation on the local region.
Revised: In the ﬁrst proposed architecture, we focus on addressing the inﬂuence from the middle word to the context words. For example, in the sentence The food is not very good in this hotel , the occurrence of word not might bring a semantic reversal to the local region.

Original: Hence we use the local context kernel of the center word and word embeddings of all words in the region to compose the region embedding, and call this model word-context region embedding.
Revised: Hence we compose the region embedding with word embeddings of all words in the region as well as the local context unit of the central word. we call this model word-context region embedding.

Original: Formally, for a region ( i, c ) , given the kernel K w i of center word w i and the embeddings of words in the region, the projected embeddings can be computed as the following matrix:
Revised: Formally, for a region ( i, c ) , given the context unit K w i of center word w i and the embeddings of words in the region, the projected embeddings can be computed as the following matrix:

Original: [Equation]
Revised: [Equation]

Original: To compose the representation of a region region ( i, c ) noted as r i,c , we simply max pooling on the column dimension of the projected embedding matrix.
Revised: To compose the representation of a region region ( i, c ) noted as r i,c , we apply a max pooling on the column dimension of the projected embedding matrix.

Original: Figure 1a shows the details of the ﬁrst model architecture.
Revised: Figure 1a shows the details of the ﬁrst model architecture.

Original: For instance, in the sentence The food is not very good in this hotel , the projected word embeddings in the region (3 , 5) are produced by the element-wise multiplies between columns in the local context kernel of not and word embeddings of words food , is , not , very and good , and the embedding r 3 , 5 of region (3 , 5) can be obtain by max pooling onthese projected word embeddings.
Revised: For instance, in the sentence The food is not very good in this hotel , the projected word embeddings in the region (3 , 5) are produced by the element-wise multiplications between columns in the local context unit of not and word embeddings of words food , is , not , very and good . The embedding r 3 , 5 of region (3 , 5) can be obtained by max pooling on projected word embedding matrix.

Original: 3. C ONTEXT - WORD REGION EMBEDDING
Revised: 3. C ONTEXT - WORD REGION EMBEDDING

Original: The second architecture goes as a different view, which address the local context words’ inﬂuence on the center word in the region, and we call this model context-word region embedding.
Revised: The second architecture takes as a different view, which address the local context words’ inﬂuence on the center word in the region, and we call this model context-word region embedding.

Original: Similarly, for a region ( i, c ) , the projected embeddings can be represent as the following matrix:
Revised: Similarly, for a region ( i, c ) , the projected embeddings can be represent as the following matrix:

Original: [Equation]
Revised: [Equation]

Original: Since the kernels of words in the local contextmay contains semantic information as well as the word embeddings, the region embedding r i,c can be composed with a max pooling on the column dimension of the projected embedding matrix.
Revised: Since the units of words in the local context contains semantic information as well as the word embeddings, the region embedding r i,c can be composed with a max pooling on the column dimension of the projected embedding matrix.

Original: Figure 1b shows the details of the second model architecture.
Revised: Figure 1b shows the details of the second model architecture.

Revised: The difference between our two models is that the region embedding is produced by different inputs the max pooling operation applied on, the Word-Context model uses the projected embedding produced by the context unit of central word and word embedding of context words, while the

Revised: Context-Word just the opposite.

Original: 3. R EGION EMBEDDING FOR TEXT CLASSIFICATION
Revised: 3. R EGION EMBEDDING FOR TEXT CLASSIFICATION

Original: The documents on text classiﬁcation tasks commonly are of variable-length, to achieve the classiﬁcation task, the documents need to be represented as ﬁxed size vectors.
Revised: On text classiﬁcation task, the documents commonly are of variable-length, which need to be represented as ﬁxed size vectors.

Original: In order to show the effectiveness of our proposed region embedding models, we simply sum the embeddings of all regions in one document with an additional non-linear activation function to represent the document. We use a Full-Connectedlinear layer as the classiﬁer for text classiﬁcation.
Revised: In order to show the effectiveness of our proposed region embedding models, we simply sum the embeddings of all regions in one document with an additional nonlinear activation function to represent the document, and an upper Full-Connected layer is used for text classiﬁcation.

Original: Formally, the model can be represented as following:
Revised: Formally, the model can be represented as following:

Original: [Equation]
Revised: [Equation]

Original: where x is the input text sequence, W and b denote the weight matrix and bias of the fully connected layer respectively, g denotes the softmax function of the output layer, σ denotes the softsign function and n denotes the number of regions in the document, r is the region embedding which can be computed by the two proposed methods.
Revised: where x denotes the input text sequence, W and b denote the weight matrix and bias of the fully connected layer respectively, g denotes the softmax function of the output layer, σ denotes the softsign function and n denotes the number of regions in the document, r is the region embedding which can be computed by the two proposed methods.

Original: E , U , W and b can be updated in the training period.
Revised: E , U , W and b can be updated in the training period.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We report the experiments with proposed models in comparison with previous models.
Revised: We report the experiments with proposed models in comparison with previous models.

Original: The Code is publicly available on the Internet.
Revised: The Code is publicly available on the Internet.

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: We use publicly available datasets from Zhang et al.
Revised: We use publicly available datasets from Zhang et al.

Original: (2015) to evaluate our models.
Revised: (2015) to evaluate our models.

Original: There are total 8 text classiﬁcation datasets, corresponding to sentiment analysis, news topics, question-answer, ontology extraction, respectively.
Revised: There are total 8 text classiﬁcation datasets, corresponding to sentiment analysis, news topics, question-answer, ontology extraction, respectively.

Original: Table 1 shows the descriptive statistics of datasets used in our experiments.
Revised: Table 1 shows the descriptive statistics of datasets used in our experiments.

Original: To guarantee comparable indications, same evaluation protocol of Zhang et al.
Revised: To guarantee comparable indications, same evaluation protocol of Zhang et al.

Original: (2015) is employed.
Revised: (2015) is employed.

Original: [Table]
Revised: [Table]

Original: Our models are compared with several widely used supervised text classiﬁcation models.
Revised: Our models are compared with several widely used supervised text classiﬁcation models.

Original: We report the n-grams and TFIDF baselines from Zhang et al.
Revised: We report the n-grams and TFIDF baselines from Zhang et al.

Original: (2015), as well as the character level convolutional model (char-CNN) of Zhang & LeCun (2015), the word level convolutional model (wordCNN) of Zhang & LeCun (2015), the character based convolution recurrent network (char-CRNN) of Xiao & Cho (2016), the very deep convolutional network (VDCNN) of Conneau et al.
Revised: (2015), as well as the character level convolutional model (char-CNN) of Zhang & LeCun (2015), the word level convolutional model (wordCNN) of Zhang & LeCun (2015), the character based convolution recurrent network (char-CRNN) of Xiao & Cho (2016), the very deep convolutional network (VDCNN) of Conneau et al.

Original: (2016), the Discriminative LSTM (D-LSTM) of Yogatama et al.
Revised: (2016), the Discriminative LSTM (D-LSTM) of Yogatama et al.

Original: (2017) and the bigram Fasttext (bigramFasttext) of Joulin et al.
Revised: (2017) and the bigram Fasttext (bigramFasttext) of Joulin et al.

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: For data preprocessing, all the texts of datasets are tokenized by Stanford tokenizer and all words are converted to lower case.
Revised: For data preprocessing, all the texts of datasets are tokenized by Stanford tokenizer and all words are converted to lower case.

Original: Words that appear only once are treated as out-of-vocabulary (OOV) items, and any stop words or symbols are kept.
Revised: Words that appear only once are treated as out-of-vocabulary (OOV) items, and any stop words or symbols are kept.

Original: Additionally, padding lenght of c 2 are added to both the head and tail of each document.
Revised: Additionally, padding lenght of c 2 are added to both the head and tail of each document.

Original: Optimal hyperparameters are tuned with 10 % of the training set on Yelp Review Full dataset, and identical hyperparameters are applied to all datasets: the dimension of word embedding is 128 , the region size is 7 which means the shape of local context kernel matrix of each word is 128 × 7 , the learning rate is set 1 × 10 − 4 , and the batch size is 16 .
Revised: Optimal hyperparameters are tuned with 10 % of the training set on Yelp Review Full dataset, and identical hyperparameters are applied to all datasets: the dimension of word embedding is 128 , the region size is 7 which means the shape of local context kernel matrix of each word is 128 × 7 , the learning rate is set 1 × 10 − 4 , and the batch size is 16 .

Original: For optimization, the embeddings of words and kernels are randomly initialized with Gaussian Distribution.
Revised: For optimization, the embeddings of words and kernels are randomly initialized with Gaussian Distribution.

Original: Adam (Kingma & Ba, 2014) are used as the optimizer, Cross-entropy loss with softmax is applied meanwhile.
Revised: Adam (Kingma & Ba, 2014) are used as the optimizer, Cross-entropy loss with softmax is applied meanwhile.

Original: We do not use any extra regularization methods, such as L2 normalization or dropout.
Revised: We do not use any extra regularization methods, such as L2 normalization or dropout.

Original: Algorithms are entirely implemented with TensorFlow and trained on NVIDIA Tesla P40 GPUs.
Revised: Algorithms are entirely implemented with TensorFlow and trained on NVIDIA Tesla P40 GPUs.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: [Table]
Revised: [Table]

Original:  2 is a performace summary of compared methods on listed datasets.
Revised: Table 2 is a performace summary of compared methods on listed datasets.

Original: We use underscores to represent the best known published results, and use the bold to indicate the new records.
Revised: We use underscores to represent the best known published results, and use the bold to indicate the new records.

Original: On six datasets of eight, our models achieve new state-of-art or equally best with a performamce gain highest to 0.7(%).
Revised: On six datasets of eight, our models achieve new state-of-art or equally best with a performamce gain highest to 0.7(%).

Original: We beat all the previous models on all datasets except VDCNN, while the latter performes almost best on all classiﬁcation tasks before the proposing of our methods.
Revised: We beat all the previous models on all datasets except VDCNN, while the latter performes almost best on all classiﬁcation tasks before the proposing of our methods.

Original: As a result, we slightly win VDCNN on six datasets and lost in two of Amazon datasets.
Revised: As a result, we slightly win VDCNN on six datasets and lost in two of Amazon datasets.

Original: Although the slight advantage, our methods are remarkable on account of its simplicity along with divergence.
Revised: Although the slight advantage, our methods are remarkable on account of its simplicity along with divergence.

Original: Concretly,
Revised: Concretly,

Original: The upper layer structure of our models only uses a summing up operation, which is more concise and robust than any other deep or complex models.
Revised: The upper layer structure of our models only uses a summing up operation, which is more concise and robust than any other deep or complex models.

Original: On the other hand, different with the fashionable convolutional kernels, which are shared across all the sequence, we use the self context-aided kernel for each word.
Revised: On the other hand, different with the fashionable convolutional kernels, which are shared across all the sequence, we use the self context-aided kernel for each word.

Original: To our best known, we are the ﬁrst to introduce word speciﬁc kernels techniques.
Revised: To our best known, we are the ﬁrst to introduce word speciﬁc kernels techniques.

Original: In fact, we list the comparation of our two proposed models.
Revised: In fact, we list the comparation of our two proposed models.

Original: Both of them perform fairly well.
Revised: Both of them perform fairly well.

Original: In general, Word-Context model is slightly better than Context-Word model.
Revised: In general, Word-Context model is slightly better than Context-Word model.

Original: The difference is that
Revised: The difference is that

Original: Context-Word model uses the same embedding of center word to get the projected word embeddings while Word-Context model uses different embeddings of words in one region, so the latter may use more decision units in each pooling operation than the former.
Revised: Context-Word model uses the same embedding of center word to get the projected word embeddings while Word-Context model uses different embeddings of words in one region, so the latter may use more decision units in each pooling operation than the former.

Original: 4. E XPLORATORY E XPERIMENTS
Revised: 4. E XPLORATORY E XPERIMENTS

Original: In this subsection we are going to do a set of exploratory experiments to study the relative effect of each component in our model.
Revised: In this subsection we are going to do a set of exploratory experiments to study the relative effect of each component in our model.

Original: Typical cases will be analyzed to validate properties of various aspects of our models.
Revised: Typical cases will be analyzed to validate properties of various aspects of our models.

Original: Without loss of generality, we only analyzed the Word-Context region embedding model in our exploratory experiments.
Revised: Without loss of generality, we only analyzed the Word-Context region embedding model in our exploratory experiments.

Original: 4.5. E FFECT OF REGION SIZE AND EMBEDDING SIZE
Revised: 4.5. E FFECT OF REGION SIZE AND EMBEDDING SIZE

Original: Our method uses a ﬁxed size of region as contextual information just like CNN.
Revised: Our method uses a ﬁxed size of region as contextual information just like CNN.

Original: So the selection of region size really matters.
Revised: So the selection of region size really matters.

Original: A small region may lose of some long-distance patterns, whereas large regions will lead into more noise.
Revised: A small region may lose of some long-distance patterns, whereas large regions will lead into more noise.

Original: Luckily, our models seem to be fairly insensitive towards kinds of datasets.
Revised: Luckily, our models seem to be fairly insensitive towards kinds of datasets.

Original: Actually, we just select consistent conﬁguration of size 7 for entire datasets and it has been able to outperform best published results ever.
Revised: Actually, we just select consistent conﬁguration of size 7 for entire datasets and it has been able to outperform best published results ever.

Original: Figure 2 describes the performance in Yelp Review Full with different region sizes, and when the size equals to 1 , the result is quite close to unigram Fasttext(Acc 60.7%), but still get 0.6%+ promotion.
Revised: Figure 2 describes the performance in Yelp Review Full with different region sizes, and when the size equals to 1 , the result is quite close to unigram Fasttext(Acc 60.7%), but still get 0.6%+ promotion.

Original: Intuitively, the central word can not inﬂuence other words except itself when the size equals to 1 .
Revised: Intuitively, the central word can not inﬂuence other words except itself when the size equals to 1 .

Original: The performance increases with the growth of region size up to 7, followed by a drop.
Revised: The performance increases with the growth of region size up to 7, followed by a drop.

Original: To further study the effectiveness of region sizes, we experimented our models with the combination of multi region sizes.
Revised: To further study the effectiveness of region sizes, we experimented our models with the combination of multi region sizes.

Original: Figure 2 shows the comparisons between single region sizes and the multi-size combination of 3,5,7 , which shows that the combination of multi region sizes is slightly better than the best single region size 7.
Revised: Figure 2 shows the comparisons between single region sizes and the multi-size combination of 3,5,7 , which shows that the combination of multi region sizes is slightly better than the best single region size 7.

Original: The effectiveness of multi-size combination can be explained by the difference of inﬂuence ranges between words, such as very only emphasizes the next word while however may lay stress on a wide range of the following words.
Revised: The effectiveness of multi-size combination can be explained by the difference of inﬂuence ranges between words, such as very only emphasizes the next word while however may lay stress on a wide range of the following words.

Original: In addition to the analysis of region sizes, we further studied the inﬂuence of word embedding dimensions.
Revised: In addition to the analysis of region sizes, we further studied the inﬂuence of word embedding dimensions.

Original: Figure 3 lists the comparative results on Yelp Review Full with different embedding dimensions.
Revised: Figure 3 lists the comparative results on Yelp Review Full with different embedding dimensions.

Original: The results show that our model is less prone to be overﬁtting than Fasttext and CNN with the word embedding dimension increasing.
Revised: The results show that our model is less prone to be overﬁtting than Fasttext and CNN with the word embedding dimension increasing.

Original: In fact, the amount of parameters in our models is relatively large.
Revised: In fact, the amount of parameters in our models is relatively large.

Original: Since we learn a speciﬁc kernel for each word, under the same word embedding dimension, our parameter size has been expanded by region size times.
Revised: Since we learn a speciﬁc kernel for each word, under the same word embedding dimension, our parameter size has been expanded by region size times.

Original: Notice that the sizes of parameters are relatively consistent among 1024 in
Revised: Notice that the sizes of parameters are relatively consistent among 1024 in

Original: Fasttext , 1024 in CNN and 128 in ours .
Revised: Fasttext , 1024 in CNN and 128 in ours .

Original: 4.5. E FFECT OF KERNELS
Revised: 4.5. E FFECT OF KERNELS

Original: In this section, we explored some comparative experiments to show the effectiveness of our proposed word kernels.
Revised: In this section, we explored some comparative experiments to show the effectiveness of our proposed word kernels.

Original: The experiments are employed based on unigram Fastext baseline, which has similar upper layer structure with our models.
Revised: The experiments are employed based on unigram Fastext baseline, which has similar upper layer structure with our models.

Original: Table 3 illustrated the interrelated results.
Revised: Table 3 illustrated the interrelated results.

Original: Firstly, we remove the entire kernels from our model, which means it is just a variant version of unigram fasttext, we call it Fasttext(Win-pool).
Revised: Firstly, we remove the entire kernels from our model, which means it is just a variant version of unigram fasttext, we call it Fasttext(Win-pool).

Original: The difference is that Fasttext sums up the word
Revised: The difference is that Fasttext sums up the word

Original: [Table]
Revised: [Table]

Original: embeddings directly while Fasttext(Win-pool) sums up the window pooled embeddings in a stride of 1.
Revised: embeddings directly while Fasttext(Win-pool) sums up the window pooled embeddings in a stride of 1.

Original: It yields a slightly accuracy gain of 0.28% than unigram fasttext.
Revised: It yields a slightly accuracy gain of 0.28% than unigram fasttext.

Original: Secondly, we apply a simpliﬁed scalar version of kernel to Fasttext(Win-pool).
Revised: Secondly, we apply a simpliﬁed scalar version of kernel to Fasttext(Win-pool).

Original: Unlike our proposed kernels, the kernel of w i has the shape with 1 × c , hence the kernel can be regarded as scalling on corresponding word embeddings in its local context.
Revised: Unlike our proposed kernels, the kernel of w i has the shape with 1 × c , hence the kernel can be regarded as scalling on corresponding word embeddings in its local context.

Original: We call this method C.W.region.emb(Scalar).
Revised: We call this method C.W.region.emb(Scalar).

Original: Compared to our models, it yields a huge parameter size reduction, however, it already yileds a signiﬁcant gain of 2.45% compared to unigram Fastext.
Revised: Compared to our models, it yields a huge parameter size reduction, however, it already yileds a signiﬁcant gain of 2.45% compared to unigram Fastext.

Original: Futermore, C.W.region.emb(our model) is the variant version of C.W.region.emb(Scalar) that the scala kernel is expanded to a dense vector, which the kernel of w i has shape with h × c .
Revised: Futermore, C.W.region.emb(our model) is the variant version of C.W.region.emb(Scalar) that the scala kernel is expanded to a dense vector, which the kernel of w i has shape with h × c .

Original: Adding the low dimensional dense kernels improves the performance by 4.17% compared to unigram Fasttext and 1.72% to C.W.region.emb(Scalar).
Revised: Adding the low dimensional dense kernels improves the performance by 4.17% compared to unigram Fasttext and 1.72% to C.W.region.emb(Scalar).

Original: The results proved that our proposed word kernels can considerably improves the performance compared to the unigram fasttext even with a simpler scalar version.
Revised: The results proved that our proposed word kernels can considerably improves the performance compared to the unigram fasttext even with a simpler scalar version.

Original: In order to further analyze what kernels actually capture, Figure 4 presents the heat maps of word kernels for chosen word samples, such as however , very , e.g. each row represent a word kernel of the center word, r i denoted as i -th columns L norm on the right in the word kernel matrix and similar as l i .
Revised: In order to further analyze what kernels actually capture, Figure 4 presents the heat maps of word kernels for chosen word samples, such as however , very , e.g. each row represent a word kernel of the center word, r i denoted as i -th columns L norm on the right in the word kernel matrix and similar as l i .

Original: Qualitative but not fully rigorous a normalized L norm used to render shade intensity
Revised: Qualitative but not fully rigorous a normalized L norm used to render shade intensity

Original: i
Revised: i

Original: What the ﬁgure reﬂects keeps consistent with intuitive priors of human beings.
Revised: What the ﬁgure reﬂects keeps consistent with intuitive priors of human beings.

Original: In the perspective of however , right contexts play the key role for classiﬁcation polarity because of the emotional turnaround, the color is indeed deeper in r i than l i , so does but .
Revised: In the perspective of however , right contexts play the key role for classiﬁcation polarity because of the emotional turnaround, the color is indeed deeper in r i than l i , so does but .

Original: For word very , r 1 is more prominent than the rest of all, which capture some modiﬁed patterns like very happy or very sad .
Revised: For word very , r 1 is more prominent than the rest of all, which capture some modiﬁed patterns like very happy or very sad .

Original: For word good , tendencies will be completely different for pattern not good , very good and not that good , which are intensive negative, intensive positive and slightly hesitated, separately, the position of lwill be strengthened as a result, so does word bad .
Revised: For word good , tendencies will be completely different for pattern not good , very good and not that good , which are intensive negative, intensive positive and slightly hesitated, separately, the position of lwill be strengthened as a result, so does word bad .

Original: Actually, from the motivation of word speciﬁed kernels, we would like to believe this feature help capture syntactic and semantic inﬂuence on the relative position of its contexts.
Revised: Actually, from the motivation of word speciﬁed kernels, we would like to believe this feature help capture syntactic and semantic inﬂuence on the relative position of its contexts.

Original: It can be regarded as a way to make use of word order in kernel-covered regions
Revised: It can be regarded as a way to make use of word order in kernel-covered regions

Original: 4.5. V ISUALIZATION
Revised: 4.5. V ISUALIZATION

Original: In this subsection, we will try to visualized evaluate the contribution of each word(or selected region) to the classiﬁcation performance.
Revised: In this subsection, we will try to visualize the contribution of each word and selected phrase to the classiﬁcation.

Original: Since matrix has the law of association, the contribution can be truthfully divided into each pooled region owing to the simple summing up layer of our model. Furthermore, window-inner pooling is indeed explained by corresponding word’s certain dimension via argument max function. Overlaying will be considered because covered word can be pooled by different windows.
Revised: Detailed visualization techniques have been introduced in Li et al.

Original: Through such a trivial backward, we can calculate each word’s proportion to the ﬁnal category logits.
Revised: Here we generalize it the to the color rendering of multi-category version.

Original: Notice that not the self embedding acts here, but the projected embedding of self one on surrounding words’ kernel.
Revised: Notice that not the self embedding acts here, but the projected embedding of self one on surrounding words’ context unit.

Original: [Table]
Revised: [Table]

Original: wallet
Revised: unit

Original: [Table]

Original: [Table]
Revised: [Table]

Original: With kernel get your wallet ready , the prices are crazy 
Revised: With C-unit get your wallet ready , the prices are crazy high

Original: [Table]
Revised: [Table]

Original: [Table] No kernel nothing remarkable , but [Table]
Revised: No C-unit nothing remarkable , but not bad either but not bad either

Original: With kernel nothing remarkable , but [Table]
Revised: With C-unit nothing remarkable , but not bad either but not bad either

Original: For clarity, we choose a binary classiﬁcation task of sentiment analysis.
Revised: For clarity, we choose a binary classiﬁcation task of sentiment analysis.

Original: In Table 4, we list two real case samples in Yelp Review Polarity dataset, in which our model performance as expected.
Revised: In Table 4, we list two real case samples in Yelp Review Polarity dataset, in which our model behave as expected.

Original: Words or artiﬁcially selected phrases are highlighted green if they are positive factors, red if they are negative.
Revised: Words or artiﬁcially selected phrases are highlighted green if they are positive factors, red if they are negative.

Original: The intensity of the color indicates the extends of the polarity.
Revised: The intensity of the color indicates the extends of the polarity.

Original: We show the visualization ofthe two dispose on samples, no kernel(Fasttext) and with kernel.
Revised: To have a better comparation, the methods of with and without context unit are both visualized. We abbreviate them as With C-unit and Without C-unit , respectively.

Original: For sentence get your wallet ready, the prices are crazy high! ,  word colors will be  only the embodiment of their own embedding if no kernel adopted.
Revised: For sentence get your wallet ready, the prices are crazy high , if no context unit is adopted, the word color will be reﬂected by its only embedding.

Original: The word crazy is colored positive, even the exclamation mark behaves slightly green. The word highis colored negative, oppositely.
Revised: The word crazy is colored positive, and high colored negative.

Original: Because the intensity of crazy is higher than high , the entire polarity of phrase  crazy high is totally positive, which is a complete mistake.
Revised: Because the intensity of crazy is higher than high , the entire polarity of phrase prices are crazy high is totally positive, which is a complete mistake.

Original: But with kernel, things have changed quite a bit.
Revised: But with context unit, things have changed quite a bit.

Original: Because of the exist of high as the context, the positive polarity of crazyand ! vanishes.
Revised: Because of the exist of high as the context, the positive polarity of crazy vanishes.

Original: The negative polarity of high is stronger, phrase  crazy high performs positive overall.
Revised: The negative polarity of high is stronger, phrase prices are crazy high performs positive overall.

Original: For another case nothing remarkable, but not bad either , things seems more interesting.
Revised: For another case nothing remarkable, but not bad either , things seem more interesting.

Original: If no kernel, remarkable will be positive, while nothing, not, bad perform negative, respectively.
Revised: If without context-unit , remarkable will be positive, while nothing, not, bad perform negative, respectively.

Original: But with kernel, the polarity of the part ahead of but is reduced, meanwhile the polarity of not and bad ﬂip attractively.
Revised: But with context-unit, the polarity of the part ahead of but is reduced, meanwhile the polarity of not and bad ﬂip attractively.

Original: As a result, phrase  not bad either performs positive overall.
Revised: As a result, phrase but not bad either performs positive overall.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper proposed two novel architectures for text classiﬁcation that learns embeddings of small text regions by introduced local context kernels.
Revised: This paper proposed two novel architectures for text classiﬁcation that learns embeddings of small text regions by introduced local context kernels.

Original: Without aid of additional resources , the embeddings and kernels are trained speciﬁcally for the task of interest.
Revised: Without aid of additional resources , the embeddings and kernels are trained speciﬁcally for the task of interest.

Original: Beneﬁting the simplicity of our models, we achieved state-of-the-art performances on six benchmark text classiﬁcation datasets with identical hyperparameters .
Revised: Beneﬁting the simplicity of our models, we achieved state-of-the-art performances on six benchmark text classiﬁcation datasets with identical hyperparameters .