Feature Projection for Improved Text Classification (2020)

Abstract: In classification, there are usually some good features that are indicative of class labels. For example, in sentiment classification, words like good and nice are indicative of the positive sentiment and words like bad and terrible are indicative of the negative sentiment. However, there are also many common features (e.g., words) that are not indicative of any specific class (e.g., voice and screen, which are common to both sentiment classes and are not discriminative for classification). Although deep learning has made significant progresses in generating discriminative features through its powerful representation learning, we believe there is still room for improvement. In this paper, we propose a novel angle to further improve this representation learning, i.e., feature projection. This method projects existing features into the orthogonal space of the common features. The resulting projection is thus perpendicular to the common features and more discriminative for classification. We apply this new method to improve CNN, RNN, Transformer, and Bert based text classification and obtain markedly better results.

## introduction text classification is an important task in natural language processing and text mining. it has a very wide range of applications, such as sentiment classification @xcite , question classification @xcite , and deception detection @xcite . in recent years, deep learning models have been shown to outperform traditional classification methods @xcite @xcite @xcite @xcite . given the input document, the system applies a mapping function (e.g., averaging or summation, a convolution neural network (cnn), recurrent neural network (rnn), and so on) to learn a dense representation of the document and then uses this representation to perform the final classification. representation learning is one of the key strengthes of deep learning. in this paper, we propose to further improve the representation learning, i.e., to make the representation more discriminative for classification. note that throughout the paper we will use sentence sentiment classification as an example to explain different ideas, but in our experiments, non-sentiment classification datasets are also used to show the generality of the proposed method. for text classification, many neural networks and embedding techniques have been devised and applied, e.g., rnn, cnn, transformer @xcite and bert @xcite . for example, rnn can model the whole sentence and also capture the long-term dependencies within the sentence. however, modeling the entire sequence may neglect some key local contexts that are important for classification @xcite . cnn is able to extract more local and position-invariant features @xcite . however, these methods may not give enough weights to some special or discriminative words. to solve this problem, the attention mechanism was introduced. for example, by exploiting attention, transformer and bert (which maximizes transformer's ability to extract sentence semantic information) can achieve even better results than both cnn and rnn on many tasks. we will see some other related methods to produce effective representations in the related work section. although the existing models are already able to produce excellent representations, we will show that these representations can still be improved. this paper explores in an entirely different direction, i.e., feature projection. in a typical sentence or document, there are usually some words or features that are correlated with some class labels, but there are also many other common features that cannot distinguish different classes. for example, in sentiment classification, words like good and nice are indicative of the positive sentiment, and words like bad and terrible are indicative of the negative sentiment. words like picture, price, and battery are not indicative of any sentiment, i.e., they are not discriminative. however, they may still interfere the representation learning to produce sub-optimal feature representations for the final classification. even though the attention mechanism can alleviate this problem to some extent by giving higher weights to words associated with classes and lower weights to the other words that are not indicative of any specific classes. however, due to the idiosyncrasy of the data and the inaccuracy of the attention mechanism, the problem remains. in this paper, we propose a novel feature projection method to improve feature representation learning to make it more discriminative for classification. the proposed method is called feature purification network (fp-net). specifically, fp-net consists of two sub-networks, a common feature learning network referred to as the c-net and a projection network referred to as the p-net. cnet uses a gradient reverse layer (grl) @xcite to extract common features b (i.e., invariant features @xcite ) that are shared by multiple classes and have little discriminative power for classification. at the same time, p-net uses a traditional feature extractor to learn the feature vector a for the input sentence or document. then the feature (or representation) vector a is projected onto the vector of the common features b (i.e., vector b) to get a projection vector c, which represents the input sentence's own common features. then, we project the feature vector a onto the orthogonal direction of the vector of the common features c to produce the final purer features for classification. it is quite clear and intuitive that this orthogonal project is to get rid of the common features and make the system focusing on those discriminative features only. we will explain why two projections are used in section 3. in summary, the key contribution of this paper is the improvement to representation learning through feature vector projection. to the best of our knowledge, this is the first such technique. specifically, an orthogonal projection layer (opl) is proposed to map the features obtained by a traditional feature extractor to the classification-specific semantic space, which is orthogonal to the common features such that we obtain a more relevant and discriminative (or purer) feature representation from the original document for classification. extensive experiments have been conducted to verify the effectiveness of the proposed method on two sentence sentiment classification datasets mr and sst2, a natural language inference dataset snli, and a question classification dataset trec. the results show that the proposed method can improve the classification accuracy of rnn, cnn, transformer and bert based classification methods markedly, which shows that feature projection is a highly promising direction to explore. ## related work it is well known that one of the key strengths of deep neural networks is their superb ability to learn highly effective representations or features from the raw data, which have been shown to be very successful for all kinds of applications including natural language processing tasks such as text classification @xcite , machine translation @xcite dialogue @xcite , etc. previous work on learning representations broadly falls in two main categories: supervised and unsupervised methods. our work focuses on improving the representation of text for supervised classification. supervised methods: these methods improve data utilization efficiency and discriminative feature distillation as they can obtain better training signals from the labeled data. sequence models such as recurrent neural networks (rnn), long short-term memory (lstm) @xcite and gated recurrent unit (gru) @xcite networks are suitable for handling text because a sentence or document can be regarded as a sequence. therefore, a large amount of work based on rnn and its variants for feature extraction and downstream tasks has been done @xcite @xcite . unlike rnn's sequence modeling approach, cnn (convolutional neural network) uses different sized windows to capture local correlations and position-invariant information @xcite @xcite @xcite . a common approach of these methods is to create an instance-level representation by using the final hidden state of the rnn, the maximum (or average) pooling of the rnn hidden states, or convolutional n-grams. however, they may ignore the importance of special words that are highly discriminative for classification. after @xcite introduced the attention mechanism in machine translation, attention mechanism has been exploited in many natural language processing tasks including text classification to solve the above problem. for example, @xcite introduced attention as an integral part of the model for text classification. @xcite proposed a new model for extracting interpretable sentence embeddings using self-attention. @xcite showed that attention mechanism is also effective for sentiment classification. @xcite further illustrated that they can get a stronger sentence-level representation by stacking multiple blocks of selfattention. bert @xcite combines transformer and a large corpus to produce an even more complete and better sentence-level representation. some other studies improved the representation of sentences from the perspective of language structures (e.g., parse trees and dependency trees) @xcite . @xcite utilized a single multi-task framework to combine the benefits of diverse sentence representation learning objectives. however, to the best of our knowledge, these existing works and others have not used feature projection to improve (or purify) representations for supervised learning, which we believe is a promising direction to explore. unsupervised methods: these methods utilize a large unlabeled text corpus to learn word representations which are then composed into sentence and document representations. for example, @xcite constructed sentence representations by trying to reconstruct neighbouring sentences. @xcite proposed a log-linear bag-of-words models for sentence representation. the unsupervised smooth inverse frequency method in (ethayarajh, 2018) built on this but used a weighted average of word embeddings and principal component removal for sentence representations. our work is again clearly different from these unsupervised methods as the proposed method works under supervised learning. existing unsupervised methods also do not use feature projection. some other works have also been done for semi-supervised representation learning (kevin clark, 2018) and transfer learning @xcite . jason phang (2019) also proposed to use some data-rich intermediate supervised tasks for pre-training to help produce better representation for the end task. to the best of our knowledge, all these previous studies tried to improve representations using external data or knowledge, which are quite different from our method as we don't use any external information. also, the philosophy of our approach is entirely different as we try to eliminate commonalities among classes through feature projection, which is orthogonal to existing representation learning approaches. finally, our work is related to several other works. @xcite introduced the gradient reverse layer (grl) for extracting common features in the context of domain adaptation. it embeds domain adaptation into the process of learning representations so that the final classification decision has more discriminative and invariant characteristics for domain changes. we also use grl to extract irrelevant or common features. however, we do not work on domain adaptation and they do not use feature projection. @xcite used adversarial learning to encourage models to learn representations free of hypothesis-only biases in the snli dataset. @xcite combined grl and aspect attention to study cross-domain sentiment classification. they found common features across domains and then extracted information from the aspects (which are product features) with the help of common features to do classifications. our work is clearly different because none of these existing works improve representation learning through feature projection. ## feature purification network the overall framework of our model is shown in figure 1 . the whole model consists of two parts, the first part is the projection network (i.e., p-net) and the other is the common feature learning network (i.e., c-net). as mentioned earlier, the goal of c-net is to extract common features and the goal of p-net is to compute the purified features for classification, which is done by projecting the learned full information vector of the input document into a more discriminative semantic space to eliminate the influence of the common features. p-net consists of four parts: the input layer x, the feature extractor f p , orthogonal projection layer (opl), and the final classification layer c p . c-net is also composed of four parts: the input layer x, the feature extractor f c (f p and f c 's parameters are not shared) 1 , the gradient reverse layer (grl) and the classification layer c c . the key idea of the proposed technique is as follows: the feature vector f p computed by the feature extractor f p is projected to the orthogonal direction of the feature vector f c extracted by f c of the c-net. that is, f p (the full information extracted from the input document) is projected to the discriminative semantic space to be purified for the final classification. however, in order to perform the orthogonal projection, two operations are required, which we will explain shortly. next we use cnn as an example feature extractor to detail each component of the proposed fp-net. ## conclusion in this paper, we proposed a novel feature purification network (fp-net) to improve the representation for text classification. the method is based on feature projection. the proposed model uses two sub-networks, one for identifying common features that are not discriminative for classification, and the other for feature projection that projects the traditional features to the orthogonal direction of the common features. to the best of our knowledge, this is the first method that uses feature projection to improve text classification. through a large number of comparative experiments, we showed the effectiveness of the proposed feature projection method. our current method is designed only for traditional text classification methods such as lstm, cnn, and transformer. in our future work, we will consider extending it to graph-based methods such as gcn for graph data, and to generation-based methods such as gan for adversarial learning.