Complicate Then Simplify: A Novel Way to Explore Pre-trained Models for Text Classification (2022)

Abstract: With the development of pre-trained models (PTMs), the performance of text classification has been continuously improved by directly employing the features generated by PTMs. However such way might not fully explore the knowledge in PTMs as it is constrained by the difficulty of the task. Compared to difficult task, the learning algorithms tend to saturate early on the simple task. Moreover, the native sentence representations derived from BERT are prone to be collapsed and directly employing such representation for text classification might fail to fully capture discriminative features. In order to address these issues, in this paper we propose a novel framework for text classification which implements a two-stage training strategy. In the pre-training stage, auxiliary labels are introduced to increase the task difficulties and to fully exploit the knowledge in the pre-trained model. In the fine-tuning stage, the textual representation learned in the pre-training stage is employed and the classifier is fine-tuned to obtain better classification performance. Experiments were conducted on six text classification corpora and the results showed that the proposed framework outperformed several state-of-the-art baselines.

## introduction text classification is a fundamental task in the field of natural language processing and is widely employed in various tasks such as question answering, sentiment analysis, and information retrieval. with the continuous development of machine learning algorithms, especially the success of deep learning methods, text classification has been significantly improved, e.g. cnns @xcite , rnns @xcite , berts @xcite @xcite , etc. recently, pre-trained models have been shining in classification-based natural language processing tasks @xcite @xcite . the advent of bert has led to an effective enhancement of textual feature representation. a series of improved pre-trained models have been proposed, e.g. roberta @xcite , macbert @xcite , ernie @xcite , chine-sebert @xcite . for example, ernie learns real-world semantic knowledge by modeling words, entities and entity relationships in massive amounts of data @xcite . @xcite proposed a large-scale chinese pre-trained model that incorporates glyph and pinyin information. the above works improve the textual representation of the pre-trained model by introducing external knowledge, without fully exploring the semantic representation in the existing pre-trained model. the recent work, prompt learning, is the technique of making better use of the knowledge from the pre-trained model by adding additional texts to the input @xcite . like prompt learning, we propose to further extract more meaningful textual representations from the pre-trained model, i.e. to make the extracted textual semantic representation more discriminative for classification. although the current pre-trained model already obtain relatively good textual representation, it is still possible to further explore the information in the pre-trained representation based on our observation. as shown in figure 1 , for sentiment analysis, there are two sets of features in a sentence that indicate its positive emotional polarity, "很 不 错(very good)" and "下 次 选 择(next time choose)". words like "很 不错(very good)" explicitly express positive sentiment. features like these, which are closely related to category labels, are given higher weight during training. while more implicit features like "下次(next time)" and "选择(choose)" are easily ignored as shown by the green curve in figure 1 . however, by using the two-stage framework proposed in this paper, the weights of implicit discriminative features are further highlighted without weakening the weights of the most discriminative feature as shown in the black curve in figure 1 . meanwhile, we notice that @xcite found the word representation space of bert to be anisotropic, with high-frequency words clustered together and close to the origin, while lowfrequency words were sparsely scattered. when averaging token embeddings, those high-frequency words dominate the sentence representation, inducing a bias against their actual semantics. such phenomenon has also been observed in some previous work @xcite @xcite . therefore, directly employing such representation for text classification might fail to fully capture discriminative features. therefore, in this paper, we consider the extracting of semantic features from a cognitive perspective by introducing auxiliary labels and constructing pre-training and fine-tuning strategies based on pre-trained models. we devise a novel approach to perform a secondary pre-training based on the pre-trained model and then fine-tuning for text classification which is similar to the process of gaining new insights through restudying old material. in the pre-training stage, the model learns a better representation of the task under consideration. in the fine-tuning stage, the classifier is fine-tuned by applying the text feature representations obtained from the pre-training stage. to fully exploit the discriminative features in the pre-trained model, in the pre-training stage, auxiliary labels are constructed to take fine-grained semantic categories into account. the introduction of auxiliary labels makes the information entropy increase. knowledge in the pre-trained model is fully mined for a more effective discriminative semantic representation. the main contributions are listed as follows. the reminder of the paper is structured as follows. some related work is briefly reviewed in section 2. the detailed implementation of our framework is described in section 3. section 4 reports the experiments and results and section 5 shows further analysis and discussion. finally, the paper is concluded in section 6. ## related work the development of deep learning has led to significant improvements in text classification, and some of the more widely employed deep learning models for text classification tasks are cnns @xcite , rnns @xcite @xcite , and pretrained models @xcite @xcite . in recent years, pre-trained models have shown excellent performance in the field of text classification. whether cnns, rnns, or more recently pre-trained models, the purpose of employing deep models is to efficiently capture the textual semantic representation. ## model the overall framework is shown in figure 2 . the whole framework is divided into two stages: pretraining and fine-tuning. in the pre-training stage, auxiliary labels are introduced to artificially boost the training difficulty, which can better tap the knowledge from pre-trained models and obtain a more discriminative textual feature representation. in the fine-tuning stage, the textual representation pre-trained in the pre-training stage is employed and the classifier is fine-tuned to obtain better classification performance. ## conclusion for simple text classification tasks, we propose a novel framework for simple text classification tasks, which implements a two-stage training strategy, including pre-training based on pre-trained models and fine-tuning for classification. in the pretraining stage, auxiliary labels can be integrated to increase the training challenge and to fully exploit the knowledge in the pre-trained model. experiments on six datasets depict that our approach outperforms the baseline bert model for simple classification tasks. furthermore, in the sentence semantic matching task, after adding adversarial perturbations to the embedding layer only, our basic version of the mre-bert model achieves promising performance, better than or equivalent to large versions of the pre-trained model, but with much fewer parameters than them. in the future, we will investigate the generalizability of our model to other classification tasks.