FL i T ext: A Faster and Lighter Semi-Supervised Text Classification with Convolution Networks (2021)

Abstract: In natural language processing (NLP), state-of-the-art (SOTA) semi-supervised learning (SSL) frameworks have shown great performance on deep pre-trained language models such as BERT, and are expected to significantly reduce the demand for manual labeling. However, our empirical studies indicate that these frameworks are not suitable for lightweight models such as TextCNN, LSTM and etc. In this work, we develop a new SSL framework called FLiText, which stands for Faster and Lighter semi-supervised Text classification. FLiText introduces an inspirer network together with the consistency regularization framework, which leverages a generalized regular constraint on the lightweight models for efficient SSL. As a result, FLiText obtains new SOTA performance for lightweight models across multiple SSL benchmarks on text classification. Compared with existing SOTA SSL methods on TextCNN, FLiText improves the accuracy of lightweight model TextCNN from 51.00% to 90.49% on IMDb, 39.8% to 58.06% on Yelp-5, and from 55.3% to 65.08% on Yahoo! Answer. In addition, compared with the fully supervised method on the full dataset, FLiText just uses less than 1% of labeled data to improve the accuracy by 6.59%, 3.94%, and 3.22% on the datasets of IMDb, Yelp-5, and Yahoo! Answer respectively.

## introduction developments in deep learning technology have great breakthroughs in most natural language processing (nlp) tasks, such as machine translation, sentiment analysis, and reading comprehension. @xcite @xcite @xcite @xcite @xcite the success of these advancements is highly dependent on large-scale and high-quality manual labeled data. however, obtaining vast amounts of high-quality labeled data is expensive. especially in certain fields, such as finance, medicine, law, and so on, text labeling relies on the in-depth participation of field experts. the rapid development of ssl technology is expected to significantly reduce the demand for labeled data. the core goal of this technology is to use a small number of labeled data and vast amounts of unlabeled data to train a model with good generalization performance to solve machine-learning problems. @xcite @xcite @xcite @xcite unsupervised data augmentation (uda) @xcite and mix-text @xcite are sota ssl methods for text classification, and have been used to various tasks with notable success. in nlp, applying the ssl framework to deep pre-trained language models (e.g., bert, gpt, and xlnet) has been demonstrated effective. however, the good performance of these ssl methods depends on a bulky "large model". in most practical situations, due to the large-scale parameters and slow inference speed, it is difficult to implement these models with limited resources, such as mobile devices @xcite , online search engines @xcite , and edge platforms @xcite ). an intuitive idea to address the problem is to apply the ssl method on a small model, such as textcnn, of which the parameter scale is about one or two orders of magnitude lower than that of bert, as shown in figure 1 . however, many applications show that the existing sota ssl framework performs poorly on lightweight models. furthermore, there is a lack of relevant research on the implementation of ssl on lightweight models. this paper develops an ssl framework on lightweight models, for faster and lighter semisupervised text classification (flitext). we use a deep pre-trained inspirer network to learn the distribution relationship and the task-specific features of the data. next, the inspirer network provides two types of regularization constraints on a lightweight model. the intuitive explanation is as follows: "teachers not only teach results but also teach experiences in the learning process so that students can learn more effectively." to evaluate flitext, we compare flitext and sota methods on three benchmark text classification datasets. we also conduct an ablation study to verify the performance of each part of flitext. the results show that flitext can significantly improve the inference speed while maintaining or exceeding sota performance. compared with uda on textcnn, flitext improves the accuracy from 51.00% to 90.32% on the imdb dataset, from 39.80% to 58.06% on the yelp-5 dataset, and from 55.30% to 65.08% on the yahoo dataset. compared with the supervised learning on complete datasets, the performance is improved by 6.28%, 4.08%, and 3.81% on the three datasets respectively, by just using less than 1% labeled data. our contributions can be summarized as follows: knowledge distillation: @xcite uses student model to mimic teacher's prediction by soft target. @xcite distills bert into a single layer of bilstm. for the first time, the knowledge of the transformer-based model was distilled into the non transformer-based model. @xcite extracts knowledge from the intermediate layer of the bert; @xcite distills knowledge during the model's pre-trained stage; @xcite combines the above various methods and propose a two-stage distillation method. although all of these methods have achieved excellent results, the transformer has the problem of a huge amount of parameters and high computational complexity.