Transfer Learning for Text Classification via Model Risk Analysis (2024)

Abstract: It has been well recognized that text classification can be satisfactorily performed by Deep Neural Network (DNN) models, provided that there are sufficient in-distribution training data. However, in the presence of distribution drift, a well trained DNN model may not perform well on a new dataset even though class labels are aligned between training and target datasets. To alleviate this limitation, we propose a novel approach based on model risk analysis to adapt a pre-trained DNN model towards a new dataset given only a small set of representative data. We first present a solution of model risk analysis for text classification, which can effectively quantify misprediction risk of a classifier on a dataset. Built upon the existing framework of LearnRisk, the proposed solution, denoted by LearnRisk-TC, first generates interpretable risk features, then constructs a risk model by aggregating these features, and finally trains the risk model on a small set of labeled data. Furthermore, we present a transfer learning solution based on model risk analysis, which can effectively fine-tune a pre-trained model toward a target dataset by minimizing its misprediction risk. We have conducted extensive experiments on real datasets. Our experimental results show that the proposed solution performs considerably better than the existing alternative approaches. By using text classification as a test case, we demonstrate the potential applicability of risk-based transfer learning to various challenging NLP tasks. Our codes are available at https://github.com/syjcomputer/LRTC.

## introduction as a very important task in natural language processing, text classification aims to categorize a given text into multiple groups based on its contents. text classification, including the more specific tasks of sentiment analysis @xcite , news categorization @xcite and topic classification @xcite , have been extensively studied in the literature @xcite . with the emergence of large language models (e.g., bert @xcite , bae @xcite and bertgcn @xcite ), the research community has experienced a considerable shift towards how to adapt these models to the task. it has been well recognized that dnn models can usually perform well on text classification, provided that there are sufficient in-distribution training data @xcite . however, in the presence of distribution drift, a well trained dnn model may not perform well on a new dataset even though class labels are aligned between training and target datasets . as other mainstream machine learning models, the efficacy of dnn models depends on i.i.d (identically and independently distributed) assumption. unfortunately, in real applications, it is usually very laborintensive thus prohibitive to retrieve a sufficient amount of in-distribution labeled data. in these scenarios, it is desirable that a pre-trained model can be easily adapted to a new dataset with only a small amount of additional labeled data. it is noteworthy that the community has presented many transfer learning approaches, targeting many tasks including but not limited to text classification @xcite @xcite . these existing work mainly focused on how to mine the knowledge from training data and apply them on target data. however, most of them can not effectively adapt a model towards a new dataset by its particular characteristics. therefore, transfer learning generally remains very challenging, with text classification being no exception. on the other hand, we have observed that ai model risk analysis has garnered much attention in recent years due to the concern on ai model's misbehavior @xcite @xcite . the purpose of model risk analysis is to quantify a model's misprediction risk on a dataset, and then leverage the results of risk analysis for adaptive model fine-tuning @xcite . since risk analysis can be potentially effectively performed with only a small amount of labeled data, it provides a viable technical roadmap for transfer learning. in this paper, we first present a solution of model risk analysis for text classification, and then propose a corresponding risk-based transfer learning approach, which can effectively adapt a dnn model towards a target dataset by its particular characteristics. specifically, because the risk analysis framework of learnrisk, which was originally proposed for the task of entity resolution @xcite , is more interpretable and accurate than the existing alternatives, we have built the solution of risk analysis for text classification based on learnrisk. we have sketched the solution of risk analysis, which is denoted by learnrisk-tc, in figure 1 . it first generates interpretable risk features, then constructs a risk model by aggregating these features, and finally trains the risk model using a small set of representative data. furthermore, we present a corresponding solution of transfer learning based on risk analysis. the overall solution of transfer learning is shown in figure 2 . it consists of two training phases. the first phase is the same as the process of traditional model training, which trains a model based on labeled data from a source. in the second phase, it furthermore fine-tunes the model on a target dataset by minimizing its misprediction risk. our main contributions in this paper can be summarized as follows: in this section, we review related work from the orthogonal perspectives of text classification, risk analysis and transfer learning. text classification. traditional solutions for text classification were constructed based on statistics and machine learning, e.g., knn @xcite , naïve bayes @xcite and tf-idf @xcite . the hybrid approach of fusing the knn and the tf-idf was first explored in 2014 @xcite . the approach that combined knn and clustering, which could considerably reduce similarity computation, was first proposed in 2014 @xcite . the naïve bayes method was also a popular approach for text classification @xcite . with the emergence of deep learning, various dnn models have been proposed for text classification. in recent years, the research community has experienced a considerable shift towards pretrained large language models, the most prominent among which is the bert model @xcite . in the following work, the bert model has been extended into multiple variations, including bae @xcite and bertgcn @xcite . additionally, contrastive learning and adversarial learning have been employed for text classification to facilitate labeling with fewer labeled samples. ust introduced bayesian networks for uncertainty estimations to improve the classification performance on few-label tasks @xcite . nsp-bert used templates to perform inference on test samples using masked language modeling (mlm) @xcite . more recently, large language models (llms) such as llam and alpaca-7b were tuned using the techniques such as recurrently ensembling base learners (rgpt) @xcite or based on instruction through contrastive self-training @xcite , demonstrating impressive performance on zero-shot text classification tasks. it is noteworthy that despite their efficacy, these llms often require significant computational resources. ## task statement in this paper, we consider text classification as a multi-label classification problem. given a text, a classifier needs to output a label for the text, which can accurately summarize its topic. as usual, we use accuracy as the evaluation metric. formally, we define the task of text classification by: definition 1. [task of text classification.] given a dataset of text classification, d, consisting of d s , d v and d t , where d s , d v and d t denote the sets of training data, validation data and test data respectively, the task aims to learn an optimal classifier, g(w * ) based on d such that the performance of g(w * ) on d t as measured by the metric of accuracy is maximized. it is noteworthy that in this paper, we focus on the scenario of transfer learning, where the validation and test data, d v and d t , come from a source other than that of d s , and there is distribution drift between these two text sources. we also suppose that the validation dataset, d v , which are representatives of d t , contains only a limited amount of labeled data. otherwise, in real scenarios, the labeled data in d v can be directly used to fine-tune a pre-trained model, making transfer learning unnecessary. ## risk analysis solution: learnrisk-tc in this section, we first briefly introduce the existing framework of learnrisk, and then present the learnrisk-tc solution for text classification. ## empirical evaluation in this section, we evaluate the performance of our proposed solutions by a comparative study on real benchmark datasets. subsection 6.1 describes our experimental setup. subsection 6.2 presents the evaluation results of risk analysis. subsection 6.3 presents the evaluation results of transfer learning. finally, in subsection 6.4, we evaluate the performance sensitivity of the proposed solutions w.r.t the size of validation data, which serve as the representatives of a target dataset. we have used three benchmark news datasets in our experiments, which included 20news 1 , bbc2 2 and agnews @xcite . to simulate the scenario of distribution drift, our experiments suppose that training data come from a 1 http://qwone.com/ jason/20newsgroups/ 2 http://mlg.ucd.ie/datasets/tbbc.html dataset while validation and test data come from another dataset. we have summarized the statistics of our test datasets in ## conclusion in this paper, we have proposed a novel solution of model risk analysis for text classification, and a corresponding risk-based solution of transfer learning to adapt a dnn model towards a target dataset by using only a small set of representative data. our experiments on real datasets have validated the efficacy of the proposed solutions. on future work, it is interesting to point out that the proposed risk-based approach of transfer learning can be potentially applied to other nlp tasks. however, the detailed technical solutions require further investigation.