S a FER : A Robust and Efficient Framework for Fine-tuning BERT -based Classifier with Noisy Labels (2023)

Abstract: Learning on noisy datasets is a challenging problem when pre-trained language models are applied to real-world text classification tasks. In numerous industrial applications, acquiring task-specific datasets with 100% accurate labels is difficult, thus many datasets are accompanied by label noise at different levels. Previous work has shown that existing noise-handling methods could not improve the peak performance of BERT on noisy datasets, and might even deteriorate it. In this paper, we propose SaFER, a robust and efficient fine-tuning framework for BERT-based text classifiers, combating label noises without access to any clean data for training or validation. Utilizing a label-agnostic early-stopping strategy and self-supervised learning, our proposed framework achieves superior performance in terms of both accuracy and speed on multiple text classification benchmarks. The trained model is finally fully deployed in several industrial biomedical literature mining tasks and demonstrates high effectiveness and efficiency.

## introduction large language models (llms) have dominated natural language processing (nlp) in recent years and achieved state-of-the-art performance in a variety of industrial applications. among them, the most widely-adapted llms are transformer-based models, including bert, t5, gpt, etc @xcite @xcite . llms learn general natural language knowledge from large corpora and the representations for text data can be utilized in various downstream nlp tasks. such a paradigm is also extensively implemented in both industrial and research domains and achieves considerable performance improvement compared with traditional statistical approaches. text classification is one of the most important tasks in the industrial domain @xcite @xcite @xcite , including sentence classification, named entity recognition, etc. typically, these tasks can be accomplished by leveraging embeddings generated by the encoder architecture of llms. unfortunately, the performance is always limited by the data quality in either pre-training or fine-tuning stage. real-world datasets, especially those collected for industrial applications, usually contain a substantial proportion of mislabeled data @xcite . such label noise can be induced by crowd-sourcing, human mistakes, system errors, or the uncertainty itself in the weakly-supervised labeling methodology. the corrupted labels can dramatically influence the model performance and robustness, which has been validated theoretically and experimentally @xcite . worse still, re-labeling procedures can be cost-intensive and time-consuming due to lack of domain experts. that means in most cases we can only access the noisy validation set and lose the validation with ground truth. previous work addressed the label noise issue by proposing robust loss functions, recovering the noise transition matrix, and incorporating unsupervised learning strategies @xcite @xcite @xcite . however, the label noise issue of using llms in nlp tasks is still an open problem and remains unsolved, especially in text classification tasks. @xcite demonstrate that directly incorporating existing noise-encountering methods cannot consistently improve and even deteriorate the bert model performance under noisy labels in text classification. this conclusion is also supported by the results of our industrial implementation and ablation study (appendix c). by investigating the process of fine-tuning llms on noisy sets, we observe that the model is experiencing a "convex" learning curve under label noise (appendix c figure 3 ): the model gradually increases the accuracy by learning easy samples in the earlier stage, but continuously experiences performance drop by over-fitting the noise labels. hence, can we reserve the knowledge of the first stage and mitigate the over-fitting in the second stage without using any clean data validation? following this intuition, we propose safer: a noise-resistant framework for efficient and robust bert fine-tuning to perform robust llms fine-tuning under noisy classification nlp tasks, without using any clean labels. generally, this framework is compatible with any llm that contains an encoder architecture to encode sequences into latent representations. we first fine-tune the model following a typical manner but perform early stopping with a label-agnostic strategy. then, we leverage contrastive learning with an nlp-specific augmentation strategy and implement structural learning to further combat noisy labels. to evaluate our proposed framework, we perform experiments on text classification tasks using pre-trained bert models @xcite . here, we select the bert family to represent llms due to their widespread recognition in both the industry and research domains. we implement several state-of-the-art robust learning methods against label noise as our baseline methods (appendix b.1). these methods are designed to mitigate the general classification label noise issue without explicitly considering the usage of llms. the experiment results show the consistent and superior performance of our proposed learning framework. finally, we implement safer in two industrial biomedical literature mining tasks under unavoidable human labeling noise and achieve robust practical performance compared with baselines. the main contributions are as follows: