Self-supervised Regularization for Text Classification (2021)

Abstract: Text classification is a widely studied problem and has broad applications. In many real-world problems, the number of texts for training classification models is limited, which renders these models prone to overfitting. To address this problem, we propose SSL-Reg, a data-dependent regularization approach based on self-supervised learning (SSL). SSL (Devlin et al., 2019a) is an unsupervised learning approach that defines auxiliary tasks on input data without using any human-provided labels and learns data representations by solving these auxiliary tasks. In SSL-Reg, a supervised classification task and an unsupervised SSL task are performed simultaneously. The SSL task is unsupervised, which is defined purely on input texts without using any human- provided labels. Training a model using an SSL task can prevent the model from being overfitted to a limited number of class labels in the classification task. Experiments on 17 text classification datasets demonstrate the effectiveness of our proposed method. Code is available at https://github.com/UCSD-AI4H/SSReg .

## methods to alleviate overfitting in text classification, we propose ssl-reg, which is a regularization into a text encoder. encodings of these texts and their corresponding labels are fed into the head of a target task (e.g., classification) which yields a classification loss. in a self-supervised task, inputs are encodings of texts and outputs are constructed on original texts (e.g., masked tokens). the classification task and ssl task share the same text encoder and losses of these two tasks are optimized jointly to learn the text encoder. ## conclusions and future work in this paper, we propose to use self-supervised learning to alleviate overfitting in text classification problems. we propose ssl-reg, which is a regularizer based on ssl and a text encoder is trained to simultaneously minimize classification loss and regularization loss. we demonstrate the effectiveness of our methods on 17 text classification datasets. for future work, we will use other selfsupervised learning tasks to perform regularization, such as contrastive learning, which predicts whether two augmented sentences stem from the same original sentence.