Noise-injected Consistency Training and Entropy-constrained Pseudo Labeling for Semi-supervised Extractive Summarization (2022)

Abstract: Labeling large amounts of extractive summarization data is often prohibitive expensive due to time, financial, and expertise constraints, which poses great challenges to incorporating summarization system in practical applications. This limitation can be overcome by semi-supervised approaches: consistency-training and pseudo-labeling to make full use of unlabeled data. Researches on the two, however, are conducted independently, and very few works try to connect them. In this paper, we first use the noise-injected consistency training paradigm to regularize model predictions. Subsequently, we propose a novel entropy-constrained pseudo labeling strategy to obtain high-confidence labels from unlabeled predictions, which can obtain high-confidence labels from unlabeled predictions by comparing the entropy of supervised and unsupervised predictions. By combining consistency training and pseudo-labeling, this framework enforce a low-density separation between classes, which decently improves the performance of supervised learning over an insufficient labeled extractive summarization dataset.

## introduction text summarization is a challenging task that generates a condensed version of an input text that captures the original's core meaning. in this paper, we focus on extractive summarization since it usually generates semantically and grammatically correct sentences @xcite @xcite . the extractive summarization typically requires to label each sentence in massive documents for model training. however, acquiring well-annotated labels is a costly process, and labeling every sentence would be labor-intensive and error-prone due to subjective judgments of human. this motivates † these two authors contributed equally. * jianxin li is the corresponding author. towards this goal, we first revisit an effective semi-supervised method, consistency training @xcite . the consistency training leverages voluminous unlabeled data and employs advanced data augmentation methods to generate diverse and realistic noisy source text, forcing the model to be consistent with these noises. the consistency training has been extensively applied on the classification problems, such as text classification @xcite , image recognition @xcite @xcite @xcite . however, how does the consistency training work on semi-supervised extractive summarization tasks is still unclear. we investigate the noise-injected consistency training for semi-supervised extractive summarization to encourage a consistent reason of model @xcite under data perturbation. this framework makes sense intuitively because a good supervised model should be robust to any slight change in an input example. namely, encouraging local change by injecting a slight noise in a diverse perturbation manner can improve the summarization effectiveness. nevertheless, the consistency regularized semisupervised framework usually suffers from insufficient supervision. when labeled data is limited, the model is easy to over-fitting. extensive unlabeled data will then make the model suffer from the gradual drift problem and impede further improvements of the model. to address this problem, we develop new methods of selection and exploitation for pseudo labels to explore all unlabeled samples for the semi-supervised summarization cycle. prior pseudo labeling work @xcite cuses on the confidence of the predictions to pick up the class used as if they are accurate labels. however, the summarization task exposes potential problems with this approach. the confidence of the prediction results is hard to change significantly along with model convergence (e.g., using the bert-based model). as shown in the second sub-figure in fig. 1(a) , almost all prediction probability of sentences has achieved an undifferentiated score between 0.12 to 0.16 in the convergence epoch, which means almost all sentences are mapped into a small area for classification. therefore, it is challenging to set a fixed threshold to determine a proper confidence score, and it is easy to produce low-quality pseudo labels. we divert attention to entropya metric for measuring uncertainty, to address this problem. as is shown in fig 1 (b), the prediction entropy of supervised and unsupervised data are comparable, which enables us to constrain the unsupervised entropy with supervised entropy without any external threshold value. we introduce entropy-constrained pseudo labeling to avoid fixed threshold adoption, which ensures the entropy of prediction results of unlabeled data adjust that of labeled samples adaptively. the final framework is cpsum foot_0 , which combines the noise-injected consistency training with the entropy-constrained pseudo labeling for semisupervised extractive summarization. experimental results demonstrate that our approach achieves the state-of-the-art in low-resource scenarios with 10, 100, and 1000 labeled examples on the target corpus. the main contributions of our method are: ## proposed framework the overview illustration of our framework is shown in fig. 2 . it is composed of two components: noise-injected consistency training and entropyconstrained pseudo labeling. specific to extractive summarization, we use the base version of bert @xcite to implement our models in all experiments. we give detail description of two components in sec.3.1 and sec.3.2. ## conclusions in this paper, we present a new perspective on effectively using consistency training and pseudo labeling to improve low resource extractive summarization over an insufficiently labeled dataset. with substituting simple noise injection operations with advanced data augmentation and constraining pseudo label selection with average entropy, our method brings substantial improvements compared with the supervised learning frameworks. since our proposed model is orthogonal to the methods that using pre-trained models, we believe our model can be further boosted by taking other salient pretrained models to initialize the text representations. additionally, although we use ramp-up exploitation to control the adverse entropy effect brought by the early model, incorrect prediction cannot be avoided. an impeccable minimum entropy regularization method can be exploited in the future.