Open-Set Semi-Supervised Text Classification via Adversarial Disagreement Maximization (2024)

Abstract: Open-Set Semi-Supervised Text Classification (OSTC) aims to train a classification model on a limited set of labeled texts, alongside plenty of unlabeled texts that include both in-distribution and out-of-distribution examples. In this paper, we revisit the main challenge in OSTC, i.e., outlier detection, from a measurement disagreement perspective and innovatively propose to improve OSTC performance by directly maximizing the measurement disagreements. Based on the properties of in-measurement and cross-measurements, we design an Adversarial Disagreement Maximization (ADM) model that synergeticly optimizes the measurement disagreements. In addition, we develop an abnormal example detection and measurement calibration approach to guarantee the effectiveness of ADM training. Experiment results and comprehensive analysis of three benchmarks demonstrate the effectiveness of our model.

## introduction text classification is a fundamental task in natural language processing. with the development of modern deep learning techniques, text classification has achieved significant advancement. however, deep learning models usually require substantial labeled data, which is expensive in many realworld applications. to tackle this problem, semisupervised text classification (stc) has been proposed, which only needs a small set of labeled examples along with plenty of unlabeled examples @xcite @xcite @xcite @xcite . by utilizing unlabeled texts in training machine learning models, these approaches reduce the need to expensively annotate abundant data. however, the stc assumption that all unlabeled texts are sampled within the intended * corresponding author scope is impractical in real-world applications. therefore, researchers recently explored open-set semi-supervised text classification (ostc) @xcite , which allows the inclusion of out-ofdistribution examples in the unlabeled text set. the main challenge in ostc is the commonly known false positive inference problem @xcite , which indicates a phenomenon that out-ofdistribution texts are prone to be recognized as an in-distribution class, leading to unsatisfactory ostc outcomes. to address this issue, prior studies have integrated an stc model with different outlier detection techniques. for example, lmcl @xcite and softmax @xcite ) learn discriminative embeddings and utilize local outlier factor (lof) @xcite to identify in-distribution (id) and out-of-distribution (ood) examples. msp @xcite and los @xcite respectively employ maximum softmax probability and normalized entropy function to identify id and ood examples. we provide a general understanding of existing ostc models. specifically, we revisit outlier detection in existing methods from a measurement disagreement perspective. it can be concluded that existing approaches are fundamentally grounded in the assumption that the differentiation between id and ood examples hinges on the presence of disagreement in specific measurements associated with the examples. for instance, the density measurement in lof of lmcl and softmax, and the confidence and entropy measurements in msp and los. with this assumption, existing methods distinguish id and ood examples by setting certain thresholds on the measurement. the above discussion motivates us to answer a question: since id and ood examples can be more easily distinguished when the measurement disagreement is larger, can we improve outlier detection in ostc by directly maximizing such measurement disagreement between id and ood exam-ples? to answer this question, we make formal definitions and assumptions for measurement disagreements and reveal several useful properties: (1) the disagreement bounds can be increased by maximizing in-measurement disagreement between id and ood examples. (2) the comparative consistency assumption allows us to synergisticly optimize two different measurements. (3) the in-example consistency assumption enables us to detect abnormal examples and calibrate measurements. based on the above motivations, we propose an adversarial disagreement maximization (adm) model for ostc. concretely, we treat the crossentropy loss of the id softmax classifier and the outlier detection confidence as two different measurements. to perform a synergistically measurement optimization, we leverage an adversarial learning approach to iteratively enlarge the disagreements of the two measurements. in addition, to guarantee the effectiveness of disagreement maximization, we design an abnormal-example detection approach to correct the measurement optimization direction. to summarize, we make the following contributions: (1) we provide a general understanding of outlier detection in ostc and revisit it from a measurement disagreement perspective. ( 2) we propose an adm model to directly maximize measurement disagreements combined with measurement calibration. (3) we evaluate our adm model on three benchmark datasets and demonstrate its effectiveness by comprehensive analysis. ## related works semi-supervised text classification (stc) trains a model with a few labeled texts and many unlabeled texts. various regularization techniques, consistency training approach are developed for stc @xcite @xcite @xcite . among these models, uda @xcite utilizes substituting noising operations to construct data augmentations combined with consistency training. mixtext @xcite design a novel text data augmentation method with mixup and used in consistency training. some other works leverage pseudo-labeling to annotate unlabeled texts as additional training data @xcite @xcite @xcite . open-set semi-supervised learning is understudied task. the pipeline approaches first filters out ood examples and then conduct semi-supervised training with filtered data. these methods often design special outlier detectors @xcite @xcite or build new optimization process @xcite @xcite . in recent work los @xcite , the authors design a set of pipeline methods utilizing different outlier detection approaches, including msp @xcite , doc @xcite , lmcl @xcite and lsoftmax @xcite . los unify semi-supervised training and outlier detection within probabilistic latent variable modeling. we understand ostc in a new measurement disagreement perspective and leverages adversarial learning @xcite to achieve disagreement maximization. ## conclusion we reveal the potential of measurement disagreement in ostc. to fully employ the advantage of measurement disagreement in ostc, we propose to directly maximize it and design a novel adversarial disagreement maximization (adm) approach combined with abnormal-example detection to improve ostc performance. experiment results demonstrate the effectiveness of adm.