Combining Supervised Learning and Reinforcement Learning for Multi-Label Classification Tasks with Partial Labels (2024)

Abstract: Traditional supervised learning heavily relies on human-annotated datasets, especially in data-hungry neural approaches. However, various tasks, especially multi-label tasks like document-level relation extraction, pose challenges in fully manual annotation due to the specific domain knowledge and large class sets. Therefore, we address the multi-label positive-unlabelled learning (MLPUL) problem, where only a subset of positive classes is annotated. We propose Mixture Learner for Partially Annotated Classification (MLPAC), an RL-based framework combining the exploration ability of reinforcement learning and the exploitation ability of supervised learning. Experimental results across various tasks, including document-level relation extraction, multi-label image classification, and binary PU learning, demonstrate the generalization and effectiveness of our framework.

## introduction multi-label classification (mlc) task treats a problem that allows instances to take multiple labels, and traditional supervised learning (sl) methods on mlc heavily rely on human-annotated data sets, especially neural approaches that are datahungry and susceptible to over-fitting when lacking training data. however, in many mlc tasks that generally have dozens or hundreds of sizes of class sets, incompleteness in the acquired annotations frequently arises owing to the limited availability of expert annotators or the subjective nature inherent in human annotation processes. @xcite @xcite . therefore, we focus on the fundamentally important problem, typically termed multi-label positive-unlabelled learning (mlpul) @xcite , which involves learning from a multi-label * correspondence to zilong zheng. dataset in which only a subset of positive classes is definitely annotated, while all the remaining classes are unknown (which could be positives or negatives). for instance, as shown in fig. 1a&b , human annotators find it hard to completely annotate all the relations due to the confusion of understanding relation definitions and long-context semantics in document-level relation extraction (docre) task @xcite . positive and unlabelled (pu) classification has received extensive attention in binary settings, with several recent mlpul approaches adapting traditional binary pu loss functions to address multilabel classification tasks @xcite @xcite . these methods typically operate under the assumption that the prior distribution of positive labels can be inferred from fully labeled training samples or closely unbiased estimations. in a specific instance, @xcite supposed that the actual positive classes are three times the number of observed labels in the docre task, and their model's performance is heavily influenced by the prior (fig. 1c ). however, estimating the prior distribution of labels in real-world scenarios poses significant challenges, as it is rarely feasible to ensure a comprehensive data set encompassing all label types @xcite @xcite . additionally, @xcite noted that many long-tail label types tend to be omitted from training annotations. consequently, we focus on addressing mlpul without prior knowledge of class distribution. moreover, mlc generally faces the challenge of imbalanced positive and negative labels, which is severely exacerbated by missing positive class annotations under mlpul, as shown in fig. 1b . previous works typically adopted the re-balance factor to re-weight the loss functions, containing positive up-weight and negative under-weight @xcite . we simply attempt these approaches and find they partly improve the model performance but still perform 13553 results of training on incomplete docred and testing on reannotated re-docred and docgnre @xcite . ssr-pu is sensitive to prior estimation, while ours is prior agnostic. d. performance comparison in docre task. based on this intuition, we introduce a novel framework termed mixture learner for partially annotated classification (mlpac), which combines the exploratory capacity of rl in tandem with the exploitation capabilities of supervised learning. specifically, we design a policy network (as a multi-label classifier) and a critic network, along with two types of reward functions: global rewards calculated by a recall function, which evaluates the all-classes prediction performance for each instance, and local rewards provided by the critic network, which assesses the prediction quality of each individual class for a given instance. the local rewards are expected to narrow the exploration space of traditional rl and offer a preliminary yet instructive signal to guide the learning process, while the global rewards encourage the policy network to explore a broader spectrum of positive classes, consequently mitigating distribution bias stemming from imbalanced labels and incomplete annotations. in addition, inspired by the traditional actorcritic rl algorithm @xcite , we iteratively train the policy network and the critic network, which achieves dynamic reward estimation in our setting. the absence of fully annotated samples in both training and validation sets precludes the direct attainment of perfectly accurate rewards. hence, we introduce label enhancement through collaborative policy and critic network efforts during iterative training, boosting label confidence and enhancing the critic network's reward estimation accuracy. moreover, our rl framework is concise and flexible, guaranteeing its generalization and adaptation to many tasks. beyond the experiments on document-level relation extraction task ( §4.1) in natural language process (nlp) field, we also conduct sufficient experiments in multi-label image classification task ( §4.2) in computer vision (cv) field and general pu learning setting in binary case ( §4.3) to verify the generalization and effectiveness of our framework. all experimental results demonstrate the advantage and significant improvement of our framework. besides the above methodologies, chen et al. (2020); @xcite proposed regularization term based on mixup @xcite to enhance incomplete labels, alleviating the overfitting problem and increasing model robustness. the concurrent work to us, @xcite designs positive-augmentation and positive-mixup strategies to improve rank-based learning methods @xcite for pu document-level relation extraction. these strategies are not uniquely applicable to their framework, and our method can also integrate them, which we leave to future work. ## rl-based framework we propose a novel rl framework to solve the mlpul task. we formulate the multi-label prediction as the action execution in the markov decision process (mdp) @xcite . we design both local and global rewards for actions to guide the action decision process. the policy-based rl method is adopted to train our policy network. the overall rl framework is illustrated in figure 2 . we introduce the details in the following subsections. ## experiments the multi-label positive unlabelled learning (mlpul) problem is common and essential in the nlp field. there are many tasks in nlp that face incomplete annotation problems, such as finegrained entity typing, multi-label text classification, and document-level relation extraction. to verify the effectiveness of our proposed rl framework, we experiment with the positive-unlabelled document-level relation extraction (docre) task as a representative mlpul task in nlp. we also conduct experiments in multi-label image classification (mlic) tasks and binary pu learning setting 10% 30% 50% 70% 100% method p r f1 p r f1 p r f1 p r f1 p r f1 besides comparing with previous state-of-the-art (sota) models in each specific task, we construct some simple baseline methods: ## conclusion in this work, we propose an rl framework to deal with partially annotated multi-label classification tasks. we design local rewards assessed by a critic network and global rewards assessed by recall functions to guide the learning of our policy network, achieving both exploitation and exploration. with an iterative training procedure and a cautious data enhancement, our mlpac has demonstrated its effectiveness and superiority on different tasks.