Reinforced Active Learning for Low-Resource, Domain-Specific, Multi-Label Text Classification (2023)

Abstract: Text classification datasets from specialised or technical domains are in high demand, especially in industrial applications. However, due to the high cost of annotation such datasets are usually expensive to create. While Active Learning (AL) can reduce the labeling cost, required AL strategies are often only tested on general knowledge domains and tend to use information sources that are not consistent across tasks. We propose Reinforced Active Learning (RAL) to train a Reinforcement Learning policy that utilizes many different aspects of the data and the task in order to select the most informative unlabeled subset dynamically over the course of the AL procedure. We demonstrate the superior performance of the proposed RAL framework compared to strong AL baselines across four intricate multi-class, multi-label text classification datasets taken from specialised domains. In addition, we experiment with a unique data augmentation approach to further reduce the number of samples RAL needs to annotate.

## introduction modern text classification systems powered by pre-trained language models achieve excellent accuracy across a variety of tasks and corpora. consequently, there is a rapidly growing demand to build such systems for increasingly challenging problem settings, such as dealing with domain-specific, low-resource data of high categorical complexity. typically, this kind of data pertains to specialised domains such as medical or legal, and is often critical in order to realize real world, industrial scale applications. in addition, many of these datasets are multi-label, meaning each sample can have several different categories assigned to it. annotation in these specialised domains is extremely costly since experts have to be consulted in order to guarantee a reliable assignment of labels. active learning (al) provides a technique for an effective sample selection that reduces the amount of labeling required to reach a certain model performance or increases model performance given an annotation budget. while al strategies are numerous, their effectiveness tends to highly depend on the dataset and the classification model used. for example, one of the bestperforming al strategy to select samples based on model confidence can be unreliable if the model has not yet reached a certain level of performance. in this work we aim to combine the expressiveness of several active learning strategies with the capabilities of reinforcement learning (rl) in a sequential al setup. in reinforced active learning (ral), the agent evaluates samples based on several al strategies and other model-based features. the resulting decision policy can dynamically adapt to the al task, underlying classification model and dataset. selected samples are annotated and added to the labeled pool. the agent then receives a reward based on the classifiers performance given the newly labeled set and consequently, can learn connections between the presented information from the sample representations and the underlying classification model. as such, ral learns the best combination of al strategies for a particular dataset and furthermore, retains the ability to sample randomly should the al strategies be less effective. in addition, ral also tunes a variety of other parameters that are often set arbitrarily, such as the most beneficial number of samples to be labelled in each al step. our work contains several direct continuations of the extreme multi-label study in @xcite . in their experiment, the authors showed that no single al strategy would outperform all baselines across several datasets. in this work, we aim to expand on these findings and devise an adaptive al strategy that is effective across domains. the main contributions of our work are as follows: ## related work the effectiveness of al for text classification has been subject to extensive research @xcite , @xcite ) with specific solutions for deep models @xcite , @xcite ) and multi-label settings @xcite , @xcite ). al specifically for deep learning however is a topic that still requires further exploration @xcite . successful al strategies for deep models often employ only limited use of the classifiers prediction probabilities, for example by using structures within the dnn @xcite , separate selection networks @xcite , @xcite ) or properties of pre-trained word embeddings @xcite . powerful representations learned by deep models can be leveraged for reinforcement learning in order to solve tasks "previously out of reach for a machine." @xcite . while results are often difficult to reproduce @xcite , deep rl is effectively used for various applications e.g. in games and robotics @xcite as well as modeling human intelligence and behaviour, for example, in multi-task, multiagent settings @xcite , @xcite ). learning al strategies that generalize across domains, has been employed by transferring policies learned with rl from a source task to a target task @xcite , desreumaux and lemaire ( 2020 )). alternative approaches look to move beyond rl, using imitation learning @xcite or cyclic "dreaming" @xcite but keep the concept of transferring the learned policy in order to fully learn on the source task. similar to our approach @xcite and @xcite ) switch between al strategies depending on their predicted effectiveness. the term active reinforcement learning appears in works dealing with improved rl methods, such as guiding the policy search space or attaching a query cost to obtaining the reward @xcite ) @xcite , which share theoretical foundations with our work but are otherwise not related. algorithm 1 reinforced active learning for text classification using deepq reinforcement learning. in reinforcement learning an agent interacts with an environment by perceiving a state, deciding on an action and then receiving a reward from the environment based on the action. over the course of many such interactions, the agent attempts to maximize the cumulative reward and as a consequence, learns to better interact with the environment. the decision process that maps states to actions is called the policy π. an important part of learning π is to perform random actions in the early learning stages; a process also called exploration. the idea is that the agent can "try out" actions randomly and form an impression of how the newly encountered environment will react. with an increasing number of steps the agent gradually shifts to using π to decide the action. active learning strategically selects samples to be annotated from an unlabeled collection u starting from a small initial labeled set d. first, al trains a model m on d. then al improves the performance of m iteratively: samples are selected, an annotator provides labels for the selected samples and the labeled samples are used to update m . the algorithms used to decide which sentences to select are referred to as al strategies. in ral, we transform al into a environment that a rl agent can interact with. in particular, we treat each sample from the unlabeled set as an observable state which is defined by a variety of features (see section 3.2). for the action, the agent decides whether or not to annotate a sample after observing the corresponding state. similar to @xcite , we transform al into a sequence of decisions. the reward is received from the performance of m after training with the newly annotated samples. in particular, we do not train the model each time a single sample is annotated. instead, the model is only trained after a fixed number of steps have been completed (see section 3.1). we stop the al when a certain annotation budget is exceeded. π is updated each step following a rl scheme using the model accuracy as reward. when the al is finished, the environment resets and a new rl episode begins. ral using deepq-rl is illustrated in algorithm 1.