Active Curriculum Learning (2021)

Abstract: This paper investigates and reveals the relationship between two closely related machine learning disciplines, namely Active Learning (AL) and Curriculum Learning (CL), from the lens of several novel curricula. This paper also introduces Active Curriculum Learning (ACL) which improves AL by combining AL with CL to benefit from the dynamic nature of the AL informativeness concept as well as the human insights used in the design of the curriculum heuristics. Comparison of the performance of ACL and AL on two public datasets for the Named Entity Recognition (NER) task shows the effectiveness of combining AL and CL using our proposed framework.

## introduction modern deep learning architectures predominantly need large amounts of labeled data to achieve high levels of performance. in the presence of a large unlabeled corpus, data points are usually chosen randomly to be annotated. however, annotation can be a costly task and not all the annotations are equally beneficial. active learning (al) aims to reduce the number of annotations required to train a machine learning model by choosing the most "informative" unlabeled data for annotation. the informativeness is determined by querying a model or a set of models trained on the available annotated data @xcite . algorithm 1 shows al more formally. several categories of informativeness score have been developed in the literature. for example, uncertainty metrics select unlabeled data for which the model has the highest uncertainty of label prediction @xcite . examples of uncertainty measures for a classification task are the difference of the probability of prediction for the first and second most likely classes (i.e., the margin of the prediction probability) and the entropy of prediction over all classes (i.e., -∑ 𝑝 𝑖 log 𝑝 𝑖 𝑐 𝑖=1 some other examples of informativeness scoring methods for unlabeled data are the amount of prediction disagreement in a committee of models @xcite and the amount of expected change to model weights @xcite or loss value @xcite . curriculum learning (cl), on the other hand, attempts to mimic how humans learn and uses that knowledge to train better models @xcite . complex topics are taught to humans based on a curriculum which takes into account the level of difficulty of the material presented to the learner. cl borrows this idea and engages the human experts to design a metric that is used to sort the annotated training data from "easy" to "hard" to be presented to the model during training @xcite . the 1. seed labeled data 𝑫 𝑳 = {(x1, y1), …, (xk, yk)} 2. unlabeled data 𝑫 𝑼 = {xk+1, …, xm} 3. while the stopping criterion is not met: 3.1. fine-tune or train model 𝑴 on 𝑫 𝑳 3.2. 𝑰 ≔ the set of 𝑖 most informative data samples in 𝑫 𝑼 according to 𝑴 3.3. 𝑫 𝑼 ≔ 𝑫 𝑼 \ 𝑰; 𝑫 𝑳 ≔ 𝑫 𝑳 ∪ 𝑳(𝑰) algorithm 1: steps of the al algorithm where 𝐿(𝑰) denotes the set 𝑰 after annotation. an example of stopping criterion can be a minimum value for accuracy. 1. training data 𝑫 𝑻 = {} 2. available data 𝑫 𝑨 = {(𝑥 1 , 𝑦 1 ), … , (𝑥 𝑛 , 𝑦 𝑛 )} 3. repeat until 𝑫 𝑨 is empty: 3.1. 𝑰: = the set of 𝒌 easiest examples in 𝑫 𝑨 according to a fixed curriculum 3.2. 𝑫 𝑻 : = 𝑫 𝑻 ∪ 𝑰; 𝑫 𝑨 : = 𝑫 𝑨 \ 𝑰 3.3. fine-tune existing model 𝑴 on 𝑫 𝑻 algorithm 2: steps of the cl algorithm. other related approaches such as self-paced learning (spl) @xcite and self-paced curriculum learning @xcite have also been proposed to show the efficacy of a designed curriculum which adapts dynamically to the pace at which the learner progresses. other attempts at improving an al strategy include self-paced active learning @xcite in which the authors introduce practical techniques to consider informativeness, representativeness, and easiness of samples while querying for labels. such methods that only focus on designing a curriculum miss, in general, the opportunity to also leverage the ability of the predictive model which progresses as new labeled data becomes available. the addition of cl injects human expertise into learning manifested in the design of a curriculum. this is in contrast with previous studies that combined al with spl @xcite . spl is inspired by cl but, similarly to al, relies on querying the model being trained to select instances for labeling. our contributions in this paper are twofold: (i) we shed light on the relationship between al and cl by investigating if al enforces (or follows) a curriculum. to this end, we monitor and visualize a variety of novel curricula during the al simulation loop; (ii) we propose a novel method which we call active curriculum learning (acl). acl takes advantage of the benefits of both cl (i.e., designing a curriculum for the model to follow) and al (i.e., choosing samples based on the enhanced ability of the predictive model) at the same time to improve al. our preliminary experiments show that the performance of an al strategy will be improved by deliberately combining al and cl concepts. this article presents the foundation of this method accompanied by the preliminary results and in our future work we will explore its effectiveness more extensively by implementing more experiments and performing hyper parameter tuning as well as exploring other nlp tasks beyond ner. ## novel curricula other than the most explored curriculum features such as sentence length and word frequency some other curricula for measuring diversity, simplicity, and prototypicality of the samples are proposed in @xcite . our conjecture is that largescale language models and also linguistic features can be used to design nlp curricula. we design seven novel curricula which assign a score to a sentence indicating its level of difficulty for a specific nlp task. then, to acquire a curriculum, sentences are sorted by their corresponding scores. other than our 7 novel curricula, we also experiment with the following commonly used curricula: 1. sent_len: number of words in a sentence. 2. word_freq: average of frequency of the words in a sentence (e.g., frequency of the word a is calculated by 𝑁 𝐴 ∑ 𝑁 𝑤 𝑤∈𝑉 where v is the set of the unique vocabulary of the labeled dataset, and 𝑁 𝑤 is the number of times the word 𝑤 has appeared in the labeled dataset). our seven novel curricula are as follows: 1. parse_child: average of the number of children of words in the sentence parse tree. 2. gpt_score: sentence score according to the gpt2 language model (radford et al. 2019) calculated as follows: ∑ log(𝑝(𝑤 𝑘 )) 𝑘 ## the relationship between al and cl and the experimental setup we set out to answer the following question: what is the relationship between al and cl from the lens of the nine curricula? to answer this question, we simulate two al strategies as well as random strategy and monitor the curriculum metrics on the most informative samples (from the unlabeled data) chosen for annotation by each sampling strategy and compare them. we use the following two informativeness measures for unlabeled sentences in our al strategies: (i) min-margin: minimum of margin of the prediction probability for the sentence tokens is considered as the al score for that sentence. sentences with lower scores are preferred, (ii) max-entropy: maximum of entropy of the prediction probability for the sentence tokens are considered as the al score for that sentence and sentences with higher scores are preferred. for the experiments, we use a single layer bi-lstm model @xcite with the hidden state size of 768, enhanced with a 2-layer feedforward network in which the number of hidden and output layers' nodes are equal to the number of classes in the dataset. the input to the lstm model is the word2vec embedding @xcite ) of sentence words. we use adam optimizer @xcite with the batch size of 64 and the learning rate of 5e-4. we experiment with two publicly available english-language ner datasets: ontonotes5 foot_0 , and conll 2003 foot_1 and use early stopping on the loss of the provided validation sets. furthermore, we start with 500 randomly selected sentences as the seed data and choose 500 sentences to be labeled in each iteration for a total of 15 iterations. figure 1 illustrates the experimental results of monitoring gpt score during al loop. this figure clearly shows that gpt score of sentences chosen by max-entropy tends to have lower values (i.e., more complex sentences) and min-margin tends to choose sentences with higher values (i.e., simpler sentences) compared to a random strategy. similar figures for other curricula reveal peculiarities of the different al strategies compared to the random strategy and other al strategies. due to space limitations, instead of including such figures for different strategies, we calculate the following metric which we call mean normalized difference (mnd) to quantify how an al selection strategy differs from a random strategy in choosing the most informative unlabeled data based on a curriculum. this metric is defined as follows: ## active curriculum learning (acl) to improve the performance of the al strategies, we introduce a simple yet effective method leveraging both advantages of al and cl which we call active curriculum learning (acl). the goal of this proposed method is to benefit from the dynamic nature of al data selection metric while utilizing experts' knowledge in designing a fixed curriculum. to this end, in each step of the acl loop, we use the following linear combination of the al and cl scores to choose the most informative unlabeled data: the overall steps of the acl algorithm are presented in algorithm 3. similar to the al algorithm, the min-margin based strategy favors sentences with lower 𝜓 𝐴𝐶𝐿 for annotation and the opposite is true for the max-entropy based approach. ## conclusions and future work to the best of our knowledge, this is the first work to investigate and reveal the relationship between two closely related machine learning techniques namely, al and cl. we observed that al in fact follows a curriculum as it progresses through its iterations compared to the random strategy. this is also the first work to take advantage of the benefits of both cl (i.e., designing a curriculum for the model to learn) and al (i.e., choosing samples based on the improved ability of the predictive model) to improve al in a unified model. in our future work, we are interested in understanding in detail how cl helps al, and exploring model-based techniques of combining al and cl rather than a fixed set of weights for α and β. another interesting question to investigate is to conduct similar experiments for other nlp tasks or using multiple curricula together with al can be beneficial in reducing the annotation cost. we are also interested in investigating our novel curricula on their own in an isolated cl setting.