Dynamic Data Selection for Curriculum Learning via Ability Estimation (2020)

Abstract: Curriculum learning methods typically rely on heuristics to estimate the difficulty of training examples or the ability of the model. In this work, we propose replacing difficulty heuristics with learned difficulty parameters. We also propose Dynamic Data selection for Curriculum Learning via Ability Estimation (DDaCLAE), a strategy that probes model ability at each training epoch to select the best training examples at that point. We show that models using learned difficulty and/or ability outperform heuristic-based curriculum learning models on the GLUE classification tasks.

## introduction curriculum learning trains a model by using easy examples first and gradually adding more difficult examples. it can speed up learning and improve generalization in supervised learning models @xcite @xcite . a major drawback of existing curriculum learning techniques is that they rely on heuristics to measure the difficulty of data, and either ignore the competency of the model during training or rely on heuristics there as well. for example, sentence length is often used as a proxy for difficulty in nlp tasks @xcite . such heuristics can be useful but have limitations. first, the heuristic chosen may not actually be a proxy for difficulty. depending on the task, long sequences could signal easier or harder examples, or have no signal for difficulty. second, a model's notion of difficulty may not align with the heuristic imposed by a human developing the model. it may be that examples that appear difficult for the human are in fact easy for the model to learn. competency was recently introduced as a mechanism to determine when new examples should be * work performed while at umass amherst. in this study, we propose to estimate both the difficulty of examples and the ability of deep learning models as latent variables based on model performance using item response theory (irt), a well-studied methodology in psychometrics for test set construction and subject evaluation @xcite . irt models estimate latent parameters such as difficulty for examples and a latent ability parameter for individuals ("subjects"). irt models are learned by administering a test to a large number of subjects, collecting and grading their responses, and using the subject-response matrix to estimate the latent traits of the data. these learned parameters can be used to estimate the ability of future subjects, based on their graded responses to the examples. irt has not seen wide adoption in the machine learning community, primarily due to the fact that fitting irt models requires a large amount of human annotated data for each example. however, recent work has shown that irt models can be fit using machine-generated data instead of humangenerated data as input @xcite . because irt learns example difficulty and subject ability together, in this work we propose replacing heuristics for learned parameters in curriculum learning. first, we experiment with replacing a typical difficulty heuristic (sentence length) with learned difficulty parameters. second, we propose dynamic data selection for curriculum learning via ability estimation (ddaclae), a novel curriculum learning framework that uses the estimated ability of a model during the training process to dynamically identify appropriate training data. at each training epoch, the latent ability of the model is estimated using output labels generated at the current epoch. once ability is known, only training data that the model has a reasonable chance of labeling correctly is included in training. as the model improves, the estimated ability will improve, and more training examples will be added. to the best of our knowledge, this is the first work to learn a model competency during training that is directly comparable to the difficulty of the examples. our study will test the following three hypotheses: h1: using learned latent difficulties instead of difficulty heuristics leads to better heldout test set performance for models trained using curriculum learning, h2: a dynamic data selection curriculum learning strategy that probes model ability during training leads to better held-out test set performance than a static curriculum learning strategy that does not take model ability into account, h3: dynamic curriculum learning is more efficient than static curriculum learning, leading to faster convergence. we test our hypotheses on the glue classification data sets @xcite . our contributions are as follows: (i) we demonstrate that for curriculum learning, learned difficulty outperforms traditional difficulty heuristics, (ii) we introduce a novel curriculum learning framework which automatically selects training data based on the estimated ability of the model, and (iii) we show that training using ddaclae leads to better performance than both traditional curriculum learning methods and a fully supervised competitive baseline. our findings support the overall curriculum learning framework, and demonstrate that learning difficulty and ability lead to further performance improvements beyond common heuristics. 1 2 methods ## related work curriculum learning is a well-studied area of machine learning @xcite . the primary focus has been on developing new heuristics to identify easy and difficult examples in order to build a curriculum. originally, curriculum learning methods were evaluated on toy data sets with heuristic measures of difficulty @xcite . for example, on a shapes data set, shapes with more sides were considered difficult. similarly, longer sentences were considered difficult. word rarity has also been proposed as a heuristic for difficulty, with similar results to sentence length @xcite . recent work on automating curriculum learning strategies use multi-arm bandits to minimize regret with respect to curriculum selection @xcite . in that work the authors again rely on proxies for progress (loss-driven and complexity-driven). loss-driven proxies are inherently model-specific, in that the difficulty of an example is determined by a specific model's performance on the example. using a global difficulty such as one learned using irt methods allows for an interpretable difficulty metric that applies across models. the complexity-driven proxies proposed are specific to neural networks, while ddaclaeis a generic algorithm for dynamic curriculum learning. spaced repetition strategies (sr) can be effective for improving model performance @xcite . instead of using a traditional curriculum learning setup, spaced repetition bins examples based on estimated difficulty, and shows bins to the model at differing intervals so that harder examples are seen more frequently than easier ones. this method has been shown to be effective for human learning, and results demonstrate effectiveness on nlp tasks as well. similarly to traditional curriculum learning, sr uses heuristics for difficulty and rigid schedulers to determine when examples should be re-introduced to the learner. self-paced learning (spl) is another related strategy for example ordering during training @xcite . in our work the difficulty values of examples are global and static, where for spl examples do not have set difficulty values; instead, groups of examples are considered in sets. in addition, model competency is not considered in spl, it is assumed that competency improves as more difficult examples are added. there has been recent work investigating the theory behind curriculum learning @xcite , particularly around trying to define an ideal curriculum. the authors explicitly identify the two key aspects of curriculum learning, namely "sorting by difficulty" and "pacing." curriculum learning theoretically leads to a steeper optimization landscape (i.e., faster learning) while keeping the same global minimum of the task without curriculum learning. in that work there is still a reliance on "pacing functions" as opposed to an actual assessment of model ability at a point in time. this work may open interesting new areas of theoretical study linking difficulty and ability in curriculum learning. theoretical results @xcite have also demonstrated a key distinction between curriculum learning and similar methods such as self-paced learning @xcite , hard example mining @xcite , and boosting @xcite : namely that the former considers difficulty with respect to the final hypothesis space (i.e., a model trained on the full data set) while the later methods consider ranking examples according to how difficult the current model determines them to be. ddaclae bridges a gap between these methods by probing model ability at the current point in training and using this ability to identify appropriate training examples in terms of global difficulty. in addition, a key component of most prior work in curriculum learning is the notion of balance. when defining a curriculum, it is often the case that proportions are maintained between classes. that is, difficulty itself is not the only factor when building the curriculum. instead, the easiest examples for each class are added so that the model is proportionally exposed to the data consistent with the full training set. ddaclae does not consider class labels when selecting examples for training. it is important to note here that labels are used when learning difficulties, estimating ability, and actually updating parameters during training. they are not used to balance the curriculum. in this way ddaclae is more closely aligned with a pure curriculum learning strategy that considers only the easiness/hardness of an example during training. this is an added benefit to ddaclae as there is no need for class label accounting during training. ## conclusion this work validates and supports the existing literature on curriculum learning. our results confirm that curriculum learning methods for supervised learning can lead to faster convergence or better local minima, as measured by test set performance @xcite . we have shown that by replacing a heuristic for difficulty with a theoreticallybased, learned difficulty value for training examples, static curriculum learning strategies can be improved. we have also proposed ddaclae, the first curriculum learning method to dynamically probe a model during training to estimate model ability at a point in time. knowing the model's ability allows for data to be selected for training that is appropriate for the model and is not rigidly tied to a heuristic schedule. ddaclae trains more effective models in most cases that we considered, particularly for randomly initialized lstm models. based on our experiments, we report mixed results on our stated hypotheses. replacing heuristics with learned difficulty values leads to improved performance when training models with curriculum learning, supporting h1. ddaclae does outperform other training setups when used to train lstm models. results are mixed when used to fine-tune the bert base model. therefore h2 is partially supported. we see similarly mixed results when evaluating efficiency. with bert base fine-tuning, fully supervised fine-tuning is usually the most efficient, as the number of fine-tuning epochs needed is already very low. for lstm, ddaclae is more efficient than the other curriculum learning strategies, and is the most efficient overall for two of the six tasks. h3 is partially supported by these results. even though it is dynamic, ddaclae employs a simple schedule: only include examples where b @xmath0. however, being able to estimate ability on the fly with ddaclae opens up an exciting new research direction: what is the best way to build a curriculum, knowing example difficulty and model ability (e.g., the 85% rule of @xcite ?