H u C url: Human-induced Curriculum Discovery (2023)

Abstract: We introduce the problem of curriculum discovery and describe a curriculum learning framework capable of discovering effective curricula in a curriculum space based on prior knowledge about sample difficulty. Using annotation entropy and loss as measures of difficulty, we show that (i): the top-performing discovered curricula for a given model and dataset are often non-monotonic as apposed to monotonic curricula in existing literature, (ii): the prevailing easy-to-hard or hard-to-easy transition curricula are often at the risk of underperforming, and (iii): the curricula discovered for smaller datasets and models perform well on larger datasets and models respectively. The proposed framework encompasses some of the existing curriculum learning approaches and can discover curricula that outperform them across several NLP tasks.

## introduction annotation information has been extensively used by previous research in nlp to devise strategies for further data collection @xcite , model improvement and annotation analysis @xcite , pruning and weighting samples for better learning @xcite , or efficient use of monetary funds @xcite . recent studies show consistent positive correlation between difficulty of samples to the model and their level of human agreement @xcite @xcite . building on these findings, we aim to utilize such prior knowledge about sample difficulty to develop a curriculum learning (cl) framework that is capable of discovering effective curricula for nlp tasks. a curriculum is a planned sequence of learning materials and an effective one can improve training of nlp systems @xcite @xcite @xcite @xcite . cl seeks to improve model generalizability by ordering samples for training based on their latent difficulty @xcite . recent work reported efficiency and effectiveness gains through cl @xcite @xcite , especially in cases of harder tasks and limited or noisy data @xcite . existing cl approaches are designed to learn a single curriculum that works best for a given model and dataset. however, effective training could be achieved in multiple ways. in addition, existing approaches quantify sample difficulty through model behavior during training. although efficient and effective, model behavior can be affected by initialization and training dynamics @xcite , which limits the curriculum space that can be examined for finding effective curricula. this paper advocates a re-imagining of cl paradigms by introducing and formalizing the task of curriculum discovery, which aims to find effective curricula for a given model and dataset over a curriculum space. the present work specifically focuses on determining when and in which difficulty order text data samples should be learned for effective training of nlp systems. we propose a framework that employs prior knowledge about sample difficulty, such as entropy in human annotations, to inform an effective and flexible sample weighting scheme for curriculum discovery. the framework is capable of discovering optimal curricula (within the space of its weight functions) for any given model and dataset by optimizing the weight functions and adjusting the difficulty group of data samples as training progresses. the discovered curricula provide useful insights about datasets and models, such as the relative importance of different groups of samples for models or knowledge dependency among samples. we illustrate that the proposed framework has the potential to encompass some of the existing cl approaches. experimental results show that (a): the topperforming discovered curricula for the same model and dataset can be fundamentally dissimilar in their training strategies, indicating that effective training can be achieved in multiple ways; (b): the discovered curricula are often non-monotonic and greatly differ from the known strategies reported in existing literature, indicating that existing curricula, including easy-to-hard transition curricula, are at the risk of underperforming; and (c): the curricula discovered on small datasets and models perform exceptionally well on larger datasets and models respectively, illustrating the transferability of the discovered curricula. the paper presents a new curriculum learning approach that unlike existing approaches can discover multiple high-performing (and often diverse) curricula for each given nlp model and dataset, provide interpretable curricula in terms of sample difficulty, and encompass some of the existing curriculum learning approaches. foot_0 2 related work existing cl approaches are designed to learn a single curriculum that works best for a given model and dataset. they estimate sample difficulty through model behavior during training, quantified by the instantaneous loss @xcite , consistency in instantaneous loss @xcite , moving average of loss @xcite , transformations of loss @xcite @xcite , loss regularization @xcite @xcite , or learnable per-sample confidence @xcite @xcite . in terms of data ordering, subsampling approaches sample the easiest or hardest instances at every training iteration @xcite @xcite @xcite , sample weighting techniques weight instances according to their estimated difficulty @xcite @xcite @xcite , and sample pruning techniques filter hard or noisy instances from data prior to training @xcite . sub-sampling methods can be cumulative, exclusive or a combination of both. cumulative approaches add new samples to the ones that have been previously used network input features input final test loss task loss difficulty class update bayesian optimization weights task label figure 1 : the model defines a difficulty score based on prior knowledge about sample difficulty and assigns samples to k difficulty groups before training, e.g., easy, medium, and hard for @xmath0. a curriculum is defined for each difficulty group, which dynamically weights sample losses according to their difficulty groups. each curriculum is defined by a pair of parameters (r, s) that will be optimized to discover an optimized curriculum based on sample difficulty and model behavior. for training @xcite , while exclusive approaches create a new subset of the data at every training stage @xcite . in addition, previous research has developed model-driven @xcite @xcite and task-driven @xcite @xcite techniques. ## curriculum discovery framework we consider the training dataset @xmath1, y 1 ), . . . , (x n , y n )} of size n, where x i denotes the ith training sample with the groundtruth label y i and ψ ∈ [0, 1] n indicates the initial difficulty estimates of training samples, see §3.4. the data is initially clustered into k groups of increasing difficulty, e.g. {easy, medium, hard} groups for @xmath2, which can be achieved using difficulty score percentiles or 1-dimensional k-means applied to ψ. as figure 1 shows, the framework develops a separate parameterized weight function for each difficulty group ( §3.1), and dynamically weights training samples and adjust their difficulty groups according to the training progress of the downstream model ( §3.2). specifically, at training iteration t, the weighted loss li for sample i of the difficulty group @xmath3, . . . , k} will be computed as follows: ## conclusion and future work we introduce an effective curriculum learning framework that employs prior knowledge about sample difficulty in its training paradigm for curriculum discovery. the proposed framework initially partitions its input data into several groups of increasing difficulty, defines parameterized functions to weight sample losses in each difficulty group, moves samples across difficulty groups based on their learning progress, and enables tuning the parameters of the weight function to discover novel curricula. we demonstrate that this framework is capable of representing several categories of curriculum learning approaches. the task of curriculum discovery alleviates the limitations imposed by selecting a single curriculum strategy, and instead, focuses on finding and analyzing different curricula that work equally-well for a given model and dataset. in addition, the discovered curricula provide insight into how different portions of the dataset contribute toward learning at different stages of training a model, which, in turn, provide knowledge about the learning dynamics of different models. the task of curriculum discovery could be costly on large datasets, in particular, when the goal is to find optimal curricula for different models and datasets. to mitigate the computational the shaded area is the 95% ci over the datasets with five random seeds each. the curves are monotonic for most parts, and can be approximated by our framework. there are several promising areas for future work. these include approaches for learning new difficulty indicators from data (e.g., linguistic difficulty including lexical, syntactic and semantic difficulty), prioritizing medium level instances and those with greatest progress during training, and developing challenge datasets that contain diverse data samples with different levels of difficulty. finally, investigating diverse curricula that are suitable for general use and across datasets through curriculum discovery and generalization is a promising area for research.