C URRICULUM D ISCOVERY THROUGH AN E NCOMPASSING C URRICULUM L EARNING F RAMEWORK
Anonymous authors
Paper under double-blind review
A BSTRACT
We describe a curriculum learning framework capable of discovering optimal curricula in addition to performing standard curriculum learning.
We show that this framework encompasses existing curriculum learning approaches such as difﬁculty-based data sub-sampling, data pruning, and loss re-weighting.
We employ the proposed framework to address the following key questions in curriculum learning research: (a) What is the best curriculum to train a given model on a given dataset?
(b) What are the characteristics of optimal curricula for different datasets and different difﬁculty scoring functions?
We show that our framework outperforms competing state-of-the-art curriculum learning approaches in natural language inference and other text classiﬁcation tasks.
In addition, exhaustive experiments illustrate the generalizability of the discovered curricula across the three datasets and two difﬁculty scoring functions.
I NTRODUCTION
Curriculum Learning (CL) is a technique in Machine Learning that mimics human education systems.
To learn a complex subject, students must ﬁrst learn the foundational and basic materials before learning more complex ones.
Without a curriculum, learning may be intractable, inefﬁcient and learners may never reach a full understanding of a topic due to lack of the required background knowledge.
Machine learning optimization through stochastic gradient descent teaches models through experience by observing example data instances.
Some data instances are harder than others and require background knowledge.
That knowledge could be acquired by observing easier examples before harder ones.
CL techniques seek to order examples according to their difﬁculty for training to generate better models.
CL has been shown to improve performance in solving harder tasks, and in cases of limited data availability or noisy data (Wu et al., 2021)
CL research has made signiﬁcant progress in the last decade through the work of Bengio et al.
(2009), although the principle of ordering training samples from easier to harder was introduced in the 1990s (Elman, 1993; Sanger, 1994; Rohde & Plaut, 1999).
A curriculum can be deﬁned by providing a difﬁculty measurement for each sample, or by dividing the data into groups of increasing difﬁculty.
Given a measure of difﬁculty, there are different types of CL approaches for ordering the data.
For example, sub-sampling the easiest samples at every iteration and only using those to train (Zhou et al., 2020; Bengio et al., 2009; Xu et al., 2020;
Guo et al., 2018; Platanios et al., 2019), using the complete data and weighting samples differently according to their difﬁculty (Castells et al., 2020; Kumar et al., 2010; Jiang et al., 2015; 2018;
Zhou et al., 2020; Yang et al., 2019), and pruning the hard or noisy samples from the dataset prior to training (Northcutt et al., 2021; Guo et al., 2018).
Sub-sampling methods can be cumulative, or exclusive, or a combination of both; at every stage of increasing the level of difﬁculty, we can choose to keep using the examples of the previous stage, or to use only the new subset.
A curriculum can also introduce samples from easy to hard (Bengio et al., 2009; Kumar et al., 2010), or hard to easy, which can be an effective learning strategy in some speciﬁc tasks (Kocmi & Bojar, 2017; Zhang et al., 2018; 2019).
CL methods may impose a curriculum by adjusting model’s capacity according to the difﬁculty of their inputs (Karras et al., 2018; Sinha et al., 2020; Morerio et al., 2017) or schedule the order of tasks in the context of multi-task learning (Caubri`ere et al., 2019; Saraﬁanos et al., 2017; Florensa et al., 2017).
Other CL approaches such as (Graves et al., 2017; Zhou et al., 2020; Saxena et al., 2019) use O(n), where n is the number of training samples, extra parameters for learning curricula. In this work, we focus on data-oriented CL, which aims to rank and order
[Figure]
instances for effective training. Our approach can encode prior knowledge about sample difﬁculty to CL, and can be considered as a student-instructor collaborative CL. Current CL approaches calculate difﬁculty scores for training samples based on the loss of a trained model (Xu et al., 2020; Wu et al., 2021), trainable parameters to weight samples (Kumar et al., 2010; Jiang et al., 2015; Castells et al., 2020), loss value during training (Wu et al., 2021), moving average of loss during training (Zhou et al., 2020), transformations of the loss during training (Jiang et al., 2018; Castells et al., 2020), and consistency in the correct classiﬁcation of samples (Amiri et al., 2017; Xu et al., 2020).
The resulting difﬁculty scores will then be used to order samples for training.
Alternatively, examples can be ordered based on prior knowledge about their difﬁculty, which is often estimated through intrinsic sample features and with respect to the task, e.g. object shape or orientation in image classiﬁcation (Bengio et al., 2009).
We will demonstrate a CL framework that encompasses existing CL approaches through an effective and ﬂexible data partitioning and weighting scheme.
Our framework provides a new paradigm for selecting an ordering strategy.
Instead of a pre-determined strategy, the framework allows searching over the curriculum space to identify the best curriculum for a particular dataset and model.
It partitions training data into three groups of training samples, i.e. { easy , medium , hard } samples, according to a difﬁculty scoring function.
Three parameterized weighting functions will then be deﬁned, which specify the weight of each group at every point in training.
Each weight function is controlled by two parameters, which can be set empirically or adjusted using Bayesian optimization.
In addition, the framework discovers optimal curricula by optimizing the parameters of the weight functions (only 2 parameters per function) using a hyper-parameter optimization algorithm.
Furthermore, the curricula identiﬁed through this search provide useful insight about the dataset, such as the relative importance of different samples or knowledge dependency between samples, e.g. which samples should be learned ﬁrst.
We begin by explaining our framework and showing how it is capable of approximating existing CL approaches.
Then, in the context of the proposed framework, we investigate curriculum discovery, characteristics of discovered curricula, and generalizability of curricula with respect to their input datasets.
C URRICULUM L EARNING F RAMEWORK 2. W EIGHTING F UNCTIONS
We deﬁne the curriculum using generalized logistic functions (Richards, 1959) of the form:
[Equation]
where r ∈ R is the rate of change that can be a positive or negative value, specifying how fast the weight can increase or decrease, t ∈ [0 , 1] is the training progress (iteration number divided [Figure] by max iterations) and s ∈ R shifts the pivot weight ( w ( . )
= .
5 ) of the logistics function to the left or right such that at t =
s the weight is 0 .
Figure 1a illustrates the effect of varying these parameters.
Greater absolute values for the rate parameter enforce greater slope and faster rate of changes in weights, while greater values of the shift parameter enforce longer delays in reaching the pivot weight of 0 . 5 .
These parameters provide ﬂexibility in controlling sample weights during training, which is critical for deriving effective curricula.
The data is split into three partitions of increasing difﬁculty, and for each partition, we deﬁne a weight function by a pair of parameters ( r, s ).
The proposed sample weighting framework provides ﬂexibility in sample-ordering according to difﬁculty.
It can be learned to approximate existing predetermind curricula, for example, it is possible to begin with only easy instances or only difﬁcult instances (as an anti-curriculum ) or a combination of both.
Figure 1b shows a speciﬁc conﬁguration for the logistic functions based on standard
CL (Bengio et al., 2009; Kumar et al., 2010), where training starts with easier samples and gradually proceeds with harder ones.
In addition, our approach enables discovering new data-driven curricula from data.
Prior to training, the difﬁculty scores of samples are computed and each sample is assigned to a difﬁculty class c ∈ { easy, medium, hard } , see §2.2.
In addition, the hyper-parameters of our three weight functions are optimized prior to training and kept ﬁxed throughout training { ( r e , s e ) , ( r m , s m ) , ( r h , s h ) } .
During training, the weighted loss is computed as follows.
[Equation]
Where l i is the unweighted and instantaneous loss of instance i , ˆ l i is the weighted loss, t is the current training iteration divided by the maximum number of iterations, c is the difﬁculty class of instance i , and ( r c , s c ) are the corresponding weighting parameters for the difﬁculty class c .
2. S CORING F UNCTIONS
Ground-truth labels for many datasets are often obtained through human annotation and crowdsourcing.
This is achieved by collecting multiple annotations per data sample and aggregating the result, typically by majority voting.
For such datasets, we deﬁne a difﬁculty score for each sample [Figure] using Shannon entropy (Shannon, 2001), where higher disagreement among annotators corresponds to higher sample difﬁculty.
Entropy is a natural measure of difﬁculty (for the human population) and may serve as a reliable criterion for partitioning data by difﬁculty.
Nevertheless, training loss contains valuable information about the difﬁculty to the learner, which may be different among architectures and tasks, and indicative of the model’s speciﬁc needs.
We will use these two difﬁculty scores in our analysis.
Entropy of each sample x i is calculated as H ( x i ) =
− (cid:80) l p c log p c where c is a class category and p c is the fraction of annotators who chose label c for the sample.
The use of entropy is supported by experiments in Nie et al.
(2020), who also study the correlation between human agreement and model performance, and ﬁnd that there is a consistent positive correlation between model accuracy and level of human agreement, showing that model performs better on samples with a high level of agreement.
Loss at a particular step (e.g., ﬁnal loss) is dictated by the stochastic gradient descent and mini-batching dynamics and therefore is not a good indicator of difﬁculty (Zhou et al., 2020; Wu et al., 2021).
Using a baseline model, trained with no curriculum and with default hyper-parameters, we collect the loss values of all training instances at intervals of 0.5 epochs and use the average loss to estimate sample difﬁculty.
In our experiments, we obtain twenty observations of the loss and compute the average for each instance.
Such an estimation is supported by Zhou et al.
(2020) who showed that the moving average of a sample’s instantaneous loss is a good metric for difﬁculty.
Partitioning the data into three groups of increasing difﬁculty can be done using difﬁculty score percentiles, or 1-dimensional k-means clustering of the scores.
Examples of data partitions using entropy and loss are shown in Figure 2.
2. E NCOMPASSING F RAMEWORK
Curriculum learning approaches can be divided into three categories depending on how they process their input data: approaches that identify and prune noisy data that may hurt performance (Northcutt et al., 2021;
Guo et al., 2018; Rooyen et al., 2015; Patrini et al., 2016; Chen et al., 2019), approaches that use different sub-samples of data during different stages of training (Bengio et al., 2009;
Zhou et al., 2020; Xu et al., 2020; Platanios et al., 2019; Zhou & Bilmes, 2018), and approaches that re-weight loss according to sample difﬁculty, choosing to emphasize either easy or hard samples (Castells et al., 2020; Jiang et al., 2015; 2018;
Yang et al., 2019; Saxena et al., 2019).
The framework presented in this paper is capable of representing all of the three approaches.
First, data pruning can be done by assigning negative values to the rate change and shift parameters in our framework, r and s in Eq.
A negative r causes the weight to approach zero, and a negative s shifts the curve to the left, so the curve reaches zero before training begins.
The framework also allows ﬂexibility in pruning: by setting a small positive s , the noisy data can be seen by the model for a short amount of time before reaching zero weight, or by setting a positive r and a large positive s the noisy data will only be seen at the end of training after training has stabilized.
Second, data sub-sampling can be represented by the weight going to zero or increasing from zero at different stages of training.
For instance, Figure 1b illustrates a curriculum where the easy samples are sub-sampled in the beginning, and harder samples are introduced at later stages.
Third, we display in Figure 3 the conﬁdence scores assigned to our data by three loss re-weighting approaches.
The results are generated by our implementations of the three approaches, evaluated on the three datasets introduced in §3.1, where each model runs with ﬁve random seeds.
The partitioning of easy , medium , and hard is according to the entropy-based difﬁculty classes, as described in §2.2.
We record the average weight (conﬁdence) assigned to each class.
The result is averaged over all runs, and the shaded area is the 95% conﬁdence interval.
The approaches that estimate sample conﬁdence based on loss (Zhou et al., 2020; Castells et al., 2020; Kumar et al., 2010; Jiang et al., 2015; Felzenszwalb et al., 2009) tend to generate monotonic curves over the course of training because training loss tends to be non-increasing at every step.
Therefore, the conﬁdence scores assigned by these re-weighting approaches follow a monotonic curve that can be approximated by our weighting functions (§2.1, Figure 1).
We note that although the weight scale of SuperLoss (Castells et al., 2020) in Figure 3a is larger than one, this model can still be represented by our CL framework because the increased scale corresponds to a scaling of the learning rate, as shown below:
[Equation]
where l i and σ i are the loss and conﬁdence of sample i , respectively.
Therefore, our framework can also represent CL approaches with a conﬁdence scale larger then one.
2. C URRICULUM D ISCOVERY
We employ a hyperparameter optimization framework (Akiba et al., 2019) to ﬁnd the optimal value of the curriculum parameters ( r, s ) for each difﬁculty-class.
Using this method, we can learn a datadriven curriculum beyond what we could manually design through empirical settings or a choice among the limited ordering strategies.
To optimize the three pairs of parameters, we use the Treestructured Parzen Estimator (TPE) sampling algorithm (Bergstra et al., 2011).
Unlike grid or random search (Bergstra & Bengio, 2012), TPE traverses the parameter space by estimating the parameters that are most probable to perform better on a trial, based on the previous trials.
TPE deﬁnes two
Gaussian Mixture Models, l ( x ) and g ( x ) which are formed using the best and remaining observed parameters, respectively.
TPE selects the parameter x with a high probability under l ( x ) and low probability under g ( x ) ; max x l ( x ) /g ( x ) .
This choice of sampling algorithm greatly speeds up the search of our six curriculum parameters.
The search space that we deﬁne in our experiments consists of 11 possible values for r and 9 for s , for a total of 970k combinations.
Using TPE, we ﬁnd the optimal set of parameters within 100 trials, as compared to over 1000 trials by random search.
We note that the discovered curricula are optimal within this framework, constrained by the method of data partitioning and the class of weight functions.
We argue that the proposed framework is able to approximate curricula deﬁned by existing CL approaches, and outperform existing CL approaches across several datasets.
E XPERIMENTS 3. D ATASETS
We evaluate our approach on three datasets that contain multiple annotations for each sample.
First, the Stanford Natural Language Inference (SNLI) benchmark (Bowman et al., 2015), which contains 550k training samples, 10k development samples, and 10k test samples.
Within the training samples, there are 36.7k samples annotated by 5 workers and 2.6k annotated by 4 workers, which we use for our experiments and refer to as SNLI ”full.”
Furthermore, in order to control for variance due to imbalanced difﬁculty classes, we create a balanced subset of the data.
As shown in Figure 2a (notice the y-axis scale), SNLI is highly imbalanced in entropy classes.
The data is downsampled by selecting an equal number of samples from each entropy-class.
The balanced subset contains a total of 2.3k samples, i.e. 774 samples in each entropy class.
On the other hand, loss classes are fairly distributed, see Figure 2d.
The downsampled subset contains both entropy and loss classes which will be used in experiments.
The Alcohol dataset (Weitzman et al., 2020) has been developed to obtain population-level statistics of alcohol use reports through social media.
The dataset consists of more than 9k tweets.
Given an alcohol relevant tweet, annotators are asked to determine if it reports ﬁrst-person alcohol use, and if yes, the intensity of the drinking (light vs. heavy), the context of drinking (social vs. individual), and the time of drinking (past, present, or future).
All samples in the dataset are labeled by at least three workers, including over 1.3k samples labeled by ﬁve or more workers.
We deﬁne a multi-class classiﬁcation task for this dataset based on alcohol relevant, intensity and context of drinking.
The categories and their data distributions are reported in Appendix C. We randomly split the data into 5.4k training samples, 1.8k development samples, and 1.8k test samples.
The balanced version of the training set contains a total of 2.5k training samples, i.e. 863 samples in each entropy class.
The Cancer 1 dataset has been developed to obtain population-level statistics of cancer patients; it contains 3.8k Reddit posts.
Annotators are asked to determine if the post describes the experience of a cancer patient, the type of cancer, and the relation of the author of the post to the patient.
We deﬁne a multi-class classiﬁcation task based on post relevance and caner type.
The categories and their data distribution are reported in Appendix C. All samples are labeled by at least three workers, including about 1k labeled by at least ﬁve.
We randomly split the data into around 2.2k training samples, 765 development samples, and 765 test samples.
The balanced version of the training set contains a total number of 1.7k sample, i.e. 578 samples in each entropy class.
We note that the datasets are signiﬁcantly different in terms of document length, the average document length in SNLI, Alcohol and Cancer datasets are 10, 15 and 174 words.
This variation can induce signiﬁcant variance in the created models.
3. B ASELINES
We compare the performance of our CL approach with the following state-of-the-art approaches.
SuperLoss (SL) (Castells et al., 2020) is a CL approach that deﬁnes a task-agnostic conﬁdenceaware loss function.
It infers the conﬁdences of instances from the instance loss with minimal cost, providing a closed-form solution function for estimating conﬁdence.
It up-weights samples with smaller loss values (easier instances), while down-weights those with larger loss values (harder ones) (Figure 3a).
MentorNet (Jiang et al., 2018) uses an auxiliary network to generate a weight for training samples at every training iteration.
It incorporates additional signals such as epoch number and instance loss history to learn data-driven curricula and is particularly strong against noisy data.
Difﬁculty Prediction (DP) (Yang et al., 2019) deﬁnes a difﬁculty score based on multi-annotator data, and weight samples according to the following formula w = 1 − α ( d i − τ DP ) / (1 − τ DP ) , where d i is the sample difﬁculty and τ is a pre-deﬁned threshold.
As discussed before, our approach employs two scoring functions (§ 2.2) and two curriculum conﬁgurations for each dataset.
A curriculum conﬁguration refers to a particular setting of the six parameters controlling the three weight curves (§ 2.1).
The two scoring functions are labeled as
Loss and Ent (entropy).
In addition, the ﬁrst curriculum conﬁguration is a gradually increasing approach in Figure 1b named inc. , this conﬁguration is applied identically to all models.
The second conﬁguration is the specialized conﬁguration ( sp. )
that is obtained through hyper-parameters search for each triple of dataset, balance, and scoring function (illustrated in Figure 5).
3. S ETTINGS
We tune the parameters λ of SL and α and τ DP of DP using development data.
The optimal values found are λ = 1 . 2 , α = 0 .
9 and τ DP is set dynamically upon loading the dataset to the 50-percentile difﬁculty value.
Following Castells et al.
(2020), we set τ SL (instances with l i > τ SL are considered hard) to the moving average of the loss in all experiments.
We use the transformers python package (Wolf et al., 2020), using the bert-base-uncased model for SNLI and Cancer, and twitter-roberta-base for Alcohol.
We set learning rate = 1 e − 5 , batch size = 16 , epochs = 10 (we conﬁrm that this number of iterations is sufﬁcient for all models to converge), and the optimizer to Adam (Kingma & Ba, 2017).
For each experiment, we train ﬁve models using ﬁve random seeds applied to both pytorch and numpy .
Additionally, during all data pre-processing, splitting, and sub-sampling, the random seed is set to 0 , and a single
NVIDIA A100 40GB GPU is used for training.
The development set is used to determine the best training step which is used for the ﬁnal evaluation.
In addition, we conduct a hyper-parameter search over the ( r, s ) for each weight function, for the three difﬁculty classes.
We set the search space of r to be from − 10 to 10 with a step of 2 , and of s to be − 0 .
5 to 1 . 5 with a step of 0 . 25 .
We observe that changes smaller than this step size have little effect on performance.
The search is run for at least 100 trials using the method described in §2.4.
Each trial is run with 5 random seeds and the result is averaged.
The search objective is to maximize development-set accuracy.
3. P ERFORMANCE G AIN FROM C URRICULA D ISCOVERY
Results are shown in Figure 4.
Accuracy is averaged over the six datasets (full and balanced version of each dataset).
The gradually increasing curriculum ( inc ) achieves a signiﬁcant improvement over
No-CL using either of the scoring functions while being a static, off-the-shelf curriculum conﬁguration.
This shows that the effectiveness of our approach of partitioning the data using generalized logistic functions (§2.1) as weighting functions.
Moreover, the specialized ( sp ) curricula obtained through curriculum discovery perform signiﬁcantly better than state-of-the-art CL approaches.
In our three datasets, loss as a scoring function performs better on average than entropy.
This is expected as loss values can capture sample difﬁculty with respect to the downstream learner (model), as apposed to entropy values which do not take into account the model.
Appendix B includes further breakdown of the results by dataset and accuracy across samples of different difﬁculty.
3. C HARACTERISTICS OF D ISCOVERED C URRICULA
Figure 5 shows the mean and 95% CI of the top 25 performing conﬁgurations on our datasets and scoring functions.
We observe several insightful patterns: The resulting curricula are non-trivial and greatly differ from the known strategies reported in current literature, such as gradually increasing difﬁculty or anti-curriculum.
In addition, the weights of hard samples tend to approach zero, supporting the hypothesis that either these instances are too difﬁcult for the models to learn or they are noisy.
The results support the principle of pruning techniques because noisy samples induce more [Figure] noise than useful signal; in several plots, hard samples increase in weight during the last portion of training after the model has stabilized.
In addition, in SNLI and Alcohol easy samples carry the most signiﬁcant weight, unlike Cancer, where easy samples are down-weighted early during the training.
These weighting patterns reveal the relative importance of samples in each dataset.
Finally, the full SNLI dataset with entropy-class partitions provides useful information.
Figure 2a shows that entropy classes are highly imbalanced, with hard samples being much fewer than easy ones.
In the optimal curriculum shown in Figure 5c, hard samples are assigned weights around 0.5, unlike the three other cases of SNLI.
We attribute this result to the reduced presence/effect of hard samples.
3. G ENERALIZABLE C URRICULA
Figure 6 shows the accuracy obtained when training using different curriculum conﬁgurations from
Figure 5.
Each cell in the ﬁgure is the average result of 5 seeds.
We observe common characteristics among datasets that cause the curriculum to be transferable between them.
First, the three top conﬁgurations are all products of down-sampled, balanced datasets.
Second, the curricula obtained using the small balanced datasets generalize and achieve high performance on the large datasets.
This is useful as it allows performing the hyper-parameter search much faster and cheaper on smaller datasets, providing evidence that the framework can be applied to large datasets by searching for a curriculum on a small subset of the data.
Third, the inc curriculum is available off-the-shelf and performs consistently well across datasets and scoring functions.
Fourth, as noted previously, instances of the Cancer dataset consist of long paragraphs, causing high variance in models trained using the dataset.
Consequently, the curriculum obtained using the Cancer loss-class partitions is of low quality and performs the worst.
An extended version of Figure 6 is included in Appendix A with results of models trained with balanced versions of datasets.
C ONCLUSION AND F UTURE W ORK
We introduce a curriculum learning (CL) framework that partitions the data into three groups of increasing difﬁculty, deﬁnes three parameterized weight functions to weight the losses of three groups, and provides the capability of tuning the weight function parameters to discover new curricula performing better than existing baselines.
We demonstrated that this framework is capable of representing the major CL approaches.
The proposed research opens a new paradigm in CL by removing the limitations imposed by selecting a single CL strategy, and instead, using a ﬂexible framework to discover optimal curricula for given datasets and models, and extract valuable insights about the data.
Our work has the limitation of using monotonic logistic functions as the weight curves can not take arbitrary forms.
This challenge can be addressed in future work by adding an extra set of logistic functions and having every weight function be a linear combination of two or more logistic functions, which makes it possible to represent non-monotonic functions.
Nevertheless, we have demonstrated that major CL approaches that estimate difﬁculty as a function of training loss result in monotonic curves, since gradient descent cause loss to be non-increasing.
Promising areas for future work include learning new difﬁculty indicators (centered on data, e.g. text difﬁculty), prioritizing medium level instances and those with greatest progress in training, and, developing challenge datasets that contain diverse data samples with different level of difﬁculty.
E THICS S TATEMENT
This investigation included publicly available yet sensitive data from social media, developed for an important task: obtaining population-level statistics about different public health issues.
Our work does not access, use, or release any personal data, and only textual data is used for experiments.
R EPRODUCIBILITY S TATEMENT
The source code is included as supplementary material.
It provides the required details for data processing, hyper-parameter setting, random seed setting, model evaluation, etc.
A E XTENDED C ONFIGURATION G ENERALIZABLITY E XPERIMENTS
Figure 7 shows the full results including evaluation on the balanced datasets.
B A CCURACY B REAKDOWN
The accuracy achieved on each dataset using each approach is shown in Figure 8.
Models that achieve a high accuracy may be generating correct predictions for a high percentage of easy samples while failing to correctly predict the output of the medium and hard samples.
Standard evaluation benchmarks often contains artifacts that make it east to correctly predict the label of some samples (Gururangan et al., 2018;
Poliak et al., 2018).
Therefore, it is important to closely analyze the model’s performance on harder instances.
We break down the accuracy of each difﬁculty class (with entropy-class partitioning) for balanced, full, and all datasets in Figures 9, 10, and 11, respectively.
Our approach is exceptionally powerful in predicting samples of medium difﬁculty.
Furthermore, the accuracy achieved on hard samples is almost equal by all approaches, including
No-CL, supporting the hypothesis that those samples tend to be noisy or inaccurately labeled.
[Figure]
[Figure] [Figure]
[Figure]
C D ATA C ATEGORIES D ISTRIBUTION
[Table]
D F ULL C URRICULUM S EARCH R ESULTS
This section shows the top 25 performing conﬁgurations on each dataset scoring function pair.
Blue lines (circle marker) is easy , orange (x marker) is medium , and green (diamond marker) is hard .
Above each plot is the development set accuracy.
The plot legend contains the parameters ( r, s ) for each class.
[Figure]