C URRICULUM D ISCOVERY THROUGH AN E NCOMPASSING C URRICULUM L EARNING F RAMEWORK
Anonymous authors
Paper under double-blind review
A BSTRACT
We describe a curriculum learning framework capable of discovering optimal curricula in addition to performing standard curriculum learning.
We show that this framework encompasses existing curriculum learning approaches such as difﬁculty-based data sub-sampling, data pruning, and loss re-weighting.
We employ the proposed framework to address the following key questions in curriculum learning: (a) what is the best curriculum to train a given model on a given dataset?
and (b) what are the characteristics of optimal curricula for different datasets and difﬁculty metrics?
We show that our framework outperforms competing stateof-the-art curriculum learning approaches in natural language inference and two other text classiﬁcation tasks.
Exhaustive experiments illustrate the generalizability of the discovered curricula across the datasets and difﬁculty metrics.
I NTRODUCTION
Curriculum Learning (CL) is a technique in Machine Learning that mimics human education systems.
To learn a complex subject, students must ﬁrst learn the foundational and basic materials before learning more complex ones.
Without a curriculum, learning may be intractable, inefﬁcient and learners may never reach a full understanding of a topic due to lack of the required background knowledge.
Machine learning optimization through stochastic gradient descent trains models by observing example data instances.
Some data instances are harder than others and require background knowledge, which could be acquired by observing and being adept at easier examples before harder ones.
CL techniques seek to order examples according to their difﬁculty for training to generate better models.
It has been shown that CL improves performance in solving harder tasks, or in cases of limited or noisy data (Wu et al., 2021).
CL research has made signiﬁcant progress in the last decade through the work of Bengio et al.
(2009), although the principle of ordering training samples from easier to harder was introduced in the 1990s (Elman, 1993; Sanger, 1994; Rohde & Plaut, 1999).
A curriculum can be deﬁned by ordering training samples based on their difﬁculty to learn.
Given a measure of difﬁculty, there are different types of CL approaches for ordering the data: sub-sampling techniques, which sample the easiest or hardest data points at every iteration for training (Zhou et al., 2020; Bengio et al., 2009; Xu et al., 2020; Guo et al., 2018;
Platanios et al., 2019), sample weighting techniques, which use the complete data at every iteration but weight data points differently according to their difﬁculty (Castells et al., 2020; Kumar et al., 2010; Jiang et al., 2015; 2018;
Zhou et al., 2020; Yang et al., 2019), and pruning techniques, which prune the hard or noisy samples from the dataset prior to training (Northcutt et al., 2021; Guo et al., 2018).
Sub-sampling methods can be cumulative, exclusive or a combination of both. Cumulative approaches add new data points to the ones that have previously been used for training. On the other hand, exclusive approaches create a new subset of data at every training stage.
Such methods can introduce samples from easy to hard (Bengio et al., 2009; Kumar et al., 2010) or hard to easy, which can be an effective learning strategy in some speciﬁc tasks (Kocmi & Bojar, 2017; Zhang et al., 2018; 2019).
In addition, CL methods may impose a curriculum by adjusting model’s capacity according to the difﬁculty of their inputs (Karras et al., 2018; Sinha et al., 2020;
Morerio et al., 2017) or schedule the order of tasks in the context of multi-task learning (Caubri`ere et al., 2019; Saraﬁanos et al., 2017; Florensa et al., 2017).
Other CL approaches such as (Zhou et al., 2020; Saxena et al., 2019) use O ( n ) , where n is the number of training samples, extra parameters for learning curricula.
[Figure]
Current CL approaches calculate difﬁculty scores for training samples based on the loss of a trained model (Xu et al., 2020; Wu et al., 2021), trainable parameters that weight samples (Kumar et al., 2010; Jiang et al., 2015; Castells et al., 2020), loss value during training (Wu et al., 2021), moving average of loss during training (Zhou et al., 2020), transformations of the loss during training (Jiang et al., 2018; Castells et al., 2020), and consistency in the correct classiﬁcation of samples (Amiri et al., 2017; Xu et al., 2020).
The difﬁculty scores will then be used to order samples for
In this work, we focus on an alternative approach, where examples can be ordered based on prior knowledge about their difﬁculty, e.g., object shape or orientation in image classiﬁcation (Bengio et al., 2009).
We will demonstrate a CL framework that encompasses existing CL approaches through an effective and ﬂexible data partitioning and weighting scheme.
Our framework provides a new paradigm for selecting an ordering strategy.
Instead of a pre-determined strategy, the framework allows searching over the curriculum space to identify the best curriculum for a particular dataset and model.
It partitions training data into several groups of training samples, e.g. { easy , medium , hard } samples, according to a difﬁculty scoring function.
Parameterized weighting functions will then be deﬁned for each data group to specify the weight of its samples during training.
Each weight function is controlled by two parameters, which can be set empirically or adjusted using Bayesian optimization.
In addition, the framework discovers optimal curricula by optimizing the parameters of the weight functions (only 2 parameters per function) using a hyper-parameter optimization algorithm.
Furthermore, the curricula identiﬁed through this search provide useful insight about the dataset, such as the relative importance of different samples or knowledge dependency between samples, e.g. which samples should be learned ﬁrst.
We begin by explaining our framework and showing how it is capable of approximating existing CL approaches.
Then, in the context of the proposed framework, we investigate curriculum discovery, characteristics of discovered curricula, and generalizability of curricula with respect to their datasets.
C URRICULUM L EARNING F RAMEWORK 2. W EIGHTING F UNCTIONS
We deﬁne the curriculum using generalized logistic functions (Richards, 1959) of the form:
[Equation]
where r ∈ R is the rate-of-change parameter, which speciﬁes how fast the weight can increase ( r > 0 ) or decrease ( r < 0 ); t ∈ [0 , 1] is the training progress (iteration number divided by max iterations); and s ∈ R shifts the pivot weight of the logistics function ( w ( . )
= .
5 ) to the left or right such that at t =
s the weight is 0 .
Figure 1a illustrates the effect of varying these parameters.
Greater absolute values for the rate parameter enforce greater slope and faster rate of changes in weights, while greater values of the shift parameter enforce longer delays in reaching the pivot [Figure] weight of 0 . 5 .
These parameters provide ﬂexibility in controlling sample weights during training, which is critical for deriving effective curricula.
Given the weighting function, we encode prior knowledge about sample difﬁculty into the training paradigm of our CL framework by splitting the data into several partitions of increasing difﬁculty according to the prior knowledge.
Then, we deﬁne a weight function by a pair of parameters ( r, s ) for each partition,
The proposed sample weighting framework provides ﬂexibility in sample-ordering according to difﬁculty.
It can be learned to approximate existing predetermind curricula. For example, it is possible to begin with only easy instances or only difﬁcult instances (as an anti-curriculum ) or a combination of both.
Figure 1b shows a speciﬁc conﬁguration for the logistic functions based on standard CL (Bengio et al., 2009; Kumar et al., 2010), where training starts with easier samples and gradually proceeds with harder ones.
In addition, our approach enables discovering new data-driven curricula from data.
Prior to training, the difﬁculty scores of samples are computed and each sample is assigned to a difﬁculty class c ∈ { easy, medium, hard } , see §2.2.
In addition, the hyper-parameters of our three weight functions are optimized prior to training and kept ﬁxed throughout training { ( r e , s e ) , ( r m , s m ) , ( r h , s h ) } .
During training, the weighted loss is computed as follows.
[Equation]
Where l i is the unweighted and instantaneous loss of instance i , ˆ l i is the weighted loss, t is the current training iteration divided by the maximum number of iterations, c is the difﬁculty class of instance i , and ( r c , s c ) are the corresponding rate and shift parameters for the difﬁculty class c .
2. S CORING F UNCTIONS
Ground-truth labels for many datasets are often obtained through human annotation and crowdsourcing.
This is achieved by collecting multiple annotations per data sample and aggregating the results, typically by majority voting.
We use sample-level annotator disagreement to deﬁne a difﬁculty score for each sample using Shannon entropy (Shannon, 2001), where higher disagreement among annotators corresponds to higher sample difﬁculty.
Entropy is a natural measure of difﬁculty (for human population) and may serve as a reliable prior knowledge for partitioning data by difﬁculty.
Entropy of each sample x i is calculated as H ( x i ) =
− (cid:80) l p c log p c where c is a class category and p c is the fraction of annotators who chose label c for the sample.
The use of entropy is supported by Nie et al.
(2020), who studied the correlation between human agreement and model performance and reported a consistent positive correlation between model accuracy and level of human agreement, showing that model performs better on samples with a high level of agreement.
Furthermore, training loss contains valuable information about difﬁculty with respect to the model (learner), which may be different among architectures and tasks, and indicative of the model’s speciﬁc needs.
However, loss at a particular step (e.g., ﬁnal loss) is dictated by the stochastic gradient descent and mini-batching dynamics and therefore is not a good indicator of difﬁculty (Zhou et al., 2020; Wu et al., 2021).
Using a baseline model, trained with no curriculum and with default hyperparameters, we collect the loss values of all training instances at intervals of 0.5 epochs and use the average loss to estimate sample difﬁculty.
In our experiments, we obtain twenty observations of the loss and compute the average for each instance.
Such an estimation is supported by Zhou et al.
(2020) who showed that the moving average of a sample’s instantaneous loss is a good metric for difﬁculty.
Partitioning the data into three groups of increasing difﬁculty can be done using difﬁculty score percentiles, or 1-dimensional k-means clustering of the scores.
Examples of data partitions using entropy and loss are shown in Figure 2.
2. E NCOMPASSING F RAMEWORK
Curriculum learning approaches can be divided into three categories depending on how they process their input data: approaches that identify and prune noisy data that may hurt performance (Northcutt et al., 2021;
Guo et al., 2018; Rooyen et al., 2015; Patrini et al., 2016; Chen et al., 2019), approaches that use different sub-samples of data during training (Bengio et al., 2009;
Zhou et al., 2020; Xu et al., 2020; Platanios et al., 2019; Zhou & Bilmes, 2018), and approaches that re-weight loss according to sample difﬁculty, choosing to emphasize either easy or hard samples (Castells et al., 2020;
Jiang et al., 2015; 2018; Yang et al., 2019; Saxena et al., 2019).
The framework presented in this paper is capable of representing all of the three approaches.
First, data pruning can be done by assigning negative values to the rate change and shift parameters in our framework, r and s in Eq.
A negative r causes the weight to approach zero, and a negative s shifts the curve to the left, so the curve reaches zero before training begins.
The framework also allows ﬂexibility in pruning: by setting a small positive s , the noisy data can be seen by the model for a short amount of time before reaching zero weight, or by setting a positive r and a large positive s the noisy data will only be seen at the end of training (after it stabilizes).
Second, data sub-sampling can be represented by the weight going to zero or increasing from zero at different stages of training.
For instance, Figure 1b illustrates a curriculum where the easy samples are sub-sampled in the beginning, and harder samples are introduced at later stages.
Third, we display in Figure 3 the conﬁdence scores assigned to our data by three loss re-weighting approaches.
The results are generated by our implementations of the three approaches, evaluated on the three datasets introduced in §3.1, where each model runs with ﬁve random seeds.
The partitioning of easy , medium , and hard is according to the entropy-based difﬁculty classes, as described in §2.2.
We record the average weight (conﬁdence) assigned to each class.
The result is averaged over all runs, and the shaded area is the 95% conﬁdence interval.
The approaches that estimate sample conﬁdence based on loss (Castells et al., 2020; Zhou et al., 2020; Kumar et al., 2010; Jiang et al., 2015; Felzenszwalb et al., 2009) tend to generate monotonic curves over the course of training because training loss tends to be non-increasing at every step.
Therefore, the conﬁdence scores assigned by these re-weighting approaches follow a monotonic curve that can be approximated by our weighting functions (§2.1, Figure 1).
We note that although the weight scale of SuperLoss (Castells et al., 2020) in Figure 3a is larger than one, this model can still be represented by our CL framework because the increased scale corresponds to a scaling of the learning rate, as shown below:
[Equation]
where l i and σ i are the loss and conﬁdence of sample i , respectively.
Therefore, our framework can also represent CL approaches with a conﬁdence scale larger then one. [Figure]
2. C URRICULUM D ISCOVERY
We employ a hyperparameter optimization framework (Akiba et al., 2019) to ﬁnd the optimal value of the curriculum parameters ( r, s ) for each difﬁculty class.
Using this method, we can learn a datadriven curriculum beyond what we could manually design through empirical settings or a choice among the limited ordering strategies.
To optimize the three pairs of parameters, we use the Treestructured Parzen Estimator (TPE) sampling algorithm (Bergstra et al., 2011).
Unlike grid or random search (Bergstra & Bengio, 2012), TPE traverses the parameter space by estimating the parameters that are most probable to perform better on a trial, based on the previous trials.
TPE deﬁnes two
Gaussian Mixture Models, l ( x ) and g ( x ) which are formed using the best and remaining observed parameters, respectively.
TPE selects the parameter x with a high probability under l ( x ) and low probability under g ( x ) , i.e. arg max x l ( x ) /g ( x ) .
This choice of sampling algorithm greatly speeds up the search of our curriculum parameters.
We note that the discovered curricula are optimal within this framework, constrained by the method of data partitioning and the class of weight functions.
We argue that the proposed framework is able to approximate curricula deﬁned by existing CL approaches, and outperform existing CL approaches across several datasets.
E XPERIMENTS 3. D ATASETS
We evaluate our approach on three datasets that contain multiple annotations for each sample.
First, the Stanford Natural Language Inference (SNLI) benchmark (Bowman et al., 2015), which contains 550k training samples, 10k development samples, and 10k test samples.
Within the training samples, there are 36.7k samples annotated by 5 workers and 2.6k annotated by 4 workers, which we use for our experiments and refer to as SNLI “full.”
Furthermore, in order to control for variance due to imbalanced difﬁculty classes, we create a balanced subset of the data.
As shown in Figure 2a (notice the y-axis scale), SNLI is highly imbalanced in entropy classes.
The data is downsampled by selecting an equal number of samples from each entropy-class.
The balanced subset contains a total of 2.3k samples, i.e. 774 samples in each entropy class.
On the other hand, loss classes are fairly distributed, see Figure 2d.
The downsampled subset contains both entropy and loss classes which will be used in experiments.
The Alcohol dataset (Amiri et al., 2018; Weitzman et al., 2020) has been developed to obtain population-level statistics of alcohol use reports through social media.
The dataset consists of more than 9k tweets.
Given an alcohol relevant tweet, annotators are asked to determine if it reports ﬁrst- person alcohol use, and if yes, the intensity of the drinking (light vs. heavy), the context of drinking (social vs. individual), and the time of drinking (past, present, or future).
All samples in the dataset are labeled by at least three workers, including over 1.3k samples labeled by ﬁve or more workers.
We deﬁne a multi-class classiﬁcation task for this dataset based on alcohol relevance, intensity and context of drinking.
The categories and their data distributions are reported in Appendix A. We randomly split the data into 5.4k training samples, 1.8k development samples, and 1.8k test samples.
The balanced version of the training set contains a total of 2.5k training samples, i.e. 863 samples in each entropy class.
The Cancer dataset has been developed to obtain population-level statistics of cancer patients; it contains 3.8k Reddit posts.
Annotators are asked to determine if the post describes the experience of a cancer patient, the type of cancer, and the relation of the author of the post to the patient.
We deﬁne a multi-class classiﬁcation task based on post relevance and caner type.
The categories and their data distribution are reported in Appendix A. All samples are labeled by at least three workers, including about 1k labeled by at least ﬁve.
We randomly split the data into around 2.2k training samples, 765 development samples, and 765 test samples.
The balanced version of the training set contains a total number of 1.7k sample, i.e. 578 samples in each entropy class.
We note that the datasets are signiﬁcantly different in average document length, ranging from(SNLI), to 15 (Alcohol) to 174 (Cancer) words.
This variation can induce signiﬁcant variance in the created models.
3. B ASELINES
We compare the performance of our CL approach with the following state-of-the-art approaches.
SuperLoss (SL) (Castells et al., 2020) is a CL approach that deﬁnes a task-agnostic conﬁdenceaware loss function.
It infers the conﬁdences of instances from the instance loss with minimal cost, providing a closed-form solution function for estimating conﬁdence.
It up-weights samples with smaller loss values (easier instances), while down-weights those with larger loss values (harder ones) (Figure 3a).
MentorNet (Jiang et al., 2018) uses an auxiliary network to generate a weight for training samples at every training iteration.
It incorporates additional signals such as epoch number and instance loss history to learn data-driven curricula and is particularly strong against noisy data.
Difﬁculty Prediction (DP) (Yang et al., 2019) deﬁnes a difﬁculty score based on multi-annotator data, and weight samples according to the following formula w = 1 − α ( d i − τ DP ) / (1 − τ DP ) , where d i is the sample difﬁculty and τ is a pre-deﬁned threshold.
As discussed before, our approach employs two scoring functions (§ 2.2) and two curriculum conﬁgurations for each dataset.
A curriculum conﬁguration refers to a particular setting of the six parameters controlling the three weight curves (§ 2.1).
The two scoring functions are labeled as
Loss and Ent (entropy).
In addition, the ﬁrst curriculum conﬁguration is a gradually increasing approach in Figure 1b named inc. , this conﬁguration is applied identically to all models.
The second conﬁguration is the specialized conﬁguration ( sp. )
that is obtained through hyper-parameter search as discussed in § 2.4.
3. S ETTINGS
We tune the parameters λ of SL and α and τ DP of DP using development data.
The optimal values found are λ = 1 . 2 , α = 0 .
9 and τ DP is set dynamically upon loading the dataset to the 50-percentile difﬁculty value.
Following Castells et al.
(2020), we set τ SL (instances with l i > τ SL are considered hard) to the moving average of the loss in all experiments.
We use the transformers python package (Wolf et al., 2020), using the bert-base-uncased model for SNLI and Cancer, and twitter-roberta-base for Alcohol.
We set learning rate to 1 e − 5 , batch size to 16 , epochs to 10 (we conﬁrm that this number of iterations is sufﬁcient for all models to converge), and the optimizer to Adam (Kingma & Ba, 2017).
For each experiment, we train ﬁve models using ﬁve random seeds applied to both pytorch and numpy .
Additionally, during all data pre-processing, splitting, and sub-sampling, the random seed is set to 0 , and a single
NVIDIA A100 40GB GPU is used for training.
The development set is used to determine the best training step which is used for the ﬁnal evaluation.
In addition, we conduct a hyper-parameter search over the ( r, s ) for each weight function or difﬁculty classes (easy, medium, and hard).
We set the search space of r to be from − 10 to 10 with a step of 2 , and of s to be − 0 .
5 to 1 . 5 with a step of 0 . 25 .
We observe that changes smaller than this step size have little effect on performance.
This search space of 11 possible values for r and 9 for s for 970k combinations at least to over and result
3. P ERFORMANCE G AIN FROM C URRICULA D ISCOVERY
Results are shown in Figure 4.
Accuracy is averaged over the six datasets (full and balanced version of each dataset).
The gradually increasing curriculum ( inc ) achieves a signiﬁcant improvement over
No-CL using either of the scoring functions while being a static, off-the-shelf curriculum conﬁguration.
This improvement shows the effectiveness of our approach of partitioning the data using generalized logistic functions (§2.1).
Moreover, both ( inc ) and the specialized ( sp ) curricula obtained through curriculum discovery perform signiﬁcantly better than the state-of-the-art CL approaches.
In our three datasets, loss as a scoring function performs better than entropy on average.
This is expected as loss values can capture sample difﬁculty with respect to the downstream learner (model), as apposed to entropy values which do not take into account the model.
Appendix B includes further breakdown of the results by dataset and accuracy across samples of different difﬁculty.
3. C HARACTERISTICS OF D ISCOVERED C URRICULA
Figure 5 shows the mean and 95% CI of the top 25 performing conﬁgurations on our datasets and scoring functions.
We observe several insightful patterns: the resulting curricula are non-trivial and greatly differ from the known strategies reported in current literature, such as gradually increasing difﬁculty or anti-curriculum.
In addition, the weights of hard samples tend to approach zero, supporting the hypothesis that either these instances are too difﬁcult for the models to learn or they are noisy.
The results support the principle of pruning techniques because noisy samples induce more noise than useful signal; in several plots, the weight of hard samples increases only at the end of training after the model stabilizes.
In addition, in SNLI and Alcohol easy samples carry the most signiﬁcant weight, unlike Cancer, where easy samples are down-weighted early during the training.
These weighting patterns reveal the relative importance of samples in each dataset.
Finally, the full SNLI dataset with entropy-class partitions provides useful information.
Figure 2a shows that entropy classes are highly imbalanced, with hard samples being much fewer than easy ones.
In the [Figure] optimal curriculum shown in Figure 5c, hard samples are assigned weights around 0.5, unlike the three other cases of SNLI.
We attribute this result to the reduced presence/effect of hard samples.
3. G ENERALIZABLE C URRICULA
Figure 6 shows the accuracy obtained when training using different curriculum conﬁgurations from
Figure 5.
Each cell in the ﬁgure is the average result of 5 seeds.
We observe common characteristics among datasets that cause the curriculum to be transferable between them.
First, the top three conﬁgurations are all products of down-sampled, balanced datasets.
Second, the curricula obtained using the small balanced datasets generalize and achieve high performance on the large datasets.
This is useful as it allows performing the hyper-parameter search much faster and cheaper on smaller datasets, providing evidence that the framework can be applied to large datasets by searching for a curriculum on a small subset of the data.
Third, the inc curriculum is available off-the-shelf and performs consistently well across datasets and scoring functions.
Fourth, as noted previously, instances of the Cancer dataset consist of long paragraphs, causing high variance in models trained using the dataset.
Consequently, the curricula obtained using the Cancer and loss as measure of difﬁculty are of lower quality and perform the worst.
An extended version of Figure 6 is included in Appendix C with results of models trained with balanced versions of datasets.
C ONCLUSION AND F UTURE W ORK
We introduce an effective curriculum learning (CL) framework that employs prior knowledge about sample-level difﬁculty in its training paradigm, and effectively creates and explores a curricula space for curricula discovery and model generalizability.
The proposed framework partitions its input data into three groups of increasing difﬁculty, deﬁnes three parameterized logistic weight functions to weight the loss of samples in each of the three groups, and provides the capability of tuning the parameters of the weight function to discover new curricula performing better than existing baselines.
We demonstrate that this framework is capable of representing major CL approaches.
In addition, an important advantage of our approach is that the curricula that it discovers for smaller and balanced datasets work well on larger datasets, across the three datasets that we experimented with.
The proposed research opens a new paradigm in CL by removing the limitations imposed by selecting a single CL strategy, and instead, using a ﬂexible framework to discover optimal curricula for given datasets and models, and extract valuable insights about the data.
Our work has the limitation of using monotonic logistic functions because the weight curves can not take arbitrary forms.
This challenge can be addressed in future work by adding an extra set of logistic functions and having every weight function be a linear combination of two or more logistic functions, which makes it possible to represent non-monotonic functions.
Nevertheless, we have demonstrated that major CL approaches that estimate difﬁculty as a function of training loss result in monotonic curves because of gradient descent that causes training loss to be non-increasing.
There are several promising areas for future work. These include approaches for learning new difﬁculty indicators that are centered on data (e.g., text difﬁculty), prioritizing medium level instances and those with greatest progress in training, and developing challenge datasets that contain diverse data samples with different level of difﬁculty.
E THICS S TATEMENT
This investigation included publicly available yet sensitive data from social media, developed for an important task: obtaining population-level statistics about different public health issues.
Our work does not access, use, or release any personal data, and only textual data is used for experiments.
R EPRODUCIBILITY S TATEMENT
The source code is included as supplementary material.
It provides the required details for data processing, hyper-parameter setting, random seed setting, model evaluation, etc.
A D ATA C ATEGORIES D ISTRIBUTION
[Table]
B A CCURACY B REAKDOWN
The accuracy achieved on each dataset using each approach is shown in Figure 7.
Models that achieve a high accuracy may be generating correct predictions for a high percentage of easy samples while failing to correctly predict the output of the medium and hard samples.
Standard evaluation benchmarks often contains artifacts that make it east to correctly predict the label of some samples (Gururangan et al., 2018;
Poliak et al., 2018).
Therefore, it is important to closely analyze the model’s performance on harder instances.
We break down the accuracy of each difﬁculty class (with entropy-class partitioning) for balanced, full, and all datasets in Figures 8, 9, and 10, respectively.
Our approach is exceptionally powerful in predicting samples of medium difﬁculty.
Furthermore, the accuracy achieved on hard samples is almost equal by all approaches, including
No-CL, supporting the hypothesis that those samples tend to be noisy or inaccurately labeled.
[Figure] [Figure]
[Figure] [Figure]
C E XTENDED C ONFIGURATION G ENERALIZABLITY E XPERIMENTS
Figure 11 shows the full results including evaluation on the balanced datasets.
D F ULL C URRICULUM S EARCH R ESULTS
This section shows the top 25 performing conﬁgurations on each dataset scoring function pair.
Blue lines (circle marker) is easy , orange (x marker) is medium , and green (diamond marker) is hard .
Above each plot is the development set accuracy.
The plot legend contains the parameters ( r, s ) for each class.
[Figure]