Towards a Task-Agnostic Model of Difficulty Estimation for Supervised Learning Tasks (2020)

Abstract: Curriculum learning, a training strategy where training data are ordered based on their difficulty, has been shown to improve performance and reduce training time on various NLP tasks. While much work over the years has developed novel approaches for generating curricula, these strategies are typically only suited for the task they were designed for. This work explores developing a task-agnostic model for problem difficulty and applying it to the Stanford Natural Language Inference (SNLI) dataset. Using the human responses that come with the dev set of SNLI, we train both regression and classification models to predict how many annotators will answer a question correctly and then project the difficulty estimates onto the full SNLI train set to create the curriculum. We argue that our curriculum is effectively capturing difficulty for this task through various analyses of both the model and the predicted difficulty scores.

## introduction recent advances on natural language processing (nlp) benchmarks have been driven by increasingly sophisticated language models, which are pretrained on enormous amounts of data before use. refinements of this process has led to increasingly powerful language models, such as bert @xcite , roberta @xcite , and more recently t5 @xcite . such models are quickly saturating even new tasks that have undergone a rigorous adversarial filtering process @xcite . however, these downstream performance improvements also require more computational resources and data to train the models, which is not always feasible. curriculum learning @xcite , a strategy where the model is trained on easier examples before harder ones, has recently been shown to improve performance and reduce training time on a variety of nlp tasks, especially machine translation @xcite @xcite . the success of this research shows that the order in which training data is presented to a model is important, but how to best apply curriculum learning broadly to nlp and how to design effective measures of difficulty to create curricula remain relatively unexplored. prior approaches either use a hand-crafted measure of difficulty that works well for a particular task or design an architecture that automatically creates the curriculum during training. in either case, the curriculum is formed using some information-theoretic measure of difficulty (semantic distance, feedback from a separate network, etc.), and it is difficult to interpret why they work well for some tasks and not others. in a practical sense, it is seldom clear how to apply a previously investigated curricula directly to another task. in this paper, we explore how to address these shortcomings by creating what we call a taskagnostic model of difficulty, which we argue can, in principle, be applied to any supervised learning task. we use this model to investigate what makes a good difficulty measure for curricula beyond how it affects downstream performance to better explain why one curriculum should be preferred over another. we use the well-known snli dataset for our experiments @xcite , which provides a high-quality data source for the natural language inference (nli) task. given a premise sentence, the goal is to predict whether a hypothesis sentence is entailed by the premise, contradicted by it, or is neither entailed nor contradicted by it (neutral). unlike many nli datasets, snli includes in its dev set the responses from each crowd-sourced worker who attempted to answer the question, giving us a set of 10,000 annotated questions to create a difficulty model. we release all code related to our experiments on github. 1 ## related work the idea of applying concepts from developmental psychology to artificial intelligence traces as far back as @xcite . although much work over the years has explored variants of this idea, mostly under the subject title "developmental ai" @xcite @xcite @xcite @xcite , only recently has the target task been defined as that of determining how to organize training data of machine learning models so that they can benefit from the kind of scaffolded learning that a human tutor might provide to a child. @xcite demonstrated that following a curriculum can provide a more optimal solution during gradient descent. other early work includes @xcite , which experimented with shaping tasks @xcite learned by a lstm model. these early results suggest that, for many learning tasks, estimating the difficulty of problems in a training set may indeed be simpler than training on random permutations of the dataset, but determining an effective measure of difficulty remains far from trivial. previous work has addressed this problem in two fundamentally different ways: by using a separate model from that used on the main task itself to predict difficulty @xcite @xcite @xcite ); or using a manually designed measure of difficulty not obtained through learning @xcite @xcite . a crucial question which recent ai work has not sufficiently addressed is how humans design curricula, which was stressed as essential to understand by early research in this area @xcite . for example, much has been said about the use of scaffolding in child education, in which a student is provided examples by a more experienced (typically older) tutor, such that those examples are neither "too easy" nor "too hard" given their current skill level (a space called the zone of proximal development) @xcite . in reinforcement learning, there has been fruitful research that applies these ideas by using human feedback to manually create curricula @xcite ; suay and chernova, 1 https://github.com/amhrlab/supervised-cl 2011; @xcite . we seek to apply these ideas in the context of supervised learning. ## difficulty model experiments using the snli dev set's annotation data, our goal is to train a model that can estimate the difficulty of the questions in the train set, hence creating a measure of difficulty. given the prior success of human-in-the-loop training, we believe this provides a natural way to estimate difficulty for any task where human annotation data can be collected. each question in the dev set contains 5 different labels assigned by each annotator, along with the gold label deemed correct for the question. the model's specific objective is to predict how many of 5 possible respondents predicted the gold label correctly, and this percentage is the approximate difficulty. the most intuitive solution to this problem is to treat it as a regression task, and we experiment with several regression models (section 3.1). however, because the notion of difficulty among humans is a somewhat fuzzy concept, it's unclear to us if a floating-point difficulty score will be meaningful up to arbitrary precision. therefore, we also experiment with framing the objective as a classification task by mapping the 5 possible percentages into labels and then training various classification models (section 3.2). using features from prior work, we create 5 different feature sets as inputs to both sets of models. the first is sentence length, which has demonstrated to be useful to measure difficulty in machine translation tasks @xcite . we adapt this to work for snli by taking the sentence length of the premise and hypothesis. second is the roberta sentence embeddings of the premise and the hypothesis. we feed the raw embeddings in by concatenating the entire hypothesis vector to the premise vector, and also by stacking them pairwise such that dimension i of the premise vector is paired with dimension i of the hypothesis vector in the final representation. we refer to this as the flattened embedding. finally, we use cosine similarity to measure the semantic distance between the input sequences. ## conclusion and future work in humans, teaching, especially according to a curriculum, reduces the difficulty of learning complex skills by providing a simpler path for learners to easy hard decision tree model: premise: a white dog runs through a field. hypothesis: a white dog is running back to his master. neutral premise: a man is surfing in a bodysuit in beautiful blue water. hypothesis: on the beautiful blue water there is a man in a bodysuit surfing. entailment premise: a group of workers are posing for a picture. hypothesis: a group of workers are playing baseball. contradiction premise: a man is kneeling in the top step while many people are behind him sitting in chairs. hypothesis: a man sleeps comfortably at home. contradiction linear regression model: premise: a small child is running on the shore of the beach surrounded by birds. hypothesis: birds surround a child looking for food. neutral follow. while much work has explored various applications of curriculum learning over the years, the larger problem of what makes a good difficulty measure remains an important research question. in this work, we have presented initial studies on designing a task-agnostic measure of difficulty for curricula. while for practical reasons, we chose to focus specifically on an nlp task, in principle our approach can be extended to any supervised learning task where it is possible to gather human annotations. our analysis suggests that the resulting curriculum is suitable for use in a curriculum learning algorithm, firstly because it correlates well with human difficulty estimates, and secondly because it displays wide differences in the estimated difficulties across all the problems in the train set. an important question to answer next is how to actually apply the curriculum to the training of a model. while this may appear trivial, the wide array of factors that affect downstream performance means that it cannot be taken lightly. the model choice is an especially important factor; using a re-cent model which already does very well on snli @xcite ) is unlikely to have any effect because performance on the task has saturated at this point. however, if we apply a model which is too simplistic, for instance, the classic n-gram language model, then it's unlikely that even the best curriculum will help such a model to learn a task which is probably far too challenging for it. a study examining how our curriculum affects downstream performance for models ranging from the state-of-the-art to the baseline would help answer this question. another important factor is the sampling protocol. feeding examples to the model in order of difficulty, with easier ones preceding harder ones, is the most straightforward solution. however, there is no guarantee that this is the optimal way to feed examples into the model. in fact, there have been cases where feeding examples in reverse order, with hard examples preceding easier ones, has lead to optimal performance improvement @xcite . future work will also investigate the effect of various sampling strategies. finally, something which should be considered is the quality of the gold labels which our models are trying to predict. since nli is an inherently ambiguous task, determining the ground truth of a given nli question is challenging. recent work by @xcite has shown that many of the gold labels for both snli and the mnli dataset @xcite , which also includes the responses of individual annotators, will change when more annotators are used with stricter quality control protocols, though they only found this to be true when agreement on the correct label was low to begin with. our choice for using snli specifically is that, unlike more recent datasets @xcite , it also includes the label predicted by each individual annotator and not just the final gold label. we might be able to account for the fuzziness of the ground truth in the final difficulty prediction of our model. for instance, if the agreement of the gold label for a question is low, we could take this into consideration and predict the question as being even more difficult. this avenue will be explored as a way to improve the quality of the difficulty model, using both snli and similar datasets, such as mnli, as part of a more comprehensive study.