Do Data-based Curricula Work? (2022)

Abstract: Current state-of-the-art NLP systems use large neural networks that require extensive computational resources for training. Inspired by human knowledge acquisition, researchers have proposed curriculum learning - sequencing tasks (task-based curricula) or ordering and sampling the datasets (data-based curricula) that facilitate training. This work investigates the benefits of data-based curriculum learning for large language models such as BERT and T5. We experiment with various curricula based on complexity measures and different sampling strategies. Extensive experiments on several NLP tasks show that curricula based on various complexity measures rarely have any benefits, while random sampling performs either as well or better than curricula.

## introduction in the last years state-of-art results in natural language processing (nlp) are often obtained with transformer-like architectures based on the selfattention mechanism @xcite such as bert @xcite , gpt-3 @xcite , t5 @xcite , which could have billions of parameters. due to many parameters, these architectures require lots of time and hardware resources to be trained. curriculum learning (cl) is one of the popular methods to reduce training time and increase the resulting quality of the model. inspired by the importance of adequately ordering information when teaching humans @xcite , curriculum learning increases the difficulty of training samples shown to the model over time @xcite . previous studies have demonstrated that curriculum learning significantly impacts training time and quality in different machine learning domains, such as computer vision (soviany, 2020) and reinforcement learning @xcite . in nlp, some results hint that cl might be beneficial @xcite @xcite ; however, these results are not as optimistic as in reinforcement learning setup. we suggest dividing recent research in curriculum learning into two main categories: task-driven curriculum and data-driven curriculum. the idea of the task-driven curriculum was inspired by human behavior. first, the model learns how to solve a simple task, and then the difficulty is gradually increased. this type of curriculum proposed by @xcite is considered to be classical, and a majority of curriculum-related results are obtained in this framework. alternatively to the taskdriven curriculum, some curricula try to use some form of filtering or sorting of training data that could facilitate learning a model on a given task. we suggest calling these curricula data-driven and distinguishing them from the classical task-based approach. this paper attempts to understand when datadriven curriculum learning works for transformerbased language models. generally, data-driven curriculum learning is organized in two steps: first, estimating the complexity for the elements that comprise the dataset; second, designing a sampling strategy, thus forming a curriculum. in the first part of the paper, we list potentially useful natural language processing complexity measures. the second part discusses possible sampling strategies that might apply to corresponding complexity measures. we run extensive experiments with different metrics and sampling strategies on three classes of nlp tasks: unsupervised learning with masked language modeling, text classification, and machine translation. our experiments show that data-driven curriculum learning does not give quality increase or time reduction on all metric-sampling strategy setups and often makes results even worse. ## metrics the first important part of the curriculum learning pipeline is measuring the complexity of samples for a given dataset. texts could have a complex structure, and one can measure their complexity in different ways. a variety of heuristically motivated methods is accompanied by several metrics based on specific aspects of information theory. for a review of heuristic text complexity measures such as length of tf-idf @xcite we address the reader to appendix a. in this paper, we also explore the metrics initially proposed by @xcite to measure the complexity of finite systems and try to see if one could apply these metrics to nlp tasks. @xcite observes that for finite systems, a set of parts impacts the complexity of the system as well as inter-dependencies of the parts. in the context of nlp, this means that text is more than just a bag of words. the authors propose four different metrics to estimate the complexity of a system. however, one of these metrics maximizes on single-letter texts, such as "aaaaaaaaa," while the second was created to measure cyclic sequences and does not apply to texts. thus we experiment with two other metrics, namely, tononi, sporns, and edelman (tse) @xcite and excess entropy (ee), and adapt them to the complexity of texts. for the calculation of tse and ee for nlp we address the reader to appendix b. ## samplers the second important part of curriculum learning is the sampling strategy (or sampler) -the algorithm deciding which samples should be shown to the model at which moment. let us observe existing curricula and suggest some new ones. competence-based. cb a competence-based curriculum, offered by @xcite , uniformly samples data from increasing dataset's prefix. competence is a function c(t), which defines the size of the dataset prefix. where t -total number of steps, t -current step, c 0 -hyperparameter set to 0.01. hyperbolic. hyp the main idea of this sampler is to increase average batch complexity through time. all samples are split by complexity into n sequential buckets with equal size. training time is divided into n epochs and the probability of sampling the element from the j-th bucket on the i-th epoch is proportional to the distance between j and i. where p r i (j) -probability to sample from j-th bucket on the i-th epoch, c -constant to guarantee that sum of all probabilities equals to 1. ## experiments we perform our experiments on three nlp tasks: text classification, machine translation (nmt), and masked language modeling (mlm). here we discuss the first task of classification in detail. the extensive results of the experiments are available in appendix c. all the experiments are performed with the huggingface library @xcite , which provides the models with their setups, such as hyperparameters and tokenizers. we did not change default parameters in our experiment unless specifically stated otherwise. thus, the dataset and the model specify every experiment. we use the base version of the bert model @xcite for mlm and classification, and the small version of the t5 model @xcite for machine translation. experiments were performed on bookscorpus foot_0 dataset for mlm, sentiment140 foot_1 and hyperpartisan news detection foot_2 for classification, and wmt16-en-de foot_3 for machine translation. to estimate the curriculum's convergence speed, we calculate the average number of steps to reach a threshold that is 10% lower than the resulting saturation quality metric for every problem. ## discussion we try to interpret obtained results cautiously. curriculum learning depends on subtle factors, for example, a correct choice of hyperparameters. it is hard to check all possible values of hyperpa-rameters, yet to the best of our capabilities, we address this issue in appendix c.3. the results do not seem to depend on the learning rate, and once again, curriculum learning shows no benefits. at this point, we can only conclusively say two things: (1) a deeper investigation of the underlying information theoretic principles that stand behind curriculum learning is badly needed; (2) until we better understand these principles, data-based curriculum learning is a gamble with very low odds to gain either speed or resulting performance. ## conclusion in this work, we ran extensive experiments with curriculum learning for transformer-based architectures on three nlp tasks: masked language modeling, text classification, and machine translation. we demonstrate that curricula do not help in the standard training setting and sometimes even worsen results. methods in natural language processing: system demonstrations, pages 38-45, online. association for computational linguistics. benfeng xu, licheng zhang, zhendong mao, quan wang, hongtao xie, and yongdong zhang. 2020. curriculum learning for natural language understanding. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 6095-6104. xuan zhang, gaurav kumar, huda khayrallah, kenton murray, jeremy gwinnup, marianna j martindale, paul mcnamee, kevin duh, and marine carpuat.