On Curriculum Learning for Commonsense Reasoning (2022)

Abstract: Commonsense reasoning tasks follow a standard paradigm of finetuning pretrained language models on the target task data, where samples are introduced to the model in a random order during training. However, recent research suggests that data order can have a significant impact on the performance of finetuned models for natural language understanding. Hence, we examine the effect of a human-like easy-to-difficult curriculum during finetuning of language models for commonsense reasoning tasks. We use paced curriculum learning to rank data and sample training mini-batches with increasing levels of difficulty from the ranked dataset during finetuning. Further, we investigate the effect of an adaptive curriculum, i.e., the data ranking is dynamically updated during training based on the current state of the learner model. We use a teacher model to measure difficulty of each sample and experiment with three measures based on question answering probability, variability and out-of-distribution. To understand the effectiveness of curriculum learning in various scenarios, we apply it on full model fine-tuning as well as parameter-efficient prompt-tuning settings. Our results show that fixed as well as adaptive curriculum learning significantly improve performance for five commonsense reasoning tasks, i.e., SocialIQA, CosmosQA, CODAH, HellaSwag, WinoGrande in both tuning settings. Further, we find that prioritizing the difficult samples in the tail end of training improves generalization to unseen in-domain data as well as out-of-domain data. Our work provides evidence and encourages research into curriculum learning for commonsense reasoning.

## introduction curriculum learning @xcite is an alternative to the typical uniform random sampling of training data and is motivated by the gradual progression of human learning from easier to difficult concepts (see figure 1 ). in the machine learning paradigm, a 'teacher' ranks the training samples from easy to difficult and introduces them to the 'learner' in that order. @xcite show that randomly initialized training data orders can lead to large variance in model performance on the glue benchmark @xcite . in light of such evidence, we seek to answer: does a meaningful data order such as a curriculum based on model confidence or dataset distribution outperform a random data order? such experiments have been carried out for some nlp tasks like machine translation @xcite and natural language understanding @xcite with positive outcomes. while large pretrained models (ptlms) have been achieving high performance on such tasks, their commonsense reasoning abilities have been limited. moreover, the process of commonsense acquisition in humans has been shown to be informative for developing algorithms to accomplish the same in machines @xcite . hence, we study the effect of a human-like curricu-lum learning to improve the finetuning of ptlms for commonsense reasoning tasks. to impose structure on the data order for sampling training mini-batches, we adopt paced curriculum learning by transfer as proposed in @xcite . in this method, a pacing function determines the speed at which the ranked data is introduced to the model during training. ranking of the training dataset is performed using outputs from a pretrained network which has been finetuned on the target dataset using a random training order. we refer to this approach as fixed curriculum learning. during human acquisition of skill sets, a student can benefit from a curriculum that is continuously adjusted by the teacher according to the learning progress of the student. hence, we also investigate adaptive curriculum learning for commonsense reasoning tasks. the initial data order imposed by the teacher model is updated at regular intervals during training by taking the learner model's current state into account @xcite . importantly, we propose to reverse the ranking to a difficulty-to-easy curriculum in acl, in order to reinforce feedback from the hardto-learn data points, which has been shown to be beneficial for generalization @xcite . in order to measure difficulty, we explore three different data-sample informativeness scoring methods i.e. question answering probability (qap) @xcite , energy-based out-of-distribution score @xcite and cartography-based variability @xcite . our work is most related to @xcite which splits the training data into n meta-datasets, trains n models for computing the curriculum and follows a heuristically designed training regimen. in contrast, we train a single model for computing the curriculum and use bayesian optimization @xcite to find the best pacing of the curriculum for the target dataset, which is more effective than @xcite as we show in sec. 5.4, besides being computationally efficient. we analyze these methods on five commonsense reasoning datasets dealing with various tasks such as reasoning about social interactions (socialiqa; @xcite al. (2019) 2019 ). we explore curriculum learning in full-model finetuning as parameter-efficient tuning and show significant improvements using curriculum learning on each of these datasets. we also demonstrate that curriculum learning prevents the learner model from over-fitting on the training set, which leads to improved generalization to in-domain and out-of-domain data. ## related work curriculum learning (cl) is widely used in reinforcement learning @xcite @xcite and neural machine translation @xcite @xcite . @xcite xing (2018), penha and hauff (2020), xu et al. (2020) and jafarpour et al. (2021) demonstrate the effectiveness of cl for question generation, information retrieval, natural language understanding and named entity recognition respectively. to the best of our knowledge, we are the first to examine the efficacy of cl for commonsense reasoning. various task-specific measures for sample complexity have been proposed in previous works, such as inter-annotator agreement for natural language inference (laverghetta jr et al., 2020), sub-graph depth for abstract meaning representation (amr) structures @xcite , noise rate in pca jittering-based data augmentation @xcite , sentence length for sequence modelling @xcite , and semantic similarity for sentiment analysis (han and myaeng, 2017) etc. @xcite use a neural density estimator to model graph embeddings distribution for the task of graph classification. we use scores based on pretrained models @xcite @xcite to compute difficulty. @xcite and @xcite demonstrate that an adaptive curriculum can improve convergence for image classification and neural response generation respectively. we examine adaptive cl for commonsense reasoning. ## experimental setup we use a suite of large and small datasets formatted as multiple-choice question answering tasks for our experiments. socialiqa @xcite , cos-mosqa @xcite and winogrande-xl @xcite contain upto 60k training samples. additionally, we use the codah dataset @xcite and also follow the method in @xcite to create hellaswag-2k @xcite for testing our methods in low-resource scenarios. models are evaluated using the respective task-specific accuracies (see appendix for dataset statistics). we use pretrained roberta large @xcite as the teacher as well as learner model in our full-model finetuning experiments. in the first stage, the teacher model is finetuned on randomly sampled training data and used to compute the difficulty scores. models are also subjected to grid-search based tuning of training hyperparameters in the first stage whenever necessary. in the second stage, we use bayesian optimization for finding the optimum pacing function parameters for fixed as well as adaptive cl (and use the same training hyperparameters as first stage). for adaptive learning, we set @xmath0: ablation results on validation set of commonsense reasoning datasets using fixed cl with roberta and various difficulty measures. ## limitations our results are limited to the task of multiple choice question-answering based commonsense reasoning and natural language understanding, but are encouraging and warrant further research into effective adaptive cl methodologies for other nlp tasks. the adaptive curriculum learning method proposed in our paper is expensive for larger datasets since the ranking needs to be recomputed several times during training using the current version of the learner model. further work is needed to optimize curriculum learning methods for larger datasets like abductivenli @xcite and for using larger models like unifiedqa (khashabi et al., 2020) for finding the right curriculum. ## conclusion we conduct an empirical analysis of fixed and adaptive curriculum learning (cl) for five commonsense reasoning tasks using pretrained language models (ptlms). results show that cl can benefit downstream task performance when introducing a new task to a ptlm, in fully-finetuned as well prompt-tuned settings, and for in-domain as well as out-of-domain data. our work motivates future research into cl for commonsense reasoning.