Training Dynamics for Curriculum Learning: A Study on Monolingual and Cross-lingual NLU (2022)

Abstract: Curriculum Learning (CL) is a technique of training models via ranking examples in a typically increasing difficulty trend with the aim of accelerating convergence and improving generalisability. Current approaches for Natural Language Understanding (NLU) tasks use CL to improve in-distribution data performance often via heuristic-oriented or task-agnostic difficulties. In this work, instead, we employ CL for NLU by taking advantage of training dynamics as difficulty metrics, i.e., statistics that measure the behavior of the model at hand on specific task-data instances during training and propose modifications of existing CL schedulers based on these statistics. Differently from existing works, we focus on evaluating models on in-distribution (ID), out-of-distribution (OOD) as well as zero-shot (ZS) cross-lingual transfer datasets. We show across several NLU tasks that CL with training dynamics can result in better performance mostly on zero-shot cross-lingual transfer and OOD settings with improvements up by 8.5% in certain cases. Overall, experiments indicate that training dynamics can lead to better performing models with smoother training compared to other difficulty metrics while being 20% faster on average. In addition, through analysis we shed light on the correlations of task-specific versus task-agnostic metrics.

## introduction transformer-based language models @xcite have recently achieved great success in a variety of nlp tasks @xcite . however, generalisation to out-of-distribution (ood) data and zero-shot cross-lingual transfer still remain a challenge @xcite . among existing techniques, improving ood performance has been addressed by training with adversarial data @xcite , while better transfer across languages has been achieved by selecting appropriate languages to transfer from @xcite , employing meta-learning @xcite or data alignment @xcite . contrastive to such approaches that take advantage of additional training data is curriculum learning @xcite , a technique that aims to train models using a specific ordering of the original training examples. this ordering typically follows an increasing difficulty trend where easy examples are fed to the model first, moving towards harder instances. the intuition behind cl stems from human learning, as humans focus on simpler concepts before learning more complex ones, a procedure that is called shaping @xcite . although curricula have been primarily used for computer vision @xcite and machine translation @xcite , there are only a handful of approaches that incorporate cl into natural language understanding tasks @xcite @xcite . typically, cl requires a measure of difficulty for each example in the training set. existing methods using cl in nlu tasks rely on heuristics such as sentence length, word rarity, depth of the dependency tree @xcite , metrics based on item-response theory @xcite or task-agnostic model metrics such as perplexity @xcite . such metrics have been employed to either improve in-distribution performance on nlu or machine translation. however, their effect is still underexplored on other settings. in this study instead, we propose to adopt training dynamics @xcite as difficulty measures for cl and fine-tune models with curricula on downstream tasks. td were re-cently proposed as a set of statistics collected during the course of a model's training to automatically evaluate dataset quality, by identifying annotation artifacts. these statistics, offer a 3-dimensional view of a model's uncertainty towards each training example classifying them into distinct areas-easy, ambiguous and hard examples for a model to learn. we test a series of easy-to-hard curricula using td, namely td-cl, with existing schedulers as well as novel modifications of those and experiment with other task-specific and task-agnostic metrics. we show performances and training times on three settings: in-distribution (id), out-of-distribution (ood) and zero-shot (zs) transfer to languages different than english. to the best of our knowledge, no prior work on nlu considers the impact of cl on all these settings. to consolidate our findings, we evaluate models on different classification tasks, including natural language inference, paraphrase identification, commonsense causal reasoning and document classification. our findings suggest that td-cl provides better zero-shot cross-lingual transfer up to 1.2% over prior work and can gain an average speedup of 20%, up to 51% in certain cases. in id settings cl has minimal to no impact, while in ood settings models trained with td-cl can boost performance up to 8.5% on a different domain. finally, td provide more stable training compared to another taskspecific metric (cross-review). on the other hand, heuristics can also offer improvements especially when testing on a completely different domain. ## related work curriculum learning was initially mentioned in the work of @xcite who demonstrated the importance of feeding neural networks with small/easy inputs at the early stages of training. the concept was later formalised by @xcite where training in an easy-to-hard ordering was shown to result in faster convergence and improved performance. in general, curriculum learning requires a difficulty metric (also known as the scoring function) used to rank training instances, and a scheduler (known as the pacing function) that decides when and how new examples-of different difficulty-should be introduced to the model. example difficulty was initially expressed via model loss, in self-paced learning @xcite , increasing the contribution of harder training instances over time. this setting posed a challenge due to the fast-changing pace of the loss during training, thus later approaches used human-intuitive difficulty metrics, such as sentence length or the existence of rare words @xcite to pre-compute difficulties of training instances. however, as such metrics do not express difficulty of the model, modelbased metrics have been proposed over the years, such as measuring the loss difference between two checkpoints @xcite or model translation variability @xcite . in our curricula we use training dynamics to measure example difficulty, i.e. metrics that consider difficulty from the perspective of a model towards a certain task. example difficulty can be also estimated either in a static (offline) or dynamic (online) manner, where in the latter training instances are evaluated and re-ordered at certain times during training, while in the former the difficulty of each example remains the same throughout. in our experiments we adopt the first setting and consider static example difficulties. transfer teacher cl is a particular family of such approaches that use an external model (namely the teacher) to measure the difficulty of training examples. notable works incorporate a simpler model as the teacher @xcite or a larger-sized model @xcite , as well as using similar-sized learners trained on different subsets of the training data. these methods have considered as example difficulty, either the teacher model perplexity @xcite , the norm of a teacher model word embeddings @xcite , the teacher's performance on a certain task @xcite or simply regard difficulty as a latent variable in a teacher model @xcite . in the same vein, we also incorporate transfer teacher cl via teacher and student models of the same size and type. however, differently, we take into account the behavior of the teacher during the course of its training to measure example difficulty instead of considering its performance at the end of training or analysing internal embeddings. moving on to schedulers, these can be divided into discrete and continuous. discrete schedulers, often referred to as bucketing, group training instances that share similar difficulties into distinct sets. different configurations include accumulating buckets over time @xcite , sampling a subset of data from each bucket @xcite or more sophisti-cated sampling strategies @xcite . in cases where the number of buckets is not obtained in a straightforward manner, methods either heuristically split examples @xcite , adopt uniform splits @xcite or employ schedulers that are based on a continuous function. a characteristic approach is that of @xcite where at each training step a monotonically increasing function chooses the amount of training data the model has access to, sorted by increasing difficulty. as we will describe later on, we experiment with two established schedulers and propose modifications of those based on training dynamics. other tasks where cl has been employed include question answering @xcite , reading comprehension @xcite and other general nlu classification tasks @xcite . others have developed modified curricula in order to train models for code-switching @xcite , anaphora resolution @xcite , relation extraction @xcite , dialogue @xcite and self-supervised neural machine translation @xcite , while more advanced approaches combine it with reinforcement learning in a collaborative teacherstudent transfer curriculum @xcite . ## conclusion we presented a set of experiments using training dynamics @xcite as difficulty metrics for cl on several nlu tasks. differently from existing works, we focus our evaluation on indistribution, out-of-distribution and zero-shot crosslingual transfer data by testing existing discrete and continuous schedulers as well as modifications of those in a transfer-teacher curriculum setting. our findings offer evidence that simply reordering the training examples in a meaningful way has mostly an impact on zero-shot cross-lingual transfer and ood data, with no improvement on id. our proposed continuous scheduler with confidence and variability sampling provided a boost up to 8.5% on a challenging ood dataset over prior work. comparing our proposed application of training dynamics to other transfer-teacher curriculum methods that are using more than 1 teacher model, we observed greater speedups, improved performance and more stable training. in particular, we found that task-agnostic metrics do not perform better than task-specific ones on id and zs data but can offer good performance on ood settings. overall, our experiments suggest there is no curriculum outperforming others by a large margin which is consistent with findings in @xcite and that task-agnostic metrics should not be rejected when transferring to challenging new domains. however we show that training dynamics are potentially better difficulty metrics for cl in both monolingual and multilingual models even with a limited budget. although in this study we focused on using cl on a single language only (english), a reasonable extension is considering training data from other languages as well and investigate instance difficulties based on language or following efforts towards continual learning @xcite . finally, using td in a dynamic rather than a static curriculum is another interesting direction that can potentially offer further training speedups as well as ways to improve model pre-training @xcite .