Curriculum Learning for Natural Language Understanding (2020)

Abstract: With the great success of pre-trained language models, the pretrain-finetune paradigm now becomes the undoubtedly dominant solution for natural language understanding (NLU) tasks. At the fine-tune stage, target task data is usually introduced in a completely random order and treated equally. However, examples in NLU tasks can vary greatly in difficulty, and similar to human learning procedure, language models can benefit from an easy-to-difficult curriculum. Based on this idea, we propose our Curriculum Learning approach. By reviewing the trainset in a crossed way, we are able to distinguish easy examples from difficult ones, and arrange a curriculum for language models. Without any manual model architecture design or use of external data, our Curriculum Learning approach obtains significant and universal performance improvements on a wide range of NLU tasks.

## introduction natural language understanding (nlu), which requires machines to understand and reason with human language, is a crucial yet challenging problem. recently, language model (lm) pre-training has achieved remarkable success in nlu. pre-trained lms learn universal language representations from large-scale unlabeled data, and can be simply finetuned with a few adjustments to adapt to various nlu tasks, showing consistent and significant improvements in these tasks @xcite . while lots of attention has been devoted to designing better pre-training strategies @xcite @xcite , it is also valuable to explore how to more effectively solve downstream nlu tasks in the fine-tuning stage. most current approaches perform fine-tuning in a straightforward manner, i.e., all training examples are treated equally and presented in a completely random order during training. however, even in the same nlu task, the training examples could vary significantly in their difficulty levels, with some easily solvable by simple lexical clues while others requiring sophisticated reasoning. such an organization of learning materials in human learning procedure is usually referred to as curriculum. in this paper, we draw inspiration from similar ideas, and propose our approach for arranging a curriculum when learning nlu tasks. curriculum learning (cl) is first proposed by @xcite in machine learning area, where the definition of easy examples is established ahead, and an easy-to-difficult curriculum is arranged accordingly for the learning procedure. recent developments have successfully applied cl in computer vision areas @xcite @xcite . it is observed in these works that by excluding the negative impact of difficult or even noisy examples in early training stage, an appropriate cl strategy can guide learning towards a better local minima in parameter space, especially for highly non-convex deep models. we argue that language models like transformer, which is hard to train @xcite , should also benefit from cl in the context of learning nlu tasks, and such idea still remains unexplored. the key challenge in designing a successful cl strategy lies in how to define easy/difficult examples. one straightforward way is to simply predefine the difficulty in revised rules by observing the particular target task formation or training data structure accordingly @xcite @xcite . for example, @xcite utilized an easier version of shape recognition trainset which comprised of less varied shapes, before the training of complex one started. more recently, @xcite considered the paragraph length of a question answering example as its reflection of difficulty. however, such strategies are highly dependent on the target dataset itself and often fails to generalize to different tasks. to address this challenge, we propose our cross review method for evaluating difficulty. specifically, we define easy examples as those well solved by the exact model that we are to employ in the task. for different tasks, we adopt their corresponding golden metrics to calculate a difficulty score for each example in the trainset. then based on these difficulty scores, we further design a re-arranging algorithm to construct the learning curriculum in an annealing style, which provides a soft transition from easy to difficult for the model. in general, our cl approach is not constrained to any particular task, and does not rely on human prior heuristics about the task or dataset. experimental results show that our cl approach can greatly help language models learn in their finetune stage. without any task-tailored model architecture design or use of external data, we are able to obtain significant and universal improvements on a wide range of downstream nlu tasks. our contributions can be concluded as follows: ## preliminaries we describe our cl approach using bert @xcite , the most influential pre-trained lm that achieved state-of-the-art results on a wide range of nlp tasks. bert is pretrained using masked language model task and next sentence prediction task via large scale corpora. it consists of a hierarchical stack of l self-attention layers, which takes an input of a sequence with no more than 512 tokens and output the contextual representation of a h-dimension vector for each token in position i, which we denote as h l @xmath0. in natural language understanding tasks, the input sequences usually start with special token cls , and end with sep , for sequences consisting of two segments like in pairwise sentence tasks, another sep is added in between for separating usage. for target benchmarks, we employ a wide range of nlu tasks, including machine reading comprehension, sequence classification and pairwise text similarity, etc.. following @xcite , we adapt bert for nlu tasks in the most straightforward way: simply add one necessary linear layer upon the final hidden outputs, then finetune the entire model altogether. specifically, we brief the configurations and corresponding metrics for different tasks employed in our algorithms as follows: machine reading comprehension in this work we consider the extractive mrc task. given a passage p and a corresponding question q, the goal is to extract a continuous span p start , p end from p as the answer a, where the start and end are its boundaries. we pass the concatenation of the question and paragraph [ cls , q, sep , p, sep ] to the pretrained lm and use a linear classifier on top of it to predict the answer span boundaries. for the i -th input token, the probabilities that it is the start or end are calculated as: f1 is used as the golden metric. ## our cl approach we decompose our cl framework into two stages: difficulty evaluation and curriculum arrangement. for any target task, let d be the examples set used for training, and θ be our language model which is expected to fit d. in the first stage, the goal is to assign each example d j in d with a score c j which reflects its difficulty with respect to the model. we denote c as the whole difficulty score set corresponding to trainset d. in the second stage, based on these scores, d is organized into a sequence of ordered learning stages {s i : @xmath1, 2, . . . , n } with an easy-to-difficult fashion, resulting in the final curriculum where the model will be trained on. we will elaborate these two stages in section 3.1 and 3.2 respectively. ## related works the idea of training a neural network in an easyto-difficult fashion can be traced back to @xcite . @xcite revisited the idea from a cognitive perspective with the shaping procedure, in which a teacher decomposes a complete task into sub-components. based on these works, curriculum learning is first proposed in @xcite . they designed several toy experiments to demonstrate the benefits of curriculum strategy both in image classification and language modeling. they also propose that curriculum can be seen as a sequence of training criteria, and at the end of it, the reweighting of examples should be uniform with the target distribution, which inspired the design of our curriculum arrangement algorithm. although cl has been successfully applied to many areas in computer vision @xcite @xcite , it was not introduced to solve nlu tasks until @xcite . by experimenting with several heuristics, they migrated the success of cl @xcite to machine reading com-prehension tasks. @xcite further extended this work to question generation. more recently, @xcite employed cl strategy to solve reading comprehension over long narratives. apart from them, there aren't very many works that discuss cl in the context of nlu to the best of our knowledge. on the methodology of designing cl algorithms, our approach is closely related to @xcite @xcite , where a curriculum is formed via two steps: evaluating the difficulty first, then sampling the examples into batches accordingly. for different target tasks, the evaluation methods also vary greatly. @xcite first examined the examples in their feature space, and define difficulty by the distribution density, which successfully distinguished noisy images. @xcite incorporated category information into difficulty metric to address imbalanced data classification. in language tasks, @xcite and @xcite propose to consider the length of context as extent of difficulty. another line of works see curriculum construction as an optimization problem @xcite @xcite , which usually involves sophisticated design and is quite different from our approach. ## conclusion in this work we proposed a novel curriculum learning approach which does not rely on human heuristics and is simple to implement. with the help of such a curriculum, language models can significantly and universally perform better on a wide range of downstream nlu tasks. in the future, we look forward to extend cl strategy to the pretraining stage, and guide deep models like transformer from a language beginner to a language expert.