Efficient Pre-training of Masked Language Model via Concept-based Curriculum Masking (2022)

Abstract: Self-supervised pre-training has achieved remarkable success in extensive natural language processing tasks. Masked language modeling (MLM) has been widely used for pre-training effective bidirectional representations but comes at a substantial training cost. In this paper, we propose a novel concept-based curriculum masking (CCM) method to efficiently pre-train a language model. CCM has two key differences from existing curriculum learning approaches to effectively reflect the nature of MLM. First, we introduce a novel curriculum that evaluates the MLM difficulty of each token based on a carefully-designed linguistic difficulty criterion. Second, we construct a curriculum that masks easy words and phrases first and gradually masks related ones to the previously masked ones based on a knowledge graph. Experimental results show that CCM significantly improves pre-training efficiency. Specifically, the model trained with CCM shows comparative performance with the original BERT on the General Language Understanding Evaluation benchmark at half of the training cost.

## introduction self-supervised pre-training has achieved considerable performance improvements in various natural language processing (nlp) tasks @xcite @xcite @xcite . masked language modeling (mlm) @xcite , where the model predicts original tokens of a masked subset of the text using the unmasked subset as clues, has contributed significantly to these improvements. however, mlm typically requires a high number of compute operations, resulting in unrealistically large training costs @xcite . therefore, the consideration on the pre-training costs has become an increasingly important issue @xcite . ## concept-based curriculum masking our curriculum framework is inspired by the human language acquisition. people learn simple concepts (e.g., car) first and then gradually learn more complex concepts (e.g., self-driving car). by leveraging the related basic concepts, the gradual learning style enables humans to learn abstract concepts easily @xcite . to mimic this, we construct a multi-stage curriculum by gradually adding concepts relating to the initial concepts using a knowledge graph. to this end, our ccm framework consists of three processes: initial concept selection, curriculum construction, and pre-training. in the following sections, we denote a knowledge graph as @xmath0, e), where n and e are sets of nodes and edges, respectively. we define a word or phrase corresponding to a node as a concept @xmath1. we denote s i as the set of concepts to be masked in the i-th stage. the final curriculum includes multiple stages {s 1 ...s k }. ## experiment we examine the efficacy of ccm using the bert architectures @xcite . we measure the performance of the pre-trained models on the glue benchmark @xcite . ## conclusion in this work, we have proposed ccm that masks easy concepts first and gradually masks related concepts to the previously masked ones for language model pre-training. with the help of our carefully designed linguistic difficulty criteria and curriculum construction, ccm can offer an effective masking schedule for mlm. the experimental results demonstrate that our curriculum improves convergence speeds and the generalization performance of mlm.