ERNIE -Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding (2021)

Abstract: Coarse-grained linguistic information, such as named entities or phrases, facilitates adequately representation learning in pre-training. Previous works mainly focus on extending the objective of BERT’s Masked Language Modeling (MLM) from masking individual tokens to contiguous sequences of n tokens. We argue that such contiguously masking method neglects to model the intra-dependencies and inter-relation of coarse-grained linguistic information. As an alternative, we propose ERNIE-Gram, an explicitly n-gram masking method to enhance the integration of coarse-grained information into pre-training. In ERNIE-Gram, n-grams are masked and predicted directly using explicit n-gram identities rather than contiguous sequences of n tokens. Furthermore, ERNIE-Gram employs a generator model to sample plausible n-gram identities as optional n-gram masks and predict them in both coarse-grained and fine-grained manners to enable comprehensive n-gram prediction and relation modeling. We pre-train ERNIE-Gram on English and Chinese text corpora and fine-tune on 19 downstream tasks. Experimental results show that ERNIE-Gram outperforms previous pre-training models like XLNet and RoBERTa by a large margin, and achieves comparable results with state-of-the-art methods. The source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE .

## introduction pre-trained on large-scaled text corpora and finetuned on downstream tasks, self-supervised representation models @xcite @xcite @xcite have achieved remarkable improvements in natural language understanding (nlu). as one of the most prominent pre-trained models, bert @xcite employs masked language modeling (mlm) to learn representations by masking individual tokens and predicting them based on their bidirectional context. however, bert's mlm focuses on the representations of fine-grained text units (e.g. words or subwords in english and characters in chinese), rarely considering the coarse-grained linguistic information (e.g. named entities or phrases in english and words in chinese) thus incurring inadequate representation learning. many efforts have been devoted to integrate coarse-grained semantic information by independently masking and predicting contiguous sequences of n tokens, namely n-grams, such as named entities, phrases @xcite , whole words @xcite ) and text spans @xcite . we argue that such contiguously masking strategies are less effective and reliable since the prediction of tokens in masked n-grams are independent of each other, which neglects the intradependencies of n-grams. specifically, given a masked n-gram @xmath0, ..., x n }, @xmath1, we maximize p(w) = n @xmath2, where models learn to recover w in a huge and sparse prediction space @xmath3. note that v f is the fine-grained vocabulary foot_0 and c is the context. we propose ernie-gram, an explicitly ngram masked language modeling method in which n-grams are masked with single [mask] symbols, and predicted directly using explicit ngram identities rather than sequences of tokens, as depicted in figure 1 transformer encoder 1 x 2 x 3 x 5 x 4 x 6 x transformer encoder 1 x 2 y 4 y 4 x 6 x 1 2 3 4 5 1 2 3 4 5 transformer encoder 1 x 3 x 5 x 2 x 2 y 4 x 6 x 1 2 3 4 5 6 + + + + + + + + + + + + + + + + ×l ×l ×l 4 y 2 z 4 z embedding layer fine-grained classifier n-gram classifier learn the semantic of n-grams more adequately, we adopt a comprehensive n-gram prediction mechanism, simultaneously predicting masked ngrams in coarse-grained (explicit n-gram identities) and fine-grained (contained token identities) manners with well-designed attention mask metrics, as shown in figure 1(c ). in addition, to model the semantic relationships between n-grams directly, we introduce an enhanced n-gram relation modeling mechanism, masking n-grams with plausible n-grams identities sampled from a generator model, and then recovering them to the original n-grams with the pair relation between plausible and original n-grams. inspired by electra @xcite , we incorporate the replaced token detection objective to distinguish original n-grams from plausible ones, which enhances the interactions between explicit n-grams and fine-grained contextual tokens. in this paper, we pre-train ernie-gram on both base-scale and large-scale text corpora (16gb and 160gb respectively) under comparable pre-training setting. then we fine-tune ernie-gram on 13 english nlu tasks and 6 chinese nlu tasks. experimental results show that ernie-gram consistently outperforms previous well-performed pre-training models on various benchmarks by a large margin. ## proposed method in this section, we present the detailed implementation of ernie-gram, including n-gram lexicon v n extraction in section 3.5, explicitly n-gram mlm pre-training objective in section 3.2, comprehensive n-gram prediction and relation modeling mechanisms in section 3.3 and 3.4. ## experiments in this section, we first present the pre-training configuration of ernie-gram on chinese and english text corpora. then we compare ernie-gram with previous works on various downstream tasks. we also conduct several ablation experiments to access the major components of ernie-gram. ## conclusion in this paper, we present ernie-gram, an explicitly n-gram masking and predicting method to eliminate the limitations of previous contiguously masking strategies and incorporate coarse-grained linguistic information into pre-training sufficiently. ernie-gram conducts comprehensive n-gram prediction and relation modeling to further enhance the learning of semantic n-grams for pre-training. experimental results on various nlu tasks demonstrate that ernie-gram outperforms xlnet and roberta by a large margin, and achieves state-ofthe-art results on various benchmarks. future work includes constructing more comprehensive n-gram lexicon (@xmath4.