Dict- BERT : Enhancing Language Model Pre-training with Dictionary (2022)

Abstract: Pre-trained language models (PLMs) aim to learn universal language representations by conducting self-supervised training tasks on large-scale corpora. Since PLMs capture word semantics in different contexts, the quality of word representations highly depends on word frequency, which usually follows a heavy-tailed distributions in the pre-training corpus. Therefore, the embeddings of rare words on the tail are usually poorly optimized. In this work, we focus on enhancing language model pre-training by leveraging definitions of the rare words in dictionaries (e.g., Wiktionary). To incorporate a rare word definition as a part of input, we fetch its definition from the dictionary and append it to the end of the input text sequence. In addition to training with the masked language modeling objective, we propose two novel self-supervised pre-training tasks on word and sentence-level alignment between input text sequence and rare word definitions to enhance language modeling representation with dictionary. We evaluate the proposed Dict-BERT model on the language understanding benchmark GLUE and eight specialized domain benchmark datasets. Extensive experiments demonstrate that Dict-BERT can significantly improve the understanding of rare words and boost model performance on various NLP downstream tasks.

## introduction recently pre-trained language models (plms) such as bert @xcite and roberta @xcite have revolutionized the field of natural language processing (nlp), yielding remarkable performance on various downstream tasks @xcite . however, these plms suffer from lacking knowledge when completing real-world tasks. to address this issue, some methods have incorporated the knowledge to enrich language representations, ranging from linguistic @xcite , commonsense @xcite , factual @xcite , to domain knowledge @xcite . nevertheless, rare words @xcite and unseen words @xcite are still blind spots of pre-trained language models when they are fine-tuned on downstream tasks. for instance, in a dialogue system, users often talk to chatbots about recent hot topics, e.g., "covid-19", which may not appear in the pre-training corpus @xcite . since plms capture word semantics in different contexts to address the issue of polysemous and the context-dependent nature of words, consequently they usually perform poorly when a user mentions such novel words @xcite . as indicated by @xcite , the quality of word representations highly depends on the word frequency in the pre-training corpus, which typically follows a heavy-tail distribution. thus, a large proportion of words appear very few times and the embeddings of these rare words are poorly optimized @xcite . such embeddings usually carry inadequate semantic meaning, which complicate the understanding of input text, and even hurt the pre-training of the entire model. in this work, we focus on enhancing language model pre-training by leveraging rare word definitions in english dictionaries (e.g., wiktionary). definitions in dictionaries are intended to describe the meaning of a word to a human reader. we append the definitions of rare words to the end of the input text and encode the whole sequence with transformer encoder. the pre-training tasks are mainly based on the alignment between input text and the appended word definitions, some of which are randomly sampled polluted words and don't explain the input. we propose two types of pre-training objectives: 1) a word-level contrastive objective aims to maximize the mutual information between transformer representations of a rare word appeared in the input text and its dictionary definition. 2) a sentence-level discriminative objective aims at learning to differentiate between correct and polluted word definitions. during downstream fine-tuning, in order to avoid the appended rare word definitions diverting the sentence from its original meaning, we employ a knowledge attention mechanism that makes word definitions only visible to the corresponding words in the input text sequence. we name our method dict-bert. notably, dict-bert is general and model-agnostic, in the sense that any pre-trained language model (e.g., bert, roberta) suffices and can be used. overall, our main contributions in this work can be summarized as follows: language model pre-training and knowledgeenhanced methods recent years have seen substantial pre-trained language models (plms) such as bert @xcite and t5 @xcite have achieved remarkable performance in various nlp downstream tasks. however, these plms suffer from lacking domain-specific knowledge when completing many real-world tasks @xcite . for example, bert cannot give full play to its value when dealing with electronic medical record analysis tasks in the medical field @xcite . a lot of efforts have been made on investigating how to integrate knowledge into plms @xcite @xcite @xcite . overall, these approaches can be grouped into two categories: the first one is to explicitly inject knowledge representation into plms, where the representations are pre-computed from external sources @xcite . however, it has been argued that the embedding vectors of input words and knowledge are obtained in separate ways, making their vectorspace inconsistent @xcite . the second one is to implicitly model knowledge information into plms by performing knowledge-related tasks, such as concept order recovering @xcite , entity category prediction @xcite . however, none of existing work has explored using dictionary to enhance language model pre-training. ## proposed method in this section, we introduce the details of our model dict-bert. we first describe the notations and how to incorporate rare word definitions as a part of input. then we detail the two novel selfsupervised pre-training objectives. finally, we introduce the knowledge attention during fine-tuning. ## conclusions in this work, we leveraged rare word definitions in english dictionary to improve language model pre-training. when encountering a rare word in the input text during pre-training, we fetched its definition from wiktionary and appended it to the end of the input text. in order to make better interactions between the input text and rare word definitions, we proposed two novel self-supervised training tasks to help language model learn better representations for rare words. experimental on the glue benchmark and eight specialized domain datasets demonstrated that our method significantly improved the understanding of rare words and boosted model performance on various downstream tasks. in this work, we did not pay particular attention to the polysemy issue, because for most rare words, they often only have one meaning in the dictionary. for example, as mentioned in section 4.2, there are a total of 252,581 rare words in the bert pretraining corpus (i.e., bookcorpus and wikipedia). among them, 228,658 (90.52%) words only have one meaning, and 21,721 (8.6%) words have two meanings. so, words less than three meanings account for more than 99% of words. to deal with the words having more than one meaning, we use the definition of their first etymology, i.e., the most commonly used meaning, in the wiktionary. rare words during fine-tuning. one important advantage of dict-bert is that it can dynamically adjust the vocabulary of rare words, obtain and represent their definitions in a dictionary in a plugand-play manner. as different domain datasets usually follow different word distributions, the pretrained language model may still encounter many rare words when fine-tuned on the downstream tasks. to enhance the rare word representations during fine-tuning process, when encountering a rare word in the input text sequence, we append its definition to the end of input text sequence, just like what we did in pre-training.