A Self-Supervised Integration Method of Pretrained Language Models and Word Definitions (2023)

Abstract: We investigate the representation of pretrained language models and humans, using the idea of word definition modeling–how well a word is represented by its definition, and vice versa. Our analysis shows that a word representation in pretrained language models does not successfully map its human-written definition and its usage in example sentences. We then present a simple method DefBERT that integrates pretrained models with word semantics in dictionaries. We show its benefits on newly-proposed tasks of definition ranking and definition sense disambiguation. Furthermore, we present the results on standard word similarity tasks and short text classification tasks where models are required to encode semantics with only a few words. The results demonstrate the effectiveness of integrating word definitions and pretrained language models.

## introduction a word embedding vector maps a word into a fixeddimensional vector as a distributed representation. the word vectors are trained by looking at their context words and aggregating their representations in supervised ways @xcite or unsupervised ways @xcite . more recently, the representations have been learned as a form of pretrained language models @xcite . the huge success of these pretrained language models on various nlp tasks is achieved by capturing a rich semantic representation of words from their context in huge data. on the other hand, for centuries, lexicographers and linguists have created dictionaries that contain general definitions of words and examples of their usage. with these sophisticated data, there have been many applications for nlp tasks (e.g., machine translation @xcite , semantic distances between word 'love' and its definitions: the closest definition to the word 'love': "several." (definition of word 'number') @xcite in this work, we study the difference between machine-learned definitions and human-written definitions. furthermore, it is important to make general and self-indicated embeddings. for example, if we do not have pooling layers to the pretrained embedding, we need additional training data to fine-tune the pooling layer and the pretrained model. on the other hand, if we can do the same task by using the pretrained model only (without fine-tuning), this means good generalization. lastly, some researchers believe that target word token representation is better than @xcite token when the input text is short. however, we do not know 'what the short text is' or 'when the model gets short text as inputs.' thus, the fact that we can use @xcite token for a single word or short text is beneficial in that we do not need to consider the input length. to do so, we attempt to inject worddefinition-example (its usage) information into the model. to overcome the deficiency and get such a generalized model, we propose a new joint representation that combines the human-written word definition with its usage example in a dictionary entry. we show the effectiveness of this new representation on several downstream tasks. the main contributions are: more recently, glossbert @xcite used definitions for disambiguation tasks, but the approach needs context-gloss pairs and a classifier even at inference. in this work, we attempt to build a generalized model which does not require additional classifiers. definition modeling. the definition modeling task was proposed by @xcite that generates a word definition from the word's embedding. the authors considered the definition modeling as a special case of language modeling and used it for word embedding evaluation. however, @xcite found that the prior definition modeling tasks could not resolve word disambiguation because it is conditioned on only a single word. to address the issue, they also extended @xcite 's model to process context. @xcite investigated whether contextual representations can capture word definitions. unlike the prior works on definition modeling, they suggested a general framework that maps the contextualized representation into a definition embedding space and then selects top-n closest definitions. this retrieval-based approach can resolve the problems in the generative approach of definition modeling, such as the difficulty in evaluation. the major differences between the prior works and our study are as follows: first, we compare representations from pretrained language models and definitions from a lexical dictionary at embeddinglevel. second, we use word-definition pairs and definition-example pairs from the dictionary. the use of words in a sentence is similar to glossbert, but its objective is not to make definition-injected chang and chen (2019) oxford+ (ours) # words (w) 31,889 30,533 # definition (def) 79,105 93,227 # examples (exam) 707,001 1,167,055 avg./max. # def by word 10.6/65 10.5/51 avg./max. # exam by def 17.8/46 18.0/85 sense order n y ## preliminary analyses the central motivation behind our analysis is to check whether a word representation in pretrained language models (in this work, bert) can indicate the representation of its definition and vice versa. ## conclusion and further discussion we present a novel way of combining pretrained contextualized representations and human-written definitions from a dictionary. we first collect definitions and examples from an online dictionary oxford+. our analyses with the dictionary show that bert's representations do not incorporate human-written definitions. motivated by the findings, we develop a new representation defbert, by constraining bert to human-written definitions in the dictionary. in the experiments, we first proposed definition ranking (defrank) and sense disambiguation tasks (senserank) and defbert outperforms other baselines. we also presented the effectiveness of defbert in downstream tasks: word similarity benchmark and short text classification tasks. one of the contributions of this paper is to make researchers revisit the old and traditional resource, dictionaries. while resources, including synonyms, antonyms, and other relations, are widely used to improve models as a constraint, dictionaries are less frequently used. however, the dictionary is the basic form of word semantics and is a relatively objective resource compared to relational resources. furthermore, word-related resources are hard to align with pretrained language models because the weights are dynamic according to contexts. therefore, pouring resources can occur catastrophic forgetting that the information previously trained disappears. on this problem, we suggest a potential approach to enhance semantics on the pretrained weights, maintaining the nature of contextualized encoder.