Overcoming Poor Word Embeddings with Word Definitions (2021)

Abstract: Modern natural language understanding models depend on pretrained subword embeddings, but applications may need to reason about words that were never or rarely seen during pretraining. We show that examples that depend critically on a rarer word are more challenging for natural language inference models. Then we explore how a model could learn to use definitions, provided in natural text, to overcome this handicap. Our model’s understanding of a definition is usually weaker than a well-modeled word embedding, but it recovers most of the performance gap from using a completely untrained word.

## introduction the reliance of natural language understanding models on the information in pre-trained word embeddings limits these models from being applied reliably to rare words or technical vocabulary. to overcome this vulnerability, a model must be able to compensate for a poorly modeled word embedding with background knowledge to complete the required task. for example, a natural language inference (nli) model based on pre-2020 word embeddings may not be able to deduce from "jack has covid" that "jack is sick." by providing the definition, "covid is a respiratory disease," we want to assist this classification. we describe a general procedure for enhancing a classification model such as natural language inference (nli) or sentiment classification, to perform the same task on sequences including poorly modeled words using definitions of those words. from the training set t of the original model, we construct an augmented training set t for a model that may accept the same token sequence optionally concatenated with a word definition. in the case of nli, where there are two token sequences, the definition is concatenated to the premise sequence. because t has the same form as t , a model accepting the augmented information may be trained in the same way as the original model. because there are not enough truly untrained words like "covid" in natural examples, we probe performance by scrambling real words so that their word embedding becomes useless, and supplying definitions. our method recovers most of the performance lost by scrambling. moreover, the proposed technique removes biases in more ad hoc solutions like adding definitions to examples without special training. ## related work we focus on nli because it depends more deeply on word meaning than sentiment or topic classification tasks. @xcite pioneered the addition of background information to an nli model's classification on a per-example basis, augmenting a sequence of token embeddings with features encoding wordnet relations between pairs of words, to achieve a 0.6% improvement on the snli @xcite task. besides this explicit reasoning approach, implicit reasoning over background knowledge can be achieved if one updates the base model itself with background information. @xcite follows this approach to add information from conceptnet @xcite and the open mind common sense corpus @xcite through a fine-tuned adapter added to a pretrained language model, achieving better performance on subsets of nli examples that are known to require world knowledge. @xcite explore the interplay between explicitly added knowledge and implicitly stored knowledge on artificially constructed nli problems that require counting or relations from a taxonomy. in the above works, explicit background infor-mation comes from a taxonomy or knowledge base. only a few studies have worked with definition text directly, and not in the context of nli. @xcite used definitions to create embeddings for better performance on word similarity tasks, compared to word2vec @xcite and fasttext @xcite while maintaining performance on text classification. their work pushes together embeddings of words that co-occur in each other's definitions. recently, @xcite used definitions to remove biases from pretrained word embeddings while maintaining coreference resolution accuracy. in contrast, our work reasons with natural language definitions without forming a new embedding, allowing attention between a definition and the rest of an example. alternatively, schick and schütze (2020) improved classification using rare words by collecting and attending to all of the contexts in which they occur in bookcorpus @xcite combined with westbury wikipedia corpus. foot_0 like the methods above that use definitions, this method constructs a substitute or supplementary embedding for a rare word. ## conclusion this work shows how a model's training may be enhanced to support reasoning with definitions in natural text, to handle cases where word embeddings are not useful. our method forces the definitions to be considered and avoids the application of biases independent of the definition. using the approach, entailment examples like "jack has covid / jack is sick" that are misclassified by an xlnet trained on normal snli are correctly recognized as entailment when a definition "covid is a respiratory disease" is added. methods that can leverage definitions without losing the advantage of partially useful word embeddings are still needed. in an application, it also will be necessary to select the words that would benefit from definitions, and to make a model that can accept multiple definitions.