Metadata Shaping: A Simple Approach for Knowledge-Enhanced Language Models (2022)

Abstract: Popular language models (LMs) struggle to capture knowledge about rare tail facts and entities. Since widely used systems such as search and personal-assistants must support the long tail of entities that users ask about, there has been significant effort towards enhancing these base LMs with factual knowledge. We observe proposed methods typically start with a base LM and data that has been annotated with entity metadata, then change the model, by modifying the architecture or introducing auxiliary loss terms to better capture entity knowledge. In this work, we question this typical process and ask to what extent can we match the quality of model modifications, with a simple alternative: using a base LM and only changing the data. We propose metadata shaping, a method which inserts substrings corresponding to the readily available entity metadata, e.g. types and descriptions, into examples at train and inference time based on mutual information. Despite its simplicity, metadata shaping is quite effective. On standard evaluation benchmarks for knowledge-enhanced LMs, the method exceeds the base-LM baseline by an average of 4.3 F1 points and achieves state-of-the-art results. We further show the gains are on average 4.4x larger for the slice of examples containing tail vs. popular entities.

## introduction recent language models (lms) such as bert @xcite and its successors are remarkable at memorizing knowledge seen frequently during training, however performance degrades over the long tail of rare facts. given the importance of factual knowledge for tasks such as questionanswering, search, and personal assistants @xcite @xcite , there has been significant interest in injecting these base lms with factual knowledge about entities @xcite . in this work, we work we propose a simple and effective approach for enhancing lms with knowledge, called metadata shaping. existing methods to capture entity knowledge more reliably, typically use the following steps: first annotating natural language text with entity metadata, and next modifying the base lm model to learn from the tagged data. entity metadata is obtained by linking substrings of text to entries in a knowledge base such as wikidata, which stores entity ids, types, descriptions, and relations. model modifications include introducing continuous vector representations for entities or auxiliary objectives @xcite @xcite @xcite @xcite . other methods combine multiple learned modules, which are each specialized to handle fine-grained reasoning patterns or subsets of the data distribution @xcite . these knowledge-aware lms have led to impressive gains compared to base lms on entityrich tasks. that said, the new architectures are often designed by human experts, costly to pretrain and optimize, and require additional training as new entities appear. further, these lms may not use the collected entity metadata effectively -wikidata alone holds over ∼ 100m unique entities, however many of these entities fall under similar categories, e.g., "politician" entities. intuitively, if unseen entities encountered during inference share metadata with entities observed during training, the lm trained with this information may be able to better reason about the new entities using patterns learned from similar seen entities. however, the knowledge-aware lms learn from individual entity occurrences rather than learning these shared reasoning patterns. implicitly learning entity similarities for 100m entities may be challenging since 89% of the wikidata entities do not appear in wikipedia, a popular source of unstructhe fewrel benchmark involves identifying the relation between a subject and object string. the above subject and object are unseen in the fewrel training data and the tuned base lm reflects low attention weights on those words. a base lm trained with shaped data reflects high attention weights on useful metadata words such as "politician". weights are shown for words which are not stop-words, punctuation, or special-tokens. simple and effective method we propose metadata shaping and demonstrate its effectiveness on standard benchmarks that are used to evaluate knowledge-aware lms. metadata shaping, with simply an off-the-shelf base lm, exceeds the base lm trained on unshaped data by by an average of 4.3 f1 points and is competitive to state-of-theart methods, which do modify the lm. metadata shaping thus enables re-using well-studied and optimized base lms (e.g., @xcite ). ## experiments in this section, we demonstrate that metadata shaping is general and effective. ## analysis here we study the following key questions for effectively using metadata shaping: section 4.1 what are the roles of different varieties of metadata? section 4.2 what are the effects of metadata shaping on slices concerning tail versus popular entities? 4.1 framework: role of metadata types metadata effects class-discriminative metadata correlates with reduced model uncertainty. high quality metadata, as found in wikidata, results in improved classification performance. to investigate the effects of metadata on model uncertainty, we compute the entropy of pφ softmax scores over the output classes as a measure of uncertainty, and compute the average across test set examples. lower uncertainty is correlated with improved classification f1 (see figure 2 (left) ). we compute pmi scores for inserted metadata tokens as a measure of class-discriminativeness. we rank individual tokens k by pmi(y, k) (for task classes y), computed over the training dataset. on fewrel, for test examples containing a top-20 pmi word for the gold class, the accuracy is 27.6% higher when compared to the slice with no top-20 pmi words for the class. notably, 74.1% more examples contain a top-20 pmi word for their class when pmi is computed on shaped data vs. unshaped training data. metadata selection simple information theoretic heuristics are effective for selecting metadata, despite the complexity of the underlying contextual embeddings. we apply algorithm 1, which ranks metadata tags by their provided information gain, to select metadata tags for the tasks. given x i with a set benchmark strategy test f1 fewrel bert-base 84.9 random 87.2 ±0.8 popular 87.9 ±0.1 low rank 87.8 ±0.4 high rank 88.9 ±0.6 openentity bert-base 73.2 random 74.3 ±0.7 popular 74.5 ±0.4 low rank 74.1 ±0.4 high rank 74.8 ±0.1 tacred bert-base 72.0 random 73.8 ±1.6 popular 73.6 ±0.9 low rank 73.3 ±1.0 high rank 74.7 ±0.5 m (x i ) m (x i ) m (x i ) of metadata tags, our goal is to select n to use for shaping. we compare four selection approaches: using the highest ("high rank") and lowest ("low rank") ranked tokens by algorithm 1, random metadata from m (x i ) m (x i ) m (x i ) ("random"), and the most popular metadata tokens across the union of m (x i ) m (x i ) m (x i ), ∀x @xmath3, selecting the same number of metadata tags per example for each baseline. we observe that high rank consistently gives the best performance, evaluated over three seeds, and note that even random yields decent performance vs. the bert-baseline, indicating the simplicity of the method ( considering the distribution of selected category tokens under each scheme, the kl-divergence between the categories selected by low rank vs. popular is 0.2 (fewrel), 4.6 (openentity), while the kl-divergence between high rank vs. popular is 2.8 (fewrel), 2.4 (openentity). popular tokens are not simply the best candidates; instead, algorithm 1 selects discriminative metadata. for openentity, metadata are relatively sparse, so categories appear less frequently in general and it is reasonable that coarse-grained types have more overlap with high rank. for e.g., "business" is in the top-10 most frequent types under high rank, while "non-profit" (occurs in 2 train examples) is in the top-10 most frequent types for low rank. metadata tokens overall occur more frequently in fewrel (see task agnostic metadata effects using metadata correlates with reduced task-specific lm uncertainty. we observe shaping also correlates with reduced lm uncertainty in a task-agnostic way. we perform additional masked language modeling (mlm) over the shaped task training data using an off-the-shelf bert-mlm model to learn model pθ . we minimize the following loss function and evaluate the model perplexity on the task test data: metadata noise we hypothesize that noisier metadata can provide implicit regularization. noise arises from varied word choice, word order, and blank noising. feature noising @xcite is effective to prevent overfitting and while regularization is typically applied directly to model parameters, @xcite ; @xcite regularize through the data. we hypothesize that using metadata with diverse word choice and order (e.g., entity descriptions) and blank noising (e.g., by masking metadata tokens), can help reduce overfitting, and we provide initial empirical results in appendix b. ## conclusion we propose metadata shaping to improve tail performance. the method is a simple and general baseline that is competitive with sota approaches for entity-rich tasks. we empirically show that the method improves tail performance and explain why metadata can reduce generalization error. while this work focused on entity-rich tasks, metadata shaping is not limited to this setting. broadly, we hope this work motivates further research on understanding how to effectively program lms with useful and readily available side information. while modifying the lm architecture to encode the information has been a popular approach, modifying the data is a simple and effective alternative. quentin lhoest, and alexander m. rush. 2020. transformers: state-of-the-art natural language processing. in proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations (emnlp), pages 38-45, online. association for computational linguistics. qizhe xie, zihang dai, eduard hovy, minh-thang luong, and quoc v. le. 2020. unsupervised data augmentation for consistency training. 34th conference on neural information processing systems (neurips). zhang xie, sida i. wang, jiwei li, daniel levy, aiming nie, dan jurafsky, and andrew y. ng. 2017. data noising as smoothing in neural network language models. international conference on learning representations (iclr). wenhan xiong, jingfei du, william yang wang, and veselin stoyanov. 2020. pretrained encyclopedia: weakly supervised knowledge-pretrained language model. in international conference on learning representations (iclr). ikuya yamada, akari asai, hiroyuki shindo, hideaki takeda, and yuji matsumoto. 2020. luke: deep contextualized entity representations with entityaware self-attention. proceedings of the conference on empirical methods in natural language processing (emnlp). tianyi zhang and tatsunori hashimoto. 2021. on the inductive bias of masked language modeling: from statistical to syntactic dependencies. proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies (naacl-hlt). yuhao zhang, victor zhong, danqi chen, gabor angeli, and christopher d. manning. 2017. position aware attention and supervised data improve slot filling. in empirical methods in natural language processing (emnlp). zhengyan zhang, xu han, zhiyuan liu, xin jiang, maosong sun, and qun liu. 2019. ernie: enhanced language representation with informative entities. proceedings of the 57th annual meeting of the association for computational linguistics (acl). xiangxin zhu, dragomir anguelov, and deva ramanan. 2014. capturing long-tail distributions of object subcategories. in 2014 ieee conference on computer vision and pattern recognition (cvpr).