E - BERT : Efficient-Yet-Effective Entity Embeddings for BERT (2020)

Abstract: We present a novel way of injecting factual knowledge about entities into the pretrained BERT model (Devlin et al., 2019): We align Wikipedia2Vec entity vectors (Yamada et al., 2016) with BERT’s native wordpiece vector space and use the aligned entity vectors as if they were wordpiece vectors. The resulting entity-enhanced version of BERT (called E-BERT) is similar in spirit to ERNIE (Zhang et al., 2019) and KnowBert (Peters et al., 2019), but it requires no expensive further pre-training of the BERT encoder. We evaluate E-BERT on unsupervised question answering (QA), supervised relation classification (RC) and entity linking (EL). On all three tasks, E-BERT outperforms BERT and other baselines. We also show quantitatively that the original BERT model is overly reliant on the surface form of entity names (e.g., guessing that someone with an Italian-sounding name speaks Italian), and that E-BERT mitigates this problem.

## introduction bert @xcite and its successors (e.g., @xcite ; @xcite ; @xcite ) continue to achieve state of the art performance on various nlp tasks. recently, there has been interest in enhancing bert with factual knowledge about entities @xcite . to this end, we introduce e-bert: we align wikipedia2vec entity vectors @xcite ) with bert's wordpiece vector space (section 3.1) and feed the aligned vectors into bert as if they were wordpiece vectors (section 3.2). importantly, we do not make any changes to the bert encoder itself, and we do no additional pretraining. this stands in contrast to previous entity-enhanced versions of bert, such as ernie or knowbert, which require additional encoder pretraining. in section 4, we evaluate our approach on lama @xcite , a recent unsupervised qa benchmark for pretrained language models (lms). we set a new state of the art on lama, with improvements over original bert, ernie and knowbert. we also find that the original bert model is overly reliant on the surface form of entity names, e.g., it predicts that a person with an italian-sounding name speaks italian, regardless of whether this is factually correct. to quantify this effect, we create lama-uhn (unhelpfulnames), a subset of lama where questions with overly helpful entity names were deleted (section 4.4). in section 5, we show how to apply e-bert to two entity-centric downstream tasks: relation classification (section 5.1) and entity linking (section 5.2). on the former task, we feed aligned entity vectors as inputs, on the latter, they serve as inputs and outputs. in both cases, e-bert outperforms original bert and other baselines. ## becomes the native language of jean marais / jean mara ##is is @xcite . the entity id (bold) is embedded by e e-bert and all wordpieces (italics) are embedded by e bert (see figure 1 ). after the embedding operation, the sequence of vectors is combined with position and segment embeddings and fed into f bert , just like any normal sequence of wordpiece vectors. e-bert-concat is comparable to ernie or knowbert, which also represent entities as a combination of surface form (wordpieces) and entity vectors. but in contrast to ernie and knowbert, we do not change or further pretrain the bert encoder itself. ## downstream tasks we now demonstrate how to use e-bert on two downstream tasks: relation classification (rc) and entity linking (el). in both experiments, we keep the embedding layer (e bert and/or e e-bert ) fixed but finetune all other encoder parameters. we use the bert base architecture throughout. ## conclusion we introduced e-bert, an efficient yet effective way of injecting factual knowledge about entities into the bert pretrained language model. we showed how to align wikipedia2vec entity vectors with bert's wordpiece vector space, and how to feed the aligned vectors into bert as if they were wordpiece vectors. in doing so, we made no changes to the bert encoder itself. this stands in contrast to other entity-enhanced versions of bert, such as ernie or knowbert, which add encoder layers and require expensive further pretraining. we set a new state of the art on lama, a recent unsupervised qa benchmark. furthermore, we presented evidence that the original bert model sometimes relies on the surface forms of entity names (rather than "true" factual knowledge) for this task. to quantify this effect, we introduced lama-uhn, a subset of lama where questions with helpful entity names are deleted. we also showed how to apply e-bert to two supervised tasks: relation classification and entity linking. on both tasks, we achieve results competitive with or better than existing baselines, but without the need for expensive pretraining.