Interpretable Entity Representations through Large-Scale Typing (2020)

Abstract: In standard methodology for natural language processing, entities in text are typically embedded in dense vector spaces with pre-trained models. The embeddings produced this way are effective when fed into downstream models, but they require end-task fine-tuning and are fundamentally difficult to interpret. In this paper, we present an approach to creating entity representations that are human readable and achieve high performance on entity-related tasks out of the box. Our representations are vectors whose values correspond to posterior probabilities over fine-grained entity types, indicating the confidence of a typing model’s decision that the entity belongs to the corresponding type. We obtain these representations using a fine-grained entity typing model, trained either on supervised ultra-fine entity typing data (Choi et al. 2018) or distantly-supervised examples from Wikipedia. On entity probing tasks involving recognizing entity identity, our embeddings used in parameter-free downstream models achieve competitive performance with ELMo- and BERT-based embeddings in trained models. We also show that it is possible to reduce the size of our type set in a learning-based way for particular domains. Finally, we show that these embeddings can be post-hoc modified through a small number of rules to incorporate domain knowledge and improve performance.

## introduction in typical neural nlp systems, entities are embedded in the same space as other words either in context-independent @xcite or in context-dependent ways @xcite . such approaches are powerful: pre-trained language models implicitly learn factual knowledge about those entities @xcite @xcite and these representations can be grounded in structured and human-curated knowledge bases @xcite @xcite @xcite @xcite . however, these embeddings do not explicitly maintain representations of this knowledge, and dense entity representations are not directly interpretable. knowledge probing tasks can be used to measure lms' factual knowledge @xcite , but designing the right probing task is another hard problem @xcite , particularly if the probes are parameter-rich @xcite . in this work, we explore a set of interpretable entity representations that are simultaneously human and machine readable. the key idea of this approach is to use fine-grained entity typing models with large type inventories @xcite @xcite . given an entity mention and context words, our typing model outputs a highdimensional vector whose values are associated with predefined fine-grained entity types. each value ranges between 0 and 1, corresponding to the confidence of the model's decision that the entity has the property given by the corresponding type. we use pre-trained transformer-based entity typing models, trained either on a supervised entity typing dataset @xcite or on a distantlysupervised dataset derived from wikipedia categories @xcite . the type vectors from these models, which contain tens of thousands of types, are then used as contextualized entity embeddings in downstream tasks. past work has shown that such type-driven representations are useful for entity linking @xcite ; we improve the quality of these representations, broaden the scope of where they can be applied, and show techniques to extend and debug them by exploiting their interpretable na-larry fine o'brien served for the match in the third set before washington came charging back. (3) an entity embedding vector consists of probabilities for corresponding types. (4) we can reduce the size of the type set for a particular downstream task in a learning-based way (section 5.2). ( 5 ) we can also incorporate domain knowledge via rules to modify bad type probabilities and improve model performance (section 5.3). ## interpretable entity representations our approach for producing entity representations is shown in figure 1 . for an entity mention in context, 1 we compute a vector of probabilities, each 1 our approach can also embed knowledge base entities, as discussed in section 4. we define @xmath0, ..., w n ) to denote a sequence of context words, and @xmath1, ..., w j ) to denote an entity mention span in s. the input word sequence s could be naturally co-occurring context words for the mention, or descriptive words such as might be found in a definition. the output variable is a vector @xmath2, 1] |t | whose values are probabilities corresponding to fine-grained entity types t . these entity types are predefined and static, so their meanings are identical for all entities. our goal here is to learn parameters θ of a function f θ that maps the mention m and its context s to a vector t, which capture salient features of the entity mention with the context. we learn the parameters θ in a supervised manner. we use a labeled dataset @xmath3, s, t * ) (1) , ..., (m, s, t * ) (k) } to train an entity typing model. the gold labels t * are obtained by manual annotation or distant-supervision techniques @xcite . we select a predefined types t from modified wikipedia categories, or we use an existing type set such as ufet @xcite (discussed in section 4). mention and contex encoder (bert) [cls] cls dot sig. mention context [sep] [sep] entity embedding dot product element-wise sigmoid type embeddings ## embedding model our model f θ to produce these embeddings is shown in figure 2 : it takes as input the mention m and its context s and predicts probabilities for predefined entity types t . this is a transformer-based typing model following the bert model presented in @xcite . first, a transformerbased encoder @xcite maps the input variables, m and s, to an intermediate vector repre-sentation. a type embedding layer then projects the intermediate representation to a vector whose dimensions correspond to the entity types t . finally, we apply a sigmoid function on each real-valued score in the vector to obtain the posterior probabilities that form our entity representation t (top of the figure). mention and context encoder we use pretrained bert foot_0 @xcite for the mention and context encoder. this bert-based encoder accepts as input a token sequence formatted as @xmath4, where the mention m and context s are chunked into wordpiece tokens @xcite . we encode the whole sequence using bert and use the hidden vector at the [cls] token as the mention and context representation: h [cls] = bertencoder(x). type embeddings this output layer is a single linear layer whose parameter matrix can be viewed as a matrix of type embeddings @xmath5, where d is the dimension of the mention and context representation h @xcite . we obtain the output probabilities t by multiplying e by h [cls] , followed by an element-wise sigmoid function: one assumption in our approach is that the model's output probabilities are a meaningful measure of class membership. past work @xcite has observed that this is true for other models involving bert variants. training following @xcite , the loss is a sum of binary cross-entropy losses over all entity types t over all training examples d. that is, we treat each type prediction for each example as an independent binary decision, with shared parameters in the bert encoder. ## training data to train our entity typing model, we need labeled examples consisting of (m, s, t * ) triples. although there are labeled typing datasets such as ufet @xcite , getting large amounts of manually labeled data is expensive. moreover, the ufet dataset contains instances of entities in context, so it is suitable for training models for contextual embeddings, but it doesn't have examples of definitions for descriptive embeddings (following the terminology of @xcite ). therefore, we additionally use two distantly labeled entity typing datasets derived from wikipedia. we leverage past work in using types derived from wikipedia categories @xcite , which contain type information and are widely annotated across wikipedia articles. we select the appropriate dataset for each setting depending on task-specific requirements (see section 6). for all datasets, we compute entity typing macro f1 using development examples (1k) to check model convergence. wiki-context we collect a set of occurrences of typed entity mentions using hyperlinks in wikipedia. given a sentence with a hyperlink, we use the hyperlink as an entity mention m, the sentence as a context sentence s, and the wiki categories of the destination page as the gold entity types t * . we use the preprocessing of @xcite to modify the type set: they introduce more general categories into the wikipedia category set by splitting existing complex categories. following their work, we filter the resulting set to keep the 60,000 most frequent types. scraping wikipedia yields 6m training examples that cover a wide range of entities and entity types. wiki-description following a similar paradigm as for wiki-context, we create description-focused training examples from wikipedia. 4 we use the same entity type set as the wiki-context dataset. we collect lead paragraphs from all wikipedia pages and filter to keep examples that contain at least 1 entity type in the 60k entity types. we use the wikipedia page title (usually boldfaced) in the lead paragraph as the entity mention m, and retain at most 100 words on either side to form the context s. the wiki categories of the same page would be the gold entity types t * . we obtain 2m training examples after filtering. the size of entity type set is 60k. ufet this ultra-fine entity typing dataset is created by @xcite . this dataset consists of 6k manually annotated examples. the entity mention spans could be named entities, nominal expressions, and pronouns while wiki-based datasets mostly provide named entity mention spans. we use 5.5k examples for training and 500 examples for validation. note that because our goal in this work is downstream task performance, we deviate from the standard train/dev/test splits of 2k/2k/2k in favor of higher performance. ## experimental setup we evaluate the "out-of-the-box" quality of our entity representations and baselines on two entity probing tasks as discussed in the previous section. ## conclusion in this work, we presented an approach to creating interpretable entity representations that are human readable and achieve high performance on entity-related tasks out of the box. we show that it is possible to reduce the size of our type set in a learning-based way for particular domains. in addition, these embeddings can be post-hoc modified through simple rules to incorporate domain knowledge and improve performance.