Learning from Sibling Mentions with Scalable Graph Inference in Fine-Grained Entity Typing (2022)

Abstract: In this paper, we firstly empirically find that existing models struggle to handle hard mentions due to their insufficient contexts, which consequently limits their overall typing performance. To this end, we propose to exploit sibling mentions for enhancing the mention representations. Specifically, we present two different metrics for sibling selection and employ an attentive graph neural network to aggregate information from sibling mentions. The proposed graph model is scalable in that unseen test mentions are allowed to be added as new nodes for inference. Exhaustive experiments demonstrate the effectiveness of our sibling learning strategy, where our model outperforms ten strong baselines. Moreover, our experiments indeed prove the superiority of sibling mentions in helping clarify the types for hard mentions.

## introduction fine-grained entity typing (fget) aims to assign one or more fine-grained types to an entity mention given its context. for instance, the mention steve jobs should be classified as person and entrepreneur under the context "steve jobs cofounded apple ...". many tasks have witnessed the importance of fget, such as relation extraction @xcite @xcite , entity linking @xcite , and other tasks @xcite @xcite . it is challenging to learn effective representations for contextualized mentions 1 in many information extraction tasks @xcite , espe-cially in fget, since the representations are required to well distinguish fine-grained types with similar but different semantics. noticeable efforts have been made to learn type-aware representations for mentions @xcite @xcite @xcite @xcite @xcite and significant progress has been achieved. however, as supported by our empirical experiments, existing sota models perform poorly on a certain number of "hard" mentions, leading to limited overall performance. the main reasons are the following challenges. first, the structure of some contexts surrounding the hard mentions are inherently too complex to extract informative features for identifying entity types. second, the contexts of some hard mentions are ambiguous and thus it is insufficient to handle these mentions by learning from their contexts only. in this paper, we show that representation learning of such hard mentions can be well handled by learning informative knowledge from their sibling mentions. sibling mentions refer to the mentions that potentially share the same or semantically similar types (e.g., country and nation) with the target mention. we illustrate how sibling mentions assist classifying hard mentions in figure 1 . intuitively, the context of the target mention sharp is ambiguous and insufficient for inferring the ground-truth types (i.e., organization, company, and tech company), since both a person and a company can "sign a deal with qualcomm". fortunately, the sibling mentions provide rich information that works as an important supplement for the target mention sharp. by aggregating the supplementary information from siblings, it is promising to learn effective representations with less ambiguity for hard target mentions. to utilize sibling mentions, we model fget as a heterogeneous graph learning problem. the graph is composed of two kinds of nodes, namely the mentions and the types. besides, there are three kinds of edges connecting the nodes as shown in the left part of figure 1 , which represent the sibling relationship between mentions, the hierarchical relationship between types, and the islabel relationship between mentions and types, respectively. the sibling relationship is considered as the most important part in our graph. for detecting it, we propose two similarity metrics, based on which we design an effective sibling selection algorithm. upon the constructed nodes and edges, we employ an attentive graph neural module to learn their representations. particularly, the representations of mention nodes are enriched by aggregating the information from their sibling and type neighbors. it is also noteworthy that, during inference stage, our graph model is scalable to include the unseen test mentions as new nodes and connect them with their existing sibling mention nodes in the graph to derive reliable representations for predictions. extensive experiments are conducted to verify the effectiveness of our model. our experimental results demonstrate that our model outperforms several strong baselines on the standard test sets with a large margin. moreover, our model is indeed able to well handle hard mentions with the help from sibling mentions. we summarize our contributions as follows: we design two effective metrics for sibling detection and propose a scalable graph model to take advantages of sibling mentions. ## methodology overview given a mention m and the type set y, an fget model needs to predict the correct types y m (y m ⊂ y) for m based on its context. in this paper, mention representations are learned and refined with the help of sibling mentions and ground-truth types. to achieve it, we propose a heterogeneous graph model enhanced by sibling mentions for fget, as illustrated in figure 1 . first, a mention-type graph g is constructed from training samples (sec 3). then, the features for mentions and types are learned by an attentive graph neural module upon g (sec 4). during inference stage (sec 5), we add test mentions into graph g by connecting them to their sibling mentions in the training set. by aggregating sibling information, the representations of test mentions are generated and used for type prediction. ## scalable testing in the following, we describe the prediction process for test mentions. step 1: given a batch of n test mentions, we first obtain their sibling mentions. to be specific, for each test mention m t , we select a candidate set v m from the training mentions v m . then, the cosine similarity is computed between m t and each m i in v m , based on which the top k mentions are selected as siblings (see sec 3.2). step 2: we add the test mentions as nodes into the mention-type graph g, where the test mentions are connected to their sibling mentions selected at step 1. note that, in the new graph, test mentions have no type neighbors since their ground-truth types are not available. besides, there is no edge between any two test mentions in the new graph. step 3: following eq.( 6 ), the representations of test mentions {m t } are updated by aggregating the embeddings for their sibling mentions. note that y mt is empty, so no information from the groundtruth types are involved. through layers of updates, the final representations {m (l) t } are obtained. ## experiments 6.1 datasets we evaluate the proposed model on two widelyused datasets: ontonotes and bbn. ontonotes the original ontonotes dataset is annotated by distant supervision @xcite . the training, development and test samples in ontonotes are about 251k, 2k and 9k, respectively. we also conduct experiments on the augmented version foot_2 @xcite with 793k training samples foot_3 . the above two versions share the same test set and development set, as well as the same type set of size 89. bbn different from ontonotes, bbn is manually annotated @xcite . the training, development and test set contain about 84k, 2k and 14k samples respectively, and the type set contains 47 type in total. ## related work fget is an important task for the downstream nlp tasks and many efforts have been make in improving its performance @xcite . early works in fget @xcite ) mainly focus target mention: [gm officials] told workers late last week of the following moves: production of full-sized vans will be consolidated into a single plant in flint, mich. ground-truth: /person prediction from our model: /person, /person/title sibling 1: "it's been a steadily improving relationship.", says the [president]. ground-truth: /person, /person/title sibling 2: apart from those two actions, mr.sikes and the three other [commissioners] said they expect to reexamine how at&t is regulated since competition has increased. ground-truth: /person, /person/title sibling 3: hud secretary [jack kemp] backed an unsuccessful effort to strike such language last week, but received little support from the white house • • • ground-truth: /person, /person/artist, /person/artist/actor, /person/artist/author, /person/political_figure on feature extraction for mentions, which do not consider label noise introduced by distant supervision @xcite @xcite . recent years have witnessed an increasing number of researchers being dedicated to data denoising. a popular solution @xcite @xcite is to design loss functions for the clean and noisy parts of the training data separately. nevertheless, @xcite proposes an automatic relabeling framework to estimate the pseudo-truth label distribution of each sample, which treats the noisy and clean data uniformly. besides, @xcite groups mentions of the same type into a compact cluster to improve the robustness of the model. @xcite refines noisy representations by corpus-level contextual clues. @xcite introduces two additional models to delete the samples that are too noisy to be useful, and repair noisy labels for the retained examples. in addition, there are some notable work which tries to build fget with limited resources @xcite . modeling the type hierarchy is another important topic in fget. prior solutions @xcite introduce a one-hot matrix to encode the hierarchy. @xcite proposes a hierarchyaware loss function. recently, graph-based methods have been proven to be powerful in many nlp tasks @xcite @xcite . using graphs to model the type hierarchy in fget is a natural idea. @xcite models the potential type correlations for in-knowledge-base entities via hierarchical multi graph convolutional networks (gcns). further, @xcite extends gcns to a vast number of free-form types. chen et al. (2020 ) designs a multi-level learning-to-rank loss to leverage hierarchical information. recently, onoe et al. (2021) models the mention and type representations in a box space instead of the traditional vector space. ## conclusion in this paper, we firstly point out that sota typing models suffer from a bottleneck issue, i.e., they perform poorly on a certain number of hard mentions, which leads to their limited overall performance. to this end, we propose to exploit sibling information for mention representation learning and define two metrics for detecting sibling relationship between mentions. further, we model sibling learning as a graph learning problem. our model is scalable in that, once trained, it can generate sibling-aware representations for previously unseen mentions efficiently during inference stage. extensive experiments show that the proposed model indeed handles hard mentions well and thereby yields better overall performance than sota baseline models.