Fine-grained Entity Typing without Knowledge Base (2021)

Abstract: Existing work on Fine-grained Entity Typing (FET) typically trains automatic models on the datasets obtained by using Knowledge Bases (KB) as distant supervision. However, the reliance on KB means this training setting can be hampered by the lack of or the incompleteness of the KB. To alleviate this limitation, we propose a novel setting for training FET models: FET without accessing any knowledge base. Under this setting, we propose a two-step framework to train FET models. In the first step, we automatically create pseudo data with fine-grained labels from a large unlabeled dataset. Then a neural network model is trained based on the pseudo data, either in an unsupervised way or using self-training under the weak guidance from a coarse-grained Named Entity Recognition (NER) model. Experimental results show that our method achieves competitive performance with respect to the models trained on the original KB-supervised datasets.

## introduction entity typing is a fundamental task in natural language processing. traditional entity typing research focuses on a limited number of entity types while recent studies strive for finer granularity. a major challenge in fine-grained entity typing (fet) is the absence of human-annotated data. to address this problem, a common practice is to seek distant supervision from knowledge bases. typically, the training data is obtained by linking entity mentions and drawing their types from knowledge bases. for example, @xcite utilize the information encoded in anchor links from wikipedia text along with the type information in freebase @xcite . after that, classification models are trained using the resultant fet dataset, as illustrated in the left part of figure 1 . despite its success, this framework has one limitation: the type ontology for training and testing is strongly restricted by the underlying kb. as a result, the application of the trained models under this framework can be hampered by the lack or the incompleteness of the kb @xcite . in real-world applications of fet, the ontology of the fine-grained types depends on the applications instead of the kb. typically, there may not exist a one-to-one mapping from the ontology of applications to that of the existing kb. for example, a game-related application may need to identify the named entities of some specific video games, such as the heroes and items in dota 2. however, there are no knowledge bases covering these named entities. the existing fet framework can not be used when the desired type topology is not compatible with any kb. worse than this, large-scale knowledge bases are even not available in some languages. as a result, the trained fet classifiers under this framework cannot be effectively applied to real-word applications. to lift this limitation, we study a novel fet setting, fine-grained entity typing without a knowledge base, which is a common setting in real-world applications. given a fine-grained type ontology defined by an application, we aim to assign a list of labels for each mention within an input sentence, yet without accessing any knowledge base. under this setting, we propose a novel fet framework by leveraging a large unlabeled dataset, as illustrated in the right part of figure 1 . our proposed framework disentangles the type ontology from the kb and thus the given ontology can be an arbitrary one, defined by a practical application that does not correspond to any kb or by a specific kb as the conventional fet setting. therefore, our proposed framework is more flexible than the previous one. our framework consists of two steps: creating pseudo data from an unlabeled dataset and (iteratively) training the fet model based on the pseudo dataset. in the first step, we aim to create a large pseudo dataset with fine-grained type labels. we propose an automatic method using hearst patterns @xcite to achieve this goal. there exists a difference between the pseudo data and the testing data in that each sentence in the pseudo data satisfies a (relaxed) hearst pattern while few testing data do. consequently, directly training a fet model on the pseudo data may limit its generalization ability. we thereby propose a self-training method with weak supervision from a coarse-grained ner model in the second step to alleviate this issue. we do not focus on designing novel models under the conventional fet setting in this work. actually, our proposed two-step framework does not set a limit on the backbone fet models and existing fet models can be freely migrated to our framework. experimental results show that without utilizing any kb in the conventional framework, our proposed framework achieves comparable results to the conventional framework using the same backbone fet model. main contributions of our work are: ## problem statement for fetw/okb suppose l is a fine-grained entity type ontology for a real-world application as illustrated in figure 2 , which consists of a set of formal types organized as hierarchical trees, and a dictionary used to explain each node in the trees. each node in a tree is a formal entity type l and each directed edge (l 1 , l 2 ) in the tree indicates that l 2 is a subtype of l 1 . each element in the dictionary consists of a formal type (i.e., a node) and a list of informal type names explaining the meaning of the formal type. figure 2 shows an example, where the type names of the type "/organization/company" is the list " @xcite " and the type names of "/disease" is the list "[disease, illnesses]". the generic fet task aims to predict the type set t whose element is in the given ontology l. note that t may be a set with one or more type labels, so this task is characterized as a multi-class multilabel classification task. formally, the fet task is modeled by a probabilistic model p (l | x, c). a common practice is to parameterize the probabilistic model by a binary classifier p θ (l | x, c), which indicates whether l is a type label of the mention x. during inference, the classifier outputs all the labels l satisfying the condition p θ (l | x, c) > 0.5. in the conventional framework, θ is trained on a distantly supervised dataset, which is dependent on a knowledge base. as a result, the trained classifier can only predict the types corresponding to the types in that knowledge base, limiting its application in practice. problem statement in this paper, we focus on fet under a more practical setting, i.e. finegrained entity typing without knowledge base (fetw/okb). that is, given an ontology l, we aim to infer the type set t for the mention x within a sentence c when there are no kbs available. relation to zero-shot fet suppose the type ontology l is divided into two disjoint subontology l 1 and l 2 , and there is a labeled training dataset where the labels are all from l 1 . zero-shot fet aims to make a prediction on testing mentions whose labels are within l 2 . a surge of efforts have been devoted to zero-shot fet @xcite @xcite , but all of them assume that training data is obtained from knowledge bases by distant supervision, similar to the conventional fet. therefore, the proposed fetw/okb is clearly different from zero-shot fet. road map generally, it is challenging to train θ under the fetw/okb setting. in the next sections, we propose a new framework to address fetw/okb. its key idea is a two-step approach: ## pseudo data from scratch we start from the ontology l given by a real-world application, and our purpose is to create a large pseudo dataset with fine-grained labels from massive unstructured data. to this end, we propose an automatic method to achieve this goal without manual labor. at a high level, the key idea of our method is illustrated in figure 2 . in the next, we describe the key steps within our method. figure 2: the process of generating pseudo data. it could be divided into several parts: extracting pairs from the unstructured corpus, clustering the extracted pairs, and matching corpus with clusters to get the pseudo data. see section 4 for a detailed explanation. ## self-training via weak guidance note that each sentence in the above pseudo data satisfies a relaxed hearst pattern while few in the test dataset do. in other words, there exists a gap between the pseudo dataset and the testing dataset.therefore, directly using the pseudo data for training may not result in a typing model with good generalization ability. to alleviate this problem, one may try to apply self-training, where the typing model is used to expand the pseudo dataset iteratively during training. however, the typing model can only do part of the job because it is trained to predict the entity type given an entity mention but not to predict the boundary of an entity mention given an input sentence. we use an off-the-shelf coarse-grained ner model to fill the gap and propose a weakly-guided self-training method. the basic idea is to use the coarse-grained ner model to identify the entity mentions in an input sentence, which are then fed into the fine-grained typing model for self-training. typically, a coarse-grained ner model can achieve both high precision and high recall on the task of identifying mention boundaries, so the missing mentions in the pseudo data can be found by it. besides the boundaries of entity mentions, the coarse-grained ner model also predicts a coarsegrained entity type for each mention, which is used as weak supervision in our method to improve the performance of the fine-grained typing model. specifically, the process of the weakly-guided self-training is shown in algorithm 1. given the pseudo data u l obtained from section 4, an unlabeled dataset u , the fine-grained typing model p e θ , and the off-the-shelf coarse-grained ner model p o , the training process is as follows: for each sentence c i in u : ## experiments the experiments are conducted on two publicly available fine-grained typing tasks: figer @xcite and ontonotes @xcite . we evaluate our model with the following three metrics: strict accuracy, loose macro, and loose micro f-1 scores following @xcite . ontonotes it is obtained from wikipedia in a similar way as figer dataset @xcite . that is, the automatically labeled data is created by distant supervision via freebase, and manually labeled data is used as the test set. however, figer is overly dependent on types in the knowledge base and its types include some noise because an entity in freebase may include multiple types but only a few of them can be inferrable from a given mention text. as a result, the automatically labeled data is further cleaned by some heuristics to filter some incorrect types which may not be deducible from the context of a given mention. its type ontology contains three levels and 89 types in total. ## conclusion in this work, we study a novel setting of finegrained entity typing that disentangles type ontology from knowledge bases. this setting is more practical and provides larger flexibility for realworld applications. under this setting, we make an initial attempt and propose a two-step framework to address fet. experimental results show that our proposed method can achieve competitive results without accessing any kb compared to using the kb-supervised dataset for training as in the conventional framework. although we use @xcite and @xcite as the backbone models in our experiments, our proposed framework does not set a limit on the backbone classifier. the more advanced fet model architectures mentioned in section 2 can be freely mitigated to our framework to achieve better performance under our setting. we leave this to future work.