Ultra-Fine Entity Typing with Weak Supervision from a Masked Language Model (2021)

Abstract: Recently, there is an effort to extend fine-grained entity typing by using a richer and ultra-fine set of types, and labeling noun phrases including pronouns and nominal nouns instead of just named entity mentions. A key challenge for this ultra-fine entity typing task is that human annotated data are extremely scarce, and the annotation ability of existing distant or weak supervision approaches is very limited. To remedy this problem, in this paper, we propose to obtain training data for ultra-fine entity typing by using a BERT Masked Language Model (MLM). Given a mention in a sentence, our approach constructs an input for the BERT MLM so that it predicts context dependent hypernyms of the mention, which can be used as type labels. Experimental results demonstrate that, with the help of these automatically generated labels, the performance of an ultra-fine entity typing model can be improved substantially. We also show that our approach can be applied to improve traditional fine-grained entity typing after performing simple type mapping.

## introduction fine-grained entity typing @xcite has been long studied in the natural language processing community as the extracted type information is useful for downstream tasks such as entity linking @xcite , relation extraction @xcite , coreference resolution @xcite , etc. recently, ultra-fine entity typing @xcite extends the effort to using a richer set of types (e.g., @xcite to label noun phrases including not only named entity mentions, but also pronouns and nominal nouns. this task directly uses type words or phrases as tags. its tag set can contain more than 10,000 types. a challenge is that with the large type set, it is extremely difficult and time-consuming for humans to annotate samples. as a result, most existing works use weak labels that are automatically generated @xcite @xcite . there are two main approaches to obtaining weakly labeled training examples. one approach is to find the wikipedia pages that correspond to entity mentions, which can be done by using hyperlinks to wikipedia or applying entity linking. then the entity types can be obtained from knowledge bases. the other approach is to directly use the head words of nominal mentions as ultra-fine type labels. for example, if a nominal mention is "a famous actor," then the head word "actor" can be used as its type label. several problems exist when using these weak labels for the ultra-fine typing task. first, in the dataset created by @xcite , on average there are fewer than two labels (types) for each sample annotated through either entity linking or head word supervision. on the other hand, a human annotated sample has on average 5.4 labels. as a result, models trained from the automatically obtained labels have a low recall. second, neither of the above approaches can create a large number of training samples for pronoun mentions. third, it is difficult to obtain types that are highly dependent on the context. for example, in "i met the movie star leonardo dicaprio on the plane to l.a.," the type passenger is correct for "leonardo dicaprio." however, this type cannot be obtained by linking to knowledge bases. in this paper, to alleviate the problems above, we propose an approach that combines hypernym extraction patterns @xcite with a masked language model (mlm), such as bert @xcite , to generate weak labels for ultra-fine entity typing. given a sentence that contains a mention, our approach adds a short piece of text that contains a "[mask]" token into it ## related work the ultra-fine entity typing task proposed by @xcite uses a large, open type vocabulary to achieve better type coverage than the traditional fine-grained entity typing task @xcite that uses manually designed entity type ontologies. there are only limited studies on this newly proposed task: a neural model introduced by @xcite filters samples that are too noisy to be used and relabels the remaining samples to get cleaner labels. a graph propagation layer is introduced by @xcite to impose a label-relational bias on entity typing models, so as to implicitly capture type dependencies. @xcite use box embeddings to capture latent type hierarchies. there is also some work on the applications of ultra-fine entity typing: onoe and durrett (2020) apply ultra-fine entity typing to learn entity representations for two downstream tasks: coreference arc prediction and named entity disambiguation. the traditional fine-grained entity typing task @xcite is closely related to ultra-fine entity typing. automatic annotation @xcite @xcite ) is also commonly used in the studies of this task to produce large size training data. many different approaches have been proposed to improve fine-grained entity typing performance. for example, denoising the automatically generated labels @xcite , taking advantage of the entity type hierarchies or type inter-dependencies @xcite @xcite , exploiting external resources such as the information of entities provided in knowledge bases @xcite @xcite , etc. our work is also related to recent studies @xcite @xcite that probe pretrained language models to obtain knowledge or results for target tasks. different from them, we use the predictions produced by bert as intermediate results that are regarded as weak supervision to train better models. @xcite also uses hearst patterns to probe masked language models. however, they target at the entity set expansion task. ## methodology our methodology consists of two main steps. first, we obtain weak ultra-fine entity typing labels from a bert masked language model. second, we use the generated labels in model training to learn better ultra-fine entity typing models. ## application to traditional fine-grained entity typing our approach to generating weak entity type labels with bert mlm can also be applied to the traditional fine-grained entity typing task. different from ultra-fine entity typing, traditional finegrained entity typing uses a manually designed entity type ontology to annotate mentions. the types in the ontology are organized in an hierarchical structure. for example, the ontology used by the ontonotes dataset contains 89 types including /organization, /organization/company, /person, /person/politician, etc. on this dataset, our automatic annotation approach can mainly be helpful to generate better labels for nominal mentions. we still use the same method described in section 3.1 to create input for bert mlm based on the given mention. but with traditional finegrained entity typing, most mentions are assigned only one type path (e.g., a company mention will only be assigned labels {/organization, /organization/company}, which includes all the types along the path of /organization/company). thus, while generating labels, we only use the most probable word predicted by the bert mlm, which is mapped to the types used by the dataset if possible. for example, the word "company" and its plural form are both mapped to /organization/company. such a mapping from free-form entity type words to the types used by the dataset can be created manually, which does not require much effort. we mainly construct the mapping with two ways: 1) check each type used by the dataset, and think of a few words that should belong to it, if possible. for example, for the type /person/artist/author, corresponding words can be "author," "writer," etc. 2) run the bert mlm on a large number of inputs constructed with unannotated mentions, then try to map the words that are most frequently predicted as the most probable word to the entity type ontology. since only the most probable word predicted by the bert mlm is used to produce labels, we also only use one hypernym relation pattern: "m and any other h." for traditional fine-grained entity typing, we use our approach to generate labels for mentions that are not previously annotated with other automatic annotation approaches. while training, all the automatically labeled mentions are used together. the typing model is the same as the model described in 3.3. the binary cross entropy loss is directly employed as the training objective. ## experiments we conduct experiments on our primary task: ultrafine entity typing. in addition, we evaluate the performance of our approach when applied to traditional fine-grained entity typing. ## conclusion in this work, we propose a new approach to automatically generate ultra-fine entity typing labels. given a sentence that contains a mention, we insert a hypernym extraction pattern with a "[mask]" token in it, so that a pretrained bert mlm may predict hypernyms of the mention for " @xcite ." multiple patterns are used to produce better labels for each mention. we also propose to use a weighted loss and perform a self-training step to learn better entity typing models. experimental results show that our approach greatly outperforms state-of-theart systems. additionally, we also apply our approach to traditional fine-grained entity typing, and verify its effectiveness with experiments.