Ultra-fine Entity Typing with Indirect Supervision from Natural Language Inference (2022)

Abstract: The task of ultra-fine entity typing (UFET) seeks to predict diverse and free-form words or phrases that describe the appropriate types of entities mentioned in sentences. A key challenge for this task lies in the large number of types and the scarcity of annotated data per type. Existing systems formulate the task as a multi-way classification problem and train directly or distantly supervised classifiers. This causes two issues: (i) the classifiers do not capture the type semantics because types are often converted into indices; (ii) systems developed in this way are limited to predicting within a pre-defined type set, and often fall short of generalizing to types that are rarely seen or unseen in training. This work presents LITE🍻, a new approach that formulates entity typing as a natural language inference (NLI) problem, making use of (i) the indirect supervision from NLI to infer type information meaningfully represented as textual hypotheses and alleviate the data scarcity issue, as well as (ii) a learning-to-rank objective to avoid the pre-defining of a type set. Experiments show that, with limited training data, LITE obtains state-of-the-art performance on the UFET task. In addition, LITE demonstrates its strong generalizability by not only yielding best results on other fine-grained entity typing benchmarks, more importantly, a pre-trained LITE system works well on new data containing unseen types.1

## introduction entity typing, inferring the semantic types of the entity mentions in text, is a fundamental and long-lasting research problem in natural language understanding, which aims at inferring the semantic types of the entities mentioned in text. the resulted type information can help with grounding human language components to real-world concepts @xcite , and provide valuable prior knowledge for natural language understanding tasks such as entity linking @xcite , question answering @xcite , and information extraction @xcite . prior studies have mainly formulated the task as a multi-way classification problems @xcite @xcite . however, earlier efforts for entity typing are far from enough for representing real-world scenarios, where types of entities can be extremely diverse. accordingly, the community has recently paid much attention to more fine-grained modeling of types for entities. one representative work is the ultra-fine entity typing (ufet) benchmark created by @xcite . the task seeks to search for the most appropriate types for an entity among over ten thousand free-form type candidates. the drastic increase of types forces us to question whether the multi-way classification framework is still suitable for ufet. in this context, two main issues are noticed from prior work. first, prior studies have not tried to understand the target types since most classification systems converted all types into indices. without knowing the semantics of types, it is hard to match an entity mention to a correct type, especially when there is not sufficient annotated data for each type. second, existing entity typing systems are far behind the desired capability in real-world applications in which any open-form types can appear. specifically, those pre-trained multi-way classifiers cannot recognize types that are unseen in training, especially when there is no reasonable mapping from existing types to unseen type labels, unless the classifiers are re-trained to include those new types. to alleviate the aforementioned challenges, we propose a new learning framework that seeks to enhance ultra-fine entity typing with indirect supervision from natural language inference (nli) @xcite . specifically, our method lite , (language inference based typing of entities), treats each entity-mentioning sentence as a premise in nli. using simple, template-based generation techniques, a candidate type is transformed into a textual description and is treated as the hypothesis in nli. based on the premise sentence and a hypothesis description of a candidate type, the entailment score given by an nli model is regarded as the confidence of the type. on top of the pre-trained nli model, lite conducts a learning-to-rank objective, which aims at scoring hypotheses of positive types higher than the hypotheses of sampled negative types. finally, the label candidates whose hypotheses obtain scores above a threshold are given as predictions by the model. technically, lite benefits ultra-fine entity typing from three perspectives. first, the inference ability of a pre-trained nli model can provide effective indirect supervision to improve the prediction of type information. second, the hypothesis, as a type description, also provides a semantically rich representation of the type, which further benefits few-shot learning with insufficient labeled data. moreover, to handle the dependency of type labels in different granularities, we also utilize the inference ability of an nli model to learn that the finer label hypothesis of an entity mention entails its general label hypothesis. experimental results on the ufet benchmark @xcite show that lite drastically outperforms the recent state-of-the-art (sota) systems @xcite @xcite without any need of distantly supervised data as they do. in addition, our lite also yields the best performance on traditional (less) fine-grained entity typing tasks. 2 what's more, because we adopt a learning-to-rank objective to optimize the inference ability of lite rather than classification on a specified label space, it is feasible to apply the trained model across different typing data sets. we therefore test its transferability by training on ufet and evaluate on traditional fine-grained benchmarks to get promising results. moreover, we also examined the time efficiency of lite, and discussed about the trade-off between training and inference costs in comparison with prior methods. to summarize, the contributions of our work are three fold. first, to our knowledge, this is the first work that uses nli formulation and nli supervision to handle entity typing. as a result, our system is able to retain the labels' semantics and encode the label dependency effectively. second, our system offers sota performance on both ultra-fine entity typing and regular fine-grained typing tasks, being particularly strong at predicting zero-shot and few-shot cases. finally, we show that our system, once trained, can also work on different test sets that are free to have unseen types. more recently, @xcite proposed the ultra-fine entity typing (ufet) task, which involved free-form type labeling to realize the open-domain label space with much more comprehensive coverage of types. as the ufet tasks non-trivial learning and inference problems, several methods have been explored by more effectively modeling the structure of the label space. @xcite utilized a graph propagation layer to impose label-relation bias in order to capture type dependencies implicitly. @xcite trained a filtering and relabeling model with the human annotated data to denoise the automatically generated data for training. @xcite introduced box embeddings @xcite to represent the dependency among multiple levels of type labels as topology of axis-aligned hyper-rectangles (boxes). to further cope with insufficient training data, @xcite used pre-trained language model for augmenting (noisy) training data with masked entity generation. different from their strategy of augmenting training data, our approach generates type descriptions to leverage indirect supervision from nli which requires no more data samples. ## method in this section, we introduce the proposed method for (ultra-fine) entity typing with nli. we start with problem definition and an overview of our nli-based entity typing framework ( §3.1), followed by technical details of type description generation ( §3.2), label dependency modeling ( §3.3), learning objective ( §3.4), and inference ( §3.5). ## experiment in this section, we present the experimental evaluation for lite framework, based on both ufet ( §4.1) and traditional (less) fine-grained entity typing tasks ( §4.2). in addition, we also conduct comprehensive ablation studies to understand the effectiveness of the incorporated techniques ( §4.3). ## conclusion and future work we propose a new model, lite, that leverages indirect supervision from nli to type entities in texts. through template-based type hypothesis generation, lite formulates the entity typing task as a language inference task and meanwhile the semantically rich hypothesis remedies the data scarcity problem in the ufet benchmark. additionally, the learning-to-rank objective further helps lite with generalized prediction across benchmarks with disjoint type sets. our experimental results illustrate that lite promisingly offers sota on ufet, ontonotes, and figer, and yields strong performance on zero-shot and few-shot types. lite pretrained on ufet also yields strong transferability by outperforming sota baselines when directly make predictions on ontonotes and figer. for future research, as mentioned in §4.3, we first plan to investigate ways to accelerate lite by utilizing a late-binding cross-encoder @xcite for linear-complexity nli, and incorporating high-dimensional indexing techniques like ball trees in inference. to be specific, the premise and hypotheses can first be encoded respectively and the resulting representations can later be used to evaluate the confidence score of premise-hypothesis representation pairs through a trained network. with little expected loss in performance, lite can still maintain its feature of strong transferability and zero-shot prediction. in addition, we plan to extend nli-based indirect supervision to information extraction tasks such as relation extraction and event extraction. incorporating abstention-awareness @xcite for handling unknown types is another meaningful direction. additionally, @xcite recast diverse types of reasoning datasets including ner, relation extraction, and sentiment analysis into the nli structure, which we plan to incorporate as extra indirect supervision for lite to further enhance the robustness of entity typing.