Fine-grained Entity Typing via Label Reasoning (2021)

Abstract: Conventional entity typing approaches are based on independent classification paradigms, which make them difficult to recognize inter-dependent, long-tailed and fine-grained entity types. In this paper, we argue that the implicitly entailed extrinsic and intrinsic dependencies between labels can provide critical knowledge to tackle the above challenges. To this end, we propose Label Reasoning Network(LRN), which sequentially reasons fine-grained entity labels by discovering and exploiting label dependencies knowledge entailed in the data. Specifically, LRN utilizes an auto-regressive network to conduct deductive reasoning and a bipartite attribute graph to conduct inductive reasoning between labels, which can effectively model, learn and reason complex label dependencies in a sequence-to-set, end-to-end manner. Experiments show that LRN achieves the state-of-the-art performance on standard ultra fine-grained entity typing benchmarks, and can also resolve the long tail label problem effectively.

## introduction fine-grained entity typing (fet) aims to classify entity mentions to a fine-grained semantic label set, e.g., classify "fbi agents" in "they were arrested by fbi agents." as {organization, administration, force, agent, police}. by providing finegrained semantic labels, fet is critical for entity recognition @xcite @xcite and can benefit many nlp tasks, such as relation extraction @xcite , entity linking @xcite ) and question answering @xcite . the fundamental challenge of fet comes from its large-scale and fine-grained entity label set, which leads to significant difference between fet and conventional entity typing. first, due to the massive label set, it is impossible to independently recognize each entity label without considering their dependencies. for this, existing approaches use the predefined label hierarchies @xcite @xcite @xcite @xcite or label co-occurrence statistics from training data @xcite @xcite as external constraints. unfortunately, these label structures or statistics are difficult to obtain when transferring to new scenarios. second, because of the fine-grained and large-scale label set, many long tail labels are only provided with several or even no training instances. for example, in ultra-fine dataset @xcite , >80% of entity labels are @xmath0, and more seriously 25% of labels never appear in the training data. however, training data can provide very limited direct information for these labels, and therefore previous methods commonly fail to recognize these long-tailed labels. fortunately, the implicitly entailed label dependencies in the data provide critical knowledge to tackle the above challenges. specifically, the dependencies between labels exist extrinsically or intrinsically. on the one hand, the extrinsic dependencies reflect the direct connections between labels, which partially appear in the form of label hierarchy and co-occurrence. for example, in figure 1 (a) the labels person, musician, composer are with extrinsic dependencies because they form a three-level taxonomy. furthermore, singer and composer are also with extrinsic dependency because they often co-occur with each other. on the other hand, the intrinsic dependencies entail the indirect connections between labels through their underlying attributes. for the example in figure 1(b), label theorist and scientist share the same underlying attribute of scholar. such intrinsic dependencies provide an effective way to tackle the long tail labels, because many long tail labels are actually composed by non-long tail attributes which can be summarized from non-long tail labels. to this end, this paper proposes label reasoning network (lrn), which uniformly models, learns and reasons both extrinsic and intrinsic label dependencies without given any predefined label structures. specifically, lrn utilizes an auto-regressive network to conduct deductive reasoning and a bipartite attribute graph to conduct inductive reasoning between labels. both of these two kinds of mechanisms are jointly applied to sequentially generate fine-grained labels in an end-to-end, sequence-toset manner. figure 1(c ) shows several examples. to capture extrinsic dependencies, lrn introduces deductive reasoning (i.e., draw a conclusion based on premises) between labels, and formulates it using an auto-regressive network to predict labels based on both the context and previous labels. for example, given previously-generated label person of the mention they, as well as the context they theorize, lrn will deduce its new label theorist based on the extrinsic dependency between person and theorist derived from data. for intrinsic dependencies, lrn introduces inductive reasoning (i.e., gather generalized information to a conclusion), and utilizes a bipartite attribute graph to reason labels based on current activated attributes of previous labels. for example, if the attributes {expert, scholar} have been activated, lrn will induce a new label scientist based on the attribute-label relations. consequently, by decomposing labels into attributes and associating long tail labels with frequent labels, lrn can also effectively resolve the long tail label problem by leveraging their nonlong tail attributes. through jointly leveraging the extrinsic and intrinsic dependencies via deductive and inductive reasoning, lrn can effectively handle the massive label set of fet. generally, our main contributions are: we conduct experiments on standard ultra-fine @xcite and ontonotes @xcite dataset. experiments show that our method achieves new state-of-the-art performance: a 13% overall f1 improvement and a 44% f1 improvement in the ultra-fine granularity. foot_0 2 related work one main challenge for fet is how to exploit complex label dependencies in the large-scale label set. previous studies typically use predefined label hierarchy and co-occurrence structures estimated from data to enhance the models. to this end, @xcite al. (2016a); xu and barbosa (2018); wu et al. (2019); chen et al. (2020) design new loss function to exploit label hierarchies. abhishek et al. (2017) enhance the label representation by sharing parameters. @xcite ; @xcite ; @xcite embed labels into a highdimension or a new space. and the studies exploit co-occurrence structures including limiting the label range during label set prediction @xcite , enriching the label representation by introducing associated labels @xcite ## label reasoning network for fet figure 2 illustrates the framework of label reasoning network. first, we encode entity mentions through a context-sensitive encoder, then sequentially generate entity labels via two label reasoning mechanisms: deductive reasoning for exploiting extrinsic dependencies and inductive reasoning for exploiting intrinsic dependencies. in our seq2set framework, the label dependency knowledge can be effectively modeled in the parameters of lrn, automatically learned from training data, and naturally exploited during the sequential label decoding process. in the following we describe these components in detail. ## conclusions this paper proposes label reasoning network, which uniformly models, learns and reasons complex label dependencies in a sequence-to-set, endto-end manner. lrn designs two label reasoning mechanisms for effective decoding -deductive reasoning to exploit extrinsic dependencies and inductive reasoning to exploit intrinsic dependencies. experiments show that lrn can effectively cope with the massive label set on fet. and because our method uses no predefined structures, it can be easily generalized to new datasets and applied to other multi-classification tasks.