Modeling Fine-Grained Entity Types with Box Embeddings (2021)

Abstract: Neural entity typing models typically represent fine-grained entity types as vectors in a high-dimensional space, but such spaces are not well-suited to modeling these types’ complex interdependencies. We study the ability of box embeddings, which embed concepts as d-dimensional hyperrectangles, to capture hierarchies of types even when these relationships are not defined explicitly in the ontology. Our model represents both types and entity mentions as boxes. Each mention and its context are fed into a BERT-based model to embed that mention in our box space; essentially, this model leverages typological clues present in the surface text to hypothesize a type representation for the mention. Box containment can then be used to derive both the posterior probability of a mention exhibiting a given type and the conditional probability relations between types themselves. We compare our approach with a vector-based typing model and observe state-of-the-art performance on several entity typing benchmarks. In addition to competitive typing performance, our box-based model shows better performance in prediction consistency (predicting a supertype and a subtype together) and confidence (i.e., calibration), demonstrating that the box-based model captures the latent type hierarchies better than the vector-based model does.

## introduction the development of named entity recognition and entity typing has been characterized by a growth in the size and complexity of type sets: from 4 @xcite to 17 @xcite to hundreds @xcite or thousands @xcite . these types follow some kind of hierarchical structure @xcite @xcite , so effective models for these tasks frequently engage with this hierarchy explicitly. prior systems incorporate this structure via hierarchical losses @xcite @xcite or by embedding types into a high-dimensional euclidean or hyperbolic space @xcite . however, the former approach requires prior knowledge of the type hierarchy, which is unsuitable for a recent class of large type sets where the hierarchy is not explicit @xcite . the latter approaches, while leveraging the inductive bias of hyperbolic space to represent trees, lack a probabilistic interpretation of the embedding and do not naturally capture all of the complex type relationships beyond strict containment. in this paper, we describe an approach that represents entity types with box embeddings in a highdimensional space @xcite . we build an entity typing model that jointly embeds each entity mention and entity types into the same box space to determine the relation between them. volumes of boxes correspond to probabilities and taking intersections of boxes corresponds to computing joint distributions, which allows us to model mentiontype relations (what types does this mention exhibit?) and type-type relations (what is the type hierarchy?). concretely, we can compute the conditional probability of a type given the entity mention with straightforward volume calculations, allowing us to construct a probabilistic type classification model. compared to embedding types as points in euclidean space @xcite , the box space is expressive and suitable for representing entity types due to its geometric properties. boxes can nest, overlap, or be completely disjoint to capture … the hunger games, the first of 3 best selling books by suzanne collins. subtype, correlation, or disjunction relations, properties which are not explicitly manifested in euclidean space. the nature of the box computation also allows these complex relations to be represented in a lower-dimensional space than needed by vector-based models. in our experiments, we focus on comparing our box-based model against a vector-based baseline. we evaluate on four entity typing benchmarks: ultra-fine entity typing @xcite , ontonotes @xcite , bbn @xcite , and figer @xcite . to understand the behavior of box embeddings, we further analyze the model outputs in terms of consistency (predicting coherent supertypes and subtypes together), robustness (sensitivity against label noise), and calibration (i.e., model confidence). lastly, we compare entity representations obtained by the box-based and vector-based models. our box-based model outperforms the vector-based model on two benchmarks, ultra-fine entity typing and ontonotes, achieving state-ofthe-art-performance. in our other experiments, the box-based model also performs better at predicting supertypes and subtypes consistently and being robust against label noise, indicating that our approach is capable of capturing the latent hierarchical structure in entity types. ## motivation when predicting class labels like entity types that exhibit a hierarchical structure, we naturally want our model's output layer to be sensitive to this structure. previous work @xcite @xcite has fundamentally treated types as vectors, as shown in the left half of figure 1 . as is standard in multiclass or multi-label classification, the output layer of these models typically involves taking a dot product between a mention embedding and each possible type. a type could be more general and predicted on more examples by having higher norm, foot_1 but it is hard for these representations to capture that a coarse type like person will have many mutually orthogonal subtypes. by contrast, box embeddings naturally represent these kinds of hierarchies as shown in the right half of figure 1 . a box that is completely contained in another box is a strict subtype of that box: any entity exhibiting the inner type will exhibit the outer one as well. overlapping boxes like politician and author represent types that are not related in the type hierarchy but which are not mutually exclusive. the geometric structure of boxes enables complex interactions with only a moderate number of dimensions @xcite . @xcite also define a probability measure over the box space, endowing it with probabilistic semantics. if the boxes are restricted to a unit hypercube, for example, the volumes of type boxes represent priors on types and intersections capture joint probabilities, which can then be used to derive conditional probabilities. critically, box embeddings have previously been trained explicitly to reproduce a given hierarchy such as wordnet. a central question of this work is whether box embeddings can be extended to model the hierarchies and type relationships that are implicit in entity typing data: we do not assume access to explicit knowledge of a hierarchy during training. while some datasets such as ontonotes have orderly ontologies, recent work on entity typing has often focused on noisy type sets from crowdworkers @xcite or derived from wikipedia @xcite . we show that box embeddings can learn these structures organically; in fact, they are not restricted to only tree structures, but enable a natural venndiagram style of representation for concepts, as with politician and author in figure 1 . our box embeddings represent entity types as n-dimensional hyperrectangles. a box x is characterized by two points (x m , x m ), where x m , x @xmath0,@xmath1,i for each coordinate @xmath2, ..., d}. the volume of the box x is computed as vol(x) = ∏︁ i (x m,i -x m,i ). if we normalize the volume of the box space to be 1, we can interpret the volume of each box as the marginal probability of a mention exhibiting the given entity type. furthermore, the intersection volume between two boxes, x and y, is defined as vol(x ∩ y) = ∏︁ i max (min(x m,i , y m,i ) -max(x m,i , y m,i ), 0) and can be seen as the joint probability of entity types x and y. thus, we can obtain the conditional probability p (y | x) = vol(x∩y) vol(x) . soft boxes computing conditional probabilities based on hard intersection poses some practical difficulties in the context of machine learning: sparse gradients caused by disjoint or completely contained boxes prevent gradient-based optimization methods from working effectively. to ensure that gradients always flow for disjoint boxes, @xcite relax the hard edges of the boxes using gaussian convolution. we follow the more recent approach of @xcite , who further improve training of box embeddings using max and min gumbel distributions (i.e., gumbel boxes) to represent the min and max coordinates of a box. ## experimental setup our focus here is to shed light on the difference between type hierarchies learned by the box-based model and the vector-based model. to this end, we first evaluate those two models on standard entity typing datasets. then, we test models' consistency, robustness, and calibration, and evaluate the predicted types as entity representations on a downstream task (coreference resolution). see appendix a for hyperparameters. ## conclusion in this paper, we investigated a box-based model for fine-grained entity typing. by representing entity types in a box embedding space and projecting entity mentions into the same space, we can naturally capture the hierarchy of and correlations between entity types. our experiments showed several benefits of box embeddings over the equivalent vector-based model, including typing performance, calibration, and robustness to noise.