M any E nt: A Dataset for Few-shot Entity Typing (2020)

Abstract: We introduce ManyEnt, a benchmark for entity typing models in few-shot scenarios. ManyEnt offers a rich typeset, with a fine-grain variant featuring 256 entity types and a coarse-grain one with 53 entity types. Both versions have been derived from the Wikidata knowledge graph in a semi-automatic fashion. We also report results for two baselines using BERT, reaching up to 70.68% accuracy (10-way 1-shot).

## introduction information extraction is targeted at inferring knowledge from unstructured documents, most commonly in the form of entities and relations between them. like many other nlp tasks, the field has recently experienced a boost from deep neural networks @xcite @xcite . while research datasets are usually derived from wikipedia, for industrial applications it is the adaptation to new domains which matters: here, different relations may be of interest than on the source domain, and existing relations may express themselves differently (think of the "part of" relation, which differs between the medical domain and mechanical engineering). as data is often scarce on the target domain, recent work has addressed few-shot scenarios: a model is pre-trained on a training set of relations representing the source domain and is then adapted to new relations representing the target domain, usually using only few (i.e., 1-5) samples. one prominent example is the fewrel benchmark @xcite for relation classification: given a sentence in which two entity mentions are marked, the tasks is to assign the entity pair to a relation. models are first trained on a dataset of 80 relation types and are then applied to held-out test relation types. while fewrel addresses relations, we argue that -since information extraction models targeted at new domains will have to adapt both to new entities and relations -the few-shot scenario is also interesting for entity typing. this way, models pretrained on some entity types (such as computer) can be adapted to other types (such as smartphone). therefore, we suggest a novel benchmark "manyent" 1 for fewshot entity typing. the benchmark is based on fewrel and features a similar setting: given a sentence with a highlighted entity, the entity is to be assigned to an entity type. here, a rich number of types is beneficial: only if a breadth of types is known in training transfer can be expected to work well by adapting to "similar" types as the ones from the source domain. while fewrel offers a much richer typeset of relations (80 types) than other datasets, e.g. @xcite , it offers no entity types. therefore, our first contribution is a semi-automatic annotation of entities on fewrel at two granularities (53 types and 256 types) using the wikidata knowledge graph. second, we also report the results of two common baselines based on the well-known bert model @xcite , which reach an accuracy of up to 70.68% (10-way 1-shot). with respect to entity typing, models and benchmarks addressing rich typesets exist @xcite . we follow a similar approach in label acquisition to @xcite , exploiting the wikidata hierarchy. only few previous works, however, are specifically targeted at fewshot scenarios @xcite apply a bilstm with glove embeddings for ner in medical texts. they show that few-shot performance can be improved by a pre-training on related domains. in comparison with manyent, the dataset used by hofer et al. contains only a small number of entity labels (6) and is very domain specific. @xcite train a prototypical network based model on the ontonotes dataset (18 entity types, about 170k sentences). manyent is smaller regarding sentences, but contains a broader set of entity types. @xcite apply a meta-learning approach for ner domain adaption. they use six domains with homogeneous entity types, while our goal is to adapt a model to unknown entity types given only a few labeled samples. finally, ma et al. @xcite present a few-shot and zero-shot model focusing on category representations exploiting similarities between category labels (e.g., book vs. song). in contrast, our model is based on instance-level prototypes similar to fewrel. ## the manyent dataset the basis for manyent is the fewrel dataset for few-shot relation extraction @xcite . fewrel offers a large-scale annotated set of sentences extracted from wikipedia, annotated with 80 relation types but no entity types. fewrel also contains two annotated entity mentions per sentence, each linked to a wikidata foot_0 entity. we denote the set of all these entities with e. while wikidata comes with a rich concept hierarchy, we found many of its entity types to be unsuitable: they are either too specific (high school), too coarse/uninformative (physical object) or unintuitive (artificial geographic entity). we would like to select a subset t of "suitable" entity types, following common criteria for concept ontologies @xcite such as utility (practical relevance), coverage (semantic breadth), feasibility of detection, and observability in the corpus. given the subset of suitable entity types t , we map each wikidata entity from e to a type from t using a semi-automatic procedure outlined in the following section. ## experiments we adapt the commonly used few-shot setting @xcite for entities. the setting is formulated as a n-way, k-shot problem: given a set of n entity types (way), which the model did not encounter during training, the correct type of a sample must be predicted given only k examples (shots) per type. the set of entity type examples is commonly denoted as the support set and the samples that the system must classify as the query set. we design a prototypical network @xcite using the pre-trained transformer-type network bert @xcite as the encoder. prototypical networks aim at creating a concise representation (the prototype) of a target class (here, an entity type) from the support set. it is trained on randomly sampled episodes, each consisting of k support samples per n entity types and q query samples per type (see @xcite for more details). we first tokenize an input sentence, obtaining a sequence of n byte-pair encoded tokens. these tokens are then embedded with bert into a sequence of contextualized embeddings (e 1 , e 2 , ...e n ). given an entity span s := (e i , e i+1 , ..., e i+k ) with length k, an entity representation e(s) is obtained by averaging over s. the prototype representation of an entity type t is acquired by an averaging over the support set s t of t: each query sample is also encoded (again by averaging over the entity's span) and compared with each prototype by euclidean distance. a probability distribution over all entity types is obtained using a softmax function over the negated distances. we then minimize the negative log-likelihood loss of the ground truth entity type. we trained the model for 3 epochs of 4,000 episodes. we use the bert base (multilingual-cased) model for our experiments. hyperparameters were tuned on a held-out validation set (10% of training data). we use the adam optimizer with a learning rate of 3e-5 and a linear increase/decrease schedule and perform early stopping on the validation set. we evaluate our model on 200 random episodes per setting, each containing 5 query samples per entity type. the accuracy is averaged over all episodes. we also include a second baseline, where bert is not fine-tuned, to asses the inherent ability of bert to distinguish entities through its unsupervised pretraining. for the fine-grain dataset, we evaluate on a 5-, 10-, and 20-way setting with 1 or 5 shots. since me-53 contains fewer types, only a 5-way setting is evaluated. ## conclusions we have introduced a new benchmark for few-shot entity typing featuring a rich entity typeset of 256 types. results with two baselines using a bert-based prototypical network indicate that bert -particularly when fine-tuned -already provides a solid performance (62-93%). a key challenge to the system appears to be the separation of similar fine-grain types (such as "comic format" and "document"), which poses an interesting challenge for future research.