Global Bootstrapping Neural Network for Entity Set Expansion (2020)

Abstract: Bootstrapping for entity set expansion (ESE) has been studied for a long period, which expands new entities using only a few seed entities as supervision. Recent end-to-end bootstrapping approaches have shown their advantages in information capturing and bootstrapping process modeling. However, due to the sparse supervision problem, previous end-to-end methods often only leverage information from near neighborhoods (local semantics) rather than those propagated from the co-occurrence structure of the whole corpus (global semantics). To address this issue, this paper proposes Global Bootstrapping Network (GBN) with the “pre-training and fine-tuning” strategies for effective learning. Specifically, it contains a global-sighted encoder to capture and encode both local and global semantics into entity embedding, and an attention-guided decoder to sequentially expand new entities based on these embeddings. The experimental results show that the GBN learned by “pre-training and fine-tuning” strategies achieves state-of-the-art performance on two bootstrapping datasets.

## introduction bootstrapping is a classical technique for entity set expansion (ese), which starts from several seed entities of a specific category (e.g., {london, beijing, u.s.} for gpe category) and then iteratively expands the entity set to cover more entities of the category (e.g., egypt and harare). most previous ese studies @xcite @xcite adopt the pipelined paradigm (see figure 1a ), which iteratively: evaluates patterns using seeds, matches and evaluates entities using patterns, adds top entities to the seed set. such a pipelined paradigm makes it hard to represent the whole bootstrapping process as a single learnable model, and the implementation of ese systems are often very ad hoc. witnessed the drawbacks of the pipelined bootstrapping paradigm, recent studies start turning to the end-to-end paradigm. for instance, @xcite propose the first end-to-end bootstrapping neural network for ese, which uses the encoderdecoder architecture (see figure 1b ): the encoder leverages and encodes the co-occurrence relations between entities and patterns into their embeddings; the decoder models bootstrapping as a sequential entity generation process, and the generated entities are used as expansion results. compared with the pipeline paradigm, the end-to-end paradigm represents the whole bootstrapping process as a single learnable model and therefore is capable of leveraging more information and is more flexible. one of the biggest challenges of end-to-end bootstrapping is how to learn it effectively since only very sparse supervision signals (i.e., several seed entities) are provided. in general, bootstrapping systems expand entities based on the entity-pattern duality assumption that "similar entities will share similar patterns, and similar patterns will match similar entities". based on this assumption, a bootstrapping network should be able to represent entities/patterns by leveraging both their near neighborhoods (local semantics) and the information propagated via the entity/pattern co-occurrence structure in the whole corpus (global semantics). currently, using only several seeds as supervision signals, previous end-to-end bootstrapping models often only aggregate neighborhood information to represent entity/patterns, therefore the final representations of entities/patterns are mostly learned from their neighborhoods (short-sighted), rather than from global information (global-sighted). this raises a big issue because most entities are long-tail @xcite , which will only match a limited number of patterns, and as a result, the local semantics cannot provide reliable and informative representations for effective bootstrapping (see figure 2 for an example). to address the sparse supervision problem, this paper proposes a new end-to-end bootstrapping neural network for ese, called global bootstrapping network (gbn), which can effectively capture the global information of a corpus via an augmented entity-pattern bipartite graph, and learn to leverage both the local and the global semantics for bootstrapping via effective pre-training and finetuning strategies. our method is motivated by the recent success of the pre-training and fine-tuning strategy in addressing the sparse supervision challenges @xcite . concretely, the global bootstrapping network adopts the encoder-decoder architecture. the encoder is a global-sighted graph neural network, in which each layer aggregates rich information not only between directly linked entities and patterns but also the entities and patterns multi-hop away via augmented links. the decoder is an attentionguided rnn model, which efficiently generates expansion results based on the global-sighted entity representations. compared with previous methods, gbn can also effectively aggregate the global information rather than only local neighborhood information, therefore it is more reliable even for the long-tail entities/patterns with sparse links. to learn the gbn, we propose several pretraining and fine-tuning algorithms: 1) in the pretraining stage, we design both the self-supervised and the supervised pre-training strategies to learn the encoder in the gbn, which ensures the learned representations of entities/patterns will capture original link augmented link entity pattern 3-hop neighborhood (local-sighted) 3-hop neighborhood (global-sighted) harare & located in * visit to * * court * says flight from * trip to * city of * * on sept. figure 2 : an example of local/global-sighted neighborhood with/without augmented links (we use a longtail gpe-"harare" as the center entity). by adding an augmented link, "harare" can easily observe its global-sighted neighborhood such as the strong gpe patterns-"visit to *", "located in *", etc., and therefore it can be accurately expanded. to summarize, the main contributions are: ## entity-pattern bipartite graph construction this section describes how to construct the entitypattern bipartite graph, which captures the global structure of entity-pattern co-occurrences in the original ese corpus. furthermore, augmented links are added for long-tail entities/patterns. traditionally, entity-pattern duality is modeled as a set of individual entity, pattern entries @xcite @xcite , e.g., { harare, * court , london, visit to * , ...}. however, such a data model considers different entity, pattern entries to capture both the local and global semantics, we follow @xcite and use the entitypattern bipartite graph. concretely, the entities and patterns are represented as graph nodes, and an entity and a pattern will be linked if the pattern matches the entity in the corpus. finally, the entity-pattern bipartite graph is formulated as a tuple @xmath0, e, s >, where v is the node set of entities and patterns, e is the set of edges connecting entities and patterns, s is the set of seed entities with corresponding labels. ## global bootstrapping network this section describes the global bootstrapping network (gbn), which adopts the encoder-decoder architecture (see figure 3 ) and contains: ## learning gbn with pre-training and fine-tuning in this section, we describe how to learn gbn effectively using the "pre-training and fine-tuning" @xcite . in the pre-training stage, we adopt both the self-supervised and the supervised pre-training algorithms to pre-train the gbencoder; in the fine-tuning stage, we adopt the multi-view learning algorithm to fine-tune both the gbencoder and the gbdecoder. in this way, the sparse supervision problem can be effectively resolved. ## related work bootstrapping. bootstrapping is a widely studied technique in ie @xcite @xcite , as well as word sense disambiguation @xcite , entity translation @xcite , model learning @xcite , etc. currently, bootstrapping methods for ese can be categorized into two paradigms: pipelined paradigm and end-to-end paradigm @xcite . the pipelined methods @xcite ) mainly leverage direct co-occurrence information, which will easily lead to the semantic drifting problem @xcite . to resolve this problem, many pipelined methods are proposed, e.g., mutual exclusive bootstrapping @xcite @xcite , bootstrapping using negative seeds @xcite , lexical and statistical features @xcite , word embeddings @xcite @xcite , active learning @xcite , lookahead search @xcite , etc. recently @xcite propose an end-to-end bootstrapping model and show its advantages in information leveraging and flexibility. besides, there are some other studies that focus on the web-based ese @xcite @xcite , which heavily relies on the base search engine. pre-training and fine-tuning. the early pretrained models on the imagenet @xcite show its advantages in many cv tasks @xcite @xcite . in nlp, the pre-training has also been proven its effectiveness on many tasks, including the early word vectors such as word2vec or glove @xcite and recent language model pre-training such as elmo @xcite , bert @xcite and xlnet @xcite . recently, @xcite also show the advantages of graph pre-training, which directly inspires our work. ## conclusions in this paper, we propose the global bootstrapping network (gbn) and effective "pre-training and fine-tuning" strategies to learn it. specifically, we design global-sighted gbencoder to capture both local and global semantics from the corpus and an effective attention-guided gbdecoder to adaptively expand new entities. to learn gbn, we design several pre-training and fine-tuning strategies. experiments show that the proposed gbn together with "pre-training and fine-tuning" al-gorithm significantly outperforms state-of-the-art methods. for future work, we want to design more effective "pre-training and fine-tuning" strategies and apply our model on other bootstrapping tasks.