Lifelong Event Detection with Knowledge Transfer (2021)

Abstract: Traditional supervised Information Extraction (IE) methods can extract structured knowledge elements from unstructured data, but it is limited to a pre-defined target ontology. In reality, the ontology of interest may change over time, adding emergent new types or more fine-grained subtypes. We propose a new lifelong learning framework to address this challenge. We focus on lifelong event detection as an exemplar case and propose a new problem formulation that is also generalizable to other IE tasks. In event detection and more general IE tasks, rich correlations or semantic relatedness exist among hierarchical knowledge element types. In our proposed framework, knowledge is being transferred between learned old event types and new event types. Specifically, we update old knowledge with new event types’ mentions using a self-training loss. In addition, we aggregate old event types’ representations based on their similarities with new event types to initialize the new event types’ representations. Experimental results show that our framework outperforms competitive baselines with a 5.1% absolute gain in the F1 score. Moreover, our proposed framework can boost the F1 score for over 30% absolute gain on some new long-tail rare event types with few training instances. Our knowledge transfer module improves performance on both learned event types and new event types under the lifelong learning setting, showing that it helps consolidate old knowledge and improve novel knowledge acquisition.

## introduction the information extraction (ie) task aims to extract informative knowledge elements (e.g., entities, relations, and events) from natural language. in practice, we usually extract knowledge elements for a pre-defined ontology consisting of various types of knowledge elements of interest. in this setting, ie is often formulated as a classification problem over types in the ontology, with an additional not-any (na) type to identify text spans that don't belong to any ontology types (negative instances). ontology-based supervised ie methods such as @xcite can produce more accurate and structured results with annotated training data than open-domain ie @xcite while being limited to the ontology. however, the ontology of interest may change over time, adding emergent new types for various knowledge elements (e.g., change bombing to disease outbreak), or adding more fine-grained subtypes for some existing general types (e.g., break justice into acquit, arrest, charge, convict, release, sentence, and trial). in the past twenty-five years the ie community has been shifting from one old ontology to a new one once every five years based on the consumers' needs, under many shared tasks, including muc @xcite , ace foot_1 , tac-kbp @xcite , darpa aida foot_2 and darpa kairos 4 . when we face a new ontology, we need to annotate a new training set for new types and retrain a new system on the new ontology while discarding the old established system for the old ontologies. in contrast, we propose a new and more economic paradigm, lifelong event detection, which can combine old system and new resources in a neverending continual learning fashion. we show an example of lifelong event detection in figure 1 . such a paradigm is commonly referred as lifelong learning, continual learning or incremental learning @xcite . we formulate lifelong event detection by modifying the commonly studied class incremental lifelong learning, where the model incrementally learns to classify more classes with only positive instances. we add a special na type denoting negative instances that are not event triggers for any types. for example, injured in the sentence "bob is injured." is a mention of an injure event, and is is a negative instance. whenever training on new event types, the new training data includes instances of new types and negative instances. lifelong event detection differs from class incremental learning, where the new data only contains instances for the new classes. another challenge in lifelong event detection is the naturally imbalanced distribution of event types in natural language as shown in figure 2 . existing methods @xcite @xcite for class incremental learning usually study relatively balanced classification datasets, and previous attempts @xcite on incremental learning of event detection ignore this problem by only experimenting on frequent types. in light of the challenges above in lifelong event detection, we propose a knowledge transfer framework taking advantage of the rich connections between various types such as contextual and semantic similarity. for instance, mentions for both the trial and the charge events contain court entities and crime-related content frequently in context, and their respective typical triggers such as try and charge usually have similar word embeddings representing their semantics. in our proposed lifelong • we propose a new formulation for lifelong event detection; ## conclusions and future work in this paper, we formulate the lifelong event detection problem, and propose a novel framework that transfers knowledge among related event types and between old and new types to tackle the unique challenges brought by the imbalanced distribution of event types. this framework benefits learning of both old types and new types, and improves performance on long-tail types. it is worth mentioning that although we use event detection as an exemplar task, our proposed framework is applicable to other information extraction tasks. we will extend and empirically study our approach to them in the future. moreover, in lifelong learning the new types' can come from a completely different domain, for which the context distribution (such as text style and genre) changes significantly from seen training instances. we leave explicit exploration and evaluation of lifelong learning for specific domains as future work. moreover, although the knowledge transfer can improve the performance on long-tail types significantly, there is still a gap between rare types and frequent types. we can combine our framework with other efficient learning methods such as zero-shot learning, few-shot learning, and weakly-supervised learning to learn these types better.