Joint Constrained Learning for Event-Event Relation Extraction (2020)

Abstract: Understanding natural language involves recognizing how multiple event mentions structurally and temporally interact with each other. In this process, one can induce event complexes that organize multi-granular events with temporal order and membership relations interweaving among them. Due to the lack of jointly labeled data for these relational phenomena and the restriction on the structures they articulate, we propose a joint constrained learning framework for modeling event-event relations. Specifically, the framework enforces logical constraints within and across multiple temporal and subevent relations of events by converting these constraints into differentiable learning objectives. We show that our joint constrained learning approach effectively compensates for the lack of jointly labeled data, and outperforms SOTA methods on benchmarks for both temporal relation extraction and event hierarchy construction, replacing a commonly used but more expensive global inference process. We also present a promising case study to show the effectiveness of our approach to inducing event complexes on an external corpus.

## introduction human languages evolve to communicate about real-world events. therefore, understanding events plays a critical role in natural language understanding (nlu). a key challenge to this mission lies in the fact that events are not just simple, standalone predicates. rather, they are often described at different granularities and may form complex structures. consider the example in figure 1 , where the description of a storm (e 1 ) involves more finegrained event mentions about people killed (e 2 ), flights canceled (e 3 ) and passengers affected (e 4 ). some of those mentions also follow strict temporal order (e 3 , e 4 and e 5 ). our goal is to induce such an event complex that recognizes the membership of multi-granular events described in the text, as well as their temporal order. this is not only at the core of text understanding, but is also beneficial to various applications such as question answering @xcite , narrative prediction @xcite , timeline construction @xcite and summarization @xcite . recently, significant research effort has been devoted to several event-event relation extraction tasks, such as event temporal relation (temprel) extraction @xcite and subevent relation extraction @xcite . addressing such challenging tasks requires a model to recognize the inherent connection between event mentions as well as their contexts in the documents. accordingly, a few previous methods apply statistical learning methods to characterize the grounded events in the documents @xcite @xcite . such methods often require designing various features to characterize the structural, discourse and narrative aspects of the events, which are costly to produce and are often specific to a certain task or dataset. more recent works attempted to use datadriven methods based on neural relation extraction models @xcite @xcite which refrain from feature engineering and offer competent performances. while data-driven methods provide a general and tractable way for event-event relation extraction, their performance is restricted by the limited annotated resources available @xcite . for example, the largest temporal relation extraction dataset matres @xcite only has 275 articles, which is far from enough for training a well-performing supervised model. the observation that relations and, in particular, event-event relations should be constrained by their logical properties @xcite , led to employing global inference to comply with transitivity and symmetry consistency, specifically on temprel @xcite @xcite . however, in an event complex, the logical constraints may globally apply to different taskspecific relations, and form more complex conjunctive constraints. consider the example in figure 1 : given that e2:died is before e3:canceled and e3:canceled is a parent event of e4:affecting, the learning process should enforce e2:died before e4:affecting by considering the conjunctive constraints on both temprel and subevent relations. while previous works focus on preserving logical consistency through (post-learning) inference or structured learning @xcite , there was no effective way to endow neural models with the sense of global logical consistency during training. this is key to bridging the learning processes of temprel and subevent relations, which is a research focus of this paper. the first contribution of this work is proposing a joint constrained learning model for multi-faceted event-event relation extraction. the joint constrained learning framework seeks to regularize the model towards consistency with the logical constraints across both temporal and subevent relations, for which three types of consistency requirements are considered: annotation consistency, symmetry consistency and conjunction consistency. such consistency requirements comprehensively define the interdependencies among those relations, essentially unifying the ordered nature of time and the topological nature of multi-granular subevents based on a set of declarative logic rules. motivated by the logic-driven framework proposed by @xcite , the declarative logical constraints are converted into differentiable functions that can be incorporated into the learning objective for relation extraction tasks. enforcing logical constraints across temporal and subevent relations is also a natural way to combine the supervision signals coming from two different datasets, one for each of the relation extraction tasks with a shared learning objective. despite the scarce annotation for both tasks, the proposed method surpasses the sota temprel extraction method on matres by relatively 3.27% in f 1 ; it also offers promising performance on the hieve dataset for subevent relation extraction, relatively surpassing previous methods by at least 3.12% in f 1 . from the nlu perspective, the second contribution of this work lies in providing a general method for inducing an event complex that comprehensively represents the relational structure of several related event mentions. this is supported by the memberships vertically identified between multigranular events, as well as the horizontal temporal reasoning within the event complex. as far as we know, this is different from all previous works that only formulated relations along a single axis. our model further demonstrates the potent capability of inducing event complexes when evaluated on the red dataset @xcite . ## related work various approaches have been proposed to extract event temprels. early effort focused on characterizing event pairs based on various types of semantic and linguistic features, and utilizing statistical learning methods, such as logistic regression @xcite and svm @xcite , to capture the relations. those methods typically require extensive feature engineering, and do not comprehensively consider the contextual information and global constraints among event-event relations. recently, data-driven methods have been developed for temprel extraction, and have offered promising performance. @xcite addressed this problem using a system combining an lstm document encoder and a siamese multi-layer perceptron (mlp) encoder for temporal commonsense knowledge from temprob @xcite . @xcite proposed a bidirectional lstm (bil-stm) with structured prediction to extract tem-prels. both of these works incorporated global inference to facilitate constraints on temprels. besides temprels, a couple of efforts have focused on event hierarchy construction, a.k.a. subevent relation extraction. this task seeks to extract the hierarchy where each parent event contains child events that are described in the same document. to cope with this task, both @xcite and @xcite introduced a variety of features and employed logistic regression models for classifying event pairs into subevent relations (parent-child and child-parent, coreference (coref), and no relation (norel). @xcite further extended the characterization with more features on the discourse and narrative aspects. @xcite presented a data-driven method by fine-tuning a time duration-aware bert @xcite on corpora of time mentions, and used the estimation of time duration to predict subevent relations. though previous efforts have been devoted to preserving logical consistency through inference or structured learning @xcite @xcite , this is difficult to do in the context of neural networks. moreover, while it is a common strategy to combine multiple training data in multi-task learning @xcite , our work is distinguished by enhancing the learning process by pushing the model towards a coherent output that satisfies logical constraints across separate tasks. ## methods in this section, we present the joint learning framework for event-event relation extraction. we start with the problem formulation ( §3.1), followed by the techniques for event pair characterization ( §3.2), constrained learning ( §3.3) and inference ( §3.4). ## experiments in this section, we present the experiments on eventevent relation extraction. specifically, we conduct evaluation for temprel and subevent relation extraction based on two benchmark datasets ( §4.1- §4.4). to help understand the significance of each model component in the framework, we also give a detailed ablation study ( §4.5). finally, a case study on the red dataset is described to demonstrate the capability of inducing event complexes ( §4.6). ## conclusion we propose a joint constrained learning framework for extracting event complexes from documents. the proposed framework bridges temprel and subevent relation extraction tasks with a comprehensive set of logical constraints, which are enforced during learning by converting them into differentiable objective functions. on two benchmark datasets, the proposed method outperforms sota statistical learning methods and data-driven methods for each task, without using data that is jointly annotated with the two classes of relations. it also presents promising event complex extraction results on red that is external to training. thus, our work shows that the global consistency of the event complex significantly helps understanding both temporal order and event membership. for future work, we plan to extend the framework towards an end-to-end system with event extraction. we also seek to extend the conjunctive constraints along with event argument relations.