Domain Knowledge Empowered Structured Neural Net for End-to-End Event Temporal Relation Extraction (2020)

Abstract: Extracting event temporal relations is a critical task for information extraction and plays an important role in natural language understanding. Prior systems leverage deep learning and pre-trained language models to improve the performance of the task. However, these systems often suffer from two shortcomings: 1) when performing maximum a posteriori (MAP) inference based on neural models, previous systems only used structured knowledge that is assumed to be absolutely correct, i.e., hard constraints; 2) biased predictions on dominant temporal relations when training with a limited amount of data. To address these issues, we propose a framework that enhances deep neural network with distributional constraints constructed by probabilistic domain knowledge. We solve the constrained inference problem via Lagrangian Relaxation and apply it to end-to-end event temporal relation extraction tasks. Experimental results show our framework is able to improve the baseline neural network models with strong statistical significance on two widely used datasets in news and clinical domains.

## introduction extracting event temporal relations from raw text data has attracted surging attention in the nlp research community in recent years as it is a fundamental task for commonsense reasoning and natural language understanding. it facilitates various downstream applications, such as forecasting social events and tracking patients' medical history. figure 1 shows an example of this task where an event extractor first needs to identify events (buildup, say and stop) in the input and then a relation classifier predicts all pairwise relations among them, resulting in a temporal ordering as illustrated in the figure. for example, say is before stop; buildup includes say; the temporal ordering between buildup and stop cannot be decided from the context, so the relation should be vague. predicting event temporal relations is inherently challenging as it requires the system to understand each event's beginning and end times. however, these time anchors are often hard to specify within a complicated context, even for humans. as a result, there is usually a large amount of vague pairs (nearly 50% in the table of figure 1 ) in an expert-annotated dataset, resulting in heavily classimbalanced datasets. moreover, expert annotations are often time-consuming to gather, so the sizes of existing datasets are relatively small. to cope with the class-imbalance problem and the small dataset issues, recent research efforts adopt hard constraint-enhanced deep learning methods and leverage pre-trained language models @xcite and are able to establish reasonable baselines for the task. the hard-constraints used in the sota systems can only be constructed when they are nearly 100% correct and hence make the knowledge adoption restrictive. temporal relation transitivity is a frequently used hard constraint that requires if a before b and b before c, it must be that a before c. however, constraints are usually not deterministic in real-world applications. for example, a clinical treatment and test are more likely to happen after a medical problem, but not always. such probabilistic constraints cannot be encoded with the hard-constraints as in the previous systems. furthermore, deep neural models have biased predictions on dominant classes, which is particularly concerning given the small and biased datasets in event temporal extraction. for example, in figure 1 , an event pair headed and say (with relation includes) is incorrectly predicted as vague (column type pair (p)) by our baseline neural model, partially due to dominant percentage of vague label (column overall), and partially due to the complexity of the context. using the domain knowledge that headed and say have event types of occurrence and reporting, respectively, we can find a new label probability distribution (type pair (g)) for this pair. the probability mass allocated to vague would decrease by 10% and increase by 7.2% for includes, which significantly increases the chance for a correct label prediction. we propose to improve deep structured neural networks by incorporating domain knowledge such as corpus statistics in the model inference, and by solving the constrained inference problem using lagrangian relaxation. this framework allows us to benefit from the strong contextual understanding of pre-trained language models while optimizing model outputs based on probabilistic structured knowledge that previous deep models fail to consider. experimental results demonstrate the effectiveness of this framework. we summarize our contributions below: ## problem formulation the problem we focus on is end-to-end event temporal relation extraction, which takes a raw text as input, first identifies all events, and then classifies temporal relations for all predicted event pairs. the left column of figure 2 shows an example. an endto-end system is practical in a real-world setting where events are not annotated in the input and challenging because temporal relations are harder to predict after noise is introduced during event extraction. ## method in this section, we first describe the details of our deep neural networks for an end-to-end event temporal relation extraction system, then show how to formulate domain-knowledge between event types and relations as distributional constraints in integer linear programming (ilp), and finally apply lagrangian relaxation to solve the constrained inference problem. our base model is trained end-toend with cross-entropy loss and multitask learning to obtain relation scores. we need to perform an additional inference step in order to incorporate domain-knowledge as distributional constraints. ## constrained inference implementation this section explains how to construct our distributional constraints and the implementation details for inference with lr. ## experimental setup this section describes the two event temporal relation datasets used in this paper and then explains the evaluation metrics. the timebank-dense dataset mitigates this issue by forcing annotators to examine all pairs of events within the same or neighboring sentences, and this dataset has been widely evaluated on this task @xcite @xcite . temporal relations consist of before, after, includes, included, simultane-ous, and vague. moreover, each event has several properties, e.g., type, tense, and polarity. event types include occurrence, action, reporting, state, etc. event pairs that are more than 2 sentences away are not annotated. ## results and analysis ## related work news domain. early work on temporal relation extraction use local pair-wise classification with hand-engineered features @xcite @xcite . later efforts, such as cleartk @xcite , uttime @xcite , navytime @xcite , and caevo @xcite , improve earlier work with better linguistic and syntactic rules. @xcite ; @xcite ; @xcite explore structured learning for this task, and more recently, neural methods have also been shown effective @xcite @xcite . @xcite and @xcite are the most recent work leveraging neural network and pre-trained language models to build an end-to-end system. our work differs from these prior work in that we build a structured neural model with distributional constraints that combines both the benefits of both deep learning and domain knowledge. clinical domain. the 2012 i2b2 challenge ( @xcite ) is one of the earliest efforts to advance event temporal relation extraction of clinical data. the challenge hosted three tasks on event (and event property) classification, temporal relation extraction, and the end-to-end track. following this early effort, a series of clinical event temporal relation challenges were created in the following years @xcite @xcite ). however, data in these challenges are relatively hard to acquire, and therefore they are not used in this paper. as in the news data, traditional machine learning approaches @xcite @xcite @xcite that tackle the end-to-end event and temporal relation extraction problem require timeconsuming feature engineering such as collecting lexical and syntax features. some recent work @xcite @xcite apply neural network-based methods to model the temporal relations, but are not capable of incorporating prior knowledge about clinical events and temporal relations as proposed by our framework. ## conclusion in conclusion, we propose a general framework that augments deep neural networks with distributional constraints constructed using probabilistic domain knowledge. we apply it in the setting of end-to-end temporal relation extraction task with event-type and relation constraints and show that the map inference with distributional constraints can significantly improve the final results. we plan to apply the proposed framework on various event reasoning tasks and construct novel distributional constraints that could leverage domain knowledge beyond corpus statistics, such as the larger unlabeled data and rich information contained in knowledge bases.