Joint Extraction of Entities, Relations, and Events via Modeling Inter-Instance and Inter-Label Dependencies (2022)

Abstract: Event trigger detection, entity mention recognition, event argument extraction, and relation extraction are the four important tasks in information extraction that have been performed jointly (Joint Information Extraction - JointIE) to avoid error propagation and leverage dependencies between the task instances (i.e., event triggers, entity mentions, relations, and event arguments). However, previous JointIE models often assume heuristic manually-designed dependency between the task instances and mean-field factorization for the joint distribution of instance labels, thus unable to capture optimal dependencies among instances and labels to improve representation learning and IE performance. To overcome these limitations, we propose to induce a dependency graph among task instances from data to boost representation learning. To better capture dependencies between instance labels, we propose to directly estimate their joint distribution via Conditional Random Fields. Noise Contrastive Estimation is introduced to address the maximization of the intractable joint likelihood for model training. Finally, to improve the decoding with greedy or beam search in prior work, we present Simulated Annealing to better find the globally optimal assignment for instance labels at decoding time. Experimental results show that our proposed model outperforms previous models on multiple IE tasks across 5 datasets and 2 languages.

## introduction to extract structured information from unstructured text, a typical information extraction (ie) pipeline involves four major tasks: event trigger detection (etd), event argument extraction (eae), entity mention recognition (emr), and relation extraction @xcite . previous work has performed such ie tasks via pipelined approaches @xcite @xcite ), where a model for one task uses output predictions from other models performing other tasks. consequently, errors from the predictions can be propagated between the models in the pipeline. recently, etd, emr, eae, and re have been solved jointly in a single model, i.e., joint information extraction -jointie @xcite @xcite , to avoid error propagation and leverage dependency between prediction instances of the four ie tasks (i.e., event trigger, entity mention, relation, and event argument candidates in a sentence). for example, if a person entity mention is a victim argument for a die event, it is likely that the same entity mention is also a target argument for an attack event in the same sentence. to implicitly exploit instance dependency for representation learning, @xcite and @xcite employ a shared encoder to obtain representation vectors to classify instances of different ie tasks. later work heuristically captures dependency between ie task instances via explicitly connecting the task instances that share an entity mention or event trigger @xcite or aligning the task instances that share text spans with some nodes on a semantic graph @xcite to aid representation learning. while natural, these manual designs for dependency between task instances might not be optimal for representation learning of jointie. in addition to representation learning, at the prediction level, previous work tends to factorize the joint distribution of labels for all the task instances in jointie into the product of label distributions for each individual instance (i.e., performing local normalization), thus hindering the ability to fully exploit the interactions of instance labels across ie tasks. @xcite and @xcite mitigate this problem by decoding instance labels with handcrafted global features while @xcite focuses on encoding label interactions via consistency regularization over global type dependency graphs. however, these approaches still assume a factorization of the joint label distribution for prediction instances, thus unable to fundamentally address the label dependency encoding issue. recently, some works have attempted to directly model the joint distribution of instance labels by reformulating jointie tasks as text generation problems using state-of-the-art pre-trained seq2seq models, e.g., bart or t5 @xcite . in such generative models, text spans and labels for task instances are generated by the decoder in an autoregressive fashion to encode label dependency for joint distribution computation @xcite . unfortunately, this approach needs to assume an order of the task instances to be decoded (e.g., from left to right) that disallows later instances in the order to interfere/correct predictions for earlier instances, causing suboptimal performance for jointie. in this work, we aim to overcome these issues by inducing dependency between the task instances for jointie from data to boost representation learning, and directly modeling the joint distribution of the labels for all the task instances to fully enable label interactions. to this end, we consider each task instance as a node in a fully connected dependency graph; the weight for each edge is then learned to capture the dependency level between two corresponding instances. note that this is different from prior work @xcite that heuristically designs sparser dependency graphs with disconnected task instance pairs, thus failing to explore all possible interactions between instance pairs for optimal representations. in our method, the induced dependency graph for instance nodes is then employed by graph convolutional networks (gcns) @xcite to enhance the representation for each instance node with information from all the other nodes according to their dependency levels. afterwards, the enhanced instance representations and the induced dependency graph are utilized to estimate the joint distribution of instance labels via conditional random fields (crfs) @xcite . this formulation enables us to approximately maximize the intractable joint likelihood of the ground-truth instance labels via noise contrastive estimation (nce) (gutmann and hyvärinen, 2012), which converts the maximization problem into the nonlinear logistic regression discriminating between the true labels and the noise labels. finally, previous work for jointie has employed a greedy or beam search for decoding instance labels, which is not optimal due to their greedy nature. in this work, we propose a novel decoding algorithm for jointie via simulated annealing (sa) @xcite , which has been shown to be able to approximate the global optimum of a function @xcite . experimental results show that our proposed model for jointie significantly outperforms previous models on multiple tasks with large margins across 5 datasets and 2 languages. ## problem statement given an input sentence, etd aims to predict text spans and event types for event triggers based on a predefined set of event types, e.g., "attack" and "transport" @xcite . similarly, emr seeks to determine text spans and entity types (e.g., "person", "organization") for entity mentions in the sentence @xcite . different from the first two tasks, eae and re involves predictions for a pair of objects at a time. given an event trigger and an entity mention, eae aims to predict the argument role (e.g, "victim") of the entity mention for the event trigger @xcite ). an argument role can be "not-an-argument" indicating that the entity mention is not an argument for the trigger. for re @xcite , the task focuses on the classification of relation (e.g, "work for") for a given pair of entity mentions. there is also a special type "no-relation" to specify no relation between two entity mentions. as such, we call the union set c of the predefined event types, entity types, argument roles, and relation types as the information types (excluding "not-an-argument" and "no-relation"). ## model to capture dependency among task instances for jointie, an approach is to obtain all text spans for entity/event mention candidates along with their possible pairs to form the nodes for a dependency graph to improve representation learning. however, this approach will retain many text spans for non-entity/event mentions to introduce noise into the modeling. it will also entail a large dependency graph that can hinder the efficiency of the model. to this end, our model for jointie first identifies text spans for entity mentions and event triggers. afterwards, all possible pairs of evententity and entity-entity mentions are considered to identify positive pairs for event arguments and relations respectively. the detected entity mentions, event triggers, event arguments, and relations are called task instances that should be classified to obtain corresponding information types in c. in our model, a dependency graph among the detected task instances will be learned to provide inputs for gcns to compute dependency-enhanced representations for the task instances. finally, the enhanced representations will be used to compute a joint distribution over labels for all the task instances to train our model. we will also employ simulated annealing to achieve the global optimum for label assignment of the task instances in the decoding phase. ## experiments datasets: following previous work @xcite @xcite @xcite , we conduct experiments on 5 different datasets created by the 2005 automatic content extraction (ace05) @xcite and entity relation event (ere) @xcite programs. the three ace05 datasets feature ace05-r, ace05-e, and ace-e+, all in english, involving 33 event types, 7 entity types, 6 relation types, and 22 argument roles. the two ere datasets are ere-en (english portion) and ere-es (spanish portion), introducing 38 event types, 7 entity types, 5 relation types, and 20 argument roles. we use the same data processing and train/dev/test splits as the prior work for a fair comparison. detailed statistics for the datasets are shown in baselines: we compare our method, called gra-phie, with the following baselines for jointie: generative baselines: text2event @xcite and degree @xcite . the generative baselines perform etd and eae via formulating the tasks as text generation. the models receive an input sentence and generate an output text containing text spans and labels for event triggers and event arguments, structured in a way that a post-processing step can be used to extract etd and eae predictions for the models. classification baselines: oneie (lin et al., 2020), amrie @xcite @xcite . the classification baselines represent the instances for etd, emr, eae, and re via a shared encoder and perform classification for the instances based on task-specific label distributions. amrie and fourie employ a heuristic dependency graph among task instances to improve representation learning. dependency between instance labels is exploited in oneie and amrie via a beam search decoding with manually-designed hyper-parameters: prior work for jointie employs two different versions of pre-trained language models (plm), i.e., bert @xcite @xcite and roberta @xcite , which might cause incompatible compassion. to this end, we explore both bert and roberta to obtain the word representations x i for graphie for a fair comparison. for the spanish ere-es dataset, following prior work @xcite , we utilize the multilingual versions of bert and roberta. for each plm, we fine-tune the hyperparameter for graphie on the development data. in particular, the best values for the hyperparameters of the proposed model are reported as follows. we employ the learning rate of 1e -5 for the models with the bert-based plm (i.e., using bert-large-cased and bert-multilingual-cased) and the learning rate of 5e -6 for the roberta-based plm (i.e., using roberta-large and xlm-robertalarge). for other hyper-parameters, our tuning process results in the same values for bert-based and robert-based models: adam (kingma and ba, 2014) for the optimizer, batch size of 10, 100 for the size of the dependency relation embeddings, 400 for the size of the hidden vector for the feedforward networks, 200 for the hidden vector size in the gcn model, 2 for the number of layers for the feed-forward networks and gcn model, γ = 1 for the trade-off hyper-parameter for the global score, n @xmath1, @xmath2, n @xmath3, and ϵ = 0.1 for the temperature threshold for the sa decoding. ## related work capturing dependency between ie tasks has been a main focus of previous work on joint ie. early work employed feature engineering methods @xcite @xcite ## conclusion we propose a novel model for jointly solving four ie tasks (emr, etd, eae, and re). our proposed model learns a dependency graph among the in-stances of the tasks via a novel edge weighting mechanism. we also estimate the joint distribution among instance labels to fully enable interactions between instance labels for improved performance. the experimental results show that our model achieves best performance for multiple join-tie tasks across 5 datasets and 2 languages. in the future, we plan to extend our method to cover more ie tasks such as event coreference resolution.