Learning Cross-Task Dependencies for Joint Extraction of Entities, Events, Event Arguments, and Relations (2022)

Abstract: Extracting entities, events, event arguments, and relations (i.e., task instances) from text represents four main challenging tasks in information extraction (IE), which have been solved jointly (JointIE) to boost the overall performance for IE. As such, previous work often leverages two types of dependencies between the tasks, i.e., cross-instance and cross-type dependencies representing relatedness between task instances and correlations between information types of the tasks. However, the cross-task dependencies in prior work are not optimal as they are only designed manually according to some task heuristics. To address this issue, we propose a novel model for JointIE that aims to learn cross-task dependencies from data. In particular, we treat each task instance as a node in a dependency graph where edges between the instances are inferred through information from different layers of a pretrained language model (e.g., BERT). Furthermore, we utilize the Chow-Liu algorithm to learn a dependency tree between information types for JointIE by seeking to approximate the joint distribution of the types from data. Finally, the Chow-Liu dependency tree is used to generate cross-type patterns, serving as anchor knowledge to guide the learning of representations and dependencies between instances for JointIE. Experimental results show that our proposed model significantly outperforms strong JointIE baselines over four datasets with different languages.

## introduction entity mention recognition (emr), event trigger detection (etd), event argument extraction (eae), and relation extraction (re) are four main challenging tasks in information extraction (ie), which aim to extract entities (e.g., a person), events (e.g., an attack), event arguments (e.g., a victim in an attack), and relations (e.g., work-for) mentioned in text. these ie tasks have been solved mostly in pipelined approaches @xcite @xcite @xcite @xcite @xcite , where input to a model performing an ie task involves predictions from other models performing other ie tasks. as a result, errors in predictions by a model can be propagated to subsequent models in the pipeline to hurt overall performance. to avoid error propagation, the four ie tasks can be solved jointly (jointie) in a single model @xcite @xcite . as such, a key challenge for jointie models is to effectively capture dependencies between the ie tasks to boost overall extraction performance. in particular, two types of task dependencies are important for jointie, i.e., cross-instance and crosstype dependencies. first, for cross-instance dependencies, jointie models use instances to refer to word spans for event triggers/entity mentions (for emr and etd) or pair of word spans of event triggers/entity mentions (for eae and re) that should be classified according to predefined information types for ie. accordingly, an important insight from previous jointie models is to enrich the representation for one instance with those from related instances in different ie tasks to facilitate the type prediction @xcite . to this end, a typical approach to encode cross-instance dependencies for representation learning in previous work involves creating dependency graphs between instances to connect related instances to facilitate representation learning @xcite . however, as the instance dependency graphs in previous work are only created manually using some heuristics, e.g., connecting instances that share an entity mention or event trigger @xcite , they might be suboptimal for a given dataset and hinder further performance improvement for ie. consequently, to improve representation enrichment with information from related instances for jointie, our work proposes to automatically learn cross-instance dependency graphs for ie tasks from data. to enable maximal flexibility, we explore a fully connected graph between all task instances in a sentence where a dependency weight is assigned to each edge to quantify the relatedness between two instances. in our method, we argue that dependency weights between task instances should be computed over multiple sources of information to produce optimal and comprehensive dependency graphs. to this end, motivated by the encoding of different linguistic structures (e.g., semantics, syntax) in the layers of pre-trained language models (plms), e.g., bert @xcite , we propose to leverage the representations of instances at different layers of plms to compute dependency weights for the instances. in particular, given two instances for jointie, their representation vectors at each layer of a plm are consumed to produce a layer-specific dependency weight, which will be combined across layers to obtain an overall weight for our dependency graph. graph convolutional networks (gcns) @xcite will then be used to induce enriched representations for the instances based on the computed cross-instance dependency graph. in addition, cross-type dependencies/patterns in jointie systems characterize co-occurrences/corelations of information types of different ie tasks (e.g., entity/event types and argument roles) in a single input sentence. for instance, in the ace 2005 dataset @xcite , a "victim" argument for an "attack" event is likely to be the "victim" argument for a "die" event in the same sentence. accordingly, previous jointie models have leveraged cross-type dependencies either in the decoding phase, i.e., to form global type patterns/graphs to constrain the type prediction @xcite , or in the training phase, i.e., to form type dependency graphs to aid consistency regularization of golden and predicted types @xcite . however, as in cross-instance dependencies, the dependency graphs between information types in ie in previous work are also designed manually, e.g., by linking types that are involved in the same instance for some ie task @xcite . this is not desirable as manual designs might miss important cross-type patterns that cannot guarantee optimal performance for jointie. to this end, we propose to further learn crosstype dependencies/patterns from data to better support type predictions of jointie instances. as such, we view each information type in our ie tasks as a binary random variable, which is 1 if the type appears in the sentence, and 0 otherwise. this formulation enables us to employ bayesian structure learning algorithms to infer dependency structures from data. in particular, we propose to leverage the chow-liu algorithm @xcite ) that measures mutual information between any two types (variables) in training data to learn a first-order dependency tree, aiming to approximate the underlying joint distribution of the information types (types) for jointie. afterward, the resulting chow-liu tree containing induced dependencies between information types will be used to generate global cross-type patterns for jointie. to incorporate the learned cross-type dependencies into the jointie model, our goal is to leverage such global patterns to obtain additional features to further enrich the gcn-induced representations for type prediction. our intuition is to treat the induced cross-type patterns as anchor knowledge to which the information types, representations, and dependencies of ie instances in a sentence should adhere to exploit consistency and improve predictions for jointie in the data. to this end, for each learned cross-type pattern, we seek to compute a similarity score between the computed crossinstance dependency graph for an input sentence and the cross-type pattern that can be included into the representations for the instances to predict types. accordingly, we propose to leverage random walk graph kernels @xcite that facilitate similarity computation between two graphs (i.e., the cross-instance dependency graph and cross-type pattern) via counting common random walks on the graphs to enrich representations for jointie. finally, we evaluate the proposed model with induced cross-task and crosstype dependencies for jointie in both monolingual and cross-lingual learning settings. experimental results show that our model consistently outperforms strong baselines in all the settings across four different datasets and languages. ## conclusion we present a novel model to jointly solve four ie tasks (emr, etd, eae, and re). our model learns cross-instance dependencies through different layers of a plm and cross-type dependencies via the chow-liu algorithm. the cross-task dependencies are exploited via gcns and random walk kernels to improve representation learning. extensive experiments demonstrate the state-of-the-art performance of our model across four datasets with different languages and settings. in the future, we plan to extend our model to include more ie tasks.