Cross-Task Instance Representation Interactions and Label Dependencies for Joint Information Extraction with Graph Convolutional Networks (2021)

Abstract: Existing works on information extraction (IE) have mainly solved the four main tasks separately (entity mention recognition, relation extraction, event trigger detection, and argument extraction), thus failing to benefit from inter-dependencies between tasks. This paper presents a novel deep learning model to simultaneously solve the four tasks of IE in a single model (called FourIE). Compared to few prior work on jointly performing four IE tasks, FourIE features two novel contributions to capture inter-dependencies between tasks. First, at the representation level, we introduce an interaction graph between instances of the four tasks that is used to enrich the prediction representation for one instance with those from related instances of other tasks. Second, at the label level, we propose a dependency graph for the information types in the four IE tasks that captures the connections between the types expressed in an input sentence. A new regularization mechanism is introduced to enforce the consistency between the golden and predicted type dependency graphs to improve representation learning. We show that the proposed model achieves the state-of-the-art performance for joint IE on both monolingual and multilingual learning settings with three different languages.

## introduction information extraction (ie) is an important and challenging task in natural language processing (nlp) that aims to extract structured information from unstructured texts. following the terminology for ie in the popular ace 2005 program @xcite , we focus on four major ie tasks in this work: entity mention extraction (eme), relation extraction (re), event trigger detection (etd), and event argument extraction @xcite . given an input sentence, a vast majority of prior work has solved the four tasks in ie independently at both instance and task levels (called independent ## problem statement and background problem statement: the joint four-task ie problem in this work takes a sentence as the input and aims to jointly solve four tasks eae, etd, re, and eae using an unified model. as such, the goal of eme is to detect and classify entity mentions (names, nominals, pronouns) according to a set of predefined (semantic) entity types (e.g., person). similarly, etd seeks to identify and classify event triggers (verbs or normalization) that clearly evoke an event in some predefined set of event types (e.g., attack). note that event triggers can involve multiple words. for re, its concern is to predict the semantic relationship between two entity mentions in the sentence. here, the set of relations of interest is also predefined and includes a special type of none to indicate no-relation. finally, in eae, given an event trigger, the systems need to predict the roles (also in a predefined set with a special type none) that each entity mention plays in the corresponding event. entity mentions are thus also called event argument candidates in this work. figure 1 presents a sentence example where the expected outputs for each ie task are illustrated. graph convolutional networks (gcn): as gcns are used extensively in our model, we present their computation process in this section to facilitate the discussion. given a graph @xmath0, e) where @xmath1, . . . , v u } is the node set (with u nodes) and e is the edge set. in gcn, the edges in g are often captured via the adjacency matrix @xmath2. also, each node v @xmath3. as such, a gcn model involves multiple layers of abstraction in which the hidden vector v l i for the node v @xmath4: ## model given an input sentence @xmath5, w 2 , . . . , w n ] (with n words), our model for joint four-task ie on w involves three major components: (i) span detection, (ii) instance interaction, and (iii) type dependency-based regularization. ## experiments datasets. following the prior work on joint fourtask ie @xcite , we evaluate our joint ie model (fourie) on the ace 2005 @xcite and ere datasets that provide annotation for entity mentions, event triggers, relations, and argument roles. in particular, we use three different versions of the ace 2005 dataset that focus on three major joint inference settings for ie: (i) ace05-r for joint inference of eme and re, (ii) ace05-e for joint inference of eme, etd and eae, and (iii) ace05-e+ for joint inference of the four tasks eme, etd, re, and eae. ace05-e+ is our main evaluation setting as it fits to our model design with the four ie tasks of interest. for ere, following (lin et al., 2020), we combine the data from three datasets for english (i.e., ldc2015e29, ldc2015e68, and ldc2015e78) that are created under the deep exploration and ## related work the early joint methods for ie have employed feature engineering to capture the dependencies between ie tasks, including integer linear programming for global constraints @xcite , markov logic networks @xcite @xcite @xcite , and graphical models @xcite . recently, the application of deep learning has facilitated the joint modeling for ie via shared parameter mechanisms across tasks. these joint models have focused on different subsets of the ie tasks, including eme and re @xcite @xcite @xcite @xcite , event and temporal re @xcite @xcite @xcite . however, none of these work has explored joint inference for four ie tasks eme, etd, re, and eae as we do. the two most related works to ours include @xcite that leverages the bert-based information propagation via dynamic span graphs, and (lin et al., 2020) that exploits bert and global type dependency features to constrain the decoding step. our model is different from these works in that we introduce a novel interaction graph for instance representations for four ie tasks and a global type dependency graph to directly inject the knowledge into the training process. ## conclusion we present a novel deep learning framework to jointly solve four ie tasks (eme, etd, re, and eae). our model attempts to capture the interdependencies between instances of the four tasks and their types based on instance interaction and type dependency graphs. gcn models are employed to induce representation vectors to perform type predictions for task instances and regularize the training process. the experiments demonstrate the effectiveness of the proposed model, leading to sota performance over multiple datasets on english, chinese, and spanish. in the future, we plan to extend the model to include more ie tasks (e.g., coreference resolution). arzoo katiyar and claire cardie. 2017. going out on a limb: joint extraction of entity mentions and relations without dependency trees. in proceedings of the 55th annual meeting of the association for computational linguistics. thomas n. kipf and max welling. 2017. semisupervised classification with graph convolutional networks. in proceedings of the 5th international conference on learning representations. viet dac lai, tuan ngo nguyen, and thien huu nguyen. 2020. event detection: gate diversity and syntactic importance scores for graph convolution neural networks. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp). qi li, s. anzaroot, w. lin, xiang li, and h. ji. 2011. joint inference for cross-document information extraction. in proceedings of the 20th acm international conference on information and knowledge management. qi li, heng ji, yu hong, and sujian li. 2014. constructing information networks using one single model. in proceedings of the 2014 conference on empirical methods in natural language processing. qi li, heng ji, and liang huang. 2013. joint event extraction via structured prediction with global features. in proceedings of the 51th annual meeting of the association for computational linguistics. ying lin, heng ji, fei huang, and lingfei wu. 2020. a joint neural model for information extraction with global features. in proceedings of the 58th annual meeting of the association for computational linguistics. yi luan, dave wadden, luheng he, amy shah, mari ostendorf, and hannaneh hajishirzi. 2019. a general framework for information extraction using dynamic span graphs. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies. makoto miwa and mohit bansal. 2016. end-to-end relation extraction using lstms on sequences and tree structures. in proceedings of the 54th annual meeting of the association for computational linguistics. makoto miwa and yutaka sasaki. 2014. modeling joint entity and relation extraction with table representation. in proceedings of the 2014 conference on empirical methods in natural language processing. thien huu nguyen, kyunghyun cho, and ralph grishman. 2016. joint event extraction via recurrent neural networks. in proceedings of the 2016 conference of the north american chapter of the association for computational linguistics. thien huu nguyen and ralph grishman. 2015a. event detection and domain adaptation with convolutional neural networks. in the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing. thien huu nguyen and ralph grishman. 2015b. relation extraction: perspective from convolutional neural networks. in proceedings of the 1st naacl