Modeling Instance Interactions for Joint Information Extraction with Neural High-Order Conditional Random Field (2023)

Abstract: Prior works on joint Information Extraction (IE) typically model instance (e.g., event triggers, entities, roles, relations) interactions by representation enhancement, type dependencies scoring, or global decoding. We find that the previous models generally consider binary type dependency scoring of a pair of instances, and leverage local search such as beam search to approximate global solutions. To better integrate cross-instance interactions, in this work, we introduce a joint IE framework (CRFIE) that formulates joint IE as a high-order Conditional Random Field. Specifically, we design binary factors and ternary factors to directly model interactions between not only a pair of instances but also triplets. Then, these factors are utilized to jointly predict labels of all instances. To address the intractability problem of exact high-order inference, we incorporate a high-order neural decoder that is unfolded from a mean-field variational inference method, which achieves consistent learning and inference. The experimental results show that our approach achieves consistent improvements on three IE tasks compared with our baseline and prior work.

## introduction information extraction (ie) has long been considered a fundamental challenge for various downstream natural language understanding tasks, such as knowledge graph construction and reading comprehension, etc. the goal is to identify and extract structured information from unstructured natural language text, such that both users and machines can easily comprehend the entities, relations, and events within the text. typically, ie consists of a series of different tasks to recognize entities, connect coreferences, ). to effectively capture instance or task dependencies, joint ie tries to simultaneously predict instances of different ie tasks for an input text with a multitask learning scheme, which attracts lots of interest and demonstrates significant improvements over specific-task learning methods. previous work of joint ie focuses on three directions: 1) representation enrichment by sharing the token encoder between different ie tasks @xcite , up-dating shared span representations according to local task-specific predictions @xcite , creating dependency graphs between instances @xcite @xcite , or leveraging external dependency relations such as abstract meaning representation (amr) and syntactic structures @xcite ; 2) type dependency scoring by forming type patterns constraints @xcite , designing type dependency graphs @xcite , learning transition matrix of type pairs @xcite , or computing mutual information (mi) scores of each pair of types @xcite ; 3) global decoding by beam search according to global features or amr graphs @xcite , or adopting global optimization algorithms such as simulated annealing @xcite . our interest lies in the second and third directions and we find two main limitations of prior works. the first one is that they only score binary dependencies of instance types (i.e. constraint, transition, or mi scores between a pair of types). the second one is that their decoders are based on discrete local search strategies to approximate global optima, and they often employ different approximate strategies for inference and training. to alleviate aforementioned limitations, we propose a novel joint ie framework, information extraction as high-order crf (crfie), that explicitly models label correlations between different instances from the same or different tasks, and utilizes them to calculate a joint distribution for final instance label predictions. specifically, we demonstrate the effectiveness of our proposed high-order framework on three widely-explored ie tasks: entity recognition (entr), relation extraction (rele) and event extraction (evente). we formulate the three tasks as a unified graph prediction problem, further modeled as a high-order conditional random field (crf) @xcite , where variables contain node variables and edge variables representing trigger/entity instances and role/relation instances respectively. the term "high-order" refers to factors connecting two or more correlated variables. beyond the unary (first-order) factor, we design not only the binary (second-order) factor to model the interactions between a pair of edge variables but also the ternary (third-order) factor to model the interac-tions between node-edge-node variables. since the correlated instances may come from the same or different tasks, we categorize our high-order factors into two types: homogeneous factors (homo) representing correlations between instances of the same task, and heterogeneous factors (hete) representing correlations between instances of different tasks. taking entr and evente as an example, we calculate binary factor potentials of rolerole pairs (homo), and ternary factor potentials of trigger-role-entity triplets (hete). we leverage these scores to predict the labels of all instances jointly. since exact high-order inference is analytically intractable, we incorporate a neural decoder that is unfolded from the approximate mean-field variational inference (mfvi) @xcite ) method, which achieves end-to-end training and also consistent inference and learning processes. note that mfvi can be seen as a continuous relaxation for crf inference @xcite , which can often be more effective than discrete optimization used in previous work. experiments on joint ie tasks show that crfie achieves competitive or better performance compared with previous stateof-the-art models foot_0 . ## experiments datasets we evaluate our model on the ace2005 corpus @xcite evaluation we use f1 scores to evaluate our model's performance as in most previous work @xcite @xcite . for the entr task, an entity (ent) is correct if both its type and offsets match a gold entity. for the rele task, a relation (rel) is correct if both its type and the offsets of its two related entities match a gold relation. in addition, a strict relation evaluation (rel+) requires that the types of the two related entities are also correct. a trigger is correctly identified (trig-i) if its offsets match a gold trigger. it is correctly classified (trig-c) if its corresponding event type also matches the reference trigger. an argument is correctly identified (arg-i) if its offsets match a gold argument and its corresponding event type is correct. it is correctly classified (arg-c) if its role type also matches the reference argument. all experimental results of our approach shown in this paper are the average of three runs with different random seeds. implementation details for fair comparison with previous state-of-the-art systems, we use the bert-large-cased model @xcite or roberta model @xcite as our encoder for the ace05-e and ace05-e+ datasets, and al-bert model @xcite as the encoder for the ace05-r dataset. we train our model with bertadam optimizer foot_2 . when we use a single kind of factor, α is set to 1 for the used and set to 0 for others. when multiple kinds of factors are used, α of the used are tunable parameters. detailed hyperparameter values are provided in appendix b. ## related work information extraction classical ie models are typically task-specific @xcite @xcite . recent efforts develop joint methods for multiple ie tasks @xcite @xcite @xcite or general architectures for universal ie @xcite @xcite . graph-based joint ie methods formulate multiple ie tasks as a graph prediction task and aim to capture dependencies between different instances or tasks. lots of previous works leverage encoder sharing or graph convolutional networks (gcns) on instance dependency graphs to enrich instance representations @xcite @xcite . this work is more relevant to some recent works that take efforts on type interactions and global inference. lin et al. (2020) manually designs global features as constraints and leverages beam search to find approximated global optima. based on the method of lin et al. (2020), van nguyen et al. (2021) further incorporates amr graphs as external dependencies. the work of van nguyen @xcite is more similar to ours in that they adopt a crf to model type dependencies, but they learn a transition matrix that only scores binary dependencies. besides, they employ noise contrastive estimation (nce) @xcite to perform approximate training and simulated an-sentence & analysis baseline high-order #1: as well as previously (v1) holding senior positions at barclays bank (v2), bzw (v3) and kleinwort benson (v4), mccarthy was formerly a top civil servant at the department of trade and industry. analysis: sibling factor helps our high-order model find the bzw which is tied for barclays bank to be an argument of event personnel:end-position triggered by word previously. analysis: an entity with per type has less possibility to play an artifact role. ternary factor leverages messages passed by node label distributions to refine the edge label which in turn gives the message to refine node labels. analysis: there is less possibility that a per entity has phys relation with gpe entity and fac entity at the same time. sibling and ternary factors help our high-order model in this situation. nealing search to perform approximate inference. different from their work, we model both binary and ternary dependencies and leverage mfvi to achieve consistent training and inference. high-order methods previous high-order methods most focus on instance interactions in training process to get more expressive representations, such as sharing representations @xcite or using sequence-to-sequence architecture @xcite @xcite . there are some high-order inference methods that are related to us on different nlp tasks. on dependency parsing, wang and tu (2020) considered three types of second-order parts of semantic dependencies and approximate decoding with mean-field variational inference or loopy belief propagation. @xcite considered interactions between two arguments of the same predicate on semantic role labeling task. however, due to the complexity, they only did high-order inference on edge existence prediction while leaving label prediction in first-order, and they did not involve heterogeneous factors. in another line of research, @xcite integrate logic rules and neural network to leverage prior knowledge to help relation extraction and event extraction tasks. but they cannot achieve end-to-end training and inference. ## conclusion in this paper, we propose a novel framework that leverages high-order interactions across different instances and different ie tasks in both training and inference processes. we formulate ie tasks as a unified graph prediction problem, further modeled as a high-order crf. our framework consists of an identification module to identify spans as graph nodes and a node/edge labeling module with highorder modeling and inference to jointly label all nodes and edges.